Linux notes and commands

Redhat

Linux command library

How to Use Lxplus at CERN

Linux installation at CERN

Copy/paste with Windows10->XLaunch->Ubuntu
ie from a LibreOffice Impress (ppt) screen) :
Copy/paste within Ubuntu window: Highlight linux text with the left hand mouse button down, then just right hand mouse click on cmd line.

Copy/paste from Ubuntu window to Windows10 screen
Highlight linux text with the left hand mouse button down, paste into Impress using the Paste button in Impress.

Other proposals:
CTRL+SHIFT+C/V.
Alternatively you can try pasting by pressing Enter or the middle mouse button.
Also: selecting (left click and drag) to copy, and middle mouse (tap down on scroll wheel) to paste. Worked when adding the shift did not.

How to Copy & Paste in PuTTY:
In Windows, you can copy information from an application to the clipboard by highlighting it and pressing "Ctrl-C." You can paste information from the clipboard to a Windows application by pressing "Ctrl-V." In a PuTTY session, the Windows shortcut keys lose their meaning. Instead of "Ctrl-C," you can copy to the Windows clipboard by highlighting text with your mouse. Instead of pressing "Ctrl-V" to paste text from Windows, press the right mouse button.

Between Windows and PuTTY:
To copy from Windows and paste into PuTTY, highlight the text in Windows, press "Ctrl-C," select the PuTTY window, and press the right mouse button to paste. To copy from PuTTy and paste into Windows, highlight the information in PuTTY and press "Ctrl-V" in the Windows application to paste it.

Within PuTTY:
To copy and paste from within PuTTY, simply highlight the text you want to copy, position your cursor at the place you want to paste the text and press the right mouse button. If you're editing a document using a text editor, such as Nano or Vi, you can also use the editor's cut and paste functionality.

shell script
slides

#!/bin/bash
# My first script

echo "Hello World!"

The first line of the script is important. This is a special clue, called a shebang, given to the shell indicating what program is used to interpret the script. In this case, it is /bin/bash. Other scripting languages such as Perl, awk, tcl, Tk, and python also use this mechanism.

The second line is a comment. Everything that appears after a "#" symbol is ignored by bash. As your scripts become bigger and more complicated, comments become vital. They are used by programmers to explain what is going on so that others can figure it out. The last line is the echo command. This command simply prints its arguments on the display.

date
https://stackoverflow.com/questions/20361982/get-current-time-in-hours-and-minutes
date +%H:%M gives: 14:22
date; gives: Mon Oct 22 14:23:38 CEST 2018

cat command
merge hints

Append the contents of file2.txt to the end of file1.txt. 
The content of file3.txt is appended to the end of merged contents of file1.txt and file2.txt and so on.
The entire merged file is saved with the name mergedfile.txt in the current working directory:

bash$ cat file1.txt file2.txt file3.txt file4.txt > ./mergedfile.txt

The cat command accepts regular expressions as input file names, 
which means you can use them to reduce the number of arguments:

bash$ cat file*.txt my*.txt > mergedfile.txt
This will merge all the files in the current directory that start with the name file 
and has a txt extension followed by the files that start with my and has a txt 
extension. You have to be careful about using regular expressions, if you want 
to preserve the order of files. If you get the regular expression wrong, it will affect 
the exact order in which the files are merged.


A quick and easy way to make sure the files get merged in the exact order you want
is to use the output of another file listing program such as ls or find and pipe it to the 
cat command. First execute the find command with the regular expression and verify 
the file order…

bash$ find . -name "file*.txt" -o -name "my*.txt"
This will print the files in order such that you can verify it to be correct or modify it 
to match what you want. You can then pipe that output into the cat command.

bash$ find . -name "file*.txt" -o -name "my*.txt" | xargs cat > ./mergedfile.txt
When you merge multiple files into one file using regular expressions to match them, 
especially when it is piped and where the output file is not very obvious, make sure that 
the regular expression does not match the filename of the merged file. In the case 
that it does match, usually the cat command is pretty good at error-ing out with the 
message “input file is output file”. But it helps to be careful to start with.

RAL linux

• linux.pp.rl.ac.uk
• same usrn as at CERN

ssh -Y davec@linux.pp.rl.ac.uk
firefox &

Linux beginners guide

Linux Support Page

Printing from Linux

ssh faqs

• CERN Basic Linux
• What the root directories hold
• Basic shell commands
• About shells
• Writing bash scripts

Availabe printers

• type> /usr/sbin/lpadmincern --list --building XX
• alias lprpretty 'lpr -P 40-1B-HPCOR -oprettyprint'
• lprpretty fn to print

Using the convert command for animated gifs etc

• link, jpg to video
  
• convert -delay 20 -loop 0 evt*.gif animatedevts.gif
• NOTE: does not work on .png files to one animated png - need to convert to gif - can be done on the same command line:

convert -delay 10 -loop 0 *.png animation.gif

importing animated gifs into pdf
link

linux command.org

50 most used commands

List of Unix programmes

cd tricks

grep

More grep

xargs

Unix and Linux shortcut keys

find

PATH, CMSSW PATH info

Linux updates

System->Administration->Software update

• May get errors - try deselting item, or all items (rt click - drop down)
• cat /etc/redhat-release for release version
• /usr/bin/yum update ---skip-broken to try to avoid broken updates

Browsing files on host machine

Access file browser windows in the following ways:

• Click on the file browser window launcher on the top edge panel
• Choose Applications -> Browse Filesystem
• Select a folder, then choose File-> Browse Folder

SCP

Example syntax for Secure Copy (scp)

Copy the file "foobar.txt" from a remote host to the local host
$ scp your_username@remotehost.edu:foobar.txt /some/local/directory

Copy the file "foobar.txt" from the local host to a remote host
$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory

WinSCP

Trouble with WinSCP from home - big block SFTP transfer error.
Remove print statement (echo ROOTSYS = $ROOTSYS) from .tcshrc. All OK now.
Put all print statements into .login file !!!!!!!!!!

Combine files

cat file2 >> file1
The >> operator appends the output to the named file or creates the named file if it does not exist.

cat file1 file2 > file3
This concatenates two or more files to one. You can have as many source files as you need. For example,

cat *.txt >> newfile.txt

Linux commands

Command	Action
./	. denotes the current directory. Since you want to run a file in your current directory and that directory is not in your $PATH, you need the ./ bit to tell the shell where the executable is. So, ./foo means run the executable called foo that is in this directory.
source	source is a bash shell built-in command that executes the content of the file passed as argument, in the current shell. It has a synonym in '.' (period).
	source filename [arguments], or writing with its synonym, '.' (period) :
	. filename [arguments]
	ie source .tcshrc to make changes come into effect during the current session
sh files	do chmod 700 fn.sh to make executable
	do $bash ~davec/fn.sh to execute
g++	to compile with C++, ie g++ map-test.C will generate an executable, by default a.out
	g++ map-test.C -o map.out
	g++ filename -o outputfile
	g++ filename -Wall -ansi -o outputfile
	g++ main.cpp f1.cpp f2.cpp f3.cpp for multiple files, keep header files in same folder or explicitly address
	In order to compile with all warnings enabled and to produce standards-compatible C++ code
	If you want to have the compiler treat warnings as errors--meaning you don't even get an executable, you can use the -Werror flag. This will make sure you don't miss an error.
	Do ./a.out to run
	Also see g++
file	manipulate file names and attributes
display	ie display fn.png will show the image on any workstation running X
gv	ie gv file.pdf to display a pdf or eps file
cp -R	copy folders with files/folders
	cp -R /afs/cern.ch/user/d/davec/CMSSW_8_0_8/src/Reco/ /afs/cern.ch/user/d/davec/CMSSW_8_0_17/src
	many associated display functions
command line	tcsh examples
	command-line-format.txt
	set prompt="$HOST "
	note: leaves space to look good
	set prompt = "-- %T %n %~ -- \n$ " gives:
	-- 23:16 davec ~/my/tmp --
	$
\033	\033 commands etc
echo	print to standard output
	echo CMSSW_BASE is $CMSSW_BASE
	CMSSW_BASE is /afs/cern.ch/user/d/davec/HZZ/CMSSW_5_3_9
du -h	get size of directory, -h option to get output in human readable format i.e. show output in kilobytes (K), megabytes (M) and gigabytes (G)
du -sh	get size of directory but does not list size of subdirectories
	du -h /dir1/file2
Output stream	ofstream myfile;
	myfile.open ("example.txt");
	myfile << "Writing this to a file.\n";
	myfile.close();
Redirecting stdout	ie: redirect printf output
	freopen ("myfile.txt","w",stdout);
	printf ("This sentence is redirected to a file.");
	fclose (stdout);
	NOTE: "w" creates an empty file for output operations.
	If a file with the same name already exists, its contents are discarded
	and the file is treated as a new empty file.
	freopen ("myfile.txt","a",stdout);
	"a" appends: it opens file for output at the end of a file.
	Output operations always write data at the end of the file, expanding it.
	The file is created if it does not exist.
setenv	All variables set with setenv command are automatically exported to subshell.
set	All csh variables set to with set command are NOT automatically exported to subshell.
$	the shell sees a word that begins with a "$", tries to find out what was assigned to the variable and substitutes it.
$(command)	The characters "$( )" tell the shell, "substitute the results of the enclosed command."
	Don't put spaces on either side of the equal sign when assigning value to variable. For e.g. In following variable declaration there will be no error
	$ no=10
	But there will be problem for any of the following variable declaration:
	$ no =10
	$ no= 10
	$ no = 10
"	Double Quotes "Double Quotes" - Anything enclose in double quotes removed meaning of that characters (except \ and $).
'	Single quotes 'Single quotes' - Enclosed in single quotes remains unchanged.
`	Back quote `Back quote` - To execute command, ie echo "Today is `date`".
	in bash: right_now = $(date +"%x %r %Z") in tsch: set right_now = `date`
	time_stamp="Updated on $right_now by $USER" in tcsh set time_stamp="Updated on $right_now by $USER"
linux info	Linux commands page
shell	Learning the shell
	Writing shell scripts
command line	convention, UPPERCASE for constants and environment variables
	convention, lowercase names for variables
	you can always type the name of the shell to switch.(bash,csh,sh,tcsh)
	Linux shell scripts
	Working with commands
	How to intrduce arguments on the command line
sed	stream editor - to alter file contents etc
	Intro, sed by Bruce Barnett!
	Unix Sed Working methodology
	Called as one execution cycle. Cycle continues till end of file/input is reached:
	Read a entire line from stdin/file. Removes any trailing newline. Places the line, in its pattern buffer.
	Modify the pattern buffer according to the supplied commands. Print the pattern buffer to stdout.
	sed -n '3'p thegeekstuff.txt. Prints the 3rd line in the file thegeekstuff.txt
	sed -n '31445,32889'p thegeekstuff.txt. Prints lines 31445 to 32889 from the file thegeekstuff.txt
	sed '2d' fn > fn2 to delete line 2 in file fn and make the new file fn2
	Other examples
lines	in PuTTy change the lines stored in the session
lines	command_with_lots_of_output | less
screen	screen
cd tricks	alias ..="cd .." alias ..2="cd ../.." alias ..3="cd ../../.." alias ..4="cd ../../../.." alias ..5="cd ../../../../.."
cd ../../../..	navigate 4 directories up
cd -	toggle between last two directories
ls --color	directories are in blue, pictures in purple, and executables in green.
unzip	unzip fn.zip
tar -xvwf fn.tar	untar a tar file (there is no 'untar' command)
tar -xvwzf fn.tar.gz	unzip and untar a tar.gz file
tar -xvwzf fn.tgz	for tgz files
csh	will execute the top shell command .tcshrc - but this will be a new shell area
ctrl-shift t	for a new terminal in same directory !!!!!
rehash	sometimes needed to get back CMSSW commands
<ctrl+r>	to repeat a command
&!:str	repeat command containing str
Scrolling	Go into 'Edit' on Terminal window -> Profile Preferences -> Scrolling and set the number of scroll back lines
< file	means open a file for reading and associate with STDIN.
<< token	Means use the current input stream as STDIN for the program until token is seen. We will ignore this one until we get to scripting.
> file	means open "file" for writing and truncate it and associate it with STDOUT.
>	any previous "file" will be overwritten
>> file	means open a file for writing and seek to the end and associate it with STDOUT. This is how you append to a file using a redirect.
>> file	will append to "file"
n>&m	means redirect FD n to the same places as FD m. Eg, 2>&1 means send STDERR to the same place that STDOUT is going to.
diff	compare two files, diff file1 file2
	$ some-command >> /tmp/log.log 2>&1
find command	ie combine, do echo $PATH to check all the areas
	first directory: /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472
	find /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472 -iname combine, to search all subsequent directories
	combine is in /afs/cern.ch/user/d/davec/ANALYSIS/CMSSW_6_1_1/bin/slc5_amd64_gcc472/combine
	listed as an executable as -rwxr-xr-x 1 davec zh 166K Mar 17 19:19 combine
	Likewise for cmsRun
	/afs/cern.ch/cms/slc5_amd64_gcc472/cms/cmssw/CMSSW_6_1_1/bin/slc5_amd64_gcc472/cmsRun, -rwxr-xr-x 1 cmsbuild zh 166K Feb 13 13:20 cmsRun
find+grep	find -iname '*.*' -print0 | xargs -0 grep "ret" > ret.txt
	find mydir/ -name '*.C' -print0 | xargs -0 grep "examplestring"
	/afs/cern.ch/user/d/davec -name '*.csv' will search for .csv files in davec and subdirectories
	find /afs/cern.ch/user/d/davec//HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer/ -exec grep -l "iConfig" '{}' \;
	find /afs/cern.ch/user/d/davec/ -noleaf -name 'reco*.cc' -exec ls -l {} \;
	find /afs/cern.ch/user/d/davec/HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer -noleaf -name '*.cc' -exec grep 'iConfig' *.cc {} \;
	find /afs/cern.ch/user/d/davec/HEEP/CMSSW_5_2_3/ -noleaf -exec grep -l 'ESHandle' {} > ESHandle.txt \;
	find /afs/cern.ch/user/d/davec//HEEP/CMSSW_5_2_3/src/Reco/RecoAnalyzer/ -exec grep -l "iConfig\|parameter" '{}' \;
grep	if you want to use shell variables, you need double quotes, ie grep "$HOME" file searches file for the name of your home directory
	grep -r "sevLev" /afs/cern.ch/cms/slc5_amd64_gcc462/cms/cmssw/CMSSW_5_2_3/src/
	for a recursive search including all subdirectories
	ls |grep blah lists all files in the current directory whose names contain the string "blah"
	The canonical wildcard character is the dot "."
	grep b.g file will search going through all SINGLE characters between b and g, ie big, bag but not boogy
	grep '$HOME' file searches for the string $HOME
	grep "cat\|dog" file matches lines containing the word "cat" or the word "dog"
	grep "I am a \(cat\|dog\)" matches lines containing the string "I am a cat" or the string "I am a dog"
	To match a selection of characters, use [].
	[Hh]ello matches lines containing hello or Hello
	[0-3] is the same as [0123], [a-k] is the same as [abcdefghijk], [A-C] is the same as [ABC], [A-Ca-k] is the same as [ABCabcdefghijk]
	an expression consisting of a character followed by an escaped question mark, \?, matches one or zero instances of that character.
	bugg\?y matches all of the following: bugy , buggy but not bugggy
	An expression surrounded by "escaped" parentheses is treated by a single character.
	Fred\(eric\)\? Smith matches Fred Smith or Frederic Smith
Escaped character	character preceded by a backslash.
	The preceding backslash does one of the following: (a) removes an implied special meaning from a character (b) adds special meaning to a "non-special" character
	The $ character matches the end of the line.
	The ^ character matches the beginning of the line.
	ie, grep "^From.*mscharmi" /var/spool/mail/davec will match only from the very start
	The [] may be used to search for non-matches. This is done by putting a carat ^ as the first character inside the square brackets.
	? \ . [ ] ^ $ need to be escaped with \
	grep 'hello.gif' file would find hello-gif , hello1gif , helloagif , etc.
	To stop "." being a wildcard, precede with \ , ie grep 'hello\.gif' file
	$ sign loses its meaning if characters follow it (I think)
	the carat ^ loses its meaning if other characters precede it.
	Square brackets behave a little differently. The rules for square brackets go as follows:
	A closing square bracket loses its special meaning if placed first in a list. for example []12] matches ] , 1, or 2.
	A dash - loses it's usual meaning inside lists if it is placed last.
	A carat ^ loses it's special meaning if it is not placed first
	Most special characters lose their meaning inside square brackets
jobs/queues	commands
	The simplest command to submit a batch job to the default (8 minutes) queue is:
	bsub myjob.sh. To see the possible parameters, type: man bsub.
bsub –q 1nh ~user1/jobs/job1.csh	Submit a job in your home directory to the 1 hour queue
	(Remember: chmod 755 filename to make the file executable)
bsub "/bin/hostname"	Submit a command to the default (8 minutes) queue
bsub –M 20000 job2	Submit job2 with a 20 MB memory limit
	(It is recommended to specify a memory limit)
bsub -q 8nh -W 120 job4	Submit job4 to the 8 hour queue with a runtime limit of 120 minutes
	(It is recommended to specify a runtime limit)
bsub –R "type=SLC5_64" job3	Run job3 on a SLC5 host
Also:	bjobs, bkill, bqueues, bhosts, bmgroup, bmod, bchkpnt
	brestart, bgadd, bgdel, bjgroup, sh, getrlimit
	sbrk, libckpt.a, lsb.users, lsbqueues, lsb.params
	lsb.hosts, lsb.serviceclasses, mbatchd
top	see processes which consume most cpu, leave with Q only
kill n	the process with pid n will be signalled
kill -n	all processes in group n are signalled
ctrl alt F1	gives a terminal to login again
ctrl alt F7	to go back to X screen
ssh -X lxplus	To allow tunnelin for emacs etc
ssh -Y lxplus	ditto
source .bashrc	Will execute the .bashrc file
rm -rf directory name or filename	Will force removal. Dangerous!
fs lq	quota check
fs listquota	quota check
fs examine	detailed quota check
chmod u+x fn	to give user executable permission
chmod 755 fn	to give all read and exec permission, I also get write permission
chmod 700 fn	Only I get read, write and exec permission
.tcshrc	location for aliases etc. Executed at logon
<ctrl><shift> t	For a new terminal in the same directory
<ctrl>xs	save
./hello	to execute file 'hello', '.' for current directory, '/' something in this directory
ps -u davec	to list my processes that are running
printing
lpr	lpr -P 40-1B-HPCOR -oprettyprint VPT-UVA-Brunel-info.C
	CERN linux printing page
	/usr/sbin/lpadmincern --list --building XX to list available printers
	To install all printers available in building XX run as root:
	/usr/sbin/lpadmincern --add --building XX
	/usr/sbin/system-config-printer to fine tune printer after installing
	Copy all files and directories in dev recursively to subdirectory bak:
	$ cp -R dev bak

Examples

~/scratch0 > fs listquota
Volume Name                    Quota       Used %Used   Partition
s.cms.davec.0                2000000    1827235   91%<<       29%    <<WARNING

~ > fs lq
Volume Name                    Quota       Used %Used   Partition
user.davec                    888000     762767   86%         46%

~/scratch0 > fs examine
File . (1933946694.1.1) contained in volume 1933946694
Volume status for vid = 1933946694 named s.cms.davec.0
Current disk quota is 2000000
Current blocks used are 1827235
The partition has 348838756 blocks available out of 488139640

Linux compiler info

Makefiles

locale

Was getting locale errors (Jan 2020) in lxplus6.cern.ch while trying to do a cmsenv. Logged in via davec@dcockerill-pc01.
Seems lxplus6 was picking up 'locale' settings from the Ubuntu window davec@dcockerill-pc01 that could not be set.

>cmsenv
> perl: warning: Setting locale failed.
> perl: warning: Please check that your locale settings:
>         LANGUAGE = (unset),
>         LC_ALL = (unset),
>         LANG = "C.UTF-8"
>     are supported and installed on your system.
> perl: warning: Falling back to the standard locale ("C").

locale -a   on linux6 gave:
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

On davec@dcockerill-pc01:~$ locale       gave:
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=

Tried
davec@dcockerill-pc01: sudo update-locale LANG=en_US.UTF-8  without success.

Update locale area:
davec@dcockerill-pc01: sudo apt-get install language-pack-en language-pack-en-base >> this failed

davec@dcockerill-pc01:~$ sudo apt-get update, THEN
davec@dcockerill-pc01:sudo apt-get install language-pack-en language-pack-en-base   >> this worked, giving:
Generating locales (this might take a while)...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IL.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_ZA.UTF-8... done
  en_ZM.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.

Check:
locale -a            >> get
C
C.UTF-8
POSIX
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8

Now try to set up LANG:
davec@dcockerill-pc01:~$ sudo update-locale LANG=en_GB.UTF-8   >> did not work
davec@dcockerill-pc01:~$ LANG=en_GB.UTF-8   >>      THIS worked
davec@dcockerill-pc01:~$ locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

Now (Jan 2020), no more locale errors in lxplus6 when doing cmsenv.

Superuser, su

To add users etc to the PC
sudo su
then enter cern usrn and pwd
then addcernusr xxxxxx
May need to check bin for exact add command.

Otherwise, as a superuser:

• su -
• enter pc passwd, then:
• /usr/sbin/addusercern sproston
• check with:
• cat /etc/passwd | grep '/afs/cern.ch/user/'

-- DavidCockerill - 07-Apr-2011

• path-info-CMSSW.pdf: path-info-CMSSW.pdf

• Linux-PATH.pdf: Linux-PATH.pdf

• Emacs-from-putty.txt: Emacs-from-putty.txt

• command-line-format.txt: command-line-format.txt