sed to replace a string in a file and save it

#!/bin/bash

for i in $(ls -d p*); do
    echo $i
    cd $i
    echo $PWD
    grep "elseif (( hrut .gt. 29.2)" *
    sed -i 's/(( hrut .gt. 29.2) then/( hrut .gt. 29.2) then/g'  file.f;   
   cd ..

done

sed -i  's/string1/string2/g'  file:
it can replace the string1 with string 2 in the file, and save the modified file with the original name.


____ to replace strings at certain lines:
set a = 50
set b = 60
sed -e "$a,${b}s/string1/string2/g"  file

This is to replace the string1 with string2 at lines between 50 and 60 in a file

install or update a software in a server

I work in a server and uses the gnuplot for making figures. I found the gnuplot does not work well with some new functions.

First I tried to find out the version of gnuplot in the server.
1) which gnuplot
/usr/bin/gnuplot
2) /usr/bin/gnuplot --version
4.0
 this is an old version and I want to install the new version. Therefore I go to sourceforge http://sourceforge.net/projects/gnuplot/files/gnuplot/4.4.0/ to download the version 4.4,
and put the downloaded file in a temporary directory in my home in the server. Then I tried to install it in the server !

3) To build it:
     ./configure --PREFIX=$HOME/software
Because I do not have permission to install software in/usr/local, I use --PREFIX=$HOME/software  to install it in this path.
     make
6) test it
     make check
7) install it
     make install

8) finally I add an alias in the .bashrc and .tcshrc (and the .tcshrc_profile)
alias gnuplot4="/home/username/software/bin/gnuplot"
add in the .tcshrc (and the .tcshrc_profile)
alias   gnuplot4    home/username/software/bin/gnuplot
I also add the GNUPLOT_PS_DIR in the tcshrc and bashrc.
set GNUPLOT_PS_DIR = /home/username/software/

9) when I try to call a gnuplot program,
gnuplot4    histogram.plt

numeric calculation with shell or awk

1) For processing two variables with one number given to each variable

Shell language is usually not used for complex scientific calculation. But sometimes we may will use it to do some simple calculations with text files. What can we do ?

set  a = 1.25E-02
set b = 3.289E-02

There are two ways to calculate numerically with these two variables in the shell language: either use the echo + bc, or the awk.
For example, when we do the addition:
1) echo "$a+$b" | /usr/bin/bc 
2) echo "$a $b" | awk '{print $1+$2}'

For comparisons:
1) echo "$a > 0 &  $b > 0" | /usr/bin/bc
2) echo "$a  $b" | awk '{if ($1 > 0 & $2 > 0) print 1; else print 0}'
3) echo "$a" | awk '{if ($1 > 1 || $1 < -1) print 1; else print 0}'

If we have to do some complicated computation with shell,  the AWK is more accurate than the echo + bc, esp when the data is the the scientific format (eg., 1.236778E-02).


2) For processing the data from different files
---- if want to substract two columns of data which are from two files file1 and file2 respectively:
cat file1   | awk '{column1=$which_column_in_file1; getline <"file2"; print column1 - $which_column_in_file2}'   

exit in shell script

exit 0: finish a task, could logout ! attention.
exit 1: have an error and exit the current code

Get input from keyboard

set req = $<

continuation symbol in shell \

starting position index in shell and emacs

Attention: when open a file in emacs, the starting position for a line is 0 , but in shell, the starting position of a string is 1  !!!

reading and processing lines in a text file

For a text file with arrays is called toto:


76039011873490122462 2592.624526   6407821.687553863   3 0.2461  0.0002892 9667 74    1  10 5320
76039011873490122462 2708.024084   6277975.529578525   3 0.2461  0.0002892 9667 74    2  10 5320
76039011873490122462 2835.623772   6179282.694306660   3 0.2461  0.0002892 9667 74    3  10 5320
76039011873490122462 2916.223642   6142426.064870278   3 0.2461  0.0002892 9667 74    4  10 5320
76039011873490122462 3055.424598   6126591.671642607   3 0.2461  0.0002892 9667 74    5  10 5320
76039011873490122462 3186.624722   6167350.779549360   3 0.2461  0.0002892 9667 74    6  10 5320

1) if we want to read the file:
set file = ` awk '{print $0}'  toto `

2) if we want to make some actions to each row, then

set num_lines =  `cat toto | wc -l`

@ j  = 1

while ( $j  <= $num_lines)

     set eachline =  `awk -v ln=$j   '{if (NR==ln) print $0}'  toto`

     more actions to eachline

     @ j += 1

end

Attention: with foreach, we can find each element in each row, but not to get a complete line !

foreach tt ( `cat toto `)

     set eachelement = $tt

end

3) to get a column of data, ie., column 4,

cat toto   |   awk  '{print $4}'

find int, output a string with a format

set a = 26400.24567

--- find the integer of the variable and then the digital part
echo $a | awk '{print $0-int($0)}'

-----multiple the digital parts with a constant 86400
echo $a | awk '{print $0-int($0)}' | awk '{print $0*86400}'

---- output with a format

echo $a | awk '{print $0-int($0)}' | awk '{print $0*86400}' | awk '{printf ( "%12.5f" ,$0)}'

"f" should not be omitted which identifies the float number, not int

split a string with special character with awk

------To split a string separated by "/", put the results in variable a, and then print the first substring:
set c = `echo "tahiti/la1_22460_mrb01" | awk '{split($0,a,"/");print a[1] }'`

plot in shell with gnuplot

http://t16web.lanl.gov/Kawano/gnuplot/intro/working-e.html

1) create one script to plot one figure: the script plot_mrb.plt

#!/usr/bin/gnuplot

## This is to plot one figure use a file mrb18248101.txt and its 5th column

# this line imporant: will save the figures to the computer
set terminal postscript eps color enhanced
set xlabel "18248101"
set ylabel "MRB"
set title "MRB for station 18248101"
set size 1.5, 1.5 #set xrange [ 0 : 20 ]
#set yrange [ 0 : 2 ]
#set mxtics 5
#set mytics 5
#set xtics 5
#set ytics 0.5

 plot "mrb18248101.txt" using 1:5 notitle w l
set output "mrb18248101.eps"

 2) plot many figures: then use the other script

#!/bin/tcsh
foreach sta ( `cat stations` )
 echo $sta
 # to get the interested data
 grep "MRB $sta" parametres_gins | sort -k 1 > "mrb$sta.txt"
# to plot one data
./plot_mrb.plt
# to plot other data
sed "s/18248101/$sta/g" plot_mrb.plt | gnuplot

end

File-test Operators

File-test Operators
There are a number of operators you can use to test different attributes of a file:

-e file
true if file exists
-o file
true if file exists and is owned by the user
-r file
true if file exists and is readable
-w file
true if file exists and is writable
-x file
true if file exists and is executable
-z file
true if file exists and is zero size
-d dir
true if dir exists and is a directory


eg:
if ( ! -d /home/ditt ) then
    mkdir /home/ditt
endif

Attention: there must be space between ( and ! and -d !!

remove duplicate lines in linux

Always starting with sort, then uniq command. 

Attention: uniq -d to only print duplicate lines, and uniq -u to only print unique lines

Given data.txt
79417701
79417701
79417701
88341001
88341001
88341001
88341001

cat data.txt | sort | uniq -d to print
79417701
88341001


cat data.txt | sort | uniq -u to print
blank

compare float number in csh

the shell deals with the integer numbers, not with float numbers. Eg., expr 1 + 2 can produces 3, but it does not work with float.

In order to deal with float numbers, we can use either bc, or awk.

For example,
set a = 0.1
if (`echo "$a > 0" | /usr/bin/bc `) then
echo OK
endif

manimulate row and columns with awk: mean, standard deviation

--To find the lines contain the keywork, then find the 5th column, then do the sum and average



grep "MRB 7822" parametres_gins | awk '{print $5}' | awk '{sum+=$1} END {print sum/NR}'


----To find the lines containing the keyword, then find the 5th column, then do the sqrt of all the data in this column

grep "MRB 7822" parametres_gins | awk '{print $5}' | awk '{sum+=$1*$1} END {print sqrt(sum/NR)}'


---- To find the number of rows of the selected text
grep "MRB 7822" parametres_gins | awk 'END {print NR}'


---- To find the standard deviation of data which are in the 3rd column of a data array in a file :
awk  ‘{sum+=$3; sumsq+=$3*$3 } END {print sqrt(sumsq/NR – (sum/NR)^2)}’    data_file


---- to substract two columns of data which are from two files:
cat file1   | awk '{column1=$which_column_in_file1; getline <"file2"; print column1 - $which_column_in_file2}'   

give input interative in shell

1) printf "Input1\nInput2\nInput3\n1\n" | ~/script -dir dirr
THis will give 4 inputs, ie., Input1, Input2, Input3, 1, to the script. This means that the script will need these four inputs for running.

jobs in linux

jobs -option command [ args ... ]
The first form lists the active jobs. The options have the following meanings:
-l List process IDs in addition to the normal information.
-p List only the process ID of the job’s process group leader.
-n Display information only about jobs that have changed status since the user was last notified of their status.
-r Restrict output to running jobs.
-s Restrict output to stopped jobs.

If jobspec is given, output is restricted to information about that job. The return status is 0 unless an invalid
option is encountered or an invalid jobspec is supplied.

If the -option option is supplied, jobs replaces any jobspec found in command or args with the corresponding process group
ID, and executes command passing it args, returning its exit status.



To manage the submitted jobs, ie., wait to launch a new job after the current job (./script) is done:

cat > .batch_$$ < ./script
EOF

set job_id = `qsub -j oe .batch_$$`

while (`qstat -f | grep $job_id` != '')
sleep 10
end

insert some space in a specific position of a line of text

For given a line from the results of grep, we use awk to insert some space between the position of 130 and 131:

grep SX Sx_xxxxxxxxx92201M017xxx.lageos_coresta | awk '{printf("%s
%s\n",substr($0,0,130),substr($0,131))}'


---substr($0,0,130): $0 is the input string/text, 0 is the starting position, 130 is the ending position. This is to get the first 130 characters in the input string

---substr($0,131): 131 is the starting position. This is to get the substring starting from 131th column in the input string.

use sed to replace characters in strings: csh

1)set a= 'la1_001 la1_002'
set output = `echo $a| sed 's/la1/ttla1/g' `

to replace all the "la1" in a into "ttla1".

2)set a= 'la1_001 la1_002'
set b = "ttla1"
set temp = "la1"
set output = `echo $a| sed "s/$temp/$b/g" `

to replace all the "la1" in a into "ttla1". But here the double quote is used in the sed to get the value of $temp and $b. if using the single quote, it will look for the strings 'temp' and replace with 'b', which finds nothing and then changes nothing to the output.

3) to change the path
set a= 'la1_001 la1_002'
set b = "tt/la1"
set temp = "la1"
set output = `echo $a| sed "s;$temp;$b;g" `

It can replace all the 'la1' in a into 'tt/la1'. Here the ";" are used to separate the strings in sed, because the separator "/" can confuse with the slash in the path string of "tt/la1". Otherwise it will cause the errors of 'unknown option to 's''. Be attention with the special simbols such as #, / etc for the replacement.

Perhaps set output = `echo $a| sed "s:$temp:$b:g" ` works as well. a post about this question is seen in http://dbaspot.com/shell/359294-replace-sed-command-print.html.


Note: sed "s/searching_pattern/replace_pattern/g/p":
g for all the occurances of the search pattern and changing them to the replace_pattern.
p for the printing.

6) we must have a "entre" made in the end of a script on tcsh !!!!
For example,

foreach f (`ls`)
echo $f
end
(then enter the "enter" key), so that the tcsh knows the end of the script. Otherwise, it would not work.


7) to set up alias in tcsh
alias newname oldname
If the old name contains the space, then use
alias newname 'ssh ddd@sss'
(Note, here use the single quote !!!)

csh string manipulation: length, substring, index ...

1) get string length
set object = "Processing NGC 2345"
set nchar = `echo $object | awk '{print length($0)}'`

2) substring
One method uses the awk function substr(s,c,n). This returns the substring from string s starting from character position c up to a maximum length of n characters. If n is not supplied, the rest of the string from c is returned.

set caption = "Processing NGC 2345."
set object = `echo $caption | awk '{print substr($0,12,8)}'` # = "NGC 2345"
set objec_ = `echo $caption | awk '{print substr($0,16)}'` # = "2345."

Another method uses the UNIX cut command. It too can specify a range or ranges of characters. It can also extract fields separated by nominated characters.
The -d qualifier specifies the delimiter between associated data (otherwise called fields). Note the the space delimiter must be quoted. The -f qualifier selects the fields. You can also select character columns with the -c qualifier. Both -c and -f can comprise a comma-separated list of individual values and/or ranges of values separated by a hyphen. As you might expect, cut can take its input from files too.
set places = ( Jupiter "Eagle Nebula" "Gamma quadrant" )

set cut1 = `echo $places | cut -d ' ' -f1,3` # = "Jupiter Nebula"
set cut2 = `echo $places[3] | cut -d a -f2` # = "mm"
set cut3 = `echo $places | cut -c3,11` # = "pg"

3) find the index of a substring
This requires the awk function index. This returns zero if the string could not be located. Note that comparisons are case sensitive.

set catal = "NGC "
set number = 2345
set object = "Processing $catal$number."
set cind = `echo $object | awk '{print index($0,"ngc")}'` # = 0
set cind = `echo $object | awk '{print index($0,"NGC")}'` # = 12


A good csh cook book is found http://star-www.rl.ac.uk/star/docs/sc4.htx/sc4.html. I extracted a few examples here.