ObiLab - BASH Scripting

Linux Commands

Predefined and enviromental, variables

#$PATH, $USER, $HOME, $PS1, $HOSTNAME, $SHELL, $SSH_CLIENT, $SSH_CONNECTION

#!/bin/bash
echo $PATH  #it reports the location of executables 
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

echo $USER  #the user
shady

echo $HOME  #the home directory of the user
/home/shady

echo $PS1  #it shows the prompt of command
\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$

echo $HOSTNAME  #it shows the name of your computer
obideb

echo $SHELL
/bin/bash

Linux shell is "case-senstive". ls -al is not the same of ls -AL
Commands can be called with arguments (or flag) and parameters. Every command got a "manual", man ls
shows the manual of the command ls with all flag.

In order to start our course, let's download this file

had you already clicked on the red button as every common user?
I would bet on it

wget Command

Right click on the link, copy link location, so come back to you our wonderful shell

wget http://www.spinfo.it/bioinflesson2/files/script.tar.gz

Extract Gzip files

tar zxvf script.tar.gz
# z = Gzip type
# x = extract
# v = verbose (shows on standard output the files, while extraction)
# f = use archive file or device ARCHIVE

Tar, The GNU version of the tar archiving utility, saves every information about files. Is Open-Source and with Gzip is a wonderful compression method.

Now run (execute) the script we've downloaded before.
Before, we need to set permissions for this file in order to run (EXECUTE) it as user.
chmod 744 creaFiles.sh
then run it ./creaFiles.sh

# GZIP -9 is the best compression method for Gzip. Best compression = More time to compress and extract.
GZIP=-9 tar zcf namefile.tar.gz ./obiDir 

Cat

cat is used to

concatenate files and redirect it to standard output.
Cat can concatenate more then one file, but is often used to redirect to standard output a file, maybe... with some "filter".

cat /etc/passwd #will print on monitor (standard output) the content of /etc/passwd file

Cut

cut - remove sections from each line of files

cut -d: -f1 /etc/passwd
# -d to define a delimiter
# -f to select the fields divided by delimiter. 1 means field 10

Let's show to standard output the file created by the script.

cat testo1.txt

stefano,IT,unina
marco,bio,unina
serena,bio,unina
mario,IT,igb

This is a "comma separated value" format. How to show only the 1st field?

cut -d"," -f1 testo1.txt

stefano
marco
serena
mario

And what if we'd like to show more then one field?

cut -d"," -f1,3 testo1.txt
stefano,unina
marco,unina
serena,unina
mario,igb

Is it possibile to use any other command to obtain the same result?

cat testo1.txt | awk -F, '{print $1 "," $3}'
stefano,unina
marco,unina
serena,unina
mario,igb

What is the best way? And what is the strange char "|" used?
Do you really think that the strongest stuff is the vertical line? Trust me, you do not know how much power is inside the letters "awk".
Just the acronym discover its power. AWK stay for:
Aho, Weinberger, but the last name is the most famous into the IT environment... Yes, modern computers exist, just because of a REAL GENIUS! Kernighan any reference to property or persons or "jobs" is purely coincidental...
awk is a scripting language, not just "a command", gawk is a newest implementation with a wonderful documentation. It is used in order to process text file structured as record and obtain reports. Syntax of AWK is similar to C, but it require more time to be explained as well.
gawk manual

An easy sample of use of AWK

cat testo1.txt | grep unina | grep IT | awk -F, '{print $1" esperto in "$2}'

Two different ways to obtain the same result... sometimes AWK looks like awful

cat /etc/passwd | awk 'BEGIN {FS=":"} ($7=="/bin/bash") {print $1}'
root
shady

Who said that without awk we cannot use bash?

cat /etc/passwd | grep /bin/bash | cut -d: -f1 
root
shady

Even if...sometimes it is really useful

awk -F',' '{print NF; exit}' testo1.txt     #It will count the number of Column in a file, with field separator ","

#e.g. with passwd file
awk -F':' '{print NF; exit}' /etc/passwd
7

head and tail

Two really useful commands are, head and tail.
tail - output the last part of files head - output the first part of files.
Let's show the 1st and the last line of the file testo1.txt

head -1 testo1.txt && tail -1 testo1.txt ## && means AND 

stefano,IT,unina
mario,IT,igb

sort and grep

sort - sort lines of text files
alfabetical sort is default

sort testo1.txt

marco,bio,unina
mario,IT,igb
serena,bio,unina
stefano,IT,unina

sort -r testo1.txt #reverse order

stefano,IT,unina
serena,bio,unina
mario,IT,igb
marco,bio,unina

# -n order by dimension, -k5 select the field of the command ls -l sent to the pipe
ls -l | sort -k5 -n

grep, egrep, fgrep, rgrep - print lines matching a pattern if you want to filter a search or list, grep is what you need!
Let's filter our file testo1.txt, looking for the line that contain the word "marco"

grep marco testo1.txt
marco,bio,unina

Now, the same way to obtain all line that contain the word "unina"

grep unina testo1.txt 
stefano,IT,unina
marco,bio,unina
serena,bio,unina

The -v flag, exclude result that match with the pattern

grep -v unina testo1.txt 
mario,IT,igb

The -c flag, count the line for the final result

grep -c -v unina testo1.txt 
1

Pipe and wc -l, will help when you cannot use a specific flag to count

grep -v unina testo1.txt | wc -l
1

What is the difference between...

grep 'a' testo1.txt | wc -l

grep 'a' -o testo1.txt | wc -l   # -o, --only-matching       show only the part of a line matching PATTERN

wc -w, print the word counts

grep unina testo1.txt | wc -w
3

## Another way
grep -c -P '(unina)' testo1.txt  ## -P?? What's that "P" stand for?
3

Pipe and regular expressions to improve filters.
[^m] means, "that starts with letter m"

grep -e "^m" testo1.txt
marco,bio,unina
mario,IT,igb

Pipe and regular expressions to improve filters.
-r stay for recursive -n for number of line -w stands match the whole word. -e to find a specific pattern with regex

grep -rnw "/home/shady" -e "pattern"

We will talk about Regular Expressions later...

Pipes

Pipes redirect OUTPUT of a command as INPUT for another one.
If we want to filter a list, or a file, with more then one option, pipes is the way.

Let's order a file, select the first 3 line and filter it with word "unina"

sort testo1.txt | head -3 | grep unina
marco,bio,unina
serena,bio,unina

Redirection to file

In order to change the redirection of a file from standard output to a file we can use > or >>
redirect standard output to a new file and if it doesn't exists, a new file will be created, but if it exists, the file will be overwritten. >> redirect standard output into an existing file, in order to "add", or better "append", new information into the file.

ls > list.txt
#list will be redirected to a file called list.txt
#no standard output will be shown

Let's "append" the current date to this file! Take care, do not destroy it.

echo $(date) >>list.txt