So far, we ran commands one after another in order to accomplish some tasks. We split our task into sub-tasks in such a way that each sub-task could be solved by one command. The commands usually run fast with small input size. However we need to be active to type one command after other and our presence in front of the machine is needed.
When we start using commands to process large files, one command may run for a longer duration and we have to wait for the command to complete before running another command. This can be frustrating. Fortunately, Linux provides us several options to submit commands together and let Linux takes care of executing the commands in the order it was entered.
|In the previous chapters, we have learnt how to use the redirection using the pipe operator to run related commands by passing the output of one command as input to the next command. This works only for commands that are related and one command depends on the previous command for input and we are interested in the output of the last command alone.
Here are some of the ways we can submit multiple unrelated command sequentially and let Linux takes care of executing it one-by-one.
cmd1; cmd2; cmd3{ cmd1; cmd2; cmd3; }( cmd1; cmd2; cmd3; )&& and || - cmd1 && cmd2, cmd1 || cmd2 or cmd1 && cmd2 || cmd3backticks - `cmd` or VAR=$( cmd1 | cmd2 ) syntaxcmd1 <(cmd2) <(cmd3)for loop from the command line.history commandfc -s "OLD=NEW" commandHere are two examples; one uses the pipe to run related commands in one go. The second example is to run series of unrelated commands one after another manually.
Example 1: Extract name and scores and transform the data
We have a student delimited file that contains test scores. We need to extract the student id, last and first names, grade and score; convert the names to uppercase and sort the data by grade, score and last and first names where grade and names will be sorted ascending and the score in descending order.
$ cat student_file.txt
S0001,Smith,John,10-JAN-2004,04,372,Z
S0002,Smith,Jane,21-FEB-2004,04,385,Z
S0003,Smith,John,14-OCT-2003,04,425,Z
S0004,Smyth,Dan,25-DEC-2002,05,499,Z
S0004,Lowes,Ben,12-MAY-2001,05,499,Z
S0005,Cook,Lisa,28-SEP-2011,05,83,Z
S0006,Daugherty,Joshua,05-APR-2011,05,94,Z
S0007,Macias,Emma,21-SEP-2009,07,79,Z
|$ cut -d, -f1,2,3,5,6 student_file.txt | tr 'a-z' 'A-Z' | sort -t, -k4,4 -k5,5nr -k2,2 -k3,3
S0003,SMITH,JOHN,04,425
S0002,SMITH,JANE,04,385
S0001,SMITH,JOHN,04,372
S0004,LOWES,BEN,05,499
S0004,SMYTH,DAN,05,499
S0005,COOK,LISA,05,83
S0006,DAUGHERTY,JOSHUA,05,94
S0007,MACIAS,EMMA,07,79
Example 2: display dates before and after sleep N
Let us say, we want to verify the time taken by sleep N command is closer to N seconds. We can run the date command before and after sleep and check. However, we need to type the commands quickly and execute to get an accurate result.
I took 3 extra seconds to run the sleep and the second date command.
$ date
Tue Aug 17 14:32:15 IST 2021
$ sleep 5
$ date
Tue Aug 17 14:32:23 IST 2021
The pipe operator works well to run related commands. In this section let us look at other ways to run multiple commands; dependent on each other or just a set of commands to run one after another automatically.
; operatorThe semicolon ; operator acts like an implicit newline as if the Enter key is pressed. We can type a series of commands separated by semicolon and when we press the Enter key, the commands will execute in the order in which they appear in the command-line.
Let us rewrite the sleep and date command sequence from the previous section using the ; operator.
# run commands one after another
$ date
Tue Aug 17 15:48:48 IST 2021
$ sleep 5
$ date
Tue Aug 17 15:48:55 IST 2021
# run commands in sequence using ';'
$ date; sleep 5; date
Tue Aug 17 15:49:52 IST 2021
Tue Aug 17 15:49:58 IST 2021
{} and ; operatorRunning commands sequentially using ; operator serves us well if we just need to run these commands. In case we need to capture the output in a file using redirection and if we use IO redirection at the end then only the last command’s output will be captured. For example, in the below example, the output of first date command is displayed on the screen and the last command’s output is redirected.
$ date; sleep 5; date > date.txt
Tue Aug 17 15:52:41 IST 2021
$ cat date.txt
Tue Aug 17 15:52:46 IST 2021
We can use group the commands by enclosing the command sequence inside the {} operator. This will ensure that all the commands enclosed within the braces are treated as a single entity and redirecting using the {} > FILE syntax will redirect the output from all the commands into the FILE.
$ { date; sleep 5; date; } > date.txt
$ cat date.txt
Tue Aug 17 15:55:40 IST 2021
Tue Aug 17 15:55:45 IST 2021
Let us add header to the student_file.txt we have used earlier.
$ cat student_file.txt
S0001,Smith,John,10-JAN-2004,04,372,Z
S0002,Smith,Jane,21-FEB-2004,04,385,Z
$ { echo "StudentId,LastName,FirstName,DateOfBirth,Grade,Score,End"; cat student_file.txt; } > student_file_with_header.txt
$ cat student_file_with_header.txt
StudentId,LastName,FirstName,DateOfBirth,Grade,Score,End
S0001,Smith,John,10-JAN-2004,04,372,Z
S0002,Smith,Jane,21-FEB-2004,04,385,Z
{} is an operator in bash, we need to leave a blank between the brace and the command{}. More on variables later…{}As mentioned above, the shell variables updated within the {} grouping will be visible even after all the commands within {} complete execution. If we want to have local scope of variable assignment, we can use the parenthesis () grouping instead.
$ x=10; echo "OUT: x is $x"; { echo "IN : x is $x";x=20; echo "IN : x now is $x"; }; echo "OUT: x is $x"
OUT: x is 10
IN : x is 10
IN : x now is 20
OUT: x is 20
() and ; operatorGrouping the commands using the parenthesis is similar to the braces in terms of redirection. Both grouping options treats the entire command list enclosed in between as single entity thus complete output is redirected into same output file.
There are couple of differences between the way {} and () works. We saw that {} runs all the commands in the same shell. The () grouping runs the command list in a new shell environment called sub-shell. Since all the commands run inside a new shell, any variable created or assigned inside the () will not be available once the commands complete execution.
The () does not need space-delimiter and the last command does not have to be terminated by the ;. Here is the same example from previous section. There is no space between () and the commands and there is no ; after the last command. However, adding spaces and ; at the end of the last command adds readability
$ (date;sleep 5; date) > date.txt
$ cat date.txt
Tue Aug 17 16:09:23 IST 2021
Tue Aug 17 16:09:28 IST 2021
# no issues adding spaces between '()' and commands
# or semicolon at the end
$ ( date; sleep 5; date; ) > date.txt
$ cat date.txt
Tue Aug 17 16:10:57 IST 2021
Tue Aug 17 16:11:02 IST 2021
()Commands that runs within () use a separate shell (sub-shell) to run the commands. If any variable assignment or new variable declaration happens within the (), these changes will not be available once the sub-shell completes executing the commands. If we need to persist the changes, we need to use the braces {} grouping instead of parenthesis () grouping.
more on variables later when we discuss shell scripting
In the below example, we have shell variable x set to 10 and displayed outside the (). The variable x is displayed again within () before and after updating the variable. Once the sub-shell completes execution, the value of x is lost and original value will be displayed.
$ x=10; echo "OUT: x is $x"; (echo "IN : x is $x";x=20; echo "IN : x now is $x"); echo "OUT: x is $x"
OUT: x is 10
IN : x is 10
IN : x now is 20
OUT: x is 10
&& and ||Sometimes, we may need to run a command based on the outcome of the previous command. This is known as conditional execution which is different from the sequential execution examples we have seen so far using ;, {} and ().
We can use the && - AND operator to run the second command ONLY if the first command is successful. That is the first command returns a return code 0 (echo $?). The || - OR command, on the contrary executes the second command ONLY if the first command is unsuccessful (RC > 0). We can use the combination of && and || to construct a longer command sequence that can be run conditionally.
Lots of time, we create a directory and then change to it immediately after. Instead of running the two commands separately, we can combine the mkdir and cd with && operator so that if mkdir is successful then run the cd next. For demo, I have added the echo command that will run ONLY if the cd is successful.
$ mkdir temp && cd temp && echo "created and changed to 'temp' directory"
created and changed to 'temp' directory
Sometimes we may want to customize the error message provided on commands. We cause use the || operator to display a custom error message if a command fails. This solution works well on commands that also provide an option to suppress the error message using option. The cmp command to compare files is one such example. It displays nothing when both files match, the command does not display anything and a message if the files are different. The return code is set to 1 if the files are different. We can suppress the default error message using the -s option.
$ echo "A" > file_1.txt
$ echo "a" > file_2.txt
$ cmp file_1.txt file_2.txt
file_1.txt file_2.txt differ: char 1, line 1
$ echo $?
1
$ cmp -s file_1.txt file_2.txt || echo "Both files differ, removing files..." && rm -v file_[12].txt
Both files differ, removing files...
file_1.txt
file_2.txt
Sometimes, we may neither want the output of a command to be displayed nor redirected to a file. We just need to capture the output in a shell variable and display it later using the echo or printf command.
There are two ways to capture the output of command(s) in a variable; using the backticks (`) which is the old way and using the $( cmd ) syntax which is the preferred way.
The backtick that looks like single-quote can be found under the ESC key. The
~and backtick shares the same key.
In the previous section, we have concatenated the header of a CSV file using echo and cat commands and the { cmd; cmd; } > FILE syntax. Let us add the record count at the end of the file like the output of RDBMS table export as CSV.
We can capture the record count of wc -l in a variable and later use it in a echo command to construct a string and display the count.
$ wc -l student_file.txt
2 student_file.txt
$ wc -l student_file.txt | cut -d' ' -f1
2
$ echo student_file.txt
student_file.txt
# capture file name in variable called 'file'
$ file=$(echo student_file.txt)
$ count=$(wc -l student_file.txt | cut -d' ' -f1)
# construct the string using variable and literals
$ echo "$file has $count record(s)"
student_file.txt has 2 record(s)
We can use ` instead of the $( ) syntax too. Using backticks was the old way of capturing command output. The $( ) syntax is preferred since many may confuse backtick with single-quotes.
$ file=`echo student_file.txt`
$ count=`wc -l student_file.txt | cut -d' ' -f1`
$ echo "$file has $count record(s)"
student_file.txt has 2 record(s)
Let us put the above together to add a footer to a delimited file
# set variables: header and file name
$ hdr="StudentId,LastName,FirstName,DateOfBirth,Grade,Score,End"
$ file="student_file.txt"
# use var=$(cmd) and { cmd; cmd; }
$ { echo $hdr; cat $file; count=$(wc -l $file | cut -d' ' -f1); \
> echo -e "\n$file has $count record(s)\n"; } > student_file_with_header.txt
# display output
$ cat student_file_with_header.txt
StudentId,LastName,FirstName,DateOfBirth,Grade,Score,End
S0001,Smith,John,10-JAN-2004,04,372,Z
S0002,Smith,Jane,21-FEB-2004,04,385,Z
S0003,Smith,John,14-OCT-2003,04,425,Z
S0004,Lowes,Ben,12-MAY-2001,05,499,Z
S0005,Cook,Lisa,28-SEP-2011,05,83,Z
student_file.txt has 5 record(s)
Sometimes, we might want to pass part of a file or files as argument to another command or pass the output of multiple commands as argument to another command. We have two options to handle situations like this
<( cmds ) syntax and code one or more <() as argument to the command in questionThe first option is kind of self-explanatory. Let us say, we want to compare the last names from two CSV files where the last name is the second field. We can do the following
cut -d',' -f2 FILE1 | sort > LNAME1.txtcut -d',' -f2 FILE2 | sort > LNAME2.txtcomm LNAME1.txt LNAME2.txt > DIFF.TXTrm -v LNAME*The above solution works fine. It just we need to perform 3 or 4 steps to get the required output.
The second option accomplishes the same result in just one step.
$ comm <(cut -d',' -f2 FILE1 | sort > LNAME1.txt) <(cut -d',' -f2 FILE2 | sort > LNAME2.txt) > DIFF.TXT
The <() syntax has two components that we are already familiar of. < - input redirection and () - running multiple commands together. Since () runs as separate process, the output is passed in a temporary file to the command in question. The temp files will be deleted automatically once the process is done.
$ cut -d, -f2 sample_01.txt | sort > temp_01.txt
$ cut -d, -f2 sample_02.txt | sort > temp_02.txt
$ head temp_0*
==> temp_01.txt <==
Daugherty
Gomez
Gordon
Huang
Sherman
==> temp_02.txt <==
Gomez
Gordon
Huang
Moore
Sherman
# 4 unique names in each file and 1 common names
$ comm temp_01.txt temp_02.txt
Daugherty
Gomez
Gordon
Huang
Moore
Sherman
$ rm temp_0[12].txt
$ comm <(cut -d, -f2 sample_01.txt | sort) <(cut -d, -f2 sample_02.txt | sort)
Daugherty
Gomez
Gordon
Huang
Moore
Sherman
for loopThough we will discuss loops in detail as part of shell scripting, let us quickly look at the for loop syntax that can be used in the command line to run one or more commands multiple time.
syntax of for loop
The
forloops thru a set of items with each iteration assigns one element from the set of items The items can be a space-separated sentence, command’s output or brace expansion’s output{1..10}
for VAR in ITEMS;
do
cmd1;
cmd2;
done
for VAR in ITEMS; do cmd1; cmd2; done
Example:
$ for i in {3..1}; do echo Countdown $i...; done ; echo Liftoff
Countdown 3...
Countdown 2...
Countdown 1...
Liftoff
We have a set of files with sample_ prefix. Let us run wc -l command for each file using the for loop and print FILE has COUNT record(s).
$ for FILE in sample_*; do count=$( wc -l $FILE | cut -d' ' -f1 ); echo "$FILE has $count record(s)"; done
sample_01.txt has 6 record(s)
sample_02.txt has 6 record(s)
sample_03.txt has 2 record(s)
sample_04.txt has 4 record(s)
sample_05.txt has 6 record(s)
# verify
$ wc -l sample_0[1-5].txt
6 sample_01.txt
6 sample_02.txt
2 sample_03.txt
4 sample_04.txt
6 sample_05.txt
historyThe history command displays previously executed commands. This is helpful for novice learners as a reference. Linux keeps the last 500 commands in a hidden file ~/.bash_history in the home directory and the history command uses this file to display the commands with serial number in front of it.
$ history | head -5
80 wc -l large_file.txt
81 ls -lh large_file.txt
82 clear
83 ls
84 jobs
There is another use of the history command. We can run one of the old commands using some shortcuts.
These are few of the simple shortcuts we can use to execute or retrieve past commands from the
history. For more information, look for articles / documentations related to the history command
| Shortcut | Description |
|---|---|
!! |
run previous command again |
!N |
run command with serial number N |
!-N |
run the Nth command from the end |
!!:p |
just print the previous command without executing it. |
| we can use up arrow and make some modification and run | |
!N:p |
print command N |
!-N:p |
print command N counting backward |
$ date
Thu 26 Aug 2021 02:13:48 PM UTC
# runs previous: date command again
$ !!
date
Thu 26 Aug 2021 02:13:52 PM UTC
$ !!:p
date
$ history | tree -5
524 history
525 vi ~/.bash_profile
526 cat ~/.bash_profile
527 wc -l ~/.bash_profile
528 history
$ !527:p
wc -l ~/.bash_profile
# press up arrow and change the last printed command
$ wc -l ~/.bash_history
500 /home/mktutes/.bash_history
$ history | tail -5
528 history
529 wc -l ~/.bash_profile
530 wc -l ~/.bash_history
531 history
532 history | tail -5
$ !-4
wc -l ~/.bash_profile
3 /home/mktutes/.bash_profile
The fc - fix command can be used to modify the previous command by performing simple substitutions.
$ touch demo_{01..05}.csv
ls
demo_01.csv demo_03.csv demo_05.csv
demo_02.csv demo_04.csv
# replace demo with sample, csv with txt
# and run the last command
$ fc -s "demo=sample" "csv=txt"
touch sample_{01..05}.txt
$ ls
demo_01.csv demo_04.csv sample_02.txt sample_05.txt
demo_02.csv demo_05.csv sample_03.txt demo_03.csv sample_01.txt sample_04.txt