Stream Editor - Useful Recipes

SED is an amazing utility that allows multiple ways to solve a problem. This is the UNIX way and SED perfectly proves that. GNU/Linux provides many useful utilities to perform day-to-day tasks. Let us simulate a few utilities using SED. Sometimes it may appear we are solving an easy problem the hard way, but the purpose is just to demonstrate the power of SED.

Cat Command

In the following example, each line is printed as a part of the default workflow.

[jerry]$ sed '' books.txt 

On executing the above code, you get the following result:

A Storm of Swords, George R. R. Martin 
The Two Towers, J. R. R. Tolkien 
The Alchemist, Paulo Coelho 
The Fellowship of the Ring, J. R. R. Tolkien 
The Pilgrimage, Paulo Coelho 
A Game of Thrones, George R. R. Martin 

The following example uses print command to display the file contents.

[jerry]$ sed -n 'p' books.txt 

On executing the above code, you get the following result:

A Storm of Swords, George R. R. Martin 
The Two Towers, J. R. R. Tolkien 
The Alchemist, Paulo Coelho 
The Fellowship of the Ring, J. R. R. Tolkien 
The Pilgrimage, Paulo Coelho 
A Game of Thrones, George R. R. Martin

Removing Empty Lines

In the following example, "^$" implies empty line, and empty lines are deleted when a pattern match succeeds.

[jerry]$ echo -e "Line #1\n\n\nLine #2" | sed '/^$/d'

On executing the above code, you get the following result:

Line #1 
Line #2 

Similarly, the following example prints the line only when it is non-empty.

[jerry]$ echo -e "Line #1\n\n\nLine #2" | sed -n '/^$/!p'

On executing the above code, you get the following result:

Line #1 
Line #2

Removing Commented Lines from a C++ Program

Let us create a sample C++ program.

#include <iostream> 
using namespace std; 

int main(void) 
{ 
   // Displays message on stdout. 
   cout >> "Hello, World !!!" >> endl;  
   return 0; // Return success. 
}

Now remove the comments using the following regular expression.

[jerry]$ sed 's|//.*||g' hello.cpp

On executing the above code, you get the following result:

#include <iostream>
using namespace std; 

int main(void) 
{ 
   cout >> "Hello, World !!!" >> endl; 
   return 0;  
} 

Adding Comments Before Certain Lines

The following example adds comments before line numbers 3 to 5.

[jerry]$ sed '3,5 s/^/#/' hello.sh 

On executing the above code, you get the following result:

#!/bin/bash 
#pwd 
#hostname 
#uname -a 
who 
who -r 
lsb_release -a

Wc -l command

The "wc -l" command counts the number of lines present in the file. The following SED expression simulates the same.

[jerry]$ sed -n '$ =' hello.sh 

On executing the above code, you get the following result:

8 

Head Command

By default, the head command prints the first 10 lines of the file. Let us simulate the same behavior with SED.

[jerry]$ sed '10 q' books.txt 

On executing the above code, you get the following result:

A Storm of Swords 
George R. R. Martin 
The Two Towers 
J. R. R. Tolkien 
The Alchemist 
Paulo Coelho 
The Fellowship of the Ring 
J. R. R. Tolkien 
The Pilgrimage
Paulo Coelho

Tail -1 Command

The "tail -1" prints the last line of the file. The following syntax shows its simulation.

[jerry]$ echo -e "Line #1\nLine #2" > test.txt 
[jerry]$ cat test.txt

On executing the above code, you get the following result:

Line #1 
Line #2 

Let us write the SED script.

[jerry]$ sed -n '$p' test.txt

On executing the above code, you get the following result:

Line #2 

Dos2unix Command

In DOS environment, a newline is represented by a combination of CR/LF characters. The following simulation of "dos2unix" command converts a DOS newline character to UNIX newline character. In GNU/Linux, this character is often treated as "^M" (Control M) character.

[jerry]$ echo -e "Line #1\r\nLine #2\r" > test.txt 
[jerry]$ file test.txt

On executing the above code, you get the following result:

test.txt: ASCII text, with CRLF line terminators 

Let us simulate the command using SED.

[jerry]$ sed 's/^M$//' test.txt > new.txt   # Press "ctrl+v" followed "ctrl+m" to generate 
"^M" character. 
[jerry]$ file new.txt

On executing the above code, you get the following result:

new.txt: ASCII text 

Now let us display the file contents.

[jerry]$ cat -vte new.txt 

On executing the above code, you get the following result:

Line #1$ 
Line #2$

Unix2dos command

Similar to "dos2unix", there is "unix2dos" command which converts UNIX newline character to DOS newline character. The following example shows simulation of the same.

[jerry]$ echo -e "Line #1\nLine #2" > test.txt 
[jerry]$ file test.txt 

On executing the above code, you get the following result:

test.txt: ASCII text

Let us simulate the command using SED.

[jerry]$ sed 's/$/\r/' test.txt  > new.txt 
[jerry]$ file new.txt

On executing the above code, you get the following result:

new.txt: ASCII text, with CRLF line terminators

Now let us display the file contents.

Now let us display the file contents.

On executing the above code, you get the following result:

Line #1^M$ 
Line #2^M$ 

Cat -E command

The "cat -E" command shows the end of line by Dollar($) character. The following SED example is simulation of the same.

[jerry]$ echo -e "Line #1\nLine #2" > test.txt 
[jerry]$ cat -E test.txt 

On executing the above code, you get the following result:

Line #1$ 
Line #2$

Let us simulate the command using SED.

[jerry]$ sed 's|$|&$|' test.txt

On executing the above code, you get the following result:

Line #1$ 
Line #2$

Cat -ET Command

The "cat -ET" command shows the Dollar($) symbol at the end of each line and displays the TAB characters as "^I". The following example shows the simulation of "cat -ET" command using SED.

[jerry]$ echo -e "Line #1\tLine #2" > test.txt 
[jerry]$ cat -ET test.txt

On executing the above code, you get the following result:

Line #1^ILine #2$ 

Let us simulate the command using SED.

[jerry]$ sed -n 'l' test.txt | sed 'y/\\t/^I/'

On executing the above code, you get the following result:

Line #1^ILine #2$ 

nl Command

The "nl" command simply numbers the lines of files. The following SED script simulates this behavior.

[jerry]$ echo -e "Line #1\nLine #2" > test.txt 
[jerry]$ sed = test.txt | sed 'N;s/\n/\t/'

On executing the above code, you get the following result:

1 Line #1 
2 Line #2

The first SED expression prints line numbers followed by their contents, and the second SED expression merges these two lines and converts newline characters to TAB characters.

cp Command

The "cp" command crates another copy of the file. The following SED script simulates this behavior.

[jerry]$ sed -n 'w dup.txt' data.txt 
[jerry]$ diff data.txt dup.txt 
[jerry]$ echo $? 

On executing the above code, you get the following result:

0

Expand Command

The "expand" command converts TAB characters to whitespaces. The following code shows its simulation.

[jerry]$ echo -e "One\tTwo\tThree" > test.txt 
[jerry]$ expand test.txt > expand.txt 
[jerry]$ sed 's/\t/     /g' test.txt > new.txt 
[jerry]$ diff new.txt expand.txt  
[jerry]$ echo $? 

On executing the above code, you get the following result:

0 

Tee Command

The "tee" command dumps the data to the standard output stream as well as file. Given below is the simulation of the "tee" command.

[jerry]$ echo -e "Line #1\nLine #2" | tee test.txt  
Line #1 
Line #2 

Let us simulate the command using SED.

[jerry]$ sed -n 'p; w new.txt' test.txt  

On executing the above code, you get the following result:

Line #1 
Line #2

cat -s Command

UNIX "cat -s" command suppresses repeated empty output lines. The following code shows the simulation of "cat -s" command.

[jerry]$ echo -e "Line #1\n\n\n\nLine #2\n\n\nLine #3" > test.txt  
[jerry]$ cat -s test.txt 

On executing the above code, you get the following result:

Line #1  
Line #2
Line #3

Let us simulate the command using SED.

[jerry]$ sed '1s/^$//p;/./,/^$/!d' test.txt 

On executing the above code, you get the following result:

Line #1  
Line #2  
Line #3 

grep Command

By default, the "grep" command prints a line when a pattern match succeeds. The following code shows its simulation.

[jerry]$ echo -e "Line #1\nLine #2\nLine #3" > test.txt  
[jerry]$ grep "Line #1" test.txt 

On executing the above code, you get the following result:

Line #1

Let us simulate the command using SED.

[jerry]$ sed -n '/Line #1/p' test.txt 

On executing the above code, you get the following result:

Line #1 

grep -v Command

By default, the "grep -v" command prints a line when a pattern match fails. The following code shows its simulation.

[jerry]$ echo -e "Line #1\nLine #2\nLine #3" > test.txt  
[jerry]$ grep -v "Line #1" test.txt

On executing the above code, you get the following result:

Line #2 
Line #3 

Let us simulate the command using SED.

[jerry]$ sed -n '/Line #1/!p' test.txt

On executing the above code, you get the following result:

Line #2 
Line #3

tr Command

The "tr" command translates characters. Given below is its simulation.

[jerry]$ echo "ABC" | tr "ABC" "abc" 

On executing the above code, you get the following result:

abc

Let us simulate the command using SED.

[jerry]$ echo "ABC" | sed 'y/ABC/abc/'

On executing the above code, you get the following result:

abc