sed 101

Jul 23, 2023

sed, or Stream Editor, is a powerful utility in Unix/Linux. While it might seem complex at first, sed provides a powerful, flexible method for managing and manipulating text.

1. Basic command structure

The sed command is a powerful command-line tool used for text manipulation. The basic structure of a sed command is as follows:

sed 'options... s/search/replace/flags' filename

options: These are optional parameters for the sed command like -n or -e.
s: This is the substitute command, telling sed to replace text.
search: This is the text that sed will look for in the file.
replace: This is the text that sed will replace the search text with.
flags: These are optional parameters that can modify the behavior of the substitution command, such as g for global replacement.
filename: The name of the file sed will read from.

2. Substitution in sed

One of the most common uses of sed is to replace one bit of text with another.

Substitute first occurrence

echo "Hello World" | sed 's/World/Universe/'

This command will replace the first occurrence of the word “World” with “Universe”. The output will be “Hello Universe”.

Substitute all occurrences of text

echo "Hello World World" | sed 's/World/Universe/g'

By adding the g flag (which stands for global), this command will replace all occurrences of the word “World” with “Universe”. The output will be “Hello Universe Universe”.

Substitute only on lines matching a pattern

echo -e "Hello World\nHello Universe" | sed '/Universe/s/Hello/Goodbye/'

This command will replace “Hello” with “Goodbye”, but only on lines that contain “Universe”. The output will be:

Hello World
Goodbye Universe

Ignoring case during substitution

echo "Hello World" | sed 's/world/Universe/I'

This command will replace “World” with “Universe”, regardless of case. The I flag makes the substitution case-insensitive. The output will be “Hello Universe”.

Substitute in place and create a backup

sed -i.bak 's/search/replace/g' filename

This command will replace all occurrences of “search” with “replace” in the file and save the original file as filename.bak.

Suppress automatic printing with `-n` and print only if substitution occurs

sed -n 's/search/replace/p' filename

This command will replace “search” with “replace” and only print lines where the substitution has occurred.

3. Deleting Lines with sed

sed can also delete lines in a text file that match a certain pattern.

Delete the 3rd line of a file

sed '3d' filename

This command will delete the 3rd line of the file.

Delete the last line of a file

sed '$d' filename

This command will delete the last line of the file.

Delete a range of lines

sed '2,5d' filename

This command will delete lines 2 to 5 of the file.

Delete lines matching a pattern

sed '/pattern_to_delete/d' filename

This command will delete all lines containing “pattern_to_delete”.

Delete all lines not matching a pattern

sed '/pattern_to_keep/!d' filename

This command will delete all lines that do not contain “pattern_to_keep”.

Delete lines from a pattern to the end of file

sed '/pattern_to_start/, $d' filename

This command will delete all lines starting from the first occurrence of “pattern_to_start” to the end of the file.

Delete lines from the start of the file to a pattern

sed '1,/pattern_to_stop/d' filename

This command will delete all lines from the start of the file to the first occurrence of “pattern_to_stop”.

4. Inserting and Appending Text

You can use sed to insert or append text before or after a matched line.

Insert text before a line

sed '3i\This is the inserted line.' filename

This command will insert the line “This is the inserted line.” before the third line of the file.

Append text after a line

sed '3a\This is the appended line.' filename

This command will append the line “This is the appended line.” after the third line of the file.

Insert text before every line that matches a pattern

sed '/pattern_to_match/i\This is the inserted line.' filename

This command will insert “This is the inserted line.” before every line that contains “pattern_to_match”.

Append text after every line that matches a pattern

sed '/pattern_to_match/a\This is the appended line.' filename

This command will append “This is the appended line.” after every line that contains “pattern_to_match”.

Also, note the use of the \ character after the i and a commands. This is to indicate that what follows should be treated as the text to insert or append, not as more commands or flags.

5. Multi-Line Operations

sed is not limited to single line operations. Working with multiple lines in sed is made possible by using commands like N, D, P, n, and d.

Print the line after matching a pattern

echo -e "Hello\nWorld" | sed -n '/Hello/{n;p;}'

In this example, sed will only print the line after “Hello”. The -n option suppresses automatic printing, and the n command fetches the next line into the pattern space.

Delete the line after matching a pattern

echo -e "Hello\nWorld" | sed '/Hello/{n;d;}'

In this example, sed will delete the line after “Hello”.

Print the line before matching a pattern

echo -e "Hello\nWorld" | sed -n '/World/{x;p;d;}; x'

In this example, sed will only print the line before “World”. The x command exchanges the pattern space and the hold space, essentially allowing us to remember the previous line.

Replace a string over multiple lines

echo -e "Hello\nWorld" | sed '/Hello/,/World/s/Hello/Hi/'

This command will replace “Hello” with “Hi” in the range from lines containing “Hello” to lines containing “World”.

Append the next line to the pattern space and perform operation

echo -e "Hello\nWorld" | sed '/Hello/{N;s/\n/ /;}'

This command uses the N command to append the next line to the pattern space and then replaces the newline character with a space, effectively joining the two lines together.

Delete consecutive lines with the same text

echo -e "Hello\nHello" | sed '$!N; /^\(.*\)\n\1$/!P; D'

This command compares each line with the next, and if they are identical, it deletes one of them.

Remember that the N, D, and P commands work on the pattern space in sed, which is the area where sed holds the lines it is currently working on. The N command appends the next line to the pattern space, the D command deletes the earliest appended line, and the P command prints out the earliest appended line.

Note: All of these commands will print the result to the standard output. The original file will not be changed.

6. In-place Editing

If you want to edit the file in-place, you can add the -i option.

Replace ‘old’ with ’new’ in a file

sed -i 's/old/new/g' filename

This command replaces every occurrence of ‘old’ with ’new’ in the file. The changes are made directly in the file.

Delete lines matching a pattern in a file

sed -i '/pattern_to_delete/d' filename

This command deletes all lines containing ‘pattern_to_delete’ directly in the file.

Insert a line before the 3rd line

sed -i '3i\This is the inserted line.' filename

This command inserts ‘This is the inserted line.’ before the 3rd line of the file.

Append a line after the 3rd line

sed -i '3a\This is the appended line.' filename

This command appends ‘This is the appended line.’ after the 3rd line of the file.