1. Sed
Prev Appendix C. A Sed and Awk Micro-Primer Next

1. Sed

Sed is a non-interactive [141] stream editor. It receives text input, whether from stdin or from a file, performs certain operations on specified lines of the input, one line at a time, then outputs the result to stdout or to a file. Within a shell script, sed is usually one of several tool components in a pipe.

Sed determines which lines of its input that it will operate on from the address range passed to it. [142] Specify this address range either by line number or by a pattern to match. For example, 3d signals sed to delete line 3 of the input, and /Windows/d tells sed that you want every line of the input containing a match to “Windows” deleted.

Of all the operations in the sed toolkit, we will focus primarily on the three most commonly used ones. These are printing (to stdout), deletion, and substitution.

Table C.1. Basic sed operators

OperatorNameEffect
[address-range]/pprintPrint [specified address range]
[address-range]/ddeleteDelete [specified address range]
s/pattern1/pattern2/substituteSubstitute pattern2 for first instance of pattern1 in a line
[address-range]/s/pattern1/pattern2/substituteSubstitute pattern2 for first instance of pattern1 in a line, over address-range
[address-range]/y/pattern1/pattern2/transformreplace any character in pattern1 with the corresponding character in pattern2, over address-range (equivalent of tr)
[address] i pattern FilenameinsertInsert pattern at address indicated in file Filename. Usually used with -i in-place option.
gglobalOperate on every pattern match within each matched line of input

Note

Unless the g (global) operator is appended to a substitute command, the substitution operates only on the first instance of a pattern match within each line.

From the command-line and in a shell script, a sed operation may require quoting and certain options.

sed -e '/^$/d' $filename
# The -e option causes the next string to be interpreted as an editing instruction.
#  (If passing only a single instruction to sed, the "-e" is optional.)
#  The "strong" quotes ('') protect the RE characters in the instruction
#+ from reinterpretation as special characters by the body of the script.
# (This reserves RE expansion of the instruction for sed.)
#
# Operates on the text contained in file $filename.

In certain cases, a sed editing command will not work with single quotes.

filename=file1.txt
pattern=BEGIN

  sed "/^$pattern/d" "$filename"  # Works as specified.
# sed '/^$pattern/d' "$filename"    has unexpected results.
#        In this instance, with strong quoting (' ... '),
#+      "$pattern" will not expand to "BEGIN".

Note

Sed uses the -e option to specify that the following string is an instruction or set of instructions. If there is only a single instruction contained in the string, then this may be omitted.

sed -n '/xzy/p' $filename
# The -n option tells sed to print only those lines matching the pattern.
# Otherwise all input lines would print.
# The -e option not necessary here since there is only a single editing instruction.

Table C.2. Examples of sed operators

NotationEffect
8dDelete 8th line of input.
/^$/dDelete all blank lines.
1,/^$/dDelete from beginning of input up to, and including first blank line.
/Jones/pPrint only lines containing “Jones” (with -n option).
s/Windows/Linux/Substitute “Linux” for first instance of “Windows” found in each input line.
s/BSOD/stability/gSubstitute “stability” for every instance of “BSOD” found in each input line.
s/ *$//Delete all spaces at the end of every line.
s/00*/0/gCompress all consecutive sequences of zeroes into a single zero.
echo "Working on it." | sed -e '1i How far are you along?'Prints "How far are you along?" as first line, "Working on it" as second.
5i 'Linux is great.' file.txtInserts 'Linux is great.' at line 5 of the file file.txt.
/GUI/dDelete all lines containing “GUI”.
s/GUI//gDelete all instances of “GUI”, leaving the remainder of each line intact.

Substituting a zero-length string for another is equivalent to deleting that string within a line of input. This leaves the remainder of the line intact. Applying s/GUI// to the line 

The most important parts of any application are its GUI and sound effects

results in 

The most important parts of any application are its  and sound effects

A backslash forces the sed replacement command to continue on to the next line. This has the effect of using the newline at the end of the first line as the replacement string. 

s/^  */\
/g

This substitution replaces line-beginning spaces with a newline. The net result is to replace paragraph indents with a blank line between paragraphs.

An address range followed by one or more operations may require open and closed curly brackets, with appropriate newlines. 

/[0-9A-Za-z]/,/^$/{
/^$/d
}

This deletes only the first of each set of consecutive blank lines. That might be useful for single-spacing a text file, but retaining the blank line(s) between paragraphs.

Note

The usual delimiter that sed uses is /. However, sed allows other delimiters, such as %. This is useful when / is part of a replacement string, as in a file pathname. See Example 11.10, “Checking all the binaries in a directory for authorship” and Example 16.32, “Stripping comments from C program files”.

Tip

A quick way to double-space a text file is sed G filename.

For illustrative examples of sed within shell scripts, see:

  1. Example 36.1, “shell wrapper”

  2. Example 36.2, “ A slightly more complex shell wrapper”

  3. Example 16.3, “Badname, eliminate file names in current directory containing bad characters and whitespace.”

  4. Example A.2, “rn: A simple-minded file renaming utility”

  5. Example 16.17, “Emulating grep in a script”

  6. Example 16.27, “Using column to format a directory listing”

  7. Example A.12, “behead: Removing mail and news message headers”

  8. Example A.16, “tree: Displaying a directory tree”

  9. Example A.17, “tree2: Alternate directory tree script”

  10. Example 16.32, “Stripping comments from C program files”

  11. Example 11.10, “Checking all the binaries in a directory for authorship”

  12. Example 16.48, “Base Conversion”

  13. Example A.1, “mailformat: Formatting an e-mail message”

  14. Example 16.14, “Generating 10-digit random numbers”

  15. Example 16.12, “Word Frequency Analysis”

  16. Example A.10, “Game of Life”

  17. Example 19.12, “A self-documenting script”

  18. Example 16.19, “Looking up definitions in Webster's 1913 Dictionary”

  19. Example A.29, “Spammer Hunt”

  20. Example A.31, “A podcasting script”

  21. Example A.24, “Converting to HTML”

  22. Example A.43, “A command-line stopwatch”

  23. Example A.55, “Inserting text in a file using sed”

For a more extensive treatment of sed, refer to the pertinent references in the Bibliography.


[141] Sed executes without user intervention.

[142] If no address range is specified, the default is all lines.

Prev Up Next
Appendix C. A Sed and Awk Micro-Primer Home 2. Awk