 |
|
|
|
4. EXAMPLESONE-CHARACTER QUESTIONS4.1. How do I insert a newline into the RHS of a substitution?Several versions of sed permit '\n' to be typed directly into the RHS, which is then converted to a newline on output: ssed, gsed302a+, gsed103 (with the -x switch), sed15+, sedmod, and UnixDOS sed. The easiest solution is to use one of these versions. For other versions of sed, try one of the following: (a) If typing the sed script from a Bourne shell, use one backslash "\" if the script uses 'single quotes' or two backslashes "\\" if the script requires "double quotes". In the example below, note that the leading '>' on the 2nd line is generated by the shell to prompt the user for more input. The user types in slash, single-quote, and then ENTER to terminate the command:
[sh-prompt]$ echo twolines | sed 's/two/& new\
>/'
two new
lines
[bash-prompt]$
(b) Use a script file with one backslash '\' in the script, immediately followed by a newline. This will embed a newline into the "replace" portion. Example:
sed -f newline.sed files
# newline.sed
s/twolines/two new\
lines/g
Some versions of sed may not need the trailing backslash. If so, remove it. (c) Insert an unused character and pipe the output through tr:
echo twolines | sed 's/two/& new=/' | tr "=" "\n" # produces
two new
lines
(d) Use the "G" command: G appends a newline, plus the contents of the hold space to the end of the pattern space. If the hold space is empty, a newline is appended anyway. The newline is stored in the pattern space as "\n" where it can be addressed by grouping "\(...\)" and moved in the RHS. Thus, to change the "twolines" example used earlier, the following script will work:
sed '/twolines/{G;s/\(two\)\(lines\)\(\n\)/\1\3\2/;}'
(e) Inserting full lines, not breaking lines up: If one is not changing lines but only inserting complete lines before or after a pattern, the procedure is much easier. Use the "i" (insert) or "a" (append) command, making the alterations by an external script. To insert "This line is new" BEFORE each line matching a regex:
/RE/i This line is new # HHsed, sedmod, gsed 3.02a
/RE/{x;s/$/This line is new/;G;} # other seds
The two examples above are intended as "one-line" commands entered from the console. If using a sed script, "i\" immediately followed by a literal newline will work on all versions of sed. Furthermore, the command "s/$/This line is new/" will only work if the hold space is already empty (which it is by default). To append "This line is new" AFTER each line matching a regex:
/RE/a This line is new # HHsed, sedmod, gsed 3.02a
/RE/{G;s/$/This line is new/;} # other seds
To append 2 blank lines after each line matching a regex:
/RE/{G;G;} # assumes the hold space is empty
To replace each line matching a regex with 5 blank lines:
/RE/{s/.*//;G;G;G;G;} # assumes the hold space is empty
(f) Use the "y///" command if possible: On some Unix versions of sed (not GNU sed!), though the s/// command won't accept '\n' in the RHS, the y/// command does. If your Unix sed supports it, a newline after "aaa" can be inserted this way (which is not portable to GNU sed or other seds):
s/aaa/&~/; y/~/\n/; # assuming no other '~' is on the line!
4.2. How do I represent control-codes or nonprintable characters?Several versions of sed support the notation \xHH, where "HH" are two hex digits, 00-FF: ssed, GNU sed v3.02.80 and above, GNU sed v1.03, sed16 and sed15 (HHsed). Try to use one of those versions. Sed is not intended to process binary or object code, and files which contain nulls (0x00) will usually generate errors in most versions of sed. The latest versions of GNU sed and ssed are an exception; they permit nulls in the input files and also in regexes. On Unix platforms, the 'echo' command may allow insertion of octal or hex values, e.g., `echo "\0nnn"` or `echo -n "\0nnn"`. The echo command may also support syntax like '\\b' or '\\t' for backspace or tab characters. Check the man pages to see what syntax your version of echo supports. Some versions support the following:
# replace 0x1A (32 octal) with ASCII letters
sed 's/'`echo "\032"`'/Ctrl-Z/g'
# note the 3 backslashes in the command below
sed "s/.`echo \\\b`//g"
4.3. How do I convert files with toggle characters, like +this+, to look like [i]this[/i]?Input files, especially message-oriented text files, often contain toggle characters for emphasis, like ~this~, this, or =this=. Sed can make the same input pattern produce alternating output each time it is encountered. Typical needs might be to generate HMTL codes or print codes for boldface, italic, or underscore. This script accomodates multiple occurrences of the toggle pattern on the same line, as well as cases where the pattern starts on one line and finishes several lines later, even at the end of the file:
# sed script to convert +this+ to [i]this[/i]
:a
/+/{ x; # If "+" is found, switch hold and pattern space
/^ON/{ # If "ON" is in the (former) hold space, then ..
s///; # .. delete it
x; # .. switch hold space and pattern space back
s|+|[/i]|; # .. turn the next "+" into "[/i]"
ba; # .. jump back to label :a and start over
}
s/^/ON/; # Else, "ON" was not in the hold space; create it
x; # Switch hold space and pattern space
s|+|[i]|; # Turn the first "+" into "[i]"
ba; # Branch to label :a to find another pattern
}
#---end of script---
This script uses the hold space to create a "flag" to indicate whether the toggle is ON or not. We have added remarks to illustrate the script logic, but in most versions of sed remarks are not permitted after 'b'ranch commands or labels. If you are sure that the +toggle+ characters never cross line boundaries (i.e., never begin on one line and end on another), this script can be reduced to one line:
s|+\([^+][^+]*\)+|[i]\1[/i]|g
If your toggle pattern contains regex metacharacters (such as '*' or perhaps '+' or '?'), remember to quote them with backslashes. CHANGING STRINGS4.10. How do I perform a case-insensitive search?Several versions of sed support case-insensitive matching: ssed and GNU sed v3.02+ (with I flag after s/// or /regex/); sedmod with the -i switch; and sed16 (which supports both types of switches). With other versions of sed, case-insensitive searching is awkward, so people may use awk or perl instead, since these programs have options for case-insensitive searches. In gawk/mawk, use "BEGIN {IGNORECASE=1}" and in perl, "/regex/i". For other seds, here are three solutions: Solution 1: convert everything to upper case and search normally
# sed script, solution 1
h; # copy the original line to the hold space
# convert the pattern space to solid caps
y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
# now we can search for the word "CARLOS"
/CARLOS/ {
# add or insert lines. Note: "s/.../.../" will not work
# here because we are searching a modified pattern
# space and are not printing the pattern space.
}
x; # get back the original pattern space
# the original pattern space will be printed
#---end of sed script---
Solution 2: search for both cases Often, proper names will either start with all lower-case ("unix"), with an initial capital letter ("Unix") or occur in solid caps ("UNIX"). There may be no need to search for every possibility.
/UNIX/b match
/[Uu]nix/b match
Solution 3: search for all possible cases
# If you must, search for any possible combination
/[Ca][Aa][Rr][Ll][Oo][Ss]/ { ... }
Bear in mind that as the pattern length increases, this solution becomes an order of magnitude slower than the one of Solution 1, at least with some implementations of sed. 4.11. How do I match only the first occurrence of a pattern?(1) The general solution is to use GNU sed or ssed, with one of these range expressions. The first script ("print only the first match") works with any version of sed:
sed -n '/RE/{p;q;}' file # print only the first match
sed '0,/RE/{//d;}' file # delete only the first match
sed '0,/RE/s//to_that/' file # change only the first match
(2) If you cannot use GNU sed and if you know the pattern will not occur on the first line, this will work:
sed '1,/RE/{//d;}' file # delete only the first match
sed '1,/RE/s//to_that/' file # change only the first match
(3) If you cannot use GNU sed and the pattern might occur on the first line, use one of the following commands (credit for short GNU script goes to Donald Bruce Stewart):
sed '/RE/{x;/Y/!{s/^/Y/;h;d;};x;}' file # delete (one way)
sed -e '/RE/{d;:a' -e '$!N;$ba' -e '}' file # delete (another way)
sed '/RE/{d;:a;N;$ba;}' file # same script, GNU sed
sed -e '/RE/{s//to_that/;:a' -e '$!N;$!ba' -e '}' file # change
Still another solution, using a flag in the hold space. This is portable to all seds and works if the pattern is on the first line:
# sed script to change "foo" to "bar" only on the first occurrence
1{x;s/^/first/;x;}
1,/foo/{x;/first/s///;x;s/foo/bar/;}
#---end of script---
4.12. How do I parse a comma-delimited (CSV) data file?Comma-delimited data files can come in several forms, requiring increasing levels of complexity in parsing and handling. They are often referred to as CSV files (for "comma separated values") and occasionally as SDF files (for "standard data format"). Note that some vendors use "SDF" to refer to variable-length records with comma-separated fields which are "double-quoted" if they contain character values, while other vendors use "SDF" to designate fixed-length records with fixed-length, nonquoted fields! (For help with fixed-length fields, see question 4.23) The term "CSV" became a de-facto standard when Microsoft Excel used it as an optional output file format. Here are 4 different forms you may encounter in comma-delimited data: (a) No quotes, no internal commas
1001,John Smith,PO Box 123,Chicago,IL,60699
1002,Mary Jones,320 Main,Denver,CO,84100,
(b) Like (a), with quotes around each field
"1003","John Smith","PO Box 123","Chicago","IL","60699"
"1004","Mary Jones","320 Main","Denver","CO","84100"
(c) Like (b), with embedded commas
"1005","Tom Hall, Jr.","61 Ash Ct.","Niles","OH","44446"
"1006","Bob Davis","429 Pine, Apt. 5","Boston","MA","02128"
(d) Like (c), with embedded commas and quotes
"1007","Sue "Red" Smith","19 Main","Troy","MI","48055"
"1008","Joe "Hey, guy!" Hall","POB 44","Reno","NV","89504"
In each example above, we have 7 fields and 6 commas which function as field separators. Case (c) is a very typical form of these data files, with double quotes used to enclose each field and to protect internal commas (such as "Tom Hall, Jr.") from interpretation as field separators. However, many times the data may include both embedded quotation marks as well as embedded commas, as seen by case (d), above. Case (d) is the closest to Microsoft CSV format. However, the Microsoft CSV format allows embedded newlines within a double-quoted field. If embedded newlines within fields are a possibility for your data, you should consider using something other than sed to work with the data file. Before handling a comma-delimited data file, make sure that you fully understand its format and check the integrity of the data. Does each line contain the same number of fields? Should certain fields be composed only of numbers or of two-letter state abbreviations in all caps? Sed (or awk or perl) should be used to validate the integrity of the data file before you attempt to alter it or extract particular fields from the file. After ensuring that each line has a valid number of fields, use sed to locate and modify individual fields, using the \(...\) grouping command where needed. In case (a):
sed 's/^[^,]*,[^,]*,[^,]*,[^,]*,/.../'
^ ^ ^
| | |_ 3rd field
| |_______ 2nd field
|_____________ 1st field
# Unix script to delete the second field for case (a)
sed 's/^\([^,]*\),[^,]*,/\1,,/' file
# Unix script to change field 1 to 9999 for case (a)
sed 's/^[^,]*,/9999,/' file
In cases (b) and (c):
sed 's/^"[^"]*","[^"]*","[^"]*","[^"]*",/.../'
1st-- 2nd-- 3rd-- 4th--
# Unix script to delete the second field for case (c)
sed 's/^\("[^"]*"\),"[^"]*",/\1,"",/' file
# Unix script to change field 1 to 9999 for case (c)
sed 's/^"[^"]*",/"9999",/' file
In case (d): One way to parse such files is to replace the 3-character field separator "," with an unused character like the tab or vertical bar. (Technically, the field separator is only the comma while the fields are surrounded by "double quotes", but the net effect is that fields are separated by quote-comma-quote, with quote characters added to the beginning and end of each record.) Search your datafile first to make sure that your character appears nowhere in it!
sed -n '/|/p' file # search for any instance of '|'
# if it's not found, we can use the '|' to separate fields
Then replace the 3-character field separator and parse as before:
# sed script to delete the second field for case (d)
s/","/|/g; # global change of "," to bar
s/^\([^|]*\)|[^|]|/\1||/; # delete 2nd field
s/|/","/g; # global change of bar back to ","
#---end of script---
# sed script to change field 1 to 9999 for case (d)
# Remember to accommodate leading and trailing quote marks
s/","/|/g;
s/^[^|]*|/"9999|/;
s/|/","/g;
#---end of script---
Note that this technique works only if each and every field is surrounded with double quotes, including empty fields. The following solution is for more complex examples of (d), such as: not all fields contain "double-quote" marks, or the presence of embedded "double-quote" marks within fields, or extraneous whitespace around field delimiters. (Thanks to Greg Ubben for this script!)
# sed script to convert case (d) to bar-delimited records
s/^ *\(.*[^ ]\) *$/|\1|/;
s/" *, */"|/g;
: loop
s/| *\([^",|][^,|]*\) *, */|\1|/g;
s/| *, */|\1|/g;
t loop
s/ *|/|/g;
s/| */|/g;
s/^|\(.*\)|$/\1/;
#---end of script---
For example, it turns this (which is badly-formed but legal): first,"",unquoted ,""this" is, quoted " ,, sub "quote" inside, f", lone " empty: into this: first|""|unquoted|""this" is, quoted "||sub "quote" inside|f"|lone " empty: Note that the script preserves the "double-quote" marks, but changes only the commas where they are used as field separators. I have used the vertical bar "|" because it's easier to read, but you may change this to another field separator if you wish. If your CSV datafile is more complex, it would probably not be worth the effort to write it in sed. For such a case, you should use Perl with a dedicated CSV module (there are at least two recent CSV parsers available from CPAN). 4.13. How do I handle fixed-length, columnar data?Sed handles fixed-length fields via \(grouping\) and backreferences (\1, \2, \3 ...). If we have 3 fields of 10, 25, and 9 characters per field, our sed script might look like so:
s/^\(.\{10\}\)\(.\{25\}\)\(.\{9\}\)/\3\2\1/; # Change the fields
^^^^^^^^^^^~~~~~~~~~~~========== # from 1,2,3 to 3,2,1
field #1 field #2 field #3
This is a bit hard to read. By using GNU sed or ssed with the -r switch active, it can look like this:
s/^(.{10})(.{25})(.{9})/\3\2\1/; # Using the -r switch
To delete a field in sed, use grouping and omit the backreference from the field to be deleted. If the data is long or difficult to work with, use ssed with the -R switch and the /x flag after an s/// command, to insert comments and remarks about the fields. For records with many fields, use GNU awk with the FIELDWIDTHS variable set in the top of the script. For example:
awk 'BEGIN{FIELDWIDTHS = "10 25 9"}; {print $3 $2 $1}' file
This is much easier to read than a similar sed script, especially if there are more than 5 or 6 fields to manipulate. 4.14. How do I commify a string of numbers?Use the simplest script necessary to accomplish your task. As variations of the line increase, the sed script must become more complex to handle additional conditions. Whole numbers are simplest, followed by decimal formats, followed by embedded words. Case 1: simple strings of whole numbers separated by spaces or commas, with an optional negative sign. To convert this:
4381, -1222333, and 70000: - 44555666 1234567890 words
56890 -234567, and 89222 -999777 345888777666 chars
to this:
4,381, -1,222,333, and 70,000: - 44,555,666 1,234,567,890 words
56,890 -234,567, and 89,222 -999,777 345,888,777,666 chars
use one of these one-liners:
sed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
Case 2: strings of numbers which may have an embedded decimal point, separated by spaces or commas, with an optional negative sign. To change this:
4381, -6555.1212 and 70000, 7.18281828 44906982.071902
56890 -2345.7778 and 8.0000: -49000000 -1234567.89012
to this:
4,381, -6,555.1212 and 70,000, 7.18281828 44,906,982.071902
56,890 -2,345.7778 and 8.0000: -49,000,000 -1,234,567.89012
use the following command for GNU sed:
sed ':a;s/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g;ta'
and for other versions of sed:
sed -f case2.sed files
# case2.sed
s/^/ /; # add space to start of line
:a
s/\( [-0-9]\{1,\}\)\([0-9]\{3\}\)/\1,\2/g
ta
s/ //; # remove space from start of line
#---end of script---
4.15. How do I prevent regex expansion on substitutions?Sometimes you want to match regular expression metacharacters as literals (e.g., you want to match "[0-9]" or "\n"), to be replaced with something else. The ordinary way to prevent expanding metacharacters is to prefix them with a backslash. Thus, if "\n" matches a newline, "\\n" will match the two-character string of 'backslash' followed by 'n'. But doing this repeatedly can become tedious if there are many regexes. The following script will replace alternating strings of literals, where no character is interpreted as a regex metacharacter:
# filename: sub_quote.sed
# author: Paolo Bonzini
# sed script to add backslash to find/replace metacharacters
N; # add even numbered line to pattern space
s,[]/\\$*[],\\&,g; # quote all of [, ], /, \, $, or *
s,^,s/,; # prepend "s/" to front of pattern space
s,$,/,; # append "/" to end of pattern space
s,\n,/,; # change "\n" to "/", making s/from/to/
#---end of script---
Here's a sample of how sub_quote.sed might be used. This example converts typical sed regexes to perl-style regexes. The input file consists of 10 lines:
[0-9]
\d
[^0-9]
\D
\+
+
\?
?
\|
|
Run the command "sed -f sub_quote.sed input", to transform the input file (above) to 5 lines of output:
s/\[0-9\]/\\d/
s/\[^0-9\]/\\D/
s/\\+/+/
s/\\?/?/
s/\\|/|/
The above file is itself a sed script, which can then be used to modify other files. 4.16. How do I convert a string to all lowercase or capital letters?The easiest method is to use a new version of GNU sed, ssed, sedmod or sed16 and employ the \U, \L, or other switches on the right side of an s/// command. For example, to convert any word which begins with "reg" or "exp" into solid capital letters:
sed -r "s/\<(reg|exp)[a-z]+/\U&/g" # gsed4.+ or ssed
sed "s/\<reg[a-z]+/\U&/g; s/\<exp[a-z]+/\U&/g" # sed16 and sedmod
As you can see, sedmod and sed16 do not support alternation (|), but they do support case conversion. If none of these versions of sed are available to you, some sample scripts for this task are available from the Seder's Grab Bag:
http://sed.sourceforge.net/grabbag/scripts
Note that some case conversion scripts are listed under "Filename manipulation" and others are under "Text formatting." CHANGING BLOCKS (consecutive lines)4.20. How do I change only one section of a file?You can match a range of lines by line number, by regexes (say, all lines between the words "from" and "until"), or by a combination of the two. For multiple substitutions on the same range, put the command(s) between braces {...}. For example:
# replace only between lines 1 and 20
1,20 s/Johnson/White/g
# replace everywhere EXCEPT between lines 1 and 20
1,20 !s/Johnson/White/g
# replace only between words "from" and "until". Note the
# use of \<....\> as word boundary markers in GNU sed.
/from/,/until/ { s/\<red\>/magenta/g; s/\<blue\>/cyan/g; }
# replace only from the words "ENDNOTES:" to the end of file
/ENDNOTES:/,$ { s/Schaff/Herzog/g; s/Kraft/Ebbing/g; }
For technical details on using address ranges, see section 3.3 ("Addressing and Address ranges"). 4.21. How do I delete or change a block of text if the block contains a certain regular expression?The following deletes the block between 'start' and 'end' inclusively, if and only if the block contains the string 'regex'. Written by Russell Davies, with additional comments:
# sed script to delete a block if /regex/ matches inside it
:t
/start/,/end/ { # For each line between these block markers..
/end/!{ # If we are not at the /end/ marker
$!{ # nor the last line of the file,
N; # add the Next line to the pattern space
bt
} # and branch (loop back) to the :t label.
} # This line matches the /end/ marker.
/regex/d; # If /regex/ matches, delete the block.
} # Otherwise, the block will be printed.
#---end of script---
Note: When the script above reaches /regex/, the entire multi-line block is in the pattern space. To replace items inside the block, use "s///". To change the entire block, use the 'c' (change) command:
/regex/c\
1: This will replace the entire block\
2: with these two lines of text.
4.22. How do I locate a paragraph of text if the paragraph contains a certain regular expression?Assume that paragraphs are separated by blank lines. For regexes that are single terms, use one of the following scripts:
sed -e '/./{H;$!d;}' -e 'x;/regex/!d' # most seds
sed '/./{H;$!d;};x;/regex/!d' # GNU sed
To print paragraphs only if they contain 3 specific regular expressions (RE1, RE2, and RE3), in any order in the paragraph:
sed -e '/./{H;$!d;}' -e 'x;/RE1/!d;/RE2/!d;/RE3/!d'
With this solution and the preceding one, if the paragraphs are excessively long (more than 4k in length), you may overflow sed's internal buffers. If using HHsed, you must add a "G;" command immediately after the "x;" in the scripts above to defeat a bug in HHsed (see section 7.9(5), below, for a description). 4.23. How do I match a block of specific consecutive lines?There are three ways to approach this problem:
(1) Try to use a "/range/, /expression/"
(2) Try to use a "/multi-line\nexpression/"
(3) Try to use a block of "literal strings"
We describe each approach in the following sections. 4.23.1. Try to use a "/range/, /expression/"If the block of lines are strings that never change their order and if the top line never occurs outside the block, like this:
Abel
Baker
Charlie
Delta
then these solutions will work for deleting the block:
sed 's/^Abel$/{N;N;N;d;}' files # for blocks with few lines
sed '/^Abel$/, /^Zebra$/d' files # for blocks with many lines
sed '/^Abel$/,+25d' files # HHsed, sedmod, ssed, gsed 3.02.80
To change the block, use the 'c' (change) command instead of 'd'. To print that block only, use the -n switch and 'p' (print) instead of 'd'. To change some things inside the block, try this:
/^Abel$/,/^Delta$/ {
:ack
N;
/\nDelta$/! b ack
# At this point, all the lines in the block are collected
s/ubstitute /somethin/g;
}
4.23.2. Try to use a "multi-line\nexpression"If the top line of the block sometimes appears alone or is sometimes followed by other lines, or if a partial block may occur somewhere in the file, a multi-line expression may be required. In these examples, we give solutions for matching an N-line block. The expression "/^RE1\nRE2\nRE3...$/" represents a properly formed regular expression where \n indicates a newline between lines. Note that the 'N' followed by the 'P;D;' commands forms a "sliding window" technique. A window of N lines is formed. If the multi-line pattern matches, the block is handled. If not, the top line is printed and then deleted from the pattern space, and we try to match at the next line.
# sed script to delete 2 consecutive lines: /^RE1\nRE2$/
$b
/^RE1$/ {
$!N
/^RE1\nRE2$/d
P;D
}
#---end of script---
# sed script to delete 3 consecutive lines. (This script
# fails under GNU sed v2.05 and earlier because of the 't'
# bug when s///n is used; see section 7.5(1) of the FAQ.)
: more
$!N
s/\n/&/2;
t enough
$!b more
: enough
/^RE1\nRE2\nRE3$/d
P;D
#---end of script---
For example, to delete a block of 5 consecutive lines, the previous script must be altered in only two places: (1) Change the 2 in "s/\n/&/2;" to a 4 (the trailing semicolon is needed to work around a bug in HHsed v1.5). (2) Change the regex line to "/^RE1\nRE2\nRE3\nRE4\nRE5$/d", modifying the expression as needed. Suppose we want to delete a block of two blank lines followed by the word "foo" followed by another blank line (4 lines in all). Other blank lines and other instances of "foo" should be left alone. After changing the '2' to a '3' (always one number less than the total number of lines), the regex line would look like this: "/^\n\nfoo\n$/d". (Thanks to Greg Ubben for this script.) As an alternative to work around the 't' bug in older versions of GNU sed, the following script will delete 4 consecutive lines:
# sed script to delete 4 consecutive lines. Use this if you
# require GNU sed 2.05 and below.
/^RE1$/!b
$!N
$!N
:a
$b
N
/^RE1\nRE2\nRE3\nRE4$/d
P
s/^.*\n\(.*\n.*\n.*\)$/\1/
ba
#---end of script---
Its drawback is that it must be modified in 3 places instead of 2 to adapt it for more lines, and as additional lines are added, the 's' command is forced to work harder to match the regexes. On the other hand, it avoids a bug with gsed-2.05 and illustrates another way to solve the problem of deleting consecutive lines. 4.23.3. Try to use a block of "literal strings"If you need to match a static block of text (which may occur any number of times throughout a file), where the contents of the block are known in advance, then this script is easy to use. It requires an intermediate file, which we will call "findrep.txt" (below):
A block of several consecutive lines to
be matched literally should be placed on
top. Regular expressions like .* or [a-z]
will lose their special meaning and be
interpreted literally in this block.
----
Four hyphens separate the two sections. Put
the replacement text in the lower section.
As above, sed symbols like &, \n, or \1 will
lose their special meaning.
This is a 3-step process. A generic script called "blockrep.sed" will read "findrep.txt" (above) and generate a custom script, which is then used on the actual input file. In other words, "findrep.txt" is a simplified description of the editing that you want to do on the block, and "blockrep.sed" turns it into actual sed commands. Use this process from a Unix shell or from a DOS prompt:
sed -nf blockrep.sed findrep.txt >custom.sed
sed -f custom.sed input.file >output.file
erase custom.sed
The generic script "blockrep.sed" follows below. It's fairly long. Examining its output might help you understanding how to use the sliding window technique.
# filename: blockrep.sed
# author: Paolo Bonzini
# Requires:
# (1) blocks to find and replace, e.g., findrep.txt
# (2) an input file to be changed, input.file
#
# blockrep.sed creates a second sed script, custom.sed,
# to find the lines above the row of 4 hyphens, globally
# replacing them with the lower block of text. GNU sed
# is recommended but not required for this script.
#
# Loop on the first part, accumulating the `from' text
# into the hold space.
:a
/^----$/! {
# Escape slashes, backslashes, the final newline and
# regular expression metacharacters.
s,[/\[.*],\\&,g
s/$/\\/
H
#
# Append N cmds needed to maintain the sliding window.
x
1 s,^.,s/,
1! s/^/N\
/
x
n
ba
}
#
# Change the final backslash to a slash to separate the
# two sides of the s command.
x
s,\\$,/,
x
#
# Until EOF, gather the substitution into hold space.
:b
n
s,[/\],\\&,g
$! s/$/\\/
H
$! bb
#
# Start the RHS of the s command without a leading
# newline, add the P/D pair for the sliding window, and
# print the script.
g
s,/\n,/,
s,$,/\
P\
D,p
#---end of script---
4.24. How do I address all the lines between RE1 and RE2, excluding the lines themselves?Normally, to address the lines between two regular expressions, RE1 and RE2, one would do this: '/RE1/,/RE2/{commands;}'. Excluding those lines takes an extra step. To put 2 arrows before each line between RE1 and RE2, except for those lines:
sed '1,/RE1/!{ /RE2/,/RE1/!s/^/>>/; }' input.fil
The preceding script, though short, may be difficult to follow. It also requires that /RE1/ cannot occur on the first line of the input file. The following script, though it's not a one-liner, is easier to read and it permits /RE1/ to appear on the first line:
# sed script to replace all lines between /RE1/ and /RE2/,
# without matching /RE1/ or /RE2/
/RE1/,/RE2/{
/RE1/b
/RE2/b
s/^/>>/
}
#---end of script---
Contents of input.fil: Output of sed script: aaa aaa
bbb bbb
RE1 RE1
aaa >>aaa
bbb >>bbb
ccc >>ccc
RE2 RE2
end end
4.25. How do I join two lines if line #1 ends in a [certain string]?This question appears in the section on one-line sed scripts, but it comes up so many times that it needs a place here also. Suppose a line ends with a particular string (often, a line ends with a backslash). How do you bring up the second line after it, even in cases where several consecutive lines all end in a backslash?
sed -e :a -e '/\\$/N; s/\\\n//; ta' file # all seds
sed ':a; /\\$/N; s/\\\n//; ta' file # GNU sed, ssed, HHsed
Note that this replaces the backslash-newline with nothing. You may want to replace the backslash-newline with a single space instead. 4.26. How do I join two lines if line #2 begins in a [certain string]?The inverse situation is another FAQ. Suppose a line begins with a particular string. How do you bring that line up to follow the previous line? In this example, we want to match the string "<<=" at the beginning of one line, bring that line up to the end of the line before it, and replace the string with a single space:
sed -e :a -e '$!N;s/\n<<=/ /;ta' -e 'P;D' file # all seds
sed ':a; $!N;s/\n<<=/ /;ta;P;D' file # GNU, ssed, sed15+
4.27. How do I change all paragraphs to long lines?A frequent request is how to convert DOS-style textfiles, in which each line ends with "paragraph marker", to Microsoft-style textfiles, in which the "paragraph" marker only appears at the end of real paragraphs. Sometimes this question is framed as, "How do I remove the hard returns at the end of each line in a paragraph?" The problem occurs because newer word processors don't work the same way older text editors did. Older text editors used a newline (CR/LF in DOS; LF alone in Unix) to end each line on screen or on disk, and used two newlines to separate paragraphs. Certain word processors wanted to make paragraph reformatting and reflowing work easily, so they use one newline to end a paragraph and never allow newlines within a paragraph. This means that textfiles created with standard editors (Emacs, vi, Vedit, Boxer, etc.) appear to have "hard returns" at inappropriate places. The following sed script finds blocks of consecutive nonblank lines (i.e., paragraphs of text), and converts each block into one long line with one "hard return" at the end.
# sed script to change all paragraphs to long lines
/./{H; $!d;} # Put each paragraph into hold space
x; # Swap hold space and pattern space
s/^\(\n\)\(..*\)$/\2\1/; # Move leading \n to end of PatSpace
s/\n\(.\)/ \1/g; # Replace all other \n with 1 space
# Uncomment the following line to remove excess blank lines:
# /./!d;
#---end of sed script---
If the input files have formatting or indentation that conveys special meaning (like program source code), this script will remove it. But if the text still needs to be extended, try 'par' (paragraph reformatter) or the 'fmt' utility with the -t or -c switches and the width option (-w) set to a number like 9999. SHELL AND ENVIRONMENT4.30. How do I read environment variables with sed?4.30.1. - on Unix platformsIn Unix, environment variables begin with a dollar sign, such as $TERM, $PATH, $var or $i. In sed, the dollar sign is used to indicate the last line of the input file, the end of a line (in the LHS), or a literal symbol (in the RHS). Sed cannot access variables directly, so one must pay attention to shell quoting requirements to expand the variables properly. To ALLOW the Unix shell to interpret the dollar sign, put the script in double quotes:
sed "s/_terminal-type_/$TERM/g" input.file >output.file
To PREVENT the Unix shell from interpreting the dollar sign as a shell variable, put the script in single quotes:
sed 's/.$//' infile >outfile
To use BOTH Unix $environment_vars and sed /end-of-line$/ pattern matching, there are two solutions. (1) The easiest is to enclose the script in "double quotes" so the shell can see the $variables, and to prefix the sed metacharacter ($) with a backslash. Thus, in
sed "s/$user\$/root/" file
the shell interpolates $user and sed interprets \$ as the symbol for end-of-line. (2) Another method--somewhat less readable--is to concatenate the script with 'single quotes' where the $ should not be interpolated and "double quotes" where variable interpolation should occur. To demonstrate using the preceding script:
sed "s/$user"'$/root/' file
Solution #1 seems easier to remember. In either case, we search for the user's name (stored in a variable called $user) when it occurs at the end of the line ($), and substitute the word "root" in all matches. For longer shell scripts, it is sometimes useful to begin with single quote marks ('), close them upon encountering the variable, enclose the variable name in double quotes ("), and resume with single quotes, closing them at the end of the sed script. Example:
#! /bin/sh
# sed script to illustrate 'quote'"matching"'usage'
FROM='abcdefgh'
TO='ABCDEFGH'
sed -e '
y/'"$FROM"'/'"$TO"'/; # note the quote pairing
# some more commands go here . . .
# last line is a single quote mark
'
Thus, each variable named $FROM is replaced by $TO, and the single quotes are used to glue the multiple lines together in the script. (See also section 4.10, "How do I handle shell quoting in sed?") 4.30.2. - on MS-DOS and 4DOS platformsUnder 4DOS and MS-DOS version 7.0 (Win95) or 7.10 (Win95 OSR2), environment variables can be accessed from the command prompt. Under MS-DOS v6.22 and below, environment variables can only be accessed from within batch files. Environment variables should be enclosed between percent signs and are case-insensitive; i.e., %USER% or %user% will display the USER variable. To generate a true percent sign, just enter it twice. DOS versions of sed require that sed scripts be enclosed by double quote marks "..." (not single quotes!) if the script contains embedded tabs, spaces, redirection arrows or the vertical bar. In fact, if the input for sed comes from piping, a sed script should not contain a vertical bar, even if it is protected by double quotes (this seems to be bug in the normal MS-DOS syntax). Thus,
echo blurk | sed "s/^/ |foo /" # will cause an error
sed "s/^/ |foo /" blurk.txt # will work as expected
Using DOS environment variables which contain DOS path statements (such as a TMP variable set to "C:\TEMP") within sed scripts is discouraged because sed will interpret the backslash '\' as a metacharacter to "quote" the next character, not as a normal symbol. Thus,
sed "s/^/%TMP% /" somefile.txt
will not prefix each line with (say) "C:\TEMP ", but will prefix each line with "C:TEMP "; sed will discard the backslash, which is probably not what you want. Other variables such as %PATH% and %COMSPEC% will also lose the backslash within sed scripts. Environment variables which do not use backslashes are usually workable. Thus, all the following should work without difficulty, if they are invoked from within DOS batch files:
sed "s/=username=/%USER%/g" somefile.txt
echo %FILENAME% | sed "s/\.TXT/.BAK/"
grep -Ei "%string%" somefile.txt | sed "s/^/ /"
while from either the DOS prompt or from within a batch file,
sed "s/%%/ percent/g" input.fil >output.fil
will replace each percent symbol in a file with " percent" (adding the leading space for readability). 4.31. How do I export or pass variables back into the environment?4.31.1. - on Unix platformsSuppose that line #1, word #2 of the file 'terminals' contains a value to be put in your TERM environment variable. Sed cannot export variables directly to the shell, but it can pass strings to shell commands. To set a variable in the Bourne shell:
TERM=`sed 's/^[^ ][^ ]* \([^ ][^ ]*\).*/\1/;q' terminals`;
export TERM
If the second word were "Wyse50", this would send the shell command "TERM=Wyse50". 4.31.2. - on MS-DOS or 4DOS platformsSed cannot directly manipulate the environment. Under DOS, only batch files (.BAT) can do this, using the SET instruction, since they are run directly by the command shell. Under 4DOS, special 4DOS commands (such as ESET) can also alter the environment. Under DOS or 4DOS, sed can select a word and pass it to the SET command. Suppose you want the 1st word of the 2nd line of MY.DAT put into an environment variable named %PHONE%. You might do this:
@echo off
sed -n "2 s/^\([^ ][^ ]*\) .*/SET PHONE=\1/p;3q" MY.DAT > GO_.BAT
call GO_.BAT
echo The environment variable for PHONE is %PHONE%
:: cleanup
del GO_.BAT
The sed script assumes that the first character on the 2nd line is not a space and uses grouping \(...\) to save the first string of non-space characters as \1 for the RHS. In writing any batch files, make sure that output filenames such as GO_.BAT don't overwrite preexisting files of the same name. 4.32. How do I handle Unix shell quoting in sed?To embed a literal single quote (') in a script, use (a) or (b): (a) If possible, put the script in double quotes:
sed "s/cannot/can't/g" file
(b) If the script must use single quotes, then close-single-quote the script just before the SPECIAL single quote, prefix the single quote with a backslash, and use a 2nd pair of single quotes to finish marking the script. Thus:
sed 's/cannot$/can'\''t/g' file
Though this looks hard to read, it breaks down to 3 parts:
's/cannot$/can' \' 't/g'
--------------- -- ----- To embed a literal double quote (") in a script, use (a) or (b): (a) If possible, put the script in single quotes. You don't need to prefix the double quotes with anything. Thus:
sed 's/14"/fourteen inches/g' file
(b) If the script must use double quotes, then prefix the SPECIAL double quote with a backslash (\). Thus,
sed "s/$length\"/$length inches/g" file
To embed a literal backslash (\) into a script, enter it twice:
sed 's/C:\\DOS/D:\\DOS/g' config.sys
FILES, DIRECTORIES, AND PATHS4.40. How do I read (insert/add) a file at the top of a textfile?Normally, adding a "header" file to the top of a "body" file is done from the command prompt before passing the file on to sed. (MS-DOS below version 6.0 must use COPY and DEL instead of MOVE in the following example.)
copy header.txt+body temp # MS-DOS command 1
echo Y | move temp body # MS-DOS command 2
#
cat header.txt body >temp; mv temp body # Unix commands
However, if inserting the file must occur within sed, there is a way. The sed command "1 r header.txt" will not work; it will print line 1 and then insert "header.txt" between lines 1 and 2. The following script solves this problem; however, there must be at least 2 lines in the target file for the script to work properly.
# sed script to insert "header.txt" above the first line
1{h; r header.txt
D; }
2{x; G; }
#---end of sed script---
4.41. How do I make substitutions in every file in a directory, or in a complete directory tree?4.41.1. - ssed and Perl solutionThe best solution for multiple files in a single directory is to use ssed or gsed v4.0 or higher:
sed -i.BAK 's|foo|bar|g' files # -i does in-place replacement
If you don't have ssed, there is a similar solution in Perl. (Yes, we know this is a FAQ file for sed, not perl, but perl is more common than ssed for many users.)
perl -pi.bak -e 's|foo|bar|g' files # or
perl -pi.bak -e 's|foo|bar|g' `find /pathname -name "filespec"`
For each file in the filelist, sed (or Perl) renames the source file to "filename.bak"; the modified file gets the original filename. Remove '.bak' if you don't need backup copies. (Note the use of "s|||" instead of "s///" here, and in the scripts below. The vertical bars in the 's' command let you replace '/some/path' with '/another/path', accommodating slashes in the LHS and RHS.) To recurse directories in Unix or GNU/Linux:
# We use xargs to prevent passing too many filenames to sed, but
# this command will fail if filenames contain spaces or newlines.
find /my/path -name '*.ht' -print | xargs sed -i.BAK 's|foo|bar|g'
To recurse directories under Windows 2000 (CMD.EXE or COMMAND.COM):
# This syntax isn't supported under Windows 9x COMMAND.COM
for /R c:\my\path %f in (*.htm) do sed -i.BAK "s|foo|bar|g" %f
4.41.2. - Unix solutionFor all files in a single directory, assuming they end with *.txt and you have no files named "[anything].txt.bak" already, use a shell script:
#! /bin/sh
# Source files are saved as "filename.txt.bak" in case of error
# The '&&' after cp is an additional safety feature
for file in *.txt
do
cp $file $file.bak &&
sed 's|foo|bar|g' $file.bak >$file
done
To do an entire directory tree, use the Unix utility find, like so (thanks to Jim Dennis <jadestar@rahul.net> for this script):
#! /bin/sh
# filename: replaceall
# Backup files are NOT saved in this script.
find . -type f -name '*.txt' -print | while read i
do
sed 's|foo|bar|g' $i > $i.tmp && mv $i.tmp $i
done
This previous shell script recurses through the directory tree, finding only files in the directory (not symbolic links, which will be encountered by the shell command "for file in *.txt", above). To preserve file permissions and make backup copies, use the 2-line cp routine of the earlier script instead of "sed ... && mv ...". By replacing the sed command 's|foo|bar|g' with something like
sed "s|$1|$2|g" ${i}.bak > $i
using double quotes instead of single quotes, the user can also employ positional parameters on the shell script command tail, thus reusing the script from time to time. For example,
replaceall East West
would modify all your *.txt files in the current directory. 4.41.3. - DOS solution:MS-DOS users should use two batch files like this:
@echo off
:: MS-DOS filename: REPLACE.BAT
::
:: Create a destination directory to put the new files.
:: Note: The next command will fail under Novel Netware
:: below version 4.10 unless "SHOW DOTS=ON" is active.
if not exist .\NEWFILES\NUL mkdir NEWFILES
for %%f in (*.txt) do CALL REPL_2.BAT %%f
echo Done!!
:: ---End of first batch file---
@echo off
:: MS-DOS filename: REPL_2.BAT
::
sed "s/foo/bar/g" %1 > NEWFILES\%1
:: ---End of the second batch file---
When finished, the current directory contains all the original files, and the newly-created NEWFILES subdirectory contains the modified *.TXT files. Do not attempt a command like
for %%f in (*.txt) do sed "s/foo/bar/g" %%f >NEWFILES\%%f
under any version of MS-DOS because the output filename will be created as a literal '%f' in the NEWFILES directory before the %%f is expanded to become each filename in (*.txt). This occurs because MS-DOS creates output filenames via redirection commands before it expands "for..in..do" variables. To recurse through an entire directory tree in MS-DOS requires a batch file more complex than we have room to describe. Examine the file SWEEP.BAT in Timo Salmi's great archive of batch tricks, located at <ftp://garbo.uwasa.fi/pc/link/tsbat.zip> (this file is regularly updated). Another alternative is to get an external program designed for directory recursion. Here are some recommended programs for directory recursion. The first one, FORALL, runs under either OS/2 or DOS. Unfortunately, none of these supports Win9x long filenames.
http://hobbes.nmsu.edu/pub/os2/util/disk/forall72.zip
ftp://garbo.uwasa.fi/pc/filefind/target15.zip
4.42. How do I replace "/some/UNIX/path" in a substitution?Technically, the normal meaning of the slash can be disabled by prefixing it with a backslash. Thus,
sed 's/\/some\/UNIX\/path/\/a\/new\/path/g' files
But this is hard to read and write. There is a better solution. The s/// substitution command allows '/' to be replaced by any other character (including spaces or alphanumerics). Thus,
sed 's|/some/UNIX/path|/a/new/path|g' files
and if you are using variable names in a Unix shell script,
sed "s|$OLDPATH|$NEWPATH|g" oldfile >newfile
4.43. How do I replace "C:\SOME\DOS\PATH" in a substitution?For MS-DOS users, every backslash must be doubled. Thus, to replace "C:\SOME\DOS\PATH" with "D:\MY\NEW\PATH":
sed "s|C:\\SOME\\DOS\\PATH|D:\\MY\\NEW\\PATH|g" infile >outfile
Remember that DOS pathnames are not case sensitive and can appear in upper or lower case in the input file. If this concerns you, use a version of sed which can ignore case when matching (gsed, ssed, sedmod, sed16).
@echo off
:: sample MS-DOS batch file to alter path statements
:: requires GNU sed with the /i flag for s///
set old=C:\\SOME\\DOS\\PATH
set new=D:\\MY\\NEW\\PATH
gsed "s|%old%|%new%|gi" infile >outfile
:: or
:: sedmod -i "s|%old%|%new%|g" infile >outfile
set old=
set new=
Also, remember that under Windows long filenames may be stored in two formats: e.g., as "C:\Program Files" or as "C:\PROGRA~1". 4.44. How do I emulate file-includes, using sed?Given an input file with file-include statements, similar to C-style includes or "server-side includes" (SSI) of this format:
This is the source file. It's short.
Its name is simply 'source'. See the script below.
<!--#include file="ashe.inc"-->
And this is any amount of text between
<!--#include file="jesse.inc"-->
This is the last line of the file.
How do we direct sed to import/insert whichever files are at the point of the 'file="filename"' token? First, use this file:
#n
# filename: incl.sed
# Comments supported by GNU sed or ssed. Leading '#n' should
# be on line 1, columns 1-2 of the line.
/<!--#include file="/ { # For each "include file" command,
=; # print the line number
s/^[^"]*"/{r /; # change pattern to 'r{ '
s/".*//p; # delete rest to EOL, print
# and a(ppend) a delete command
a\
d;}
}
#---end of sed script---
Second, use the following shell script or DOS batch file (if running a DOS batch file, use "double quotes" instead of 'single quotes', and use "del" instead of "rm" to remove the temp file):
sed -nf incl.sed source | sed 'N;N;s/\n//' >temp.sed
sed -f temp.sed source >target
rm temp.sed
If you have GNU sed or ssed, you can reduce the script even further (thanks to Michael Carmack for the reminder):
sed -nf incl.sed source | sed 'N;N;s/\n//' | sed -f - source >target
In brief, the script replaces each filename with a 'r filename' command to insert the file at that point, while omitting the extraneous material. Two important things to note with this script: (1) There should be only one '#include file' directive per line, and (2) each '#include file' directive must be the only thing on that line, because everything else on the line will be deleted. Though the script uses GNU sed or ssed because of the great support for embedded script comments, it should run on any version of sed. If not, write me and let me know.
|
Site
Links
|
|
Site copyright of domain owner. All rights reserved. |