vi search and replace

There are many tips and trick in vi, so it is not reasonable to try to teach them all, but there is one complicated usage which I have used often enough, that I hope will serve as an example. If you get walked through something harder than what you are prepared for, I hope you will find the easy things easier.

first things first: replace a text string with a text string

Search and replacing in vi when you are replacing a text string with another text string is not really that hard. From command mode you type

:s/search_string/replace_string/

by default this command only operates on the line where your cursor resides. If you wanted to act on multiple lines, just like all the other commands in vi, you preface it with the line numbers on which you want ot act.

:1,10s/search_string/replace_string/

will perform the search and replace on lines 1 through 10. If you wanted to act on the whole document, but don't want to count the lines, 1,$ gets all the lines in the document, from first ( 1 ) to last ( $ ). There is also the shortcut % which you can use when you want every line in a document.
There is another detail you should know about, by default the search and replace only acts on the first match which it finds on each line. If you need to match each instance on a line, you add a "g" for global to the end of the command so,

:1,$s/search_term/replace_term/g

will replace each occurence of "search_term" with "replace_term" on every line of the document. An example. I am writing these notes in html. There are several tags for denoting computer code in html. I have been using <code> and </code> to just change the font inline, but when I viewed the result in my browser this result was not clear enough, so I decided to change all the <code> and </code> to <pre> and </pre> which puts the code on its own line. easy :

 :1,$s/code/pre/g

got them all.

More complicated example:

Searching and replacing can actually be much more powerful than the above example. What I labeled as search_string and replace_string are actually regular expressions. The simplest case of a regular expression is just like the above example, a character matches itself, so "code" in the example matched exactly that: a "c" followed by an "o" followed by a "d" followed by an "e". There are more complex regular expressions which allow for pattern matching. (see:Link regexps). You can even call back the regular expression which was matched in the first "search" part in the second "replace" part.

Try it out (we are now doing the example we did in class):

The problem is that we want to change all the names of files in a directory to back them up, from unix_notes.html to unix_notes_old.html, you can't do this with shell wildcards. To keep this example as clear as possible, I suggest we do this on a test directory full of empty files, which you will also create using vi and the shell. 1) make a test directory :

mkdir test
cd test

2) touch a bunch of files in this directory (touch creates an empty file or updates the time stamp on a file which already exists). so either type

touch 1.html
touch 2.html
...
touch 10.html

which isn't very unix like, as there must be a thousand ways of automating this you can try this line if you want: (copy it exactly)

foo=1 ; while [ "$foo" -le 10 ] ; do  touch files_$foo.html ; foo=$(($foo+1)) ; done

so now you have a directory which looks like this:

> ls
files_10.html  files_2.html  files_4.html  files_6.html  files_8.html
files_1.html   files_3.html  files_5.html  files_7.html  files_9.html

You want to get these command names into a file. Cat the output of less into a file:

ls >testfile

which puts a list of the files and directory names into the file testfile. Now open testfile in vi to edit it. You have one filename per line.

 vi testfile

      1 files_10.html
      2 files_1.html
      3 files_2.html
      4 files_3.html
      5 files_4.html
      6 files_5.html
      7 files_6.html
      8 files_7.html
      9 files_8.html
     10 files_9.html
     11 testfile
~
~
~

remove the listing for testfile. 11Gdd You want to your file to look like this, which will be the list of commands which will produce the result which we set out to solve. Remember? change all files from *.html to *.old.html which shell wildcards cannot do for us.

      1 mv files_10.html files_10.old.html
      2 mv files_1.html files_1.old.html
      3 mv files_2.html files_2.old.html
      4 mv files_3.html files_3.old.html
      5 mv files_4.html files_4.old.html
      6 mv files_5.html files_5.old.html
      7 mv files_6.html files_6.old.html
      8 mv files_7.html files_7.old.html
      9 mv files_8.html files_8.old.html
     10 mv files_9.html files_9.old.html
~

So combining what we remember from the first part of this lesson, we want to search and replace on every line. so type,

:%s/.*/mv & &/

Looks confusing? Let's take it apart. First there is,

:%s/

what do we have here?:

• colon (enters command mode)
• percent (we want to work on every line of this file) </>
• s (we want a search and replace) </>
• / (the delimiter for our regular expression )</>

Now the regular expression:

:%s/.*/

we want to match the whole line. We could be more explicit:

:%s/^.*$/

But this is not necessary because by default a regular expression will match as much as possible. Now the replace part of the expression:

:%s/.*/mv & &/

The ampersand means the whole line which was matched by the first part of the expression. So the replace part breaks down like this:

• mv --the unix shell command we are going to call, right now it is just text though, the two characters "m" and "v"
• space --this is super important right? the command lines we are producing are delimited by spaces
• & --ampersand, the whole line which was matched in the first part will be repeated here
• space --super important (see above)
• & --ampersand, another repetition of the whole line

so what does our vi file look like now?

      1 mv files_10.html files_10.html
      2 mv files_1.html files_1.html
      3 mv files_2.html files_2.html
      4 mv files_3.html files_3.html
      5 mv files_4.html files_4.html
      6 mv files_5.html files_5.html
      7 mv files_6.html files_6.html
      8 mv files_7.html files_7.html
      9 mv files_8.html files_8.html
     10 mv files_9.html files_9.html
~
~
~

almost what we wanted we just have to change the ".html" at the end of each line to ".old.html". This line will do that:

:%s/html$/old.html/

Can you parse this yourself? It's just a simple text replacement right? Why is the dollar sign "$" important? So now our file looks perfect.

      1 mv files_10.html files_10.old.html
      2 mv files_1.html files_1.old.html
      3 mv files_2.html files_2.old.html
      4 mv files_3.html files_3.old.html
      5 mv files_4.html files_4.old.html
      6 mv files_5.html files_5.old.html
      7 mv files_6.html files_6.old.html
      8 mv files_7.html files_7.old.html
      9 mv files_8.html files_8.old.html
     10 mv files_9.html files_9.old.html
~
~
~

Each line is a shell command which does exactly what we set out to do. You can save the file and quit vi :wq and cat this file to a shell cat testfile | sh which I find the most readable way of understanding what is going on. The shell understands lines of input as commands. You have just prepared these lines of commands in a editor rather than keep banging away at the commandline until carapal tunnel set in. The command sh <testfile is equivalent. It tells the shell to take as it's input the file testfile. You can also call the shell from within vi by typing :w !sh write where? to the shell (the ! allows you to run shell commands from within vi :! and the whole shell is at your fingertips).

Not enough?

We could have collapsed the two calls to search and replace into one command, but this is less easy to understand, so I kept it till the end. It is also one of the most powerful parts of regular expression matching, so I couldn't go without mentioning it.
It is possible put parentheses around a part of the regular expression match and call back exactly that matched part in the second part of the expression. But, one must use special regular expression parentheses \( and \) those are back-slashed parentheses or "escaped" parentheses. Each set of parentheses in the first part of a regular expression will be held in a variable which can be called in the second part using the operators \1, \2, \3 ...etc. The counting starts on the opening parentheses.
HELP!!! I'm going to die tyring to remember all these characters !!!!
Iknow, I know, but bear with me it is really not that hard. Here is the example:

:%s/^\(.*\)html/mv & \1old.html/

Looks almost like before right, except that I have changed the first part to a parenthesized expression : /^\(.*\)html/ which breaks down like so:

• ^ --caret at the start of the regular expression denotes the beginning of a line, so I want to match everything starting from the beginning of the line
• \(.*\) --this is the parenthesized expression. It is a .* the string set asside by the parentheses will be everything form the beginning of the line to the four characters "html"

The beginning of the second part of the expression looks just like the firsttime we went through this :

/mv & \1old.html/

"mv" space and then the ampersand, but now we call the variable "\1" which is everthing upto the "html" in each filename, and add "old.html" to that.
Do you understand?

Really asking for punishment ?

A unix guru, wouldn't need to make the file, he knows about sed which can do these types of substitutions on data streams:

 ls | sed -n 's/\(.*\)html/mv & \1old.html/p'

will print the list of commands to the screen. And :

 ls | sed -n 's/\(.*\)html/mv & \1old.html/p' | sh

will pipe those commands to the shell. Pretty powerful huh?

Copyright Marco Scoffier, released under the GFDL