Advanced Bash-Scripting Guide: An in-depth exploration of the art of shell scripting
Prev		Next

Chapter 3. Special Characters

Special Characters Found In Scripts and Elsewhere

#

Comments. Lines beginning with a # (with the exception of #!) are comments.

1 # This line is a comment.

Comments may also occur at the end of a command.

1 echo "A comment will follow." # Comment here.

Comments may also follow whitespace at the beginning of a line.

1 # A tab precedes this comment.

A command may not follow a comment on the same line. There is no method of terminating the comment, in order for "live code" to begin on the same line. Use a new line for the next command.

Of course, an escaped # in an echo statement does not begin a comment. Likewise, a # appears in certain parameter substitution constructs and in numerical constant expressions.
1 echo "The # here does not begin a comment." 2 echo 'The # here does not begin a comment.' 3 echo The \# here does not begin a comment. 4 echo The # here begins a comment. 5 6 echo ${PATH#*:} # Parameter substitution, not a comment. 7 echo $(( 2#101011 )) # Base conversion, not a comment. 8 9 # Thanks, S.C.
The standard quoting and escape characters (" ' \) escape the #.

Certain pattern matching operations also use the #.

;

Command separator. [Semicolon] Permits putting two or more commands on the same line.

1 echo hello; echo there 2 3 4 if [ -x "$filename" ]; then # Note that "if" and "then" need separation. 5 # Why? 6 echo "File $filename exists."; cp $filename $filename.bak 7 else 8 echo "File $filename not found."; touch $filename 9 fi; echo "File test complete."

Note that the ";" sometimes needs to be escaped.

;;

Terminator in a case option. [Double semicolon]

1 case "$variable" in 2 abc) echo "$variable = abc" ;; 3 xyz) echo "$variable = xyz" ;; 4 esac

.

"dot" command. [period] Equivalent to source (see Example 11-19). This is a bash builtin.

.

"dot", as a component of a filename. When working with filenames, a dot is the prefix of a "hidden" file, a file that an ls will not normally show.
bash$ touch .hidden-file bash$ ls -l total 10 -rw-r--r-- 1 bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo 877 Dec 17 2000 employment.addressbook bash$ ls -al total 14 drwxrwxr-x 2 bozo bozo 1024 Aug 29 20:54 ./ drwx------ 52 bozo bozo 3072 Aug 29 20:51 ../ -rw-r--r-- 1 bozo bozo 4034 Jul 18 22:04 data1.addressbook -rw-r--r-- 1 bozo bozo 4602 May 25 13:58 data1.addressbook.bak -rw-r--r-- 1 bozo bozo 877 Dec 17 2000 employment.addressbook -rw-rw-r-- 1 bozo bozo 0 Aug 29 20:54 .hidden-file

When considering directory names, a single dot represents the current working directory, and two dots denote the parent directory.

bash$ pwd /home/bozo/projects bash$ cd . bash$ pwd /home/bozo/projects bash$ cd .. bash$ pwd /home/bozo/

The dot often appears as the destination (directory) of a file movement command.

bash$ cp /home/bozo/current_work/junk/* .

.

"dot" character match. When matching characters, as part of a regular expression, a "dot" matches a single character.

"

partial quoting. [double quote] "STRING" preserves (from interpretation) most of the special characters within STRING. See also Chapter 5.

'

full quoting. [single quote] 'STRING' preserves all special characters within STRING. This is a stronger form of quoting than using ". See also Chapter 5.

,

comma operator. The comma operator links together a series of arithmetic operations. All are evaluated, but only the last one is returned.
1 let "t2 = ((a = 9, 15 / 3))" # Set "a" and calculate "t2".

\

escape. [backslash] \X "escapes" the character X. This has the effect of "quoting" X, equivalent to 'X'. The \ may be used to quote " and ', so they are expressed literally.

See Chapter 5 for an in-depth explanation of escaped characters.

/

Filename path separator. [forward slash] Separates the components of a filename (as in /home/bozo/projects/Makefile).

This is also the division arithmetic operator.

`

command substitution. [backticks] `command` makes available the output of command for setting a variable. This is also known as backticks or backquotes.

:

null command. [colon] This is the shell equivalent of a "NOP" (no op, a do-nothing operation). It may be considered a synonym for the shell builtin true. The ":" command is a itself a Bash builtin, and its exit status is "true" (0).

1 : 2 echo $? # 0

Endless loop:

1 while : 2 do 3 operation-1 4 operation-2 5 ... 6 operation-n 7 done 8 9 # Same as: 10 # while true 11 # do 12 # ... 13 # done

Placeholder in if/then test:

1 if condition 2 then : # Do nothing and branch ahead 3 else 4 take-some-action 5 fi

Provide a placeholder where a binary operation is expected, see Example 8-2 and default parameters.

1 : ${username=`whoami`} 2 # ${username=`whoami`} without the leading : gives an error 3 # unless "username" is a command or builtin...

Provide a placeholder where a command is expected in a here document. See Example 17-10.

Evaluate string of variables using parameter substitution (as in Example 9-13).
1 : ${HOSTNAME?} ${USER?} ${MAIL?} 2 #Prints error message if one or more of essential environmental variables not set.

Variable expansion / substring replacement.

In combination with the > redirection operator, truncates a file to zero length, without changing its permissions. If the file did not previously exist, creates it.
1 : > data.xxx # File "data.xxx" now empty. 2 3 # Same effect as cat /dev/null >data.xxx 4 # However, this does not fork a new process, since ":" is a builtin.
See also Example 12-11.

In combination with the >> redirection operator, updates a file access/modification time (: >> new_file). If the file did not previously exist, creates it. This is equivalent to touch.

This applies to regular files, not pipes, symlinks, and certain special files.

May be used to begin a comment line, although this is not recommended. Using # for a comment turns off error checking for the remainder of that line, so almost anything may be appear in a comment. However, this is not the case with :.
1 : This is a comment that generates an error, ( if [ $x -eq 3] ).

The ":" also serves as a field separator, in /etc/passwd, and in the $PATH variable.
bash$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/sbin:/usr/sbin:/usr/games

!

reverse (or negate) the sense of a test or exit status. The ! operator inverts the exit status of the command to which it is applied (see Example 6-2). It also inverts the meaning of a test operator. This can, for example, change the sense of "equal" ( = ) to "not-equal" ( != ). The ! operator is a Bash keyword.

In a different context, the ! also appears in indirect variable references.

In yet another context, from the command line, the ! invokes the Bash history mechanism (see Appendix G). Note that within a script, the history mechanism is disabled.

*

wild card. [asterisk] The * character serves as a "wild card" for filename expansion in globbing. By itself, it matches every filename in a given directory.

bash$ echo * abs-book.sgml add-drive.sh agram.sh alias.sh

The * also represents any number (or zero) characters in a regular expression.

*

arithmetic operator. In the context of arithmetic operations, the * denotes multiplication.

A double asterisk, **, is the exponentiation operator.

?

test operator. Within certain expressions, the ? indicates a test for a condition.

In a double parentheses construct, the ? serves as a C-style trinary operator. See Example 9-29.

In a parameter substitution expression, the ? tests whether a variable has been set.

?

wild card. The ? character serves as a single-character "wild card" for filename expansion in globbing, as well as representing one character in an extended regular expression.

$

Variable substitution.
1 var1=5 2 var2=23skidoo 3 4 echo $var1 # 5 5 echo $var2 # 23skidoo

A $ prefixing a variable name indicates the value the variable holds.

$

end-of-line. In a regular expression, a "$" addresses the end of a line of text.

${}

Parameter substitution.

$*, $@

positional parameters.

$?

exit status variable. The $? variable holds the exit status of a command, a function, or of the script itself.

$$

process id variable. The $$ variable holds the process id of the script in which it appears.

()

command group.
1 (a=hello; echo $a)

A listing of commands within parentheses starts a subshell.

Variables inside parentheses, within the subshell, are not visible to the rest of the script. The parent process, the script, cannot read variables created in the child process, the subshell.
1 a=123 2 ( a=321; ) 3 4 echo "a = $a" # a = 123 5 # "a" within parentheses acts like a local variable.

array initialization.
1 Array=(element1 element2 element3)

{xxx,yyy,zzz,...}

Brace expansion.
1 grep Linux file*.{txt,htm*} 2 # Finds all instances of the word "Linux" 3 # in the files "fileA.txt", "file2.txt", "fileR.html", "file-87.htm", etc.

A command may act upon a comma-separated list of file specs within braces. [1] Filename expansion (globbing) applies to the file specs between the braces.

No spaces allowed within the braces unless the spaces are quoted or escaped.

echo {file1,file2}\ :{\ A," B",' C'}

file1 : A file1 : B file1 : C file2 : A file2 : B file2 : C

{}

Block of code. [curly brackets] Also referred to as an "inline group", this construct, in effect, creates an anonymous function. However, unlike a function, the variables in a code block remain visible to the remainder of the script.

bash$ { local a; a=123; } bash: local: can only be used in a function

1 a=123 2 { a=321; } 3 echo "a = $a" # a = 321 (value inside code block) 4 5 # Thanks, S.C.

The code block enclosed in braces may have I/O redirected to and from it.

Example 3-1. Code blocks and I/O redirection

   1 #!/bin/bash
   2 # Reading lines in /etc/fstab.
   3 
   4 File=/etc/fstab
   5 
   6 {
   7 read line1
   8 read line2
   9 } < $File
  10 
  11 echo "First line in $File is:"
  12 echo "$line1"
  13 echo
  14 echo "Second line in $File is:"
  15 echo "$line2"
  16 
  17 exit 0

Example 3-2. Saving the results of a code block to a file

   1 #!/bin/bash
   2 # rpm-check.sh
   3 
   4 # Queries an rpm file for description, listing, and whether it can be installed.
   5 # Saves output to a file.
   6 # 
   7 # This script illustrates using a code block.
   8 
   9 SUCCESS=0
  10 E_NOARGS=65
  11 
  12 if [ -z "$1" ]
  13 then
  14   echo "Usage: `basename $0` rpm-file"
  15   exit $E_NOARGS
  16 fi  
  17 
  18 { 
  19   echo
  20   echo "Archive Description:"
  21   rpm -qpi $1       # Query description.
  22   echo
  23   echo "Archive Listing:"
  24   rpm -qpl $1       # Query listing.
  25   echo
  26   rpm -i --test $1  # Query whether rpm file can be installed.
  27   if [ "$?" -eq $SUCCESS ]
  28   then
  29     echo "$1 can be installed."
  30   else
  31     echo "$1 cannot be installed."
  32   fi  
  33   echo
  34 } > "$1.test"       # Redirects output of everything in block to file.
  35 
  36 echo "Results of rpm test in file $1.test"
  37 
  38 # See rpm man page for explanation of options.
  39 
  40 exit 0

Unlike a command group within (parentheses), as above, a code block enclosed by {braces} will not normally launch a subshell. [2]

{} \;

pathname. Mostly used in find constructs. This is not a shell builtin.

The ";" ends the -exec option of a find command sequence. It needs to be escaped to protect it from interpretation by the shell.

[ ]

test.

Test expression between [ ]. Note that [ is part of the shell builtin test (and a synonym for it), not a link to the external command /usr/bin/test.

[[ ]]

test.

Test expression between [[ ]] (shell keyword).

See the discussion on the [[ ... ]] construct.

[ ]

array element.

In the context of an array, brackets set off the numbering of each element of that array.
1 Array[1]=slot_1 2 echo ${Array[1]}

[ ]

range of characters.

As part of a regular expression, brackets delineate a range of characters to match.

(( ))

integer expansion.

Expand and evaluate integer expression between (( )).

See the discussion on the (( ... )) construct.

> &> >& >> <

redirection.

scriptname >filename redirects the output of scriptname to file filename. Overwrite filename if it already exists.

command &>filename redirects both the stdout and the stderr of command to filename.

command >&2 redirects stdout of command to stderr.

scriptname >>filename appends the output of scriptname to file filename. If filename does not already exist, it will be created.

process substitution.

(command)>

<(command)

In a different context, the "<" and ">" characters act as string comparison operators.

In yet another context, the "<" and ">" characters act as integer comparison operators. See also Example 12-6.

<<

redirection used in a here document.

<, >

ASCII comparison.
1 veg1=carrots 2 veg2=tomatoes 3 4 if [[ "$veg1" < "$veg2" ]] 5 then 6 echo "Although $veg1 precede $veg2 in the dictionary," 7 echo "this implies nothing about my culinary preferences." 8 else 9 echo "What kind of dictionary are you using, anyhow?" 10 fi

\<, \>

word boundary in a regular expression.

bash$ grep '\<the\>' textfile

|

pipe. Passes the output of previous command to the input of the next one, or to the shell. This is a method of chaining commands together.

1 echo ls -l | sh 2 # Passes the output of "echo ls -l" to the shell, 3 #+ with the same result as a simple "ls -l". 4 5 6 cat *.lst | sort | uniq 7 # Merges and sorts all ".lst" files, then deletes duplicate lines.

The output of a command or commands may be piped to a script.
1 #!/bin/bash 2 # uppercase.sh : Changes input to uppercase. 3 4 tr 'a-z' 'A-Z' 5 # Letter ranges must be quoted 6 #+ to prevent filename generation from single-letter filenames. 7 8 exit 0
Now, let us pipe the output of ls -l to this script.
bash$ ls -l | ./uppercase.sh -RW-RW-R-- 1 BOZO BOZO 109 APR 7 19:49 1.TXT -RW-RW-R-- 1 BOZO BOZO 109 APR 14 16:48 2.TXT -RW-R--R-- 1 BOZO BOZO 725 APR 20 20:56 DATA-FILE

The stdout of each process in a pipe must be read as the stdin of the next. If this is not the case, the data stream will block, and the pipe will not behave as expected.
1 cat file1 file2 | ls -l | sort 2 # The output from "cat file1 file2" disappears.

A pipe runs as a child process, and therefore cannot alter script variables.
1 variable="initial_value" 2 echo "new_value" | read variable 3 echo "variable = $variable" # variable = initial_value

If one of the commands in the pipe aborts, this prematurely terminates execution of the pipe. Called a broken pipe, this condition sends a SIGPIPE signal.

>|

force redirection (even if the noclobber option is set). This will forcibly overwrite an existing file.

||

OR logical operator. In a test construct, the || operator causes a return of 0 (success) if either of the linked test conditions is true.

&

Run job in background. A command followed by an & will run in the background.

bash$ sleep 10 & [1] 850 [1]+ Done sleep 10

Within a script, commands and even loops may run in the background.

Example 3-3. Running a loop in the background

   1 #!/bin/bash
   2 # background-loop.sh
   3 
   4 for i in 1 2 3 4 5 6 7 8 9 10            # First loop.
   5 do
   6   echo -n "$i "
   7 done & # Run this loop in background.
   8        # Will sometimes execute after second loop.
   9 
  10 echo   # This 'echo' sometimes will not display.
  11 
  12 for i in 11 12 13 14 15 16 17 18 19 20   # Second loop.
  13 do
  14   echo -n "$i "
  15 done  
  16 
  17 echo   # This 'echo' sometimes will not display.
  18 
  19 # ======================================================
  20 
  21 # The expected output from the script:
  22 # 1 2 3 4 5 6 7 8 9 10 
  23 # 11 12 13 14 15 16 17 18 19 20 
  24 
  25 # Sometimes, though, you get:
  26 # 11 12 13 14 15 16 17 18 19 20 
  27 # 1 2 3 4 5 6 7 8 9 10 bozo $
  28 # (The second 'echo' doesn't execute. Why?)
  29 
  30 # Occasionally also:
  31 # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
  32 # (The first 'echo' doesn't execute. Why?)
  33 
  34 # Very rarely something like:
  35 # 11 12 13 1 2 3 4 5 6 7 8 9 10 14 15 16 17 18 19 20 
  36 # The foreground loop preempts the background one.
  37 
  38 exit 0

A command run in the background within a script may cause the script to hang, waiting for a keystroke. Fortunately, there is a remedy for this.

&&

AND logical operator. In a test construct, the && operator causes a return of 0 (success) only if both the linked test conditions are true.

-

option, prefix. Option flag for a command or filter. Prefix for an operator.

COMMAND -[Option1][Option2][...]

ls -al

sort -dfu $filename

set -- $variable

1 if [ $file1 -ot $file2 ] 2 then 3 echo "File $file1 is older than $file2." 4 fi 5 6 if [ "$a" -eq "$b" ] 7 then 8 echo "$a is equal to $b." 9 fi 10 11 if [ "$c" -eq 24 -a "$d" -eq 47 ] 12 then 13 echo "$c equals 24 and $d equals 47." 14 fi

-

redirection from/to stdin or stdout. [dash]

1 (cd /source/directory && tar cf - . ) | (cd /dest/directory && tar xpvf -) 2 # Move entire file tree from one directory to another 3 # [courtesy Alan Cox <a.cox@swansea.ac.uk>, with a minor change] 4 5 # 1) cd /source/directory Source directory, where the files to be moved are. 6 # 2) && "And-list": if the 'cd' operation successful, then execute the next command. 7 # 3) tar cf - . The 'c' option 'tar' archiving command creates a new archive, 8 # the 'f' (file) option, followed by '-' designates the target file as stdout, 9 # and do it in current directory tree ('.'). 10 # 4) | Piped to... 11 # 5) ( ... ) a subshell 12 # 6) cd /dest/directory Change to the destination directory. 13 # 7) && "And-list", as above 14 # 8) tar xpvf - Unarchive ('x'), preserve ownership and file permissions ('p'), 15 # and send verbose messages to stdout ('v'), 16 # reading data from stdin ('f' followed by '-'). 17 # 18 # Note that 'x' is a command, and 'p', 'v', 'f' are options. 19 # Whew! 20 21 22 23 # More elegant than, but equivalent to: 24 # cd source-directory 25 # tar cf - . | (cd ../target-directory; tar xzf -) 26 # 27 # cp -a /source/directory /dest also has same effect.

1 bunzip2 linux-2.4.3.tar.bz2 | tar xvf - 2 # --uncompress tar file-- | --then pass it to "tar"-- 3 # If "tar" has not been patched to handle "bunzip2", 4 # this needs to be done in two discrete steps, using a pipe. 5 # The purpose of the exercise is to unarchive "bzipped" kernel source.

Note that in this context the "-" is not itself a Bash operator, but rather an option recognized by certain UNIX utilities that write to stdout, such as tar, cat, etc.

bash$ echo "whatever" | cat - whatever

Where a filename is expected, - redirects output to stdout (sometimes seen with tar cf), or accepts input from stdin, rather than from a file. This is a method of using a file-oriented utility as a filter in a pipe.

bash$ file Usage: file [-bciknvzL] [-f namefile] [-m magicfiles] file...
By itself on the command line, file fails with an error message.

Add a "-" for a more useful result. This causes the shell to await user input.
bash$ file - abc standard input: ASCII text bash$ file - #!/bin/bash standard input: Bourne-Again shell script text executable
Now the command accepts input from stdin and analyzes it.

The "-" can be used to pipe stdout to other commands. This permits such stunts as prepending lines to a file.

Using diff to compare a file with a section of another:

grep Linux file1 | diff file2 -

Finally, a real-world example using - with tar.

Example 3-4. Backup of all files changed in last day

   1 #!/bin/bash
   2 
   3 #  Backs up all files in current directory modified within last 24 hours
   4 #+ in a "tarball" (tarred and gzipped file).
   5 
   6 BACKUPFILE=backup
   7 archive=${1:-$BACKUPFILE}
   8 #  If no backup-archive filename specified on command line,
   9 #+ it will default to "backup.tar.gz."
  10 
  11 tar cvf - `find . -mtime -1 -type f -print` > $archive.tar
  12 gzip $archive.tar
  13 echo "Directory $PWD backed up in archive file \"$archive.tar.gz\"."
  14 
  15 
  16 #  Stephane Chazelas points out that the above code will fail
  17 #+ if there are too many files found
  18 #+ or if any filenames contain blank characters.
  19 
  20 # He suggests the following alternatives:
  21 # -------------------------------------------------------------------
  22 #   find . -mtime -1 -type f -print0 | xargs -0 tar rvf "$archive.tar"
  23 #      using the GNU version of "find".
  24 
  25 
  26 #   find . -mtime -1 -type f -exec tar rvf "$archive.tar" '{}' \;
  27 #         portable to other UNIX flavors, but much slower.
  28 # -------------------------------------------------------------------
  29 
  30 
  31 exit 0

Filenames beginning with "-" may cause problems when coupled with the "-" redirection operator. A script should check for this and add an appropriate prefix to such filenames, for example ./-FILENAME, $PWD/-FILENAME, or $PATHNAME/-FILENAME.

If the value of a variable begins with a -, this may likewise create problems.
1 var="-n" 2 echo $var 3 # Has the effect of "echo -n", and outputs nothing.

-

previous working directory. [dash] cd - changes to the previous working directory. This uses the $OLDPWD environmental variable.

Do not confuse the "-" used in this sense with the "-" redirection operator just discussed. The interpretation of the "-" depends on the context in which it appears.

-

Minus. Minus sign in an arithmetic operation.

=

Equals. Assignment operator
1 a=28 2 echo $a # 28

In a different context, the "=" is a string comparison operator.

+

Plus. Addition arithmetic operator.

In a different context, the + is a Regular Expression operator.

+

Option. Option flag for a command or filter.

Certain commands and builtins use the + to enable certain options and the - to disable them.

%

modulo. Modulo (remainder of a division) arithmetic operation.

In a different context, the % is a pattern matching operator.

~

home directory. [tilde] This corresponds to the $HOME internal variable. ~bozo is bozo's home directory, and ls ~bozo lists the contents of it. ~/ is the current user's home directory, and ls ~/ lists the contents of it.
bash$ echo ~bozo /home/bozo bash$ echo ~ /home/bozo bash$ echo ~/ /home/bozo/ bash$ echo ~: /home/bozo: bash$ echo ~nonexistent-user ~nonexistent-user

~+

current working directory. This corresponds to the $PWD internal variable.

~-

previous working directory. This corresponds to the $OLDPWD internal variable.

^

beginning-of-line. In a regular expression, a "^" addresses the beginning of a line of text.

Control Characters

change the behavior of the terminal or text display. A control character is a CONTROL + key combination.

Ctl-B
Backspace (nondestructive).
Ctl-C
Terminate a foreground job.
Ctl-D
Log out from a shell (similar to exit).
"EOF" (end of file). This also terminates input from stdin.
Ctl-G
"BEL" (beep).

Ctl-H

"Rubout" (destructive backspace).

1 #!/bin/bash 2 # Embedding Ctl-H in a string. 3 4 a="^H^H" # Two Ctl-H's (backspaces). 5 echo "abcdef" # abcdef 6 echo -n "abcdef$a " # abcd f 7 # Space at end ^ ^ Backspaces twice. 8 echo -n "abcdef$a" # abcdef 9 # No space at end Doesn't backspace (why?). 10 # Results may not be quite as expected. 11 echo; echo

Ctl-I
Horizontal tab.
Ctl-J
Newline (line feed).
Ctl-K
Vertical tab.
Ctl-L
Formfeed (clear the terminal screen). This has the same effect as the clear command.

Ctl-M

Carriage return.

1 #!/bin/bash 2 # Thank you, Lee Maschmeyer, for this example. 3 4 read -n 1 -s -p $'Control-M leaves cursor at beginning of this line. Press Enter. \x0d' 5 # Of course, '0d' is the hex equivalent of Control-M. 6 echo >&2 # The '-s' makes anything typed silent, 7 #+ so it is necessary to go to new line explicitly. 8 9 read -n 1 -s -p $'Control-J leaves cursor on next line. \x0a' 10 echo >&2 # Control-J is linefeed. 11 12 read -n 1 -s -p $'And Control-K\x0bgoes straight down.' 13 echo >&2 # Control-K is vertical tab. 14 15 exit 0

Ctl-S
Suspend (XOFF).
This freezes stdin in a terminal.
Ctl-U
Erase a line of input.
Ctl-Z
Pause a foreground job.

Whitespace

functions as a separator, separating commands or variables. Whitespace consists of either spaces, tabs, blank lines, or any combination thereof. In some contexts, such as variable assignment, whitespace is not permitted, and results in a syntax error.

Blank lines have no effect on the action of a script, and are therefore useful for visually separating functional sections.

$IFS, the special variable separating fields of input to certain commands, defaults to whitespace.

Prev	Home	Next
Basics	Up	Introduction to Variables and Parameters

Chapter 3. Special Characters

Notes