Bash’s Crazy Config Files

In the ‘/etc` directory are some default bash config files that affect bash’ behavior unless countermanded by user set config files.

/etc/profile

This file is read and executed at initial login time by all users. I think that subshells ignore this. It also seems like this gets executed before the the xterm box is ready, so output from this script, like a message of the day, doesn’t seem to make it to the screen. It will make it to a log file however. This means that if you want to print a message or set the PS1 prompt variable, it won’t carry over when X subsequently opens up an xterm.

/etc/bashrc

This file is read and executed by all bash initialization activity such as logging in by any user from anywhere and any sub shell that the user spawns. I think that this is referred to in a lot of unexpected places too. So it’s not smart to have anything too busy here. This file might actually be a fake. It isn’t mentioned in any documentation. It seems to be called from ${HOME}/.bashrc. Therefore, if that file doesn’t exist, then this one is useless. It’s not really a global catch-all if users can pull the reference from their ${HOME}/.bashrc, so I don’t really see the point to it. Although it would take up a trivial amount of extra disk space, a better way would be to have the things that you wanted in here, in a skeleton for creating ${HOME}/.bashrc files when you create new users.

$HOME/.bash_profile

This is the user’s chance to redo the initial login settings established by /etc/profile, if there is one. This does things once only on log in. If the user wants something run every time he logs in, like a back up or something, here’s the good place for it. Be aware that this file has no less than two functionally equivalent synonyms, 1.) ${HOME}/.bash_login and 2.) ${HOME}/.profile which are searched for in order.

$HOME/.bashrc

This file is executed for all the owning user’s new shell activity. This means that these commands will be reissued for each and all subsequent subshells. This is where users can put custom aliases that should be very persistent. This file seems to execute quite frequently - 3 times on initial login for me and once for each subshell.

Pattern-Matching Operators:

  • ${variable#pattern} if pattern matches beginning, del shortest part

  • ${variable##pattern} if pattern matches beginnnig, del longest part

  • ${variable%pattern} if pattern matches end, del shortest part

  • ${variable%%pattern} if pattern matches end, del longest part

Here’s an example - this changes a big long list of x001.gif, x002.gif,etc to this b001.gif, b002.gif, b003.gif etc.

for ce in x???.gif; do mv $ce b${ce#x}; done

Another example - this converts a series of tif images with the names tb01.tif, tb02.tif,tbwhatever.tif to tb01.gif, tb02.gif, etc. It also changes the images to 15 color grayscale.

[~/xfile/project/pix]$ for ce in tb*.tif; \
> do convert -colorspace GRAY -colors 15 $ce ${ce%tif}gif; done

Arithmetic Expansion

I don’t know how long bash has had this, but it wasn’t a useful feature when I learned Bash a long time ago. But these days it is quite handy and elevates Bash to the status of quite a sensible general purpose language. Basically in the distant Unix past one had to use bc and dc and expr to get any kind of arithmetic done, even trivial things like incrementing a variable involved spawning a new shell and casting text strings every time. Now you can pretty much use the (( expression )) syntax and the expression will be evaluated in a pretty sensible way. If you need the expression to stand by itself as a process which produces its own exit code (like the old bc but I’m not sure a new process is actually spawned), then you can do something like this:

if (($RANDOM%2)); then echo Tails; else echo Heads; fi

If you need to produce the result for further consideration, you need to expand the expression like this:

echo $(($RANDOM%6+1)) # 6 sided dice
# Print a random line from a file:
F=file; sed -n "$(($RANDOM%`sed -n $= $F`+1))p" $F

Note that the performance of these operations can be suboptimal. Often spawning new shells for sub commands beats the arithmetic expansion.

$ X=0; time while (( X < 100000 )); do true $((X++)); done
real    0m2.794s
$ X=0; time for X in `seq 100000`; do true; done
real    0m1.176s

Oh and check out this:

$ X=0;time while (( X < 100000 )); do true $(( X++ )); done
real    0m2.875s
$ X=0;time while ((X<100000)); do true $((X++)); done
real    0m2.510s

Normally in Bash, it’s good to help the tokenizer by being explicit about where things are, but in this case, spaces just waste time.

Mass updating of files that contain similar things that need changing

Here is a method to massively update lots of files that all contain some bad thing and replace it with some good thing. There might be a smoother way to use the sed command, but redirecting to the same file causes the file to simply disappear. Using a tempfile works just fine.

$ for cxe in project??.type ; do cp $cxe temp.temp; \
sed s/bad/good/g temp.temp > $cxe; done

Here, all files that match the project??.type pattern (project01.type, project1a.type, etc) will be copied into the temporary file temp.temp and then they’ll be pulled back out into their real name line by line by sed which will also make substitutions as requested. In this case, it will change all references of "bad" to "good". Don’t forget to erase the residual temp.temp file when you’re done.

Another note - if the text is long, you need quotes around it so bash doesn’t get wise with it: sed s/"This is very bad."/Nowitsgood/g file

If the text has quotes in it:

$ for X in *.html; do cp $X temp.temp;\
> sed s/height=\"66\"/height=\"100\"/ temp.temp > $X; done

Here’s another example-

$ for XXX in *html ; do cp $XXX temp.temp; \
> sed -e "/---/,/---/s/<table>/<table align=center>/" temp.temp > $XXX; done

This only does the substitution after it finds 3 dashs and only until it finds 3 more dashes.

Functions

Bash functions are quite useful. They are like bash aliases with the added ability to take parameters. Here’s an example of function use that I have in my ~/.bashrc:

# Break a document into words.
function words { cat $1 | tr ' ' '\n' ; }
function wordscounted { cat $1 | tr ' ' '\n' | sort | uniq -c | sort -n ; }

Redirection

How to pipe those nasty errors off to Neverneverland.
grep hattrick * 2> /dev/null

This will do the expected thing with stdout, but stderr will head off to the trash.

To get all of the garbage a command spits out to go to a file:

make &> compile.error
How to pipe those handy errors off to another command.
cmd1 2>&1 | cmd2
cdparanoia -Q 2>&1 | grep "no   no"

Much more fancy things can be done with pipes and redirection. It is possible to use the test command (which is related to the [ … ] syntax since [ is an alias for test) to check for where a file descriptor is from. That’s confusing but this makes it clear:

    if [ -t 0 ]; then
        echo Interactive
    else
        echo File descriptor 0 coming from a pipe
    fi

The exec command replaces the current process with a subshell containing the specified process as in exec takeoverthisjob -a args. An interesting effect and demonstration of the exec command is seen in the following example.

echo This is being sent to standard output like normal.
exec > logfile.txt
echo This is now being put in the logfile.

Complete Chat Utility Using Named Pipes

Here is a nice example of how to use named pipes to implement a complete private chat system. Imagine your user name is "primary" and your friend’s is "guest".

Put this in /home/primary/.bashrc
TOPIPE='/home/guest/toguest'

This is the named pipe which will contain your messages to the guest. Note that it goes in the guest’s home directory which you’ll need access to. These could go anywhere (somewhere in /tmp might be good).

Put this in /home/guest/.bashrc
TOPIPE='/home/guest/toprimary'

Here’s the named pipe which you will look at to see the guest’s chat. After you specify where these will go, you need to create these named pipes once with this command:

$ mkfifo /home/guest/toprimary; mkfifo /home/guest/toguest

Next put these macros into both .bashrc files.

Chat macros
SEDSUB="s/^/ [$USER]: /"
alias listen='/bin/cat </home/guest/to$USER &'
alias chat="/usr/bin/sed -u '$SEDSUB' >$TOPIPE"

Now to chat both parties simply log in (SSH makes the whole thing pretty secure) and each types listen to start listening for the other’s text. And then they type chat to start sending. Now both parties can type and read the other’s typing.

Loops

Often you’ll want to use xargs or Gnu Parallel but you will be frightened by very messy syntax. One way to get around such things is to use a bash loop to receive the output. Here’s how it would work:

find . | while read N; do md5sum $N; done
find . | while read N; do echo -n $N; md5 $N |sed -n 's/^.* / /p'; done

This will produce a list of check sums for all files in the current directory (and below). This command can be useful to create an inventory of what’s in a directory tree and compare with a directory tree elsewhere to see what has changed.

Here’s example showing how to avoid xargs:

cat excludes | while read X; do sed -i "/$X/d" mybiglist ; done

This takes a big list of things and a smaller list of things you wish were not in the big list and it looks through each item in the excludes list and removes it in place from the big list (if you’re using GNU sed, otherwise make your own temp file).

Logging

If you ever need to log something anywhere or do any kind of error monitoring or debugging, check out man logger. This is very handy and it’s universally available as an old school program yet I had never heard of it.

Here’s a clever way to turn off logging selectively (credit to DaveTaylorOnline.com):

if [ $DEBUG ] ; then
    logger="/usr/bin/logger"+$DEBUG_FILE
else
    logger="echo >/dev/null"
fi

Signal Trapping

Trapping a signal is kind of exotic but when it’s a good idea, it’s a good idea. It can be good for things like cleaning up a mess before letting a user cancel an operation with Ctrl-C. For example (more from Dave Taylor):

trap '{ echo "You pressed Ctrl-C"; rm $TEMPFILE ; exit 1; }' INT
for C in 1 2 3 4 5 6 7 8 9 10; do echo $C; sleep 5; done

It can also be used as a timeout enforcer. Basically spawn another background process that just waits the maximum allowed time and then raises a STOP signal. Then have the main thing that’s being limited trap for that. Here’s how:

trap '{ echo Too slow. ; exit 1 ; }' SIGALRM
( sleep 5 ; kill -s SIGALRM $$ > /dev/null )&
echo "What is the airspeed velocity of an unladen swallow?"
read OK
trap '' SIGALRM
echo "${OK} sounds reasonable to me."

Note that this little script will have problems if the main script exits cleanly and its process ID is taken by something else, something important that should not get a SIGALRM. Just keep that in mind.

To turn off a trap for the INT signal (as used in the previous example):

trap '' INT

General Bash Syntax

Syntax for using the "for" structure.
for variablename [in <list of textstrings or files>]
    do
        commands that take advantage of $variablename go here
    done

Here’s another example. This little routine calculates (via exit code) whether a number is a power of 2 or not. Apparently sometimes big search engine companies care about such things.

$ for N in `factor 16384 |cut -d: -f2-`;do if expr $N \!= 2 >/dev/null;\
then break;fi; done; expr $N == 2 >/dev/null
Using Here Documents
#!/bin/bash
#cxe- Heredocs are a way to put a lot of arbitrary text into a commands
#cxe- standard input stream. Can be good for CGI scripts and the like.
#cxe- Here is a sample program that illustrates the idea.
cat > test1 <<HEREDOC
<html>
<head><title>Here Doc Fun!</title><head>
<body>
HEREDOC
#cxe- Note that the heredoc closer needs to be alone on the line.
echo Here are commands that aren\'t part of the heredoc
date
IF/THEN constructions
#cxe- Here are two good examples - the first one justs waits for a
#cxe- confirmation message.
read REP
if [ CXE${REP} != "CXEy" ]; then
  echo "Ok, maybe next time."
  exit
fi

#cxe- The second one, makes some new directories if they need to be made.
if [ ! -d ${TND} ]; then
  echo "Creating the directories:"
  echo "mkdir "${WSD}" "${TND}
  mkdir ${WSD} ${TND}
fi
Option Handling With getopts
#!/bin/bash
# getopts_example - Chris X Edwards
# Note that the options must come before the arguments. For example:
#     getopts_example king queen -c jack
# Will make "-c" an argument.

function show_usage {
    echo "Usage: getopts_example [-h] [-a alpha] [-b beta] [-c] <arguments>"
    echo "       -h = usage message"
    echo "       -a = set alpha value [default 'First']"
    echo "       -b = set beta value [default not set]"
    echo "       -c = set gamma flag [default not set]"
    exit
}

ALPHA="DefaultValue"     # Set default value.
# Option letters followed by ":" mean has an OPTARG.
while getopts "a:b:ch" OPTION
do
    case ${OPTION} in
        a) ALPHA=${OPTARG};;
        b) BETA=${OPTARG};;
        c) GAMMA="true";;
        h) show_usage;;
    esac
done
shift $((OPTIND - 1)) # Leave behind remaining arguments.

[ "$ALPHA" ] && echo "Alpha is set to: $ALPHA"
[ "$BETA" ] && echo "Beta is set to: $BETA"
[ "$GAMMA" ] && echo "Gamma is set."
echo "Arguments:"
for OP in $* ; do echo $OP ; done