When it comes to debugging shell scripts probably single most valuable piece of advice is to use built in set -x command.

It’s available in most shell interpreters and sometimes it can save you hours of eyeballing at the code you didn’t write in the first place .. but you still have to fix it 😉

Lets assume you have this simple script:

#!/bin/sh

echo "Hello World!"
echo "running with pid $$"
ITER=$(seq -s ' '  3 -1 1)
for i in $ITER; do
  echo $i
  sleep 1
done
echo "bye!"

Its output would look like this:

$ ./debug.sh 
Hello World!
running with pid 27087
3
2
1
bye!

Now you want to see what is being executed in relation to the script output. Add set -x in the begging of your script:

#!/bin/sh

set -x

echo "Hello World!"
echo "running with pid $$"
ITER=$(seq -s ' '  3 -1 1)
for i in $ITER; do
  echo $i
  sleep 1
done
echo "bye!"

And now the output is a bit more verbose. You see what exactly is being executed and its output (if any):

$ ./debug.sh 
+ echo Hello World!
Hello World!
+ echo running with pid 27194
running with pid 27194
+ seq -s   3 -1 1
+ ITER=3 2 1
+ echo 3
3
+ sleep 1
+ echo 2
2
+ sleep 1
+ echo 1
1
+ sleep 1
+ echo bye!
bye!

If a script runs under bash then you can combine set -x with PS4 parameter which will make set -x even more verbose:

#!/bin/bash

PS4='$0 +$LINENO: '
set -x

echo "Hello World!"
echo "running with pid $$"
ITER=$(seq -s ' '  3 -1 1)
for i in $ITER; do
  echo $i
  sleep 1
done
echo "bye!"

PS4 in line 3 added file name and line number which produces very informative output:

$ ./debug.sh 
./debug.sh +6: echo 'Hello World!'
Hello World!
./debug.sh +7: echo 'running with pid 27496'
running with pid 27496
../debug.sh +8: seq -s ' ' 3 -1 1
./debug.sh +8: ITER='3 2 1'
./debug.sh +9: for i in $ITER
./debug.sh +10: echo 3
3
./debug.sh +11: sleep 1
./debug.sh +9: for i in $ITER
./debug.sh +10: echo 2
2
./debug.sh +11: sleep 1
./debug.sh +9: for i in $ITER
./debug.sh +10: echo 1
1
./debug.sh +11: sleep 1
./debug.sh +13: echo 'bye!'
bye!

Bingo!

This matters in real life cases when there are many files involved and execution path is not that obvious. What is more important set -x can be enabled in any place in the code – not just in the beginning – and disabled with set +x when no longer needed. You can narrow down your investigation of problematic part of a code with set -x and set +x sequence and not be overwhelmed by the output.

Both set and PS4 concepts are described in bash man pages (SHELL BUILTIN COMMANDS section).