Assemble and Run Shell Commands Dynamically

02 March 2016

In a previous post I explained that understanding what shell does under the hood (i.e. "there are three steps" section) is the key to understand most issues that are related with quotes. This blog is to reiterate it, with another scenario.

A use case

In shell scripting, it is not uncommon to compose and run commands dynamically like this:

1: options=""
2: if a; then
3:     options+="-a xyz"
4: fi
5: mycmd $options

Though this works in many cases, please be warned that it may fail when options is complicated. For exmaple, this is a real-world case we met in our project:

base_dn='cn=Request Processors,cn=config'
options="-h localhost -b '$base_dn' 'objectclass=*'"
result=$(ldapsearch $options)
# OOPS, error message: The provided search filter contains an invalid attribute
# type 'Processors,cn' with invalid character ',' at position 10"

The Three Steps

To understand why sometimes it works sometimes it does not work, it= is necessary to really understand the three steps undergone to run a command.

Any command line you typed in an interactive shell command line or shell scripts is dealt with in the following steps:

The shell does expansion and substitution.
The shell interprets the cooked command line, i.e. evokes designated command and feeds it with inputs.
The command runs and handles the inputs it gets.

A Case Study

Now, let's examine a made-up case to see how those three steps work. I wrote a command demo that accepts an optional argument description and prints arguments it receives.

Normal usage

$ ./demo --description "hello world"
Arg 1 -> --description
Arg 2 -> hello world
Desc  -> hello world

The shell does expansion and substitution etc.
- There is NO expansion and substitution.
The shell interprets the command line

The shell sees the double quotes, therefore it removes the quotes and takes hello world as a literal string. As a result, the shell regards the command line as:
```
demo + --description + hello world
```
That is command demo plus two arguments (--description and hello world)
Command runs

demo receives two arguments and parse them accordingly as expected.

With Expansion/Substitution

$ options='--description "hello world"'

$ ./demo $options
Arg 1 -> --description
Arg 2 -> "hello
Arg 3 -> world"
usage: demo [--description DESCRIPTION]
demo: error: unrecognized arguments: world"

Expansion and substitution
- It sees $options and replaces it with the value of it, that is string --description "hello world". In this case, since the quotes are the result of substitution, they do not have special meaning and are just part of the string. Therefore, the command line is actually equivalent to this:
```
./demo --description \"hello world\"
```
- The shell does NOT see any quotes.
Interpretation

Hence, what the shell sees is:
```
demo + --description + "hello + world"
```
I.e. command demo with three arguments.
Command run

The demo sees --description and then takes "hello (with quote) as the value of description. Then, it sees the unexpected third argument world" (with quote) and emits an error.

Solutions

eval

NOTE In my opinion, compared with eval, using shell array might be a better solution.

$ options='--description "hello world"'

$ eval "./demo $options"
Arg 1 -> --description
Arg 2 -> hello world
Desc  -> hello world

The eval basically goes through the 3-step process one more time.

Expansion and substitution
- The shell sees a pair of double quotes, hence take everything (after expansion and substitution) enclosed by them as a string literal.
- It replace $options with value of it.

Interpretation

The shell sees:

eval + ./demo --description "hello world"

Command run

"eval" gets one arguments, that is string ./demo --description "hello world". It then goes through the three-step procedure against this string, which is basically the same as Normal usage.

Use shell array

$ options=("--description" "hello world")

$ ./demo "${options[@]}"
Arg 1 -> --description
Arg 2 -> hello world
Desc  -> hello world

Quiz

Write a demo script that prints every and each argument it receives. Then, try the following commands and explain the outcome of each command:

# case 1
options='--description "hello world"'
./demo "$options"

# case 2,3
options=("--description" "hello world")
./demo "${options[*]}"
./demo ${options[@]}

computer 89