Module 0339: Asynchronous CLI command execution

Although NodeJs has an extensive library of modules, there are certain times when a CLI command is not only more convenient, but also more eﬃcient to complete certain tasks. This module introduces the basic concepts and code to run CLI commands from a NodeJS script.

3 The "child_process" module

The term ’process’ is technical. A process is a program in execution. A program is nothing more than just the speciﬁcations of how something is done, like a cookbook. However, a process is an instance of the instructions being followed, like a person following a cookbook to actually cook food.

As you can imagine, when your script is running, it is in a process. When a program in execution needs to start another program, it has two choices. The ﬁrst one is to give up the original program, and the new program gets to reuse the same process.

However, if the purpose is to run the new program, and then to continue with the existing process, then a new process (the child process) needs to be started to run the new program.

4 Synchronous versus asynchronous

Depending on the CLI command and what it is supposed to do, it may take a command some time to complete. Since we are using the context of an Express script, "busy waiting" for a command to ﬁnish means a single HTTP request can end up hogging time unnecessarily and making the Express server unresponsive.

As a result, the asynchronous method is often preferred when an Express script executes a CLI command.

5 Shell or no shell?

A "shell" is a utility program that provides a rich context for any CLI command to run in. While a shell has a slight overhead, it is very minimal. The advantage of having a NodeJS script to execute a CLI command within the context of a shell is that most commands that run interactively can be speciﬁed the same way when run from a NodeJS script.

Furthermore, a shell, such as BASH (the default shell on many Linux systems) oﬀers its own built-in commands and control structures. For example, the CLI command cd is actually a built-in command of BASH (most other shells oﬀer the same command).

One of the conveniences oﬀered by a shell is the PATH environment variable. This environment variable makes it easy to run commands that are not built-in to a shell (external commands). For example, du is not a built-in command, but it is within the PATH of a shell. Without this environment variable, it becomes necessary to ﬁnd the path of every external command.

The du command shows the storage consumed by a folder structure. Depending on the depth of folder hierarchy as well as the number of ﬁles included, du can take a long time to execute.

6 Call-back, promises, and async/await

Now we are getting to the details of how to run a CLI command in a shelled environment from a NodeJS script.

Listing 1:Exec using call-back function

001"use strict";
002
003const child_process = require(’child_process’)
004
005function execHandler(error, stdout, stderr)
006{
007  console.log(‘exec is done with an error = "${error}"‘)
008  console.log(‘stdout is ${stdout}‘)
009  console.log(‘stderr is ${stderr}‘)
010}
011
012child_process.exec(
013  "du",  // this is the command
014  {},  // no special options to specify
015  execHandler // this is the call-back
016)
017
018console.log(’right after async exec’)

While simple, this sample program illustrates the most basic use of asynchronous exec. When this script runs, note how the method exec returns right away. Only when the CLI command du is completed did the function execHandler gets called (by node itself) to report any error.

stdout is a string parameters that is the actual output of the command, whereas stderr is a string parameter that is the "error output" of the command. The parameter error reports back issues of attempting to the command.

In other words, if there is a problem starting the command, the error parameter will not be null. If the command runs, but encounters problems, then stderr will contain error messages from the command itself. stdout is the standard (normal) output of running the command.

Next, we parametrize and make a function so that it is easy to specify the execution of any CLI command.

Listing 2:Parametrized function to run a shelled CLI command

001"use strict";
002
003const child_process = require(’child_process’)
004
005
006function shelledCommand(command)
007{
008  function execHandler(error, stdout, stderr)
009  {
010    console.log(‘${command} is done with an error = "${error}"‘)
011    console.log(‘stdout is ${stdout}‘)
012    console.log(‘stderr is ${stderr}‘)
013  }
014  child_process.exec(command, {}, execHandler)
015}
016shelledCommand("du")
017console.log(’right after shelledCommand’)

This is a slightly dressed up version of the previous one. Note how the output can now specify what command was run because execHandler is deﬁned within shelledCommand, gaining access to all the parameters of shelledCommand.

With this version, the three pieces of information after running a CLI command, error, stdout, and stderr are visible only within execHandler. It would be nice to be able to pass these parameters along to code that is outside of shelledCommand. This is where promises can be useful.

When a promise is resolved by having the resolve call back function invoked, the handler speciﬁed in the then method of the process has access to the resolved value.

Listing 3:Using a Promise to pass results of a CLI command along

001"use strict";
002
003const child_process = require(’child_process’)
004
005
006function shelledCommand(command)
007{
008  function whatToDo(resolve, reject)
009  {
010    function execHandler(error, stdout, stderr)
011    {
012      resolve(
013        {
014          command: command,
015          error: error,
016          stdout: stdout,
017          stderr: stderr
018        }
019      )
020    }
021    child_process.exec(command, {}, execHandler)
022  }
023  return new Promise(
024    whatToDo
025  )
026}
027
028function afterShelledCommand(value)
029{
030      console.log(‘${value.command} is done with an error = "${value.error}"‘)
031      console.log(‘stdout is ${value.stdout}‘)
032      console.log(‘stderr is ${value.stderr}‘)
033}
034
035shelledCommand("du").then(afterShelledCommand)
036console.log(’right after shelledCommand’)

In this revision, shelledCommand has one new level of function called whatToDo. whatToDo captions the action to perform when a Promise object is created and returned from shelledCommand. whatToDo only has one thing to do: to call the exec method of child_process. However, because whatToDo is the action call-back of a Promise, it is automatically passed two call-back functions. One is called resolve, which is a function to be called when the action associated with whatToDo (the exec) has ﬁnished and execHandler is called.

Instead of using console.log within execHandler, execHandler calls the resolve call-back function given to whatToDo. resolve can only use one value, but there are four pieces of information. This is why an object is created to pass all four pieces of information along.

Note that the named functions whatToDo, execHandler and afterShelledCommand are often anonymous in commercial code. They are named in this sample program for clarify and ease of reference.

The use of Promises is great, but it fragments the code of a sequence of operations. At this point, shelledCommand is exactly what we need because it returns a Promise. We need to deﬁne a main function that is async so that we can throw all the sequential logic that may need to utilize asynchronous operations into it.

Listing 4:Using async and await to hide the use of Promises

001"use strict";
002
003const child_process = require(’child_process’)
004
005
006function shelledCommand(command)
007{
008  function whatToDo(resolve, reject)
009  {
010    function execHandler(error, stdout, stderr)
011    {
012      resolve(
013        {
014          command: command,
015          error: error,
016          stdout: stdout,
017          stderr: stderr
018        }
019      )
020    }
021    child_process.exec(command, {}, execHandler)
022  }
023  return new Promise(
024    whatToDo
025  )
026}
027
028async function main()
029{
030  let result = await shelledCommand("du")
031  console.log(’right after shelledCommand’)
032  console.log(‘${result.command} is done with an error = "${result.error}"‘)
033  console.log(‘stdout is ${result.stdout}‘)
034  console.log(‘stderr is ${result.stderr}‘)
035}
036
037main()

Note how afterShelledCommand is no longer necessary. This function, as a call-back function, was necessary because it is needed by the then method of a Promise to specify what to do with a resolve value. Using await, however, the use of Promise is hidden because the await expression returns the resolve value of the Promise object speciﬁed right after the word await. Execution is also blocked until the said Promise resolves.

This last revision of the sample programs contains the ﬁnished form of shelledCommand as well as the template of how it can used in an async function. This sets up the basic method of running CLI commands from async-enabled Express handlers for clarify, eﬃciency and responsiveness.