General Information

Introduction to sending commands in serial and parallel to a large number of systems.

Checklist

• Distro(s): Any
• Other: Use of SSH Public/Private keys so nothing prompts for a password in the loop.

For some things, it is better to do a rolling serial change…such as restarting a web server.

This example restarts the Apache webserver on a list of systems serially, and sleeps 10 seconds in between each.

1) Populate a text file with the list of systems, one per line. Example:

host_list.txt

webserver01
webserver02
webserver03
...

2) Execute a for loop through that list:

for NODE in $(cat host_list.txt)
do echo -e "\nWorking on ${NODE}..."
ssh admin@${NODE} 'service httpd restart'
echo "---- Done with ${NODE} ----"
sleep 10
done

If you would rather get something done very quickly, commands can be sent in parallel.

1) Create a script that will operate on 1 system.

This script takes 1 argument; a system hostname.

single_host.sh

#!/bin/bash
 
HOST=$1
echo "Executing on ${HOST}..."
ssh admin@${HOST} 'command1;command2;command3'

2) Create a script that will call that script.

This script takes two+ arguments:

• Max Number of forks
• Hostnames (space seperated)

parallel_hosts.sh

#!/bin/bash
 
MAX_FORKS=$1
shift
FORKS=0
 
for HOST in $@; do
  if [ ${FORKS} -ge ${MAX_FORKS} ]; then
    wait
    FORKS=0
  fi
  (single_host.sh ${HOST}) &
  FORKS=$(( ${FORKS} + 1 ))
done
wait

3) Call the parallel_hosts.sh script.

In this example, parallel_hosts.sh will spawn a max of 10 processes at a time:

parallel_hosts.sh 10 $(cat host_list.txt)

• parallel_hosts.sh is called with two arguments: 10 and an expanded list of host names
• The first argument (10) is assigned to MAX_FORKS
• shift (with no number) moves all arguments down a position, so the 10 drops off, and now $1 is the first host name from the text file.
• The for loop goes through each argument ($@) starting at $1
• If FORKS is ever equal to the MAX_FORKS, wait is used to pause until all child processes have completed.
• Otherwise, call single_host.sh ${HOST}, creating a child process and sending it the current host in the loop to execute on.
• Finally, after the loop is complete, a final wait is used to ensure that any other single_host.sh child processes can finish cleanly.