====== Command Many Systems: Part1 Basics ======

**General Information**

Introduction to sending commands in serial and parallel to a large number of systems.

**Checklist** 
  * Distro(s): Any
  * Other: Use of [[linux_wiki:ssh_pub_priv_keys|SSH Public/Private keys]] so nothing prompts for a password in the loop.

----

====== Systems in Serial ======

For some things, it is better to do a rolling serial change...such as restarting a web server.

This example restarts the Apache webserver on a list of systems serially, and sleeps 10 seconds in between each.

1) Populate a text file with the list of systems, one per line. Example:

host_list.txt
<code>
webserver01
webserver02
webserver03
...
</code>

2) Execute a for loop through that list:
<code bash>
for NODE in $(cat host_list.txt)
do echo -e "\nWorking on ${NODE}..."
ssh admin@${NODE} 'service httpd restart'
echo "---- Done with ${NODE} ----"
sleep 10
done
</code>

----

====== Systems in Parallel ======

If you would rather get something done very quickly, commands can be sent in parallel.

1) Create a script that will operate on 1 system.

This script takes 1 argument; a system hostname.
<code bash single_host.sh>
#!/bin/bash

HOST=$1
echo "Executing on ${HOST}..."
ssh admin@${HOST} 'command1;command2;command3'
</code>

2) Create a script that will call that script.

This script takes two+ arguments:
  * Max Number of forks
  * Hostnames (space seperated)
<code bash parallel_hosts.sh>
#!/bin/bash

MAX_FORKS=$1
shift
FORKS=0

for HOST in $@; do
  if [ ${FORKS} -ge ${MAX_FORKS} ]; then
    wait
    FORKS=0
  fi
  (single_host.sh ${HOST}) &
  FORKS=$(( ${FORKS} + 1 ))
done
wait
</code>

3) Call the parallel_hosts.sh script.

In this example, parallel_hosts.sh will spawn a max of 10 processes at a time:
<code bash>
parallel_hosts.sh 10 $(cat host_list.txt)
</code>

===== Parallel Explanation =====

  * parallel_hosts.sh is called with two arguments: 10 and an expanded list of host names
  * The first argument (10) is assigned to MAX_FORKS
  * shift (with no number) moves all arguments down a position, so the 10 drops off, and now $1 is the first host name from the text file.
  * The for loop goes through each argument ($@) starting at $1
  * If FORKS is ever equal to the MAX_FORKS, wait is used to pause until all child processes have completed.
  * Otherwise, call single_host.sh ${HOST}, creating a child process and sending it the current host in the loop to execute on.
  * Finally, after the loop is complete, a final wait is used to ensure that any other single_host.sh child processes can finish cleanly.

----