Command Many Systems: Part1 Basics
General Information
Introduction to sending commands in serial and parallel to a large number of systems.
Checklist
- Distro(s): Any
- Other: Use of SSH Public/Private keys so nothing prompts for a password in the loop.
Systems in Serial
For some things, it is better to do a rolling serial changeā¦such as restarting a web server.
This example restarts the Apache webserver on a list of systems serially, and sleeps 10 seconds in between each.
1) Populate a text file with the list of systems, one per line. Example:
host_list.txt
webserver01 webserver02 webserver03 ...
2) Execute a for loop through that list:
for NODE in $(cat host_list.txt) do echo -e "\nWorking on ${NODE}..." ssh admin@${NODE} 'service httpd restart' echo "---- Done with ${NODE} ----" sleep 10 done
Systems in Parallel
If you would rather get something done very quickly, commands can be sent in parallel.
1) Create a script that will operate on 1 system.
This script takes 1 argument; a system hostname.
- single_host.sh
#!/bin/bash HOST=$1 echo "Executing on ${HOST}..." ssh admin@${HOST} 'command1;command2;command3'
2) Create a script that will call that script.
This script takes two+ arguments:
- Max Number of forks
- Hostnames (space seperated)
- parallel_hosts.sh
#!/bin/bash MAX_FORKS=$1 shift FORKS=0 for HOST in $@; do if [ ${FORKS} -ge ${MAX_FORKS} ]; then wait FORKS=0 fi (single_host.sh ${HOST}) & FORKS=$(( ${FORKS} + 1 )) done wait
3) Call the parallel_hosts.sh script.
In this example, parallel_hosts.sh will spawn a max of 10 processes at a time:
parallel_hosts.sh 10 $(cat host_list.txt)
Parallel Explanation
- parallel_hosts.sh is called with two arguments: 10 and an expanded list of host names
- The first argument (10) is assigned to MAX_FORKS
- shift (with no number) moves all arguments down a position, so the 10 drops off, and now $1 is the first host name from the text file.
- The for loop goes through each argument ($@) starting at $1
- If FORKS is ever equal to the MAX_FORKS, wait is used to pause until all child processes have completed.
- Otherwise, call single_host.sh ${HOST}, creating a child process and sending it the current host in the loop to execute on.
- Finally, after the loop is complete, a final wait is used to ensure that any other single_host.sh child processes can finish cleanly.