linux_wiki:spacewalk

This is an old revision of the document!

Spacewalk

General Information

Spacewalk is a centralized system update and config server.
Official Site: https://fedorahosted.org/spacewalk/

Checklist

  • Spacewalk server installed

Spacecmd

Spacecmd is the command line interface to Spacewalk.
Details here: Spacecmd

Register System with Spacewalk

A Spacewalk registration script has been created to ease registration.

Re-Register

If you need to re-register a client for any reason, you need the “–force” option when executing rhnreg_ks.

  • Delete system from Spacewalk
    spacecmd system_delete <SYSTEM>
  • Register system with the –force option
    sw_activation_key="1-my-system-key"
rhnreg_ks --force --serverUrl=https://my-spacewalk-server.local/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=${sw_activation_key}

rhn_check

By default, a system checks into Spacewalk via rhn_check every 4 hours.

If systems are not picking up the scheduled action from the Spacewalk portal in a timely manner with the osad (such as a config deploy, package upgrade, etc), you can force a group of systems to check in by running the “rhn_check” command locally on that system.

To loop through a group of systems and have them check in:

Example: Loop through the dev system group and have them check in 

for NODE in $(spacecmd group_listsystems dev); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /usr/sbin/rhn_check"; done

Channel Management

About Channels

  • Systems are subscribed to “Channels”
  • Channel subscriptions can be changed at any time in the Spacewalk portal, across any amount of systems.
  • Channels have Repositories assigned to them
  • This allows for a single repo to back multiple channels

Channel Freezing/Snapshots

In order to facilitate the same updates being applied to the Development, System Test, and the Production environments, it is necessary to clone the original Channels.
This creates a “snapshot in time” of the available packages/errata on the date of clone.
Note: This copies metadata of the Channel and does not duplicate repo packages

To Clone an entire Channel tree:

  • Login to a system with spacecmd installed
  • Clone the original base tree to a “snapshot-<date>_” prefix tree
  • Clone can be performed with spacecmd in batch or interactive mode:
    • Batch Clone Example ⇒ Clone the CentOS 6 tree, giving it the prefix “ss-20151215_” (for the snapshot date)
      spacecmd softwarechannel_clonetree centos6_x86-64_base --prefix "ss-20151103_" --gpg-copy
      • The above will clone the entire tree(base and child channels), give the shown prefix, copy gpg data, and copy errata data.
    • Interactive Clone Example
      spacecmd {SSM:0}> softwarechannel_clonetree
Source Channels:
centos6_x86-64_base
centos7_x86-64_base
 
Select source channel: centos6_x86-64_base
Prefix: ss-20151215_
 
Copy source channel GPG details? [y/N]: y
 
Original State (No Errata) [y/N]: N

Errata Setup

As of 12/15/2015, CentOS does not generate an “updateinfo.xml” file in their repodata directories. This file is responsible for the package to errata mappings. (RHEL, Fedora, EPEL, and supposedly Oracle all do this)

For a workaround, use a script to scrape the CentOS mailing archive lists for the errata.

The “spacewalk-centos-errata” project is installed to:

  • Main Dir: /opt/spacewalk-centos-errata/
    • com.redhat.rhsa-all.xml ⇒ File downloaded by the “errata-import.pl” script that contains Red Hat errata info.
    • errata-import.pl ⇒ main perl script that does the work
    • errata.latest.xml ⇒ File downloaded by the “errata-import.pl” script that contains CentOS errata info. (from the mailing lists)
    • errata-sync.sh ⇒ Configuration file and parent script that launches “errata-import.pl” with the proper Spacewalk credentials and channel IDs to scan.
      • Edit this file to make login credential changes or to include other channels for inclusion in errata scanning.
    • install.sh ⇒ Downloads the latest “errata-import” script, extracts, and creates the cron job
  • Cron Job installed to:
    • /etc/cron.d/spacewalk-centos-errata
      MAILTO=""
00 01 * * * root /bin/bash /opt/spacewalk-centos-errata/errata-sync.sh 2>&1 > /opt/spacewalk-centos-errata/errata.log

Config Management

A system is automatically subscribed to the proper configuration channels when it is registered via its Activation Key.

  • Configuration is NOT pushed to the system automatically.
  • The config files can be deployed while on the client system or pushed to the client using the Spacewalk server portal or spacecmd.

Compare Configs

To compare the centrally managed files to a system's local config files:

  • Login to the Spacewalk Web Portal
  • Find the target system via one of these methods:
    • Searching in the top right
    • Browsing all systems by clicking “Systems” on the left navigation
    • Browsing system groups
  • Click the systems name
  • On the systems Overview page, click on the “Configuration” tab underneath the system name.
  • On the right under “Configurable Actions”, click on “Compare all managed files to system”
  • Click “Schedule Compare”
    • Refresh the Configuration Overview page or click on the systems “Events” tab to watch status.
  • On the systems Configuration > Overview page, at the bottom under “Recent Events”, locate the “Last Spacewalk System Comparison”
    • Click the “View Details” link to see which files differ.
  • Under the Config Files list, click on the “Differences exist” link to view the differences.

Download (Pull) Configs

The various ways to download config files while on the client system.

Download all config files, from all subscribed config channels 

rhncfg-client get

Download a specific managed config file 

rhncfg-client get /etc/resolv.conf

Download all config files from a specific Config Channel ID 

for FILE in $(rhncfg-client list | awk /config-channel-id/'{print $3}'); do rhncfg-client get ${FILE}; done

Deploy (Push) Configs

To deploy configs from the server to a client.

Portal Deploy

  • Login to the Spacewalk Web Portal.
  • At the top, click on the “Systems” tab.
  • Find the target system via one of these methods:
    • Searching in the top right
    • Browsing all systems by clicking “Systems” on the left navigation
    • Browsing system groups
  • Click the systems name
  • On the systems Overview page, click on the “Configuration” tab underneath the system name.
  • Click “Deploy Files”
  • Check the config files to deploy, then click “Deploy Files” at the bottom right.
  • On the “Confirm Deploy Files” page, set a scheduled day/time or click “Schedule Deploy” to deploy immediately.

Spacecmd Deploy

List config channels a system is subscribed to 

spacecmd system_listconfigchannels

List config files that a system is subscribed to 

spacecmd system_listconfigfiles

Deploy all of those config files 

spacecmd system_deployconfigfiles <SYSTEMS>
  • <SYSTEMS> can be:
    • a single system name
    • multiple system names space separated
    • “group:GROUPNAME”

Create a Local Managed File Override

Some systems will need to have different config files than the centrally managed ones.

To create exceptions, or local managed overrides:

  • Login to the Spacewalk Web Portal.
  • Find the system (Systems tab at the top)
  • Click on the system name to go to its Overview page.

On the system's Details > Overview page:

  • Click the “Configuration” tab underneath the hostname (not the main Configuration tab in black up top) > “View/Modify Files”
  • To to right of the File Name to override, click “Override this file”
  • Click “Import Files” > scroll down under “Import Existing Files”
  • Check the file to override that exists on the system, click “Import Configuration Files”
  • After file has been successfully imported, click “Configuration” > “View/Modify Files” > “Local Sandbox”
  • Check the file > click “Copy Latest to System Channel”
  • The file will now show up under “Locally-Managed Files” and will NOT be over written by any centrally managed config file deploys.

Server Services

Normal Status of Spacewalk Services 

/usr/sbin/spacewalk-service status
 
postmaster (pid  29875) is running...
router (pid 31614) is running...
sm (pid 31622) is running...
c2s (pid 31630) is running...
s2s (pid 31638) is running...
tomcat6 (pid 29992) is running...                          [  OK  ]
httpd (pid  30115) is running...
osa-dispatcher (pid  31659) is running...
rhn-search is running (30168).
cobblerd (pid 30204) is running...
RHN Taskomatic is running (30236).

osa-dispatcher dead but pid file exists

If osa-dispatcher shows the following: 

/etc/init.d/osa-dispatcher status
 
osa-dispatcher dead but pid file exists

And the following error messages are in its log file: 

tail /var/log/rhn/osa-dispatcher.log
 
2015/11/03 07:38:05 -05:00 30144 0.0.0.0: osad/jabber_lib.__init__
2015/11/03 07:38:05 -05:00 30144 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'my-spacewalk-server.local')
2015/11/03 07:38:05 -05:00 30144 0.0.0.0: osad/jabber_lib.register('ERROR', 'Invalid password')

Fix this by stopping jabberd and osa-dispatcher (osa-dispatcher will probably show “Failed”): 

service jabberd stop
service osa-dispatcher stop

Remove jabberd database files: 

rm -rf /var/lib/jabberd/db/*

Start jabberd and osa-dispatcher 

service jabberd start
service osa-dispatcher start

Logs should now show the “Connected to jabber server” message: 

tail /var/log/rhn/osa-dispatcher.log
 
2015/11/03 08:19:43 -05:00 31657 0.0.0.0: osad/jabber_lib.__init__
2015/11/03 08:19:43 -05:00 31657 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'my-spacewalk-server.local')
2015/11/03 08:19:43 -05:00 31657 0.0.0.0: osad/osa_dispatcher.fix_connection('Upstream notification server started on port', 1290)
2015/11/03 08:19:43 -05:00 31657 0.0.0.0: osad/jabber_lib.process_forever

Warning

  • After recovering the jabberdb in this way, the osad clients on each system need to re-establish a connection. This is done by stopping the osad service on the clients, removing the osad-auth.conf file and starting osad again.
  • From a system that has spacecmd installed:
    for NODE in $(spacecmd system_list); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /sbin/service osad stop; sudo rm -vf /etc/sysconfig/rhn/osad-auth.conf; sudo /sbin/service osad start"; done

Spacewalk SSL Certificates

Updating the SSL Certificates on the Spacewalk server is more complex than just updating Apache, as the SSL certs are used for:

  • Spacewalk Portal (Apache httpd server)
  • Jabber local daemon components communication
  • Jabber Spacewalk client to Spacewalk server communication

Using the following RPM method will allow you to update all applications correctly at the same time.

Before manipulating either client or CA cert

  • SSH to the Spacewalk server and switch to root
  • Backup the current ssl-build directory (if it exists already)
    • cp -R /root/ssl-build /root/ssl-build.bak

Client Certificate

Client Certificate locations:

  • /etc/httpd/conf/ssl.crt/server.crt
  • /etc/httpd/conf/ssl.csr/server.csr
  • /etc/httpd/conf/ssl.key/server.key

Client Certificate Update Procedure

  • Order certificate renewal from certificate provider
  • Download certificate, copy to server's /root/ directory
  • SSH to the Spacewalk server and switch to root
  • Copy the current CA cert in use to the ssl-build directory
    • cp /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/
  • Copy NEW client certificate into ssl-build/my-spacewalk-server
    • cp server.crt /root/ssl-build/my-spacewalk-server/
  • Copy existing client key and CSR into ssl-build/my-spacewalk-server
    • cp /etc/httpd/conf/ssl.key/server.key /root/ssl-build/my-spacewalk-server/
cp /etc/httpd/conf/ssl.csr/server.csr /root/ssl-build/my-spacewalk-server/
  • Verify that NEW client cert will work with CA cert
    • openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
  • Generate the new client cert RPM
    • rhn-ssl-tool --gen-server --rpm-only --dir /root/ssl-build
  • Remove old SSL key pair package
    • rpm -e rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-1.noarch
  • Install new SSL key pair package
    • rpm -ivh /root/ssl-build/my-spacewalk-server/rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-2.noarch.rpm
  • Stop Spacewalk services, clear jabberd's scratch database, start the services
    • spacewalk-service stop
rm -rf /var/lib/jabberd/db/*
spacewalk-service start
  • Force an OSAD client re-authentication on each client
    for NODE in $(spacecmd system_list); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /sbin/service osad stop; sudo rm -vf /etc/sysconfig/rhn/osad-auth.conf; sudo /sbin/service osad start"; done

CA Certificate

CA Chain Certificate locations

  • RPM build location: /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Locally installed location: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
  • Publicly available for clients to download: /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
    • Also packaged in: /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm

Updating the CA certificate will not have to be done very often; only when:

  • CA cert expires
  • You change certificate providers

WARNING

  • Updating the CA certificate on the Spacewalk server will break all communication between the server and the clients.
  • Each client will need to update to the new CA cert individually before communication can be restored.

CA Certificate Update Procedure

  • Download the new single .pem file containing all the certs from the certificate provider.
  • Copy the PEM file to the Spacewalk server
  • SSH to the Spacewalk server and switch to root
  • Cat/view the contents of the PEM file
    • The top BEGIN/END block is the client cert (server.crt)
    • The rest is the certificate chain
      • Copy this into a new file; “RHN-ORG-TRUSTED-SSL-CERT”
  • Copy into ssl-build directory
    • cp RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Verify CA cert with the server cert
    • openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
  • Generate CA chain RPM
    • rhn-ssl-tool --gen-ca --rpm-only --dir /root/ssl-build
  • Copy new CA chain cert and RPM into Spacewalk's public directory (for client installation later)
    • cp /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub/
cp ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm /var/www/html/pub/
  • Install new CA chain cert on the Spacewalk server
    • rpm -ivh /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
  • Update the database
    • rhn-ssl-dbstore -vvv --ca-cert /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Stop the Spacewalk services, clear the jabberd scratch database, start services
    • spacewalk-service stop
rm -rf /var/lib/jabberd/db/*
spacewalk-service start
  • Login to each client and update the CA chain
    • rpm -ivh https://my-spacewalk-server.local/pub/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
      • Each client will have no communication to the Spacewalk server until this is complete.
  • Force an OSAD client re-authentication on each client
    for NODE in $(spacecmd system_list); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /sbin/service osad stop; sudo rm -vf /etc/sysconfig/rhn/osad-auth.conf; sudo /sbin/service osad start"; done
  • linux_wiki/spacewalk.1450757080.txt.gz
  • Last modified: 2019/05/25 23:50
  • (external edit)