linux_wiki:spacewalk

This is an old revision of the document!

Spacewalk

General Information

Spacewalk is a centralized system update and config server.

Checklist

Spacecmd

Spacecmd is the command line interface to Spacewalk.
Details here: Spacecmd

Register System with Spacewalk

A Spacewalk registration script has been created to ease registration.

Re-Register

If you need to re-register a client for any reason, you need the “–force” option when executing rhnreg_ks.

  • Delete system from Spacewalk
    spacecmd system_delete <SYSTEM>
  • Register system with the –force option
    sw_activation_key="1-my-system-key"
sw_server="my-spacewalk-server.local"
rhnreg_ks --force --serverUrl=https://${sw_server}/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=${sw_activation_key}

Client Check Ins

  • rhnsd ⇒ By default, a system will run the rhnsd daemon.
    • This daemon checks into Spacewalk via rhn_check every 240 mins (4 hours). It is tunable down to a minimum of 60 mins (1 hour).
  • cron ⇒ An alternative is to create a cron job to execute every 30 mins and run rhn_check.
    • If you do this, you can disable rhnsd, as it provides no other functionality than to run “run_check”.
  • osad ⇒ Using osad on the client and osa-dispatcher on the server (with a jabberd daemon as well) is also available.
    • This setup is fragile and not recommended.

If you do not want to wait for the next automatic check in (via rhnsd or cron), you can force a group of systems to check in by running the “rhn_check” command locally on that system.
To loop through a group of systems and have them check in:

Example: Loop through the dev system group and have them check in 

for NODE in $(spacecmd group_listsystems dev); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /usr/sbin/rhn_check"; done

rhnsd config

rhnsd is a daemon that will run rhn_check every 240 mins (by default).

Configure: /etc/sysconfig/rhn/rhnsd 

INTERVAL=240
  • Change INTERVAL time to check in time in minutes.

Ensure it is enabled and started

  • EL7
    systemctl enable rhnsd
systemctl start rhnsd
  • EL6
    chkconfig rhnsd on
service rhnsd start

rhn_check Cron Job

The alternative to using rhnsd (if you do not want a daemon running or desire more frequent check ins) is a cron job.

Configure: /etc/cron.d/spacewalk-checkin 

# Do not e-mail root/anyone about this job
MAILTO=""
 
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
# Check in every 30 mins
*/30 * * * * root /usr/sbin/rhn_check

Optional: Disable rhnsd

  • EL7
    systemctl disable rhnsd
systemctl stop rhnsd
  • EL6
    chkconfig rhnsd off
service rhnsd stop

osad

Another option for client communication is using the osad daemon (xmpp/jabber) on the client and osa-dispatcher on the server.

  • This type of setup is very fragile and needs constant babysitting.
  • The amount of administration overhead and how often it breaks is not worth the effort for faster execution of scheduled jobs.
  • It is HIGHLY recommended to not go this route and the rest of this documentation assumes rhnsd or the cron job with rhn_check.

Channel Management

About Channels

  • Systems are subscribed to “Channels”
  • Channel subscriptions can be changed at any time in the Spacewalk portal, across any amount of systems.
  • Channels have Repositories assigned to them
  • This allows for a single repo to back multiple channels

Channel Freezing/Snapshots

In order to facilitate the same updates being applied to the Development, System Test, and the Production environments, it is necessary to clone the original Channels.
This creates a “snapshot in time” of the available packages/errata on the date of clone.
Note: This copies metadata of the Channel and does not duplicate repo packages

To Clone an entire Channel tree:

  • Login to a system with spacecmd installed
  • Clone the original base tree to a “snapshot-<date>_” prefix tree
  • Clone can be performed with spacecmd in batch or interactive mode:
    • Batch Clone Example ⇒ Clone the CentOS 6 tree, giving it the prefix “ss-20151215_” (for the snapshot date)
      spacecmd softwarechannel_clonetree centos6_x86-64_base --prefix "ss-20151103_" --gpg-copy
      • The above will clone the entire tree(base and child channels), give the shown prefix, copy gpg data, and copy errata data.
    • Interactive Clone Example
      spacecmd {SSM:0}> softwarechannel_clonetree
Source Channels:
centos6_x86-64_base
centos7_x86-64_base
 
Select source channel: centos6_x86-64_base
Prefix: ss-20151215_
 
Copy source channel GPG details? [y/N]: y
 
Original State (No Errata) [y/N]: N

Errata Setup

As of 12/15/2015, CentOS does not generate an “updateinfo.xml” file in their repodata directories. This file is responsible for the package to errata mappings. (RHEL, Fedora, EPEL, and Oracle all do this)

For a workaround, use a script to scrape the CentOS mailing archive lists for the errata.

The “spacewalk-centos-errata” project is installed to:

  • Main Dir: /opt/spacewalk-centos-errata/
    • com.redhat.rhsa-all.xml ⇒ File downloaded by the “errata-import.pl” script that contains Red Hat errata info.
    • errata-import.pl ⇒ main perl script that does the work
    • errata.latest.xml ⇒ File downloaded by the “errata-import.pl” script that contains CentOS errata info. (from the mailing lists)
    • errata-sync.sh ⇒ Configuration file and parent script that launches “errata-import.pl” with the proper Spacewalk credentials and channel IDs to scan.
      • Edit this file to make login credential changes or to include other channels for inclusion in errata scanning.
    • install.sh ⇒ Downloads the latest “errata-import” script, extracts, and creates the cron job
  • Cron Job installed to:
    • /etc/cron.d/spacewalk-centos-errata
      MAILTO=""
00 01 * * * root /bin/bash /opt/spacewalk-centos-errata/errata-sync.sh 2>&1 > /opt/spacewalk-centos-errata/errata.log

Config Management

A system is automatically subscribed to the proper configuration channels when it is registered via its Activation Key.

  • Configuration is NOT pushed to the system automatically.
  • The config files can be deployed while on the client system or pushed to the client using the Spacewalk server portal or spacecmd.

Compare Configs

To compare the centrally managed files to a system's local config files:

  • Login to the Spacewalk Web Portal
  • Find the target system via one of these methods:
    • Searching in the top right
    • Browsing all systems by clicking “Systems” on the left navigation
    • Browsing system groups
  • Click the systems name
  • On the systems Overview page, click on the “Configuration” tab underneath the system name.
  • On the right under “Configurable Actions”, click on “Compare all managed files to system”
  • Click “Schedule Compare”
    • Refresh the Configuration Overview page or click on the systems “Events” tab to watch status.
  • On the systems Configuration > Overview page, at the bottom under “Recent Events”, locate the “Last Spacewalk System Comparison”
    • Click the “View Details” link to see which files differ.
  • Under the Config Files list, click on the “Differences exist” link to view the differences.

Download (Pull) Configs

The various ways to download config files while on the client system.

Download all config files, from all subscribed config channels 

rhncfg-client get

Download a specific managed config file 

rhncfg-client get /etc/resolv.conf

Download all config files from a specific Config Channel ID 

for FILE in $(rhncfg-client list | awk /config-channel-id/'{print $3}'); do rhncfg-client get ${FILE}; done

Deploy (Push) Configs

To deploy configs from the server to a client.

Portal Deploy

  • Login to the Spacewalk Web Portal.
  • At the top, click on the “Systems” tab.
  • Find the target system via one of these methods:
    • Searching in the top right
    • Browsing all systems by clicking “Systems” on the left navigation
    • Browsing system groups
  • Click the systems name
  • On the systems Overview page, click on the “Configuration” tab underneath the system name.
  • Click “Deploy Files”
  • Check the config files to deploy, then click “Deploy Files” at the bottom right.
  • On the “Confirm Deploy Files” page, set a scheduled day/time or click “Schedule Deploy” to deploy immediately.

Spacecmd Deploy

List config channels a system is subscribed to 

spacecmd system_listconfigchannels

List config files that a system is subscribed to 

spacecmd system_listconfigfiles

Deploy all of those config files 

spacecmd system_deployconfigfiles <SYSTEMS>
  • <SYSTEMS> can be:
    • a single system name
    • multiple system names space separated
    • “group:GROUPNAME”

Create a Local Managed File Override

Some systems will need to have different config files than the centrally managed ones.

To create exceptions, or local managed overrides:

  • Login to the Spacewalk Web Portal.
  • Find the system (Systems tab at the top)
  • Click on the system name to go to its Overview page.

On the system's Details > Overview page:

  • Click the “Configuration” tab underneath the hostname (not the main Configuration tab in black up top) > “View/Modify Files”
  • To to right of the File Name to override, click “Override this file”
  • Click “Import Files” > scroll down under “Import Existing Files”
  • Check the file to override that exists on the system, click “Import Configuration Files”
  • After file has been successfully imported, click “Configuration” > “View/Modify Files” > “Local Sandbox”
  • Check the file > click “Copy Latest to System Channel”
  • The file will now show up under “Locally-Managed Files” and will NOT be over written by any centrally managed config file deploys.

Server Services

Spacewalk server services.

Removing osa/jabber

We won't be using osa-dispatcher or jabberd services, so these can safely be disabled.

  • EL7
    systemctl disable osa-dispatcher
systemctl disable jabberd
systemctl stop osa-dispatcher
systemctl stop jabberd
  • EL6
    chkconfig osa-dispatcher off
chkconfig jabberd off
service osa-dispatcher stop
service jabberd stop

Remove osa and jabber from the main spacewalk-service script.

  • Edit the script: vim /usr/sbin/spacewalk-service
  • Find the variable “SERVICES=” and remove “osa-dispatcher” and “jabberd”

Normal Status of Spacewalk Services

After removing osa-dispatcher and jabberd, the status output looks like this: 

/usr/sbin/spacewalk-service status
 
postmaster (pid  29875) is running...
tomcat6 (pid 29992) is running...                          [  OK  ]
httpd (pid  30115) is running...
rhn-search is running (30168).
cobblerd (pid 30204) is running...
RHN Taskomatic is running (30236).
  • postmaster ⇒ Spacewalk Postgres Database
  • tomcat6 ⇒ Spacewalk application
  • httpd ⇒ Spacewalk portal website
  • rhn-search ⇒ Searching functionality within the portal
  • cobblerd ⇒ Provisioning capability
  • RHN Taskomatic ⇒ Scheduled jobs viewable in the Spacewalk portal

Spacewalk SSL Certificates

The SSL Certificates on the Spacewalk server is used for:

  • Spacewalk Portal (Apache httpd server)

Before manipulating either client or CA cert

  • SSH to the Spacewalk server and switch to root
  • Backup the current ssl-build directory (if it exists already)
    • cp -R /root/ssl-build /root/ssl-build.bak

Client Certificate

Client Certificate default locations:

  • /etc/httpd/conf/ssl.crt/server.crt
  • /etc/httpd/conf/ssl.csr/server.csr
  • /etc/httpd/conf/ssl.key/server.key

Client Certificate Update Procedure

  • Order certificate renewal from certificate provider
  • Download certificate, copy to server's /root/ directory
  • SSH to the Spacewalk server and switch to root
  • Copy the current CA cert in use to the ssl-build directory
    • cp /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/
  • Copy NEW client certificate into ssl-build/my-spacewalk-server
    • cp server.crt /root/ssl-build/my-spacewalk-server/
  • Copy existing client key and CSR into ssl-build/my-spacewalk-server
    • cp /etc/httpd/conf/ssl.key/server.key /root/ssl-build/my-spacewalk-server/
cp /etc/httpd/conf/ssl.csr/server.csr /root/ssl-build/my-spacewalk-server/
  • Verify that NEW client cert will work with CA cert
    • openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
  • Generate the new client cert RPM
    • rhn-ssl-tool --gen-server --rpm-only --dir /root/ssl-build
  • Remove old SSL key pair package
    • rpm -e rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-1.noarch
  • Install new SSL key pair package
    • rpm -ivh /root/ssl-build/my-spacewalk-server/rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-2.noarch.rpm
  • Restart Spacewalk services
    • spacewalk-service restart

CA Certificate

CA Chain Certificate locations

  • RPM build location: /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Locally installed location: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
  • Publicly available for clients to download: /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
    • Also packaged in: /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm

Updating the CA certificate will not have to be done very often; only when:

  • CA cert expires
  • You change certificate providers

WARNING

  • Updating the CA certificate on the Spacewalk server will break all communication between the server and the clients.
  • Each client will need to update to the new CA cert individually before communication can be restored.

CA Certificate Update Procedure

  • Download the new single .pem file containing all the certs from the certificate provider.
  • Copy the PEM file to the Spacewalk server
  • SSH to the Spacewalk server and switch to root
  • Cat/view the contents of the PEM file
    • The top BEGIN/END block is the client cert (server.crt)
    • The rest is the certificate chain
      • Copy this into a new file; “RHN-ORG-TRUSTED-SSL-CERT”
  • Copy into ssl-build directory
    • cp RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Verify CA cert with the server cert
    • openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
  • Generate CA chain RPM
    • rhn-ssl-tool --gen-ca --rpm-only --dir /root/ssl-build
  • Copy new CA chain cert and RPM into Spacewalk's public directory (for client installation later)
    • cp /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub/
cp ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm /var/www/html/pub/
  • Install new CA chain cert on the Spacewalk server
    • rpm -ivh /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
  • Update the database
    • rhn-ssl-dbstore -vvv --ca-cert /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
  • Restart the Spacewalk services
    • spacewalk-service restart
  • Login to each client and update the CA chain
    • rpm -ivh https://my-spacewalk-server.local/pub/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
      • Each client will have no communication to the Spacewalk server until this is complete.
  • linux_wiki/spacewalk.1458612826.txt.gz
  • Last modified: 2019/05/25 23:50
  • (external edit)