====== Spacewalk ======
**General Information**
Spacewalk is a centralized system update and config server.\\
* Official Site: https://github.com/spacewalkproject/spacewalk
**Checklist**
* Distro(s): Enterprise Linux
* Other: [[https://github.com/spacewalkproject/spacewalk/wiki/HowToInstall|Spacewalk server]] setup
----
====== Spacecmd ======
Spacecmd is the command line interface to Spacewalk.\\
Details here: [[https://www.owlbearconsulting.com/doku.php?id=linux_wiki:spacecmd_spacewalk_cmd_line|Spacecmd]]
----
====== Register System with Spacewalk ======
A [[linux_wiki:spacewalk_register|Spacewalk registration script]] has been created to ease registration.
----
===== Re-Register =====
If you need to re-register a client for any reason, you need the "--force" option when executing rhnreg_ks.
* Delete system from Spacewalkspacecmd system_delete
* Register system with the --force option
sw_activation_key="1-my-system-key"
sw_server="my-spacewalk-server.local"
rhnreg_ks --force --serverUrl=https://${sw_server}/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=${sw_activation_key}
----
====== Client Check Ins ======
* rhnsd => By default, a system will run the rhnsd daemon.
* This daemon checks into Spacewalk via rhn_check every 240 mins (4 hours). It is tunable down to a minimum of 60 mins (1 hour).
* cron => An alternative is to create a cron job to execute every 30 mins and run rhn_check.
* If you do this, you can disable rhnsd, as it provides no other functionality than to run "run_check".
* osad => Using osad on the client and osa-dispatcher on the server (with a jabberd daemon as well) is also available.
* **This setup is fragile and not recommended**.
If you do not want to wait for the next automatic check in (via rhnsd or cron), you can force a group of systems to check in by running the "rhn_check" command locally on that system.\\
To loop through a group of systems and have them check in:
Example: Loop through the dev system group and have them check in
for NODE in $(spacecmd group_listsystems dev); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /usr/sbin/rhn_check"; done
----
===== rhnsd config =====
rhnsd is a daemon that will run rhn_check every 240 mins (by default).
Configure: /etc/sysconfig/rhn/rhnsd
INTERVAL=240
* Change INTERVAL time to check in time in minutes.
Ensure it is enabled and started
* EL7systemctl enable rhnsd
systemctl start rhnsd
* EL6chkconfig rhnsd on
service rhnsd start
----
===== rhn_check Cron Job =====
The alternative to using rhnsd (if you do not want a daemon running or desire more frequent check ins) is a cron job.
Configure: /etc/cron.d/spacewalk-checkin
# Do not e-mail root/anyone about this job
MAILTO=""
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
# Check in every 30 mins
*/30 * * * * root /usr/sbin/rhn_check
Optional: Disable rhnsd
* EL7systemctl disable rhnsd
systemctl stop rhnsd
* EL6chkconfig rhnsd off
service rhnsd stop
----
===== osad =====
Another option for client communication is using the osad daemon (xmpp/jabber) on the client and osa-dispatcher on the server.
* This type of setup is very fragile and needs constant babysitting.
* The amount of administration overhead and how often it breaks is not worth the effort for faster execution of scheduled jobs.
* **It is HIGHLY recommended to not go this route** and the rest of this documentation assumes rhnsd or the cron job with rhn_check.
----
====== Channel Management ======
About Channels
* Systems are subscribed to "Channels"
* Channel subscriptions can be changed at any time in the Spacewalk portal, across any amount of systems.
* Channels have Repositories assigned to them
* This allows for a single repo to back multiple channels
----
===== Channel Freezing/Snapshots =====
In order to facilitate the same updates being applied to the Development, System Test, and the Production environments, it is necessary to clone the original Channels.\\
This creates a "snapshot in time" of the available packages/errata on the date of clone.\\
**Note: This copies metadata of the Channel and does not duplicate repo packages**
To Clone an entire Channel tree:
* Login to a system with spacecmd installed
* Clone the original base tree to a "snapshot-_" prefix tree
* Clone can be performed with spacecmd in batch or interactive mode:
* Batch Clone Example => Clone the CentOS 6 tree, giving it the prefix "ss-20151215_" (for the snapshot date)spacecmd softwarechannel_clonetree centos6_x86-64_base --prefix "ss-20151103_" --gpg-copy
* The above will clone the entire tree(base and child channels), give the shown prefix, copy gpg data, and copy errata data.
* Interactive Clone Examplespacecmd {SSM:0}> softwarechannel_clonetree
Source Channels:
centos6_x86-64_base
centos7_x86-64_base
Select source channel: centos6_x86-64_base
Prefix: ss-20151215_
Copy source channel GPG details? [y/N]: y
Original State (No Errata) [y/N]: N
----
===== Errata Setup =====
As of 12/15/2015, CentOS does not generate an "updateinfo.xml" file in their repodata directories. This file is responsible for the package to errata mappings. (RHEL, Fedora, EPEL, and Oracle all do this)
For a workaround, use a script to scrape the CentOS mailing archive lists for the errata.
* Git hub project: https://github.com/mike-wendt/spacewalk-centos-errata
* This is a bash based project that is a wrapper for the perl based project, making it easy to implement.
* Original perl project's git page: https://github.com/stevemeier/cefs
* Original perl project's main web page: http://cefs.steve-meier.de/
The "spacewalk-centos-errata" project is installed to:
* Main Dir: /opt/spacewalk-centos-errata/
* com.redhat.rhsa-all.xml => File downloaded by the "errata-import.pl" script that contains Red Hat errata info.
* errata-import.pl => main perl script that does the work
* errata.latest.xml => File downloaded by the "errata-import.pl" script that contains CentOS errata info. (from the mailing lists)
* **errata-sync.sh** => Configuration file and parent script that launches "errata-import.pl" with the proper Spacewalk credentials and channel IDs to scan.
* **Edit this file to make login credential changes or to include other channels for inclusion in errata scanning.**
* install.sh => Downloads the latest "errata-import" script, extracts, and creates the cron job
* Cron Job installed to:
* /etc/cron.d/spacewalk-centos-errataMAILTO=""
00 01 * * * root /bin/bash /opt/spacewalk-centos-errata/errata-sync.sh 2>&1 > /opt/spacewalk-centos-errata/errata.log
----
====== Config Management ======
A system is automatically subscribed to the proper configuration channels when it is registered via its Activation Key.
* Configuration is NOT pushed to the system automatically.
* The config files can be deployed while on the client system or pushed to the client using the Spacewalk server portal or spacecmd.
----
===== Compare Configs =====
To compare the centrally managed files to a system's local config files:
* Login to the Spacewalk Web Portal
* Find the target system via one of these methods:
* Searching in the top right
* Browsing all systems by clicking "Systems" on the left navigation
* Browsing system groups
* Click the systems name
* On the systems Overview page, click on the "Configuration" tab underneath the system name.
* On the right under "Configurable Actions", click on "Compare all managed files to system"
* Click "Schedule Compare"
* Refresh the Configuration Overview page or click on the systems "Events" tab to watch status.
* On the systems Configuration > Overview page, at the bottom under "Recent Events", locate the "Last Spacewalk System Comparison"
* Click the "View Details" link to see which files differ.
* Under the Config Files list, click on the "Differences exist" link to view the differences.
----
===== Download (Pull) Configs =====
The various ways to download config files while on the client system.
Download all config files, from all subscribed config channels
rhncfg-client get
Download a specific managed config file
rhncfg-client get /etc/resolv.conf
Download all config files from a specific Config Channel ID
for FILE in $(rhncfg-client list | awk /config-channel-id/'{print $3}'); do rhncfg-client get ${FILE}; done
----
===== Deploy (Push) Configs =====
To deploy configs from the server to a client.
==== Portal Deploy ====
* Login to the Spacewalk Web Portal.
* At the top, click on the "Systems" tab.
* Find the target system via one of these methods:
* Searching in the top right
* Browsing all systems by clicking "Systems" on the left navigation
* Browsing system groups
* Click the systems name
* On the systems Overview page, click on the "Configuration" tab underneath the system name.
* Click "Deploy Files"
* Check the config files to deploy, then click "Deploy Files" at the bottom right.
* On the "Confirm Deploy Files" page, set a scheduled day/time or click "Schedule Deploy" to deploy immediately.
==== Spacecmd Deploy ====
List config channels a system is subscribed to
spacecmd system_listconfigchannels
List config files that a system is subscribed to
spacecmd system_listconfigfiles
Deploy all of those config files
spacecmd system_deployconfigfiles
* can be:
* a single system name
* multiple system names space separated
* "group:GROUPNAME"
----
===== Create a Local Managed File Override =====
Some systems will need to have different config files than the centrally managed ones.
\\
To create exceptions, or local managed overrides:
* Login to the Spacewalk Web Portal.
* Find the system (Systems tab at the top)
* Click on the system name to go to its Overview page.
On the system's Details > Overview page:
* Click the "Configuration" tab underneath the hostname (not the main Configuration tab in black up top) > "View/Modify Files"
* To to right of the File Name to override, click "Override this file"
* Click "Import Files" > scroll down under "Import Existing Files"
* Check the file to override that exists on the system, click "Import Configuration Files"
* After file has been successfully imported, click "Configuration" > "View/Modify Files" > "Local Sandbox"
* Check the file > click "Copy Latest to System Channel"
* The file will now show up under "Locally-Managed Files" and **will NOT be over written by any centrally managed config file deploys**.
----
====== Server Services ======
Spacewalk server services.
===== Removing osa/jabber =====
We won't be using osa-dispatcher or jabberd services, so these can safely be disabled.
* EL7systemctl disable osa-dispatcher
systemctl disable jabberd
systemctl stop osa-dispatcher
systemctl stop jabberd
* EL6chkconfig osa-dispatcher off
chkconfig jabberd off
service osa-dispatcher stop
service jabberd stop
Remove osa and jabber from the main spacewalk-service script.
* Edit the script: vim /usr/sbin/spacewalk-service
* Find the variable "SERVICES=" and remove "osa-dispatcher" and "jabberd"
----
===== Normal Status of Spacewalk Services =====
After removing osa-dispatcher and jabberd, the status output looks like this:
/usr/sbin/spacewalk-service status
postmaster (pid 29875) is running...
tomcat6 (pid 29992) is running... [ OK ]
httpd (pid 30115) is running...
rhn-search is running (30168).
cobblerd (pid 30204) is running...
RHN Taskomatic is running (30236).
* postmaster => Spacewalk Postgres Database
* tomcat6 => Spacewalk application
* httpd => Spacewalk portal website
* rhn-search => Searching functionality within the portal
* cobblerd => Provisioning capability
* RHN Taskomatic => Scheduled jobs viewable in the Spacewalk portal
----
====== Spacewalk SSL Certificates ======
The SSL Certificates on the Spacewalk server is used for:
* Spacewalk Portal (Apache httpd server)
**Before manipulating either client or CA cert**
* SSH to the Spacewalk server and switch to root
* Backup the current ssl-build directory (if it exists already)
* cp -R /root/ssl-build /root/ssl-build.bak
----
===== Client Certificate =====
Client Certificate default locations:
* /etc/httpd/conf/ssl.crt/server.crt
* /etc/httpd/conf/ssl.csr/server.csr
* /etc/httpd/conf/ssl.key/server.key
Client Certificate Update Procedure
* Order certificate renewal from certificate provider
* Download certificate, copy to server's /root/ directory
* SSH to the Spacewalk server and switch to root
* Copy the current CA cert in use to the ssl-build directory
* cp /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/
* Copy NEW client certificate into ssl-build/my-spacewalk-server
* cp server.crt /root/ssl-build/my-spacewalk-server/
* Copy existing client key and CSR into ssl-build/my-spacewalk-server
* cp /etc/httpd/conf/ssl.key/server.key /root/ssl-build/my-spacewalk-server/
cp /etc/httpd/conf/ssl.csr/server.csr /root/ssl-build/my-spacewalk-server/
* Verify that NEW client cert will work with CA cert
* openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
* Generate the new client cert RPM
* rhn-ssl-tool --gen-server --rpm-only --dir /root/ssl-build
* Remove old SSL key pair package
* rpm -e rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-1.noarch
* Install new SSL key pair package
* rpm -ivh /root/ssl-build/my-spacewalk-server/rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-2.noarch.rpm
* Restart Spacewalk services
* spacewalk-service restart
----
===== CA Certificate =====
CA Chain Certificate locations
* RPM build location: /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
* Locally installed location: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
* Publicly available for clients to download: /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT
* Also packaged in: /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm
Updating the CA certificate will not have to be done very often; only when:
* CA cert expires
* You change certificate providers
**WARNING**
* Updating the CA certificate on the Spacewalk server will break all communication between the server and the clients.
* Each client will need to update to the new CA cert individually before communication can be restored.
CA Certificate Update Procedure
* Download the new single .pem file containing all the certs from the certificate provider.
* Copy the PEM file to the Spacewalk server
* SSH to the Spacewalk server and switch to root
* Cat/view the contents of the PEM file
* The top BEGIN/END block is the client cert (server.crt)
* The rest is the certificate chain
* Copy this into a new file; "RHN-ORG-TRUSTED-SSL-CERT"
* Copy into ssl-build directory
* cp RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
* Verify CA cert with the server cert
* openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt
* Generate CA chain RPM
* rhn-ssl-tool --gen-ca --rpm-only --dir /root/ssl-build
* Copy new CA chain cert and RPM into Spacewalk's public directory (for client installation later)
* cp /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub/
cp ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm /var/www/html/pub/
* Install new CA chain cert on the Spacewalk server
* rpm -ivh /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
* Update the database
* rhn-ssl-dbstore -vvv --ca-cert /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT
* Restart the Spacewalk services
* spacewalk-service restart
* **Login to each client and update the CA chain**
* rpm -ivh https://my-spacewalk-server.local/pub/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm
* Each client will have no communication to the Spacewalk server until this is complete.
----