====== Spacewalk ====== **General Information** Spacewalk is a centralized system update and config server.\\ * Official Site: https://github.com/spacewalkproject/spacewalk **Checklist** * Distro(s): Enterprise Linux * Other: [[https://github.com/spacewalkproject/spacewalk/wiki/HowToInstall|Spacewalk server]] setup ---- ====== Spacecmd ====== Spacecmd is the command line interface to Spacewalk.\\ Details here: [[https://www.owlbearconsulting.com/doku.php?id=linux_wiki:spacecmd_spacewalk_cmd_line|Spacecmd]] ---- ====== Register System with Spacewalk ====== A [[linux_wiki:spacewalk_register|Spacewalk registration script]] has been created to ease registration. ---- ===== Re-Register ===== If you need to re-register a client for any reason, you need the "--force" option when executing rhnreg_ks. * Delete system from Spacewalkspacecmd system_delete * Register system with the --force option sw_activation_key="1-my-system-key" sw_server="my-spacewalk-server.local" rhnreg_ks --force --serverUrl=https://${sw_server}/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=${sw_activation_key} ---- ====== Client Check Ins ====== * rhnsd => By default, a system will run the rhnsd daemon. * This daemon checks into Spacewalk via rhn_check every 240 mins (4 hours). It is tunable down to a minimum of 60 mins (1 hour). * cron => An alternative is to create a cron job to execute every 30 mins and run rhn_check. * If you do this, you can disable rhnsd, as it provides no other functionality than to run "run_check". * osad => Using osad on the client and osa-dispatcher on the server (with a jabberd daemon as well) is also available. * **This setup is fragile and not recommended**. If you do not want to wait for the next automatic check in (via rhnsd or cron), you can force a group of systems to check in by running the "rhn_check" command locally on that system.\\ To loop through a group of systems and have them check in: Example: Loop through the dev system group and have them check in for NODE in $(spacecmd group_listsystems dev); do echo "=>${NODE}"; ssh -qt ${NODE} "sudo /usr/sbin/rhn_check"; done ---- ===== rhnsd config ===== rhnsd is a daemon that will run rhn_check every 240 mins (by default). Configure: /etc/sysconfig/rhn/rhnsd INTERVAL=240 * Change INTERVAL time to check in time in minutes. Ensure it is enabled and started * EL7systemctl enable rhnsd systemctl start rhnsd * EL6chkconfig rhnsd on service rhnsd start ---- ===== rhn_check Cron Job ===== The alternative to using rhnsd (if you do not want a daemon running or desire more frequent check ins) is a cron job. Configure: /etc/cron.d/spacewalk-checkin # Do not e-mail root/anyone about this job MAILTO="" # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed # Check in every 30 mins */30 * * * * root /usr/sbin/rhn_check Optional: Disable rhnsd * EL7systemctl disable rhnsd systemctl stop rhnsd * EL6chkconfig rhnsd off service rhnsd stop ---- ===== osad ===== Another option for client communication is using the osad daemon (xmpp/jabber) on the client and osa-dispatcher on the server. * This type of setup is very fragile and needs constant babysitting. * The amount of administration overhead and how often it breaks is not worth the effort for faster execution of scheduled jobs. * **It is HIGHLY recommended to not go this route** and the rest of this documentation assumes rhnsd or the cron job with rhn_check. ---- ====== Channel Management ====== About Channels * Systems are subscribed to "Channels" * Channel subscriptions can be changed at any time in the Spacewalk portal, across any amount of systems. * Channels have Repositories assigned to them * This allows for a single repo to back multiple channels ---- ===== Channel Freezing/Snapshots ===== In order to facilitate the same updates being applied to the Development, System Test, and the Production environments, it is necessary to clone the original Channels.\\ This creates a "snapshot in time" of the available packages/errata on the date of clone.\\ **Note: This copies metadata of the Channel and does not duplicate repo packages** To Clone an entire Channel tree: * Login to a system with spacecmd installed * Clone the original base tree to a "snapshot-_" prefix tree * Clone can be performed with spacecmd in batch or interactive mode: * Batch Clone Example => Clone the CentOS 6 tree, giving it the prefix "ss-20151215_" (for the snapshot date)spacecmd softwarechannel_clonetree centos6_x86-64_base --prefix "ss-20151103_" --gpg-copy * The above will clone the entire tree(base and child channels), give the shown prefix, copy gpg data, and copy errata data. * Interactive Clone Examplespacecmd {SSM:0}> softwarechannel_clonetree Source Channels: centos6_x86-64_base centos7_x86-64_base Select source channel: centos6_x86-64_base Prefix: ss-20151215_ Copy source channel GPG details? [y/N]: y Original State (No Errata) [y/N]: N ---- ===== Errata Setup ===== As of 12/15/2015, CentOS does not generate an "updateinfo.xml" file in their repodata directories. This file is responsible for the package to errata mappings. (RHEL, Fedora, EPEL, and Oracle all do this) For a workaround, use a script to scrape the CentOS mailing archive lists for the errata. * Git hub project: https://github.com/mike-wendt/spacewalk-centos-errata * This is a bash based project that is a wrapper for the perl based project, making it easy to implement. * Original perl project's git page: https://github.com/stevemeier/cefs * Original perl project's main web page: http://cefs.steve-meier.de/ The "spacewalk-centos-errata" project is installed to: * Main Dir: /opt/spacewalk-centos-errata/ * com.redhat.rhsa-all.xml => File downloaded by the "errata-import.pl" script that contains Red Hat errata info. * errata-import.pl => main perl script that does the work * errata.latest.xml => File downloaded by the "errata-import.pl" script that contains CentOS errata info. (from the mailing lists) * **errata-sync.sh** => Configuration file and parent script that launches "errata-import.pl" with the proper Spacewalk credentials and channel IDs to scan. * **Edit this file to make login credential changes or to include other channels for inclusion in errata scanning.** * install.sh => Downloads the latest "errata-import" script, extracts, and creates the cron job * Cron Job installed to: * /etc/cron.d/spacewalk-centos-errataMAILTO="" 00 01 * * * root /bin/bash /opt/spacewalk-centos-errata/errata-sync.sh 2>&1 > /opt/spacewalk-centos-errata/errata.log ---- ====== Config Management ====== A system is automatically subscribed to the proper configuration channels when it is registered via its Activation Key. * Configuration is NOT pushed to the system automatically. * The config files can be deployed while on the client system or pushed to the client using the Spacewalk server portal or spacecmd. ---- ===== Compare Configs ===== To compare the centrally managed files to a system's local config files: * Login to the Spacewalk Web Portal * Find the target system via one of these methods: * Searching in the top right * Browsing all systems by clicking "Systems" on the left navigation * Browsing system groups * Click the systems name * On the systems Overview page, click on the "Configuration" tab underneath the system name. * On the right under "Configurable Actions", click on "Compare all managed files to system" * Click "Schedule Compare" * Refresh the Configuration Overview page or click on the systems "Events" tab to watch status. * On the systems Configuration > Overview page, at the bottom under "Recent Events", locate the "Last Spacewalk System Comparison" * Click the "View Details" link to see which files differ. * Under the Config Files list, click on the "Differences exist" link to view the differences. ---- ===== Download (Pull) Configs ===== The various ways to download config files while on the client system. Download all config files, from all subscribed config channels rhncfg-client get Download a specific managed config file rhncfg-client get /etc/resolv.conf Download all config files from a specific Config Channel ID for FILE in $(rhncfg-client list | awk /config-channel-id/'{print $3}'); do rhncfg-client get ${FILE}; done ---- ===== Deploy (Push) Configs ===== To deploy configs from the server to a client. ==== Portal Deploy ==== * Login to the Spacewalk Web Portal. * At the top, click on the "Systems" tab. * Find the target system via one of these methods: * Searching in the top right * Browsing all systems by clicking "Systems" on the left navigation * Browsing system groups * Click the systems name * On the systems Overview page, click on the "Configuration" tab underneath the system name. * Click "Deploy Files" * Check the config files to deploy, then click "Deploy Files" at the bottom right. * On the "Confirm Deploy Files" page, set a scheduled day/time or click "Schedule Deploy" to deploy immediately. ==== Spacecmd Deploy ==== List config channels a system is subscribed to spacecmd system_listconfigchannels List config files that a system is subscribed to spacecmd system_listconfigfiles Deploy all of those config files spacecmd system_deployconfigfiles * can be: * a single system name * multiple system names space separated * "group:GROUPNAME" ---- ===== Create a Local Managed File Override ===== Some systems will need to have different config files than the centrally managed ones. \\ To create exceptions, or local managed overrides: * Login to the Spacewalk Web Portal. * Find the system (Systems tab at the top) * Click on the system name to go to its Overview page. On the system's Details > Overview page: * Click the "Configuration" tab underneath the hostname (not the main Configuration tab in black up top) > "View/Modify Files" * To to right of the File Name to override, click "Override this file" * Click "Import Files" > scroll down under "Import Existing Files" * Check the file to override that exists on the system, click "Import Configuration Files" * After file has been successfully imported, click "Configuration" > "View/Modify Files" > "Local Sandbox" * Check the file > click "Copy Latest to System Channel" * The file will now show up under "Locally-Managed Files" and **will NOT be over written by any centrally managed config file deploys**. ---- ====== Server Services ====== Spacewalk server services. ===== Removing osa/jabber ===== We won't be using osa-dispatcher or jabberd services, so these can safely be disabled. * EL7systemctl disable osa-dispatcher systemctl disable jabberd systemctl stop osa-dispatcher systemctl stop jabberd * EL6chkconfig osa-dispatcher off chkconfig jabberd off service osa-dispatcher stop service jabberd stop Remove osa and jabber from the main spacewalk-service script. * Edit the script: vim /usr/sbin/spacewalk-service * Find the variable "SERVICES=" and remove "osa-dispatcher" and "jabberd" ---- ===== Normal Status of Spacewalk Services ===== After removing osa-dispatcher and jabberd, the status output looks like this: /usr/sbin/spacewalk-service status postmaster (pid 29875) is running... tomcat6 (pid 29992) is running... [ OK ] httpd (pid 30115) is running... rhn-search is running (30168). cobblerd (pid 30204) is running... RHN Taskomatic is running (30236). * postmaster => Spacewalk Postgres Database * tomcat6 => Spacewalk application * httpd => Spacewalk portal website * rhn-search => Searching functionality within the portal * cobblerd => Provisioning capability * RHN Taskomatic => Scheduled jobs viewable in the Spacewalk portal ---- ====== Spacewalk SSL Certificates ====== The SSL Certificates on the Spacewalk server is used for: * Spacewalk Portal (Apache httpd server) **Before manipulating either client or CA cert** * SSH to the Spacewalk server and switch to root * Backup the current ssl-build directory (if it exists already) * cp -R /root/ssl-build /root/ssl-build.bak ---- ===== Client Certificate ===== Client Certificate default locations: * /etc/httpd/conf/ssl.crt/server.crt * /etc/httpd/conf/ssl.csr/server.csr * /etc/httpd/conf/ssl.key/server.key Client Certificate Update Procedure * Order certificate renewal from certificate provider * Download certificate, copy to server's /root/ directory * SSH to the Spacewalk server and switch to root * Copy the current CA cert in use to the ssl-build directory * cp /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/ * Copy NEW client certificate into ssl-build/my-spacewalk-server * cp server.crt /root/ssl-build/my-spacewalk-server/ * Copy existing client key and CSR into ssl-build/my-spacewalk-server * cp /etc/httpd/conf/ssl.key/server.key /root/ssl-build/my-spacewalk-server/ cp /etc/httpd/conf/ssl.csr/server.csr /root/ssl-build/my-spacewalk-server/ * Verify that NEW client cert will work with CA cert * openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt * Generate the new client cert RPM * rhn-ssl-tool --gen-server --rpm-only --dir /root/ssl-build * Remove old SSL key pair package * rpm -e rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-1.noarch * Install new SSL key pair package * rpm -ivh /root/ssl-build/my-spacewalk-server/rhn-org-httpd-ssl-key-pair-my-spacewalk-server-1.0-2.noarch.rpm * Restart Spacewalk services * spacewalk-service restart ---- ===== CA Certificate ===== CA Chain Certificate locations * RPM build location: /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT * Locally installed location: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT * Publicly available for clients to download: /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT * Also packaged in: /var/www/html/pub/rhn-org-trusted-ssl-cert-1.0-1.noarch.rpm Updating the CA certificate will not have to be done very often; only when: * CA cert expires * You change certificate providers **WARNING** * Updating the CA certificate on the Spacewalk server will break all communication between the server and the clients. * Each client will need to update to the new CA cert individually before communication can be restored. CA Certificate Update Procedure * Download the new single .pem file containing all the certs from the certificate provider. * Copy the PEM file to the Spacewalk server * SSH to the Spacewalk server and switch to root * Cat/view the contents of the PEM file * The top BEGIN/END block is the client cert (server.crt) * The rest is the certificate chain * Copy this into a new file; "RHN-ORG-TRUSTED-SSL-CERT" * Copy into ssl-build directory * cp RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT * Verify CA cert with the server cert * openssl verify -CAfile /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /root/ssl-build/my-spacewalk-server/server.crt * Generate CA chain RPM * rhn-ssl-tool --gen-ca --rpm-only --dir /root/ssl-build * Copy new CA chain cert and RPM into Spacewalk's public directory (for client installation later) * cp /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT /var/www/html/pub/ cp ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm /var/www/html/pub/ * Install new CA chain cert on the Spacewalk server * rpm -ivh /root/ssl-build/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm * Update the database * rhn-ssl-dbstore -vvv --ca-cert /root/ssl-build/RHN-ORG-TRUSTED-SSL-CERT * Restart the Spacewalk services * spacewalk-service restart * **Login to each client and update the CA chain** * rpm -ivh https://my-spacewalk-server.local/pub/rhn-org-trusted-ssl-cert-1.0-2.noarch.rpm * Each client will have no communication to the Spacewalk server until this is complete. ----