Show pageOld revisionsBacklinksBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ====== Ansible AWX ====== **General Information** Installation and operational notes for Ansible AWX (Tower). "AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived." Resources * Github Project: https://github.com/ansible/awx * Ansible AWX FAQ: https://www.ansible.com/products/awx-project/faq * Copr RPM Repo: https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/ * Github for Copr RPM repo: https://github.com/MrMEEE/awx-build ---- ====== Install ====== Start with a CentOS 7 minimal install. \\ Postgresql 9.6 required for AWX. Add the repo <code bash>yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm</code> \\ Install Postgresql 9.6 Database and other required packages <code bash> yum install postgresql96-server rabbitmq-server wget memcached nginx </code> \\ Add AWX Dev Repo <code bash> wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo </code> \\ Install AWX <code bash> yum install awx </code> ---- ====== Initial Setup ====== Initialize the database <code bash> /usr/pgsql-9.6/bin/postgresql96-setup initdb </code> \\ Configure memcached to listen locally <code bash> vim /etc/sysconfig/memcached # Daemon USER="memcached" # Reserved Cache in MBs (Default: 64) CACHESIZE="512" # Memcached Options - Listen on localhost only OPTIONS="-l 127.0.0.1" # Networking PORT="11211" MAXCONN="1024" </code> \\ Configure rabbitmq-server to listen locally <code bash> vim /etc/rabbitmq/rabbitmq.config # Uncomment the following (and delete trailing comma in ipv6 line) {tcp_listeners, [{"127.0.0.1", 5672}, {"::1", 5672}]} </code> ---- ===== Services ===== Start and Enable some services. \\ Start/Enable Rabbit <code bash> systemctl start rabbitmq-server systemctl enable rabbitmq-server </code> \\ Start/Enable Memcached <code bash> systemctl start memcached systemctl enable memcached </code> \\ Start/Enable Postgresql <code bash> systemctl start postgresql-9.6 systemctl enable postgresql-9.6 </code> ---- ===== Database Setup ===== Create Postgres user (awx) and database <code bash> su - postgres createuser -S awx createdb -O awx awx exit </code> \\ Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line<code bash>vim /etc/awx/settings.py #CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),) </code> * If you don't comment out the above line, migrations next will fail with an error. \\ Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines<code bash>vim /etc/awx/settings.py #CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID} #CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}</code> \\ Migrate AWX App data into the database (fyi; this is a Django app) <code bash> sudo -u awx /opt/awx/bin/awx-manage migrate </code> \\ Initialize the AWX Django App: Create admin user <code bash> echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell </code> \\ Initialize the AWX Django App: Add tower instance (AWX Server) <code bash> sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname) </code> \\ Initialize the AWX Django App: Create some pre-loaded organization data <code bash> sudo -u awx /opt/awx/bin/awx-manage create_preload_data </code> \\ Initialize the AWX Django App: Create a queue group <code bash> sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname) </code> ---- ===== Proxy Setup ===== Nginx will act as the proxy to the AWX application. \\ Configure Nginx - Main Config (/etc/nginx/nginx.conf) <code bash> ## NGINX - Main Configuration ## # Context: Main - General Server Configuration # User that worker processes run as user nginx; # Number of worker processes (auto = set to number of CPUs) worker_processes auto; # Error Log and PID of main process error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; # Context: Events - Connection Processing events { # Max number of connections per worker process worker_connections 1024; } # Context: HTTP - HTTP Server Directives http { # MIME - Include file and default type include /etc/nginx/mime.types; default_type application/octet-stream; # Logging: Format and Main Access Log log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; # server_tokens off - Disable nginx version on error pages and response headers server_tokens off; ## Headers - Add additional headers ## # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin add_header X-Frame-Options SAMEORIGIN; # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks add_header X-Content-Type-Options nosniff; # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting # 1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected add_header X-XSS-Protection "1; mode=block" always; # Content-Security-Policy -> Prevent XSS, clickjacking, code injection add_header Content-Security-Policy "default-src 'self';" always; # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off, # then tcp_nodelay forces the fast sending of the last data # Sendfile - Send files directly in kernel space # on -> keep on for locally stored files # off -> turn off for files served over network mounted storage sendfile on; # tcp_nopush - Do not send data until packet reaches MSS # Dependency: sendfile MUST be on for this to work #tcp_nopush on; # tcp_nodelay - Send packets in buffer as soon as they are available #tcp_nodelay on; # Server side keepalive timeout in seconds (default: 75) keepalive_timeout 65; # Gzip - Compress responses using gzip #gzip on; # AWX ADDED: Connection upgrade map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Include enabled configurations include /etc/nginx/conf.d/enabled/*.conf; # AWX ADDED: Upstream Apps upstream uwsgi { server 127.0.0.1:8050; } upstream daphne { server 127.0.0.1:8051; } } </code> \\ Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf) <code bash> ## Default Config - Catch All Matches ## # HTTP (Port 80) server { listen 80 default_server; server_name _; # Redirect everything to HTTPS return 301 https://$http_host$request_uri; } # HTTPS (Port 443) server { listen 443 ssl default_server; listen [::]:443 ssl default_server; server_name _; # HSTS (HTTPS Strict Transport Security) # 63072000 seconds = 2 years add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; "; # SSL - Certificate Config ssl on; ssl_certificate /etc/pki/tls/current_cert.crt; ssl_certificate_key /etc/pki/tls/current_key.key; ssl_client_certificate /etc/pki/tls/current_ca.crt; # SSL - Session Config ssl_session_timeout 5m; ssl_session_cache shared:SSL:50m; # SSL - Protocols and Ciphers ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM"; # Locations for AWX location /static/ { alias /opt/awx/static/; } location /favicon.ico { alias /opt/awx/static/favicon.ico; } location /websocket { # Pass request to the upstream alias proxy_pass http://daphne; # Require http version 1.1 to allow for upgrade requests proxy_http_version 1.1; # We want proxy_buffering off for proxying to websockets. proxy_buffering off; # http://en.wikipedia.org/wiki/X-Forwarded-For proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # enable this if you use HTTPS: proxy_set_header X-Forwarded-Proto https; # pass the Host: header from the client for the sake of redirects proxy_set_header Host $http_host; # We've set the Host header, so we don't need Nginx to muddle # about with redirects proxy_redirect off; # Depending on the request value, set the Upgrade and # connection headers proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; } # Location: Webserver root location / { # autoindex off - Disable directory listing output autoindex off; uwsgi_read_timeout 120s; uwsgi_pass uwsgi; include /etc/nginx/uwsgi_params; } } </code> \\ Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file) * /etc/pki/tls/current_cert.crt * /etc/pki/tls/current_key.key * /etc/pki/tls/current_ca.crt \\ Start/Enable Nginx <code bash> systemctl start nginx systemctl enable nginx </code> ---- ===== AWX Services ===== Start/Enable AWX Services <code bash> systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web </code> ---- ====== Upgrade Steps ====== **WARNING:** Upgrading to the newest AWX is **not guaranteed to work and might break your install**. The project is fast moving and does not currently support upgrade paths. ===== AWX RPM Method ===== To upgrade: * Stop all services<code bash>systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code> * Upgrade AWX<code bash>yum update awx</code> * Make Migrations<code bash>sudo -u awx /opt/awx/bin/awx-manage makemigrations</code> * Migrate Database changes<code bash>sudo -u awx /opt/awx/bin/awx-manage migrate</code> * Ensure other users still have read/execute permissions on awx directory<code bash>chmod o+rx /var/lib/awx</code> * Clear RabbitMQ Queues<code bash>rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app</code> * Start all services<code bash>systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code> ---- ===== AWX Team Suggested Upgrade Steps ===== To Upgrade: * Install cli tool<code bash>yum install ansible-tower-cli</code> * Export all data<code bash>awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata</code> * Stop all AWX services<code bash>systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code> * Upgrade AWX<code bash>yum update awx</code> * Drop and then re-create the database<code bash>su - postgres -c "dropdb awx" su - postgres -c "createdb -O awx awx"</code> * Migrate app data back in<code bash>sudo -u awx /opt/awx/bin/awx-manage migrate</code> * Initial app data setup in db<code bash>echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname) sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)</code> * Import saved data<code bash>awx-cli send alldata</code> ---- ====== Configuration ====== Other configuration steps. ---- ===== SSH Client Settings ===== Changes to ssh client settings (/etc/ssh/ssh_config)<code bash># Disable ProxyCommand for Ansible AWX. #ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h</code> ---- ===== Logos ===== Get rid of the angry potato pictures. \\ Download AWX logos <code bash>wget https://github.com/ansible/awx-logos/archive/master.zip</code> \\ Install unzip utility (if not installed) <code bash> yum install unzip </code> \\ Unzip archive <code bash>unzip master.zip</code> \\ Copy Logos to installed asset directory <code bash> cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/ </code> * **Note**: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files. ---- ===== LDAP Authentication ===== Example configuration for LDAP. **Tip**: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like. \\ Configure LDAP * On the left navigation bar: * SETTINGS * SUB CATEGORY -> LDAP FreeIPA Example * LDAP Server URI<code bash>ldap://serverldap01.mycorps.domain.org</code> * LDAP BIND DN<code bash>uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org</code> * LDAP BIND PASSWORD<code bash>password here</code> * LDAP USER DN TEMPLATE<code bash>uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org</code> * LDAP Group Type<code bash>MemberDNGroupType</code> * LDAP Require Group<code bash>cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org</code> * LDAP Deny Group<code bash>#leave blank</code> * LDAP Start TLS: **On** * LDAP User Search<code bash>[]</code> * LDAP Group Search<code bash> [ "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org", "SCOPE_SUBTREE", "(objectClass=groupofnames)" ]</code> * LDAP User Attribute Map<code bash> { "first_name": "givenname", "last_name": "sn", "email": "mail" }</code> * LDAP Group Type Parameters<code bash> { "member_attr": "member", "name_attr": "cn" }</code> * LDAP User Flags by group<code bash> { "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org" }</code> * LDAP Organization Map<code bash>{}</code> * LDAP Team Map<code bash>{}</code> ---- ===== Configure for Inventory ===== * **Create an Organization** * "Organizations" -> "Add" * Fill in: * Name * Description * Instance Groups (what AWX instances the Organization will use) * Save * **Add credentials for source control** * "Credentials" * Fill in: * Name * Description * Organization * Credential Type: Source Control * Username * Password (will be stored encrypted) * Save * **Add credentials for running jobs/playbooks** on the remote hosts * "Credentials" * Fill in: * Name * Description * Organization * Credential Type: Machine * Username * Password (will be stored encrypted) * SSH Private Key: Copy/Paste from "-----BEGIN RSA PRIVATE KEY-----" to "-----END RSA PRIVATE KEY-----" * Privilege escalation method: sudo * Save * **Add a project** * "Projects" * Fill in: * Name * Description * Organization * SCM Type (Git) * SCM URL * SCM Credential: select previously created * SCM Update Options * Clean * Save * Note: Initial sync begins immediate, watch progress on the "Jobs" page * **Schedule Regular Project Syncs** * "Projects" * To the right of the target Project, under "Actions", click the calendar ("Schedule SCM revision updates") * Click the green "+ADD" button * Fill in * Name (unique) * Start Date * Start Time * Time zone * Repeat (hourly, etc) * Every X hours * End (never) * **Sync Inventory File from Project** (Git source) * Create a new inventory for EACH Ansible inventory file; ie dev, test, prod. * "Inventories" -> "Add" -> "Inventory" * Fill in: * Name * Description * Organization * Instance Groups * Save * Within the same inventory config, click the "Sources" button link * Click the "Add Source" green button * Fill in: * Name * Description * Source: Sourced from a Project * Project (select previously created Project) * Inventory file (relative directory to project directory) * Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts * Example configured: inventories/development/hosts * Update Options * Overwrite (Keep in sync with inventory source) * Update on Project Change (Update inventory source when Project revision number is updated) ---- ===== Create a Job Template ===== AWX requires you to create a job template in order to run Playbooks cloned from source control. The templates define default run settings for the playbooks. \\ To Create a Job Template * Click "Templates" -> "Add" -> "Job Template" * Required Fields: * Name * Job Type (Run/Check) * Inventory * Project * Playbook (populated from Project) * Credential * Verbosity * Any field: check "Prompt on launch" in order to prompt the user to select a field when launching the Job. \\ ==== Example Template Fields to Use ==== * **Name**: <playbook name here> Playbook * **Description**: Run the <playbook name here> * **Job Type**: Run * **Inventory**: <leave blank>, check "**Prompt On Launch**" * **Project**: System Admin Git Project * **Playbook**: <playbook file from drop down box> * **Credential**: AWX Playbook Runner * **Forks**: 10 * **Limit**: <leave blank>, check "**Prompt On Launch**" * **Verbosity**: 0 (Normal), check "**Prompt On Launch**" * **Job Tags**: <leave blank>, check "**Prompt On Launch**" * **Skip Tags**: <leave blank>, check "**Prompt On Launch**" * **Labels**: <leave blank> * **Instance Groups**: tower * **Show Changes**: Off * **Options** * Enable Privilege Escalation: **checked** * Allow Provisioning Callbacks: not checked * Enable Concurrent Jobs: not checked * Use Fact Cache: **checked** ---- ====== Operating AWX ====== AWX operations notes. ---- ===== Service ===== **Enabled On Boot** Check to see if the service is enabled on boot <code bash> # AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web </code> \\ **Service Status** View the service status <code bash> # AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl status postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web </code> \\ **Service Start** Start the services <code bash> # AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl start postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web </code> \\ **Service Stop** Stop the services <code bash> # AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl stop postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web </code> ---- ===== Log Files ===== Log files are located: * Database (Postgres): /var/log/messages * Database caching (memcahed): /var/log/messages * Message Broker (rabbitmq): /var/log/rabbitmq/ * Web Proxy (nginx): /var/log/nginx/ * AWX Web: /var/log/awx/web.log ---- ===== Procedures ===== Common operational procedures. ==== Reboots ==== Reboot procedure and dependencies. * Ensure no jobs are running * Login to the web console: https://serverawx.mycorps.domain.org/#/login * On the left menu, navigate to: Views -> Jobs * If no job is currently running, proceed. * Reboot system ---- ==== Running Playbooks ==== To run a playbook via Ansible AWX: * Login to the web portal: https://serverawx.mycorps.domain.org/#/login * On the left side menu, navigate to: Views > Portal Mode * Under the "Job Templates", find the desired template and click the rocket picture (Start a job using this template). * Prompt window pop up * **Inventory** * Select which inventory (environment) to run against. * **Other Prompts** * Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems) * Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues. * Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run). * Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags). * **Preview** * Verify all settings are good, then click "Launch" to start the job. Jobs can be monitored a few ways * Views > Portal Mode * Right side * Click "My Jobs" to view just jobs launched by you * Click "All Jobs" to view all jobs * Views > Jobs ---- ==== Updating Playbook Runner LDAP Password ==== It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password. Examples * Username: awx-runner * Sudoers File: /etc/sudoers.d/ansible-awx * Password required for elevated privileges. When the LDAP password expires: * Update the password in LDAP. * Update the password in the Ansible AWX portal * Login to the portal * Navigate to: Resources > Credentials. * Click the "AWX Playbook Runner" machine credential * At the bottom, under "Privilege Escalation Password", click "Replace" * Type the new password ---- ===== Troubleshooting ===== Different troubleshooting scenarios and the fix. ---- ==== General Playbook Errors ==== In general, if you run into errors while running a playbook job template: * Increase the Verbosity and run it again. * Views > Portal Mode * Find Job Template to run, click the rocket (Start a job using this template) * On the "Other Prompts" screen, click the "Verbosity" drop down box and increase it to 1 or higher. ---- ==== Jobs Don't Start/Celery Workers Connection Errors ==== **Problem**: Jobs in the portal never start and the celery worker is showing connection errors in its service status<code bash>systemctl status awx-celery-worker</code> \\ **Cause**: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running. \\ **Fix**: Ensure that RabbitMQ is running <code bash>systemctl status rabbitmq-server</code> ---- ==== Jobs Don't Start/Celery Workers Unknown Tag Errors ==== **Problem**: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status<code bash>systemctl status awx-celery-worker</code> \\ **Cause**: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration. \\ **Fix**: Stop all AWX services, reset RabbitMQ, start all AWX services<code bash># Stop all AWX services systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web # Reset RabbitMQ rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app # Start all AWX services systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code> ---- linux_wiki/ansible_awx.txt Last modified: 2019/05/25 23:50(external edit)