Ansible AWX
General Information
Installation and operational notes for Ansible AWX (Tower).
“AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.”
Resources
- Github Project: https://github.com/ansible/awx
- Ansible AWX FAQ: https://www.ansible.com/products/awx-project/faq
- Copr RPM Repo: https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/
- Github for Copr RPM repo: https://github.com/MrMEEE/awx-build
Install
Start with a CentOS 7 minimal install.
Postgresql 9.6 required for AWX. Add the repo
yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
Install Postgresql 9.6 Database and other required packages
yum install postgresql96-server rabbitmq-server wget memcached nginx
Add AWX Dev Repo
wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo
Install AWX
yum install awx
Initial Setup
Initialize the database
/usr/pgsql-9.6/bin/postgresql96-setup initdb
Configure memcached to listen locally
vim /etc/sysconfig/memcached # Daemon USER="memcached" # Reserved Cache in MBs (Default: 64) CACHESIZE="512" # Memcached Options - Listen on localhost only OPTIONS="-l 127.0.0.1" # Networking PORT="11211" MAXCONN="1024"
Configure rabbitmq-server to listen locally
vim /etc/rabbitmq/rabbitmq.config # Uncomment the following (and delete trailing comma in ipv6 line) {tcp_listeners, [{"127.0.0.1", 5672}, {"::1", 5672}]}
Services
Start and Enable some services.
Start/Enable Rabbit
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
Start/Enable Memcached
systemctl start memcached
systemctl enable memcached
Start/Enable Postgresql
systemctl start postgresql-9.6 systemctl enable postgresql-9.6
Database Setup
Create Postgres user (awx) and database
su - postgres createuser -S awx createdb -O awx awx exit
Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line
vim /etc/awx/settings.py #CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
- If you don't comment out the above line, migrations next will fail with an error.
Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines
vim /etc/awx/settings.py #CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID} #CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
Migrate AWX App data into the database (fyi; this is a Django app)
sudo -u awx /opt/awx/bin/awx-manage migrate
Initialize the AWX Django App: Create admin user
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell
Initialize the AWX Django App: Add tower instance (AWX Server)
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
Initialize the AWX Django App: Create some pre-loaded organization data
sudo -u awx /opt/awx/bin/awx-manage create_preload_data
Initialize the AWX Django App: Create a queue group
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
Proxy Setup
Nginx will act as the proxy to the AWX application.
Configure Nginx - Main Config (/etc/nginx/nginx.conf)
## NGINX - Main Configuration ## # Context: Main - General Server Configuration # User that worker processes run as user nginx; # Number of worker processes (auto = set to number of CPUs) worker_processes auto; # Error Log and PID of main process error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; # Context: Events - Connection Processing events { # Max number of connections per worker process worker_connections 1024; } # Context: HTTP - HTTP Server Directives http { # MIME - Include file and default type include /etc/nginx/mime.types; default_type application/octet-stream; # Logging: Format and Main Access Log log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; # server_tokens off - Disable nginx version on error pages and response headers server_tokens off; ## Headers - Add additional headers ## # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin add_header X-Frame-Options SAMEORIGIN; # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks add_header X-Content-Type-Options nosniff; # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting # 1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected add_header X-XSS-Protection "1; mode=block" always; # Content-Security-Policy -> Prevent XSS, clickjacking, code injection add_header Content-Security-Policy "default-src 'self';" always; # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off, # then tcp_nodelay forces the fast sending of the last data # Sendfile - Send files directly in kernel space # on -> keep on for locally stored files # off -> turn off for files served over network mounted storage sendfile on; # tcp_nopush - Do not send data until packet reaches MSS # Dependency: sendfile MUST be on for this to work #tcp_nopush on; # tcp_nodelay - Send packets in buffer as soon as they are available #tcp_nodelay on; # Server side keepalive timeout in seconds (default: 75) keepalive_timeout 65; # Gzip - Compress responses using gzip #gzip on; # AWX ADDED: Connection upgrade map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Include enabled configurations include /etc/nginx/conf.d/enabled/*.conf; # AWX ADDED: Upstream Apps upstream uwsgi { server 127.0.0.1:8050; } upstream daphne { server 127.0.0.1:8051; } }
Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)
## Default Config - Catch All Matches ## # HTTP (Port 80) server { listen 80 default_server; server_name _; # Redirect everything to HTTPS return 301 https://$http_host$request_uri; } # HTTPS (Port 443) server { listen 443 ssl default_server; listen [::]:443 ssl default_server; server_name _; # HSTS (HTTPS Strict Transport Security) # 63072000 seconds = 2 years add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; "; # SSL - Certificate Config ssl on; ssl_certificate /etc/pki/tls/current_cert.crt; ssl_certificate_key /etc/pki/tls/current_key.key; ssl_client_certificate /etc/pki/tls/current_ca.crt; # SSL - Session Config ssl_session_timeout 5m; ssl_session_cache shared:SSL:50m; # SSL - Protocols and Ciphers ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM"; # Locations for AWX location /static/ { alias /opt/awx/static/; } location /favicon.ico { alias /opt/awx/static/favicon.ico; } location /websocket { # Pass request to the upstream alias proxy_pass http://daphne; # Require http version 1.1 to allow for upgrade requests proxy_http_version 1.1; # We want proxy_buffering off for proxying to websockets. proxy_buffering off; # http://en.wikipedia.org/wiki/X-Forwarded-For proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # enable this if you use HTTPS: proxy_set_header X-Forwarded-Proto https; # pass the Host: header from the client for the sake of redirects proxy_set_header Host $http_host; # We've set the Host header, so we don't need Nginx to muddle # about with redirects proxy_redirect off; # Depending on the request value, set the Upgrade and # connection headers proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; } # Location: Webserver root location / { # autoindex off - Disable directory listing output autoindex off; uwsgi_read_timeout 120s; uwsgi_pass uwsgi; include /etc/nginx/uwsgi_params; } }
Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)
- /etc/pki/tls/current_cert.crt
- /etc/pki/tls/current_key.key
- /etc/pki/tls/current_ca.crt
Start/Enable Nginx
systemctl start nginx
systemctl enable nginx
AWX Services
Start/Enable AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Upgrade Steps
WARNING: Upgrading to the newest AWX is not guaranteed to work and might break your install. The project is fast moving and does not currently support upgrade paths.
AWX RPM Method
To upgrade:
- Stop all services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
- Upgrade AWX
yum update awx
- Make Migrations
sudo -u awx /opt/awx/bin/awx-manage makemigrations
- Migrate Database changes
sudo -u awx /opt/awx/bin/awx-manage migrate
- Ensure other users still have read/execute permissions on awx directory
chmod o+rx /var/lib/awx
- Clear RabbitMQ Queues
rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app
- Start all services
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
AWX Team Suggested Upgrade Steps
To Upgrade:
- Install cli tool
yum install ansible-tower-cli
- Export all data
awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata
- Stop all AWX services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
- Upgrade AWX
yum update awx
- Drop and then re-create the database
su - postgres -c "dropdb awx" su - postgres -c "createdb -O awx awx"
- Migrate app data back in
sudo -u awx /opt/awx/bin/awx-manage migrate
- Initial app data setup in db
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname) sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
- Import saved data
awx-cli send alldata
Configuration
Other configuration steps.
SSH Client Settings
Changes to ssh client settings (/etc/ssh/ssh_config)
# Disable ProxyCommand for Ansible AWX. #ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
Logos
Get rid of the angry potato pictures.
Download AWX logos
wget https://github.com/ansible/awx-logos/archive/master.zip
Install unzip utility (if not installed)
yum install unzip
Unzip archive
unzip master.zip
Copy Logos to installed asset directory
cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
- Note: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.
LDAP Authentication
Example configuration for LDAP.
Tip: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.
Configure LDAP
- On the left navigation bar:
- SETTINGS
- SUB CATEGORY → LDAP
FreeIPA Example
- LDAP Server URI
ldap://serverldap01.mycorps.domain.org
- LDAP BIND DN
uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
- LDAP BIND PASSWORD
password here
- LDAP USER DN TEMPLATE
uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
- LDAP Group Type
MemberDNGroupType
- LDAP Require Group
cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org
- LDAP Deny Group
#leave blank
- LDAP Start TLS: On
- LDAP User Search
[]
- LDAP Group Search
[ "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org", "SCOPE_SUBTREE", "(objectClass=groupofnames)" ]
- LDAP User Attribute Map
{ "first_name": "givenname", "last_name": "sn", "email": "mail" }
- LDAP Group Type Parameters
{ "member_attr": "member", "name_attr": "cn" }
- LDAP User Flags by group
{ "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org" }
- LDAP Organization Map
{}
- LDAP Team Map
{}
Configure for Inventory
- Create an Organization
- “Organizations” → “Add”
- Fill in:
- Name
- Description
- Instance Groups (what AWX instances the Organization will use)
- Save
- Add credentials for source control
- “Credentials”
- Fill in:
- Name
- Description
- Organization
- Credential Type: Source Control
- Username
- Password (will be stored encrypted)
- Save
- Add credentials for running jobs/playbooks on the remote hosts
- “Credentials”
- Fill in:
- Name
- Description
- Organization
- Credential Type: Machine
- Username
- Password (will be stored encrypted)
- SSH Private Key: Copy/Paste from “—–BEGIN RSA PRIVATE KEY—–” to “—–END RSA PRIVATE KEY—–”
- Privilege escalation method: sudo
- Save
- Add a project
- “Projects”
- Fill in:
- Name
- Description
- Organization
- SCM Type (Git)
- SCM URL
- SCM Credential: select previously created
- SCM Update Options
- Clean
- Save
- Note: Initial sync begins immediate, watch progress on the “Jobs” page
- Schedule Regular Project Syncs
- “Projects”
- To the right of the target Project, under “Actions”, click the calendar (“Schedule SCM revision updates”)
- Click the green “+ADD” button
- Fill in
- Name (unique)
- Start Date
- Start Time
- Time zone
- Repeat (hourly, etc)
- Every X hours
- End (never)
- Sync Inventory File from Project (Git source)
- Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
- “Inventories” → “Add” → “Inventory”
- Fill in:
- Name
- Description
- Organization
- Instance Groups
- Save
- Within the same inventory config, click the “Sources” button link
- Click the “Add Source” green button
- Fill in:
- Name
- Description
- Source: Sourced from a Project
- Project (select previously created Project)
- Inventory file (relative directory to project directory)
- Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
- Example configured: inventories/development/hosts
- Update Options
- Overwrite (Keep in sync with inventory source)
- Update on Project Change (Update inventory source when Project revision number is updated)
Create a Job Template
AWX requires you to create a job template in order to run Playbooks cloned from source control.
The templates define default run settings for the playbooks.
To Create a Job Template
- Click “Templates” → “Add” → “Job Template”
- Required Fields:
- Name
- Job Type (Run/Check)
- Inventory
- Project
- Playbook (populated from Project)
- Credential
- Verbosity
- Any field: check “Prompt on launch” in order to prompt the user to select a field when launching the Job.
Example Template Fields to Use
- Name: <playbook name here> Playbook
- Description: Run the <playbook name here>
- Job Type: Run
- Inventory: <leave blank>, check “Prompt On Launch”
- Project: System Admin Git Project
- Playbook: <playbook file from drop down box>
- Credential: AWX Playbook Runner
- Forks: 10
- Limit: <leave blank>, check “Prompt On Launch”
- Verbosity: 0 (Normal), check “Prompt On Launch”
- Job Tags: <leave blank>, check “Prompt On Launch”
- Skip Tags: <leave blank>, check “Prompt On Launch”
- Labels: <leave blank>
- Instance Groups: tower
- Show Changes: Off
- Options
- Enable Privilege Escalation: checked
- Allow Provisioning Callbacks: not checked
- Enable Concurrent Jobs: not checked
- Use Fact Cache: checked
Operating AWX
AWX operations notes.
Service
Enabled On Boot
Check to see if the service is enabled on boot
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Status
View the service status
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl status postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Start
Start the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl start postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Stop
Stop the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl stop postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Log Files
Log files are located:
- Database (Postgres): /var/log/messages
- Database caching (memcahed): /var/log/messages
- Message Broker (rabbitmq): /var/log/rabbitmq/
- Web Proxy (nginx): /var/log/nginx/
- AWX Web: /var/log/awx/web.log
Procedures
Common operational procedures.
Reboots
Reboot procedure and dependencies.
- Ensure no jobs are running
- Login to the web console: https://serverawx.mycorps.domain.org/#/login
- On the left menu, navigate to: Views → Jobs
- If no job is currently running, proceed.
- Reboot system
Running Playbooks
To run a playbook via Ansible AWX:
- Login to the web portal: https://serverawx.mycorps.domain.org/#/login
- On the left side menu, navigate to: Views > Portal Mode
- Under the “Job Templates”, find the desired template and click the rocket picture (Start a job using this template).
- Prompt window pop up
- Inventory
- Select which inventory (environment) to run against.
- Other Prompts
- Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems)
- Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues.
- Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run).
- Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags).
- Preview
- Verify all settings are good, then click “Launch” to start the job.
Jobs can be monitored a few ways
- Views > Portal Mode
- Right side
- Click “My Jobs” to view just jobs launched by you
- Click “All Jobs” to view all jobs
- Views > Jobs
Updating Playbook Runner LDAP Password
It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.
Examples
- Username: awx-runner
- Sudoers File: /etc/sudoers.d/ansible-awx
- Password required for elevated privileges.
When the LDAP password expires:
- Update the password in LDAP.
- Update the password in the Ansible AWX portal
- Login to the portal
- Navigate to: Resources > Credentials.
- Click the “AWX Playbook Runner” machine credential
- At the bottom, under “Privilege Escalation Password”, click “Replace”
- Type the new password
Troubleshooting
Different troubleshooting scenarios and the fix.
General Playbook Errors
In general, if you run into errors while running a playbook job template:
- Increase the Verbosity and run it again.
- Views > Portal Mode
- Find Job Template to run, click the rocket (Start a job using this template)
- On the “Other Prompts” screen, click the “Verbosity” drop down box and increase it to 1 or higher.
Jobs Don't Start/Celery Workers Connection Errors
Problem: Jobs in the portal never start and the celery worker is showing connection errors in its service status
systemctl status awx-celery-worker
Cause: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.
Fix: Ensure that RabbitMQ is running
systemctl status rabbitmq-server
Jobs Don't Start/Celery Workers Unknown Tag Errors
Problem: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status
systemctl status awx-celery-worker
Cause: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.
Fix: Stop all AWX services, reset RabbitMQ, start all AWX services
# Stop all AWX services systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web # Reset RabbitMQ rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app # Start all AWX services systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web