General Information
Installation and operational notes for Ansible AWX (Tower).
“AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.”
Resources
Start with a CentOS 7 minimal install.
Postgresql 9.6 required for AWX. Add the repo
yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
Install Postgresql 9.6 Database and other required packages
yum install postgresql96-server rabbitmq-server wget memcached nginx
Add AWX Dev Repo
wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo
Install AWX
yum install awx
Initialize the database
/usr/pgsql-9.6/bin/postgresql96-setup initdb
Configure memcached to listen locally
vim /etc/sysconfig/memcached # Daemon USER="memcached" # Reserved Cache in MBs (Default: 64) CACHESIZE="512" # Memcached Options - Listen on localhost only OPTIONS="-l 127.0.0.1" # Networking PORT="11211" MAXCONN="1024"
Configure rabbitmq-server to listen locally
vim /etc/rabbitmq/rabbitmq.config # Uncomment the following (and delete trailing comma in ipv6 line) {tcp_listeners, [{"127.0.0.1", 5672}, {"::1", 5672}]}
Start and Enable some services.
Start/Enable Rabbit
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
Start/Enable Memcached
systemctl start memcached
systemctl enable memcached
Start/Enable Postgresql
systemctl start postgresql-9.6 systemctl enable postgresql-9.6
Create Postgres user (awx) and database
su - postgres createuser -S awx createdb -O awx awx exit
Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line
vim /etc/awx/settings.py #CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines
vim /etc/awx/settings.py #CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID} #CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
Migrate AWX App data into the database (fyi; this is a Django app)
sudo -u awx /opt/awx/bin/awx-manage migrate
Initialize the AWX Django App: Create admin user
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell
Initialize the AWX Django App: Add tower instance (AWX Server)
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
Initialize the AWX Django App: Create some pre-loaded organization data
sudo -u awx /opt/awx/bin/awx-manage create_preload_data
Initialize the AWX Django App: Create a queue group
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
Nginx will act as the proxy to the AWX application.
Configure Nginx - Main Config (/etc/nginx/nginx.conf)
## NGINX - Main Configuration ## # Context: Main - General Server Configuration # User that worker processes run as user nginx; # Number of worker processes (auto = set to number of CPUs) worker_processes auto; # Error Log and PID of main process error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; # Context: Events - Connection Processing events { # Max number of connections per worker process worker_connections 1024; } # Context: HTTP - HTTP Server Directives http { # MIME - Include file and default type include /etc/nginx/mime.types; default_type application/octet-stream; # Logging: Format and Main Access Log log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; # server_tokens off - Disable nginx version on error pages and response headers server_tokens off; ## Headers - Add additional headers ## # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin add_header X-Frame-Options SAMEORIGIN; # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks add_header X-Content-Type-Options nosniff; # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting # 1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected add_header X-XSS-Protection "1; mode=block" always; # Content-Security-Policy -> Prevent XSS, clickjacking, code injection add_header Content-Security-Policy "default-src 'self';" always; # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off, # then tcp_nodelay forces the fast sending of the last data # Sendfile - Send files directly in kernel space # on -> keep on for locally stored files # off -> turn off for files served over network mounted storage sendfile on; # tcp_nopush - Do not send data until packet reaches MSS # Dependency: sendfile MUST be on for this to work #tcp_nopush on; # tcp_nodelay - Send packets in buffer as soon as they are available #tcp_nodelay on; # Server side keepalive timeout in seconds (default: 75) keepalive_timeout 65; # Gzip - Compress responses using gzip #gzip on; # AWX ADDED: Connection upgrade map $http_upgrade $connection_upgrade { default upgrade; '' close; } # Include enabled configurations include /etc/nginx/conf.d/enabled/*.conf; # AWX ADDED: Upstream Apps upstream uwsgi { server 127.0.0.1:8050; } upstream daphne { server 127.0.0.1:8051; } }
Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)
## Default Config - Catch All Matches ## # HTTP (Port 80) server { listen 80 default_server; server_name _; # Redirect everything to HTTPS return 301 https://$http_host$request_uri; } # HTTPS (Port 443) server { listen 443 ssl default_server; listen [::]:443 ssl default_server; server_name _; # HSTS (HTTPS Strict Transport Security) # 63072000 seconds = 2 years add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; "; # SSL - Certificate Config ssl on; ssl_certificate /etc/pki/tls/current_cert.crt; ssl_certificate_key /etc/pki/tls/current_key.key; ssl_client_certificate /etc/pki/tls/current_ca.crt; # SSL - Session Config ssl_session_timeout 5m; ssl_session_cache shared:SSL:50m; # SSL - Protocols and Ciphers ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM"; # Locations for AWX location /static/ { alias /opt/awx/static/; } location /favicon.ico { alias /opt/awx/static/favicon.ico; } location /websocket { # Pass request to the upstream alias proxy_pass http://daphne; # Require http version 1.1 to allow for upgrade requests proxy_http_version 1.1; # We want proxy_buffering off for proxying to websockets. proxy_buffering off; # http://en.wikipedia.org/wiki/X-Forwarded-For proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # enable this if you use HTTPS: proxy_set_header X-Forwarded-Proto https; # pass the Host: header from the client for the sake of redirects proxy_set_header Host $http_host; # We've set the Host header, so we don't need Nginx to muddle # about with redirects proxy_redirect off; # Depending on the request value, set the Upgrade and # connection headers proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; } # Location: Webserver root location / { # autoindex off - Disable directory listing output autoindex off; uwsgi_read_timeout 120s; uwsgi_pass uwsgi; include /etc/nginx/uwsgi_params; } }
Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)
Start/Enable Nginx
systemctl start nginx
systemctl enable nginx
Start/Enable AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
WARNING: Upgrading to the newest AWX is not guaranteed to work and might break your install. The project is fast moving and does not currently support upgrade paths.
To upgrade:
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
yum update awx
sudo -u awx /opt/awx/bin/awx-manage makemigrations
sudo -u awx /opt/awx/bin/awx-manage migrate
chmod o+rx /var/lib/awx
rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
To Upgrade:
yum install ansible-tower-cli
awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
yum update awx
su - postgres -c "dropdb awx" su - postgres -c "createdb -O awx awx"
sudo -u awx /opt/awx/bin/awx-manage migrate
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname) sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
awx-cli send alldata
Other configuration steps.
Changes to ssh client settings (/etc/ssh/ssh_config)
# Disable ProxyCommand for Ansible AWX. #ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
Get rid of the angry potato pictures.
Download AWX logos
wget https://github.com/ansible/awx-logos/archive/master.zip
Install unzip utility (if not installed)
yum install unzip
Unzip archive
unzip master.zip
Copy Logos to installed asset directory
cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/ cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
Example configuration for LDAP.
Tip: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.
Configure LDAP
FreeIPA Example
ldap://serverldap01.mycorps.domain.org
uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
password here
uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
MemberDNGroupType
cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org
#leave blank
[]
[ "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org", "SCOPE_SUBTREE", "(objectClass=groupofnames)" ]
{ "first_name": "givenname", "last_name": "sn", "email": "mail" }
{ "member_attr": "member", "name_attr": "cn" }
{ "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org" }
{}
{}
AWX requires you to create a job template in order to run Playbooks cloned from source control.
The templates define default run settings for the playbooks.
To Create a Job Template
AWX operations notes.
Enabled On Boot
Check to see if the service is enabled on boot
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Status
View the service status
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl status postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Start
Start the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl start postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Service Stop
Stop the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx) systemctl stop postgresql-9.6 memcached rabbitmq-server nginx # AWX Services systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
Log files are located:
Common operational procedures.
Reboot procedure and dependencies.
To run a playbook via Ansible AWX:
Jobs can be monitored a few ways
It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.
Examples
When the LDAP password expires:
Different troubleshooting scenarios and the fix.
In general, if you run into errors while running a playbook job template:
Problem: Jobs in the portal never start and the celery worker is showing connection errors in its service status
systemctl status awx-celery-worker
Cause: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.
Fix: Ensure that RabbitMQ is running
systemctl status rabbitmq-server
Problem: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status
systemctl status awx-celery-worker
Cause: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.
Fix: Stop all AWX services, reset RabbitMQ, start all AWX services
# Stop all AWX services systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web # Reset RabbitMQ rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl start_app # Start all AWX services systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web