====== Ansible AWX ======
**General Information**
Installation and operational notes for Ansible AWX (Tower).
"AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived."
Resources
* Github Project: https://github.com/ansible/awx
* Ansible AWX FAQ: https://www.ansible.com/products/awx-project/faq
* Copr RPM Repo: https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/
* Github for Copr RPM repo: https://github.com/MrMEEE/awx-build
----
====== Install ======
Start with a CentOS 7 minimal install.
\\
Postgresql 9.6 required for AWX. Add the repo
yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm
\\
Install Postgresql 9.6 Database and other required packages
yum install postgresql96-server rabbitmq-server wget memcached nginx
\\
Add AWX Dev Repo
wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo
\\
Install AWX
yum install awx
----
====== Initial Setup ======
Initialize the database
/usr/pgsql-9.6/bin/postgresql96-setup initdb
\\
Configure memcached to listen locally
vim /etc/sysconfig/memcached
# Daemon
USER="memcached"
# Reserved Cache in MBs (Default: 64)
CACHESIZE="512"
# Memcached Options - Listen on localhost only
OPTIONS="-l 127.0.0.1"
# Networking
PORT="11211"
MAXCONN="1024"
\\
Configure rabbitmq-server to listen locally
vim /etc/rabbitmq/rabbitmq.config
# Uncomment the following (and delete trailing comma in ipv6 line)
{tcp_listeners, [{"127.0.0.1", 5672},
{"::1", 5672}]}
----
===== Services =====
Start and Enable some services.
\\
Start/Enable Rabbit
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
\\
Start/Enable Memcached
systemctl start memcached
systemctl enable memcached
\\
Start/Enable Postgresql
systemctl start postgresql-9.6
systemctl enable postgresql-9.6
----
===== Database Setup =====
Create Postgres user (awx) and database
su - postgres
createuser -S awx
createdb -O awx awx
exit
\\
Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES linevim /etc/awx/settings.py
#CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
* If you don't comment out the above line, migrations next will fail with an error.
\\
Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES linesvim /etc/awx/settings.py
#CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
#CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
\\
Migrate AWX App data into the database (fyi; this is a Django app)
sudo -u awx /opt/awx/bin/awx-manage migrate
\\
Initialize the AWX Django App: Create admin user
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell
\\
Initialize the AWX Django App: Add tower instance (AWX Server)
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
\\
Initialize the AWX Django App: Create some pre-loaded organization data
sudo -u awx /opt/awx/bin/awx-manage create_preload_data
\\
Initialize the AWX Django App: Create a queue group
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
----
===== Proxy Setup =====
Nginx will act as the proxy to the AWX application.
\\
Configure Nginx - Main Config (/etc/nginx/nginx.conf)
## NGINX - Main Configuration ##
# Context: Main - General Server Configuration
# User that worker processes run as
user nginx;
# Number of worker processes (auto = set to number of CPUs)
worker_processes auto;
# Error Log and PID of main process
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
# Context: Events - Connection Processing
events {
# Max number of connections per worker process
worker_connections 1024;
}
# Context: HTTP - HTTP Server Directives
http {
# MIME - Include file and default type
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Logging: Format and Main Access Log
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
# server_tokens off - Disable nginx version on error pages and response headers
server_tokens off;
## Headers - Add additional headers ##
# X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin
add_header X-Frame-Options SAMEORIGIN;
# X-Content-Type-Options nosniff -> Prevent MIME Type Attacks
add_header X-Content-Type-Options nosniff;
# X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting
# 1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected
add_header X-XSS-Protection "1; mode=block" always;
# Content-Security-Policy -> Prevent XSS, clickjacking, code injection
add_header Content-Security-Policy "default-src 'self';" always;
# Combined directives: sendfile, tcp_nopush, tcp_nodelay all on
# sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send
# tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off,
# then tcp_nodelay forces the fast sending of the last data
# Sendfile - Send files directly in kernel space
# on -> keep on for locally stored files
# off -> turn off for files served over network mounted storage
sendfile on;
# tcp_nopush - Do not send data until packet reaches MSS
# Dependency: sendfile MUST be on for this to work
#tcp_nopush on;
# tcp_nodelay - Send packets in buffer as soon as they are available
#tcp_nodelay on;
# Server side keepalive timeout in seconds (default: 75)
keepalive_timeout 65;
# Gzip - Compress responses using gzip
#gzip on;
# AWX ADDED: Connection upgrade
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# Include enabled configurations
include /etc/nginx/conf.d/enabled/*.conf;
# AWX ADDED: Upstream Apps
upstream uwsgi {
server 127.0.0.1:8050;
}
upstream daphne {
server 127.0.0.1:8051;
}
}
\\
Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)
## Default Config - Catch All Matches ##
# HTTP (Port 80)
server {
listen 80 default_server;
server_name _;
# Redirect everything to HTTPS
return 301 https://$http_host$request_uri;
}
# HTTPS (Port 443)
server {
listen 443 ssl default_server;
listen [::]:443 ssl default_server;
server_name _;
# HSTS (HTTPS Strict Transport Security)
# 63072000 seconds = 2 years
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
# SSL - Certificate Config
ssl on;
ssl_certificate /etc/pki/tls/current_cert.crt;
ssl_certificate_key /etc/pki/tls/current_key.key;
ssl_client_certificate /etc/pki/tls/current_ca.crt;
# SSL - Session Config
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
# SSL - Protocols and Ciphers
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM";
# Locations for AWX
location /static/ {
alias /opt/awx/static/;
}
location /favicon.ico { alias /opt/awx/static/favicon.ico; }
location /websocket {
# Pass request to the upstream alias
proxy_pass http://daphne;
# Require http version 1.1 to allow for upgrade requests
proxy_http_version 1.1;
# We want proxy_buffering off for proxying to websockets.
proxy_buffering off;
# http://en.wikipedia.org/wiki/X-Forwarded-For
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# enable this if you use HTTPS:
proxy_set_header X-Forwarded-Proto https;
# pass the Host: header from the client for the sake of redirects
proxy_set_header Host $http_host;
# We've set the Host header, so we don't need Nginx to muddle
# about with redirects
proxy_redirect off;
# Depending on the request value, set the Upgrade and
# connection headers
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
# Location: Webserver root
location / {
# autoindex off - Disable directory listing output
autoindex off;
uwsgi_read_timeout 120s;
uwsgi_pass uwsgi;
include /etc/nginx/uwsgi_params;
}
}
\\
Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)
* /etc/pki/tls/current_cert.crt
* /etc/pki/tls/current_key.key
* /etc/pki/tls/current_ca.crt
\\
Start/Enable Nginx
systemctl start nginx
systemctl enable nginx
----
===== AWX Services =====
Start/Enable AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
----
====== Upgrade Steps ======
**WARNING:** Upgrading to the newest AWX is **not guaranteed to work and might break your install**. The project is fast moving and does not currently support upgrade paths.
===== AWX RPM Method =====
To upgrade:
* Stop all servicessystemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
* Upgrade AWXyum update awx
* Make Migrationssudo -u awx /opt/awx/bin/awx-manage makemigrations
* Migrate Database changessudo -u awx /opt/awx/bin/awx-manage migrate
* Ensure other users still have read/execute permissions on awx directorychmod o+rx /var/lib/awx
* Clear RabbitMQ Queuesrabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
* Start all servicessystemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
----
===== AWX Team Suggested Upgrade Steps =====
To Upgrade:
* Install cli toolyum install ansible-tower-cli
* Export all dataawx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata
* Stop all AWX servicessystemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
* Upgrade AWXyum update awx
* Drop and then re-create the databasesu - postgres -c "dropdb awx"
su - postgres -c "createdb -O awx awx"
* Migrate app data back insudo -u awx /opt/awx/bin/awx-manage migrate
* Initial app data setup in dbecho "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
* Import saved dataawx-cli send alldata
----
====== Configuration ======
Other configuration steps.
----
===== SSH Client Settings =====
Changes to ssh client settings (/etc/ssh/ssh_config)# Disable ProxyCommand for Ansible AWX.
#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
----
===== Logos =====
Get rid of the angry potato pictures.
\\
Download AWX logos
wget https://github.com/ansible/awx-logos/archive/master.zip
\\
Install unzip utility (if not installed)
yum install unzip
\\
Unzip archive
unzip master.zip
\\
Copy Logos to installed asset directory
cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
* **Note**: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.
----
===== LDAP Authentication =====
Example configuration for LDAP.
**Tip**: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.
\\
Configure LDAP
* On the left navigation bar:
* SETTINGS
* SUB CATEGORY -> LDAP
FreeIPA Example
* LDAP Server URIldap://serverldap01.mycorps.domain.org
* LDAP BIND DNuid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
* LDAP BIND PASSWORDpassword here
* LDAP USER DN TEMPLATEuid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
* LDAP Group TypeMemberDNGroupType
* LDAP Require Groupcn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org
* LDAP Deny Group#leave blank
* LDAP Start TLS: **On**
* LDAP User Search[]
* LDAP Group Search
[
"cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org",
"SCOPE_SUBTREE",
"(objectClass=groupofnames)"
]
* LDAP User Attribute Map
{
"first_name": "givenname",
"last_name": "sn",
"email": "mail"
}
* LDAP Group Type Parameters
{
"member_attr": "member",
"name_attr": "cn"
}
* LDAP User Flags by group
{
"is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org"
}
* LDAP Organization Map{}
* LDAP Team Map{}
----
===== Configure for Inventory =====
* **Create an Organization**
* "Organizations" -> "Add"
* Fill in:
* Name
* Description
* Instance Groups (what AWX instances the Organization will use)
* Save
* **Add credentials for source control**
* "Credentials"
* Fill in:
* Name
* Description
* Organization
* Credential Type: Source Control
* Username
* Password (will be stored encrypted)
* Save
* **Add credentials for running jobs/playbooks** on the remote hosts
* "Credentials"
* Fill in:
* Name
* Description
* Organization
* Credential Type: Machine
* Username
* Password (will be stored encrypted)
* SSH Private Key: Copy/Paste from "-----BEGIN RSA PRIVATE KEY-----" to "-----END RSA PRIVATE KEY-----"
* Privilege escalation method: sudo
* Save
* **Add a project**
* "Projects"
* Fill in:
* Name
* Description
* Organization
* SCM Type (Git)
* SCM URL
* SCM Credential: select previously created
* SCM Update Options
* Clean
* Save
* Note: Initial sync begins immediate, watch progress on the "Jobs" page
* **Schedule Regular Project Syncs**
* "Projects"
* To the right of the target Project, under "Actions", click the calendar ("Schedule SCM revision updates")
* Click the green "+ADD" button
* Fill in
* Name (unique)
* Start Date
* Start Time
* Time zone
* Repeat (hourly, etc)
* Every X hours
* End (never)
* **Sync Inventory File from Project** (Git source)
* Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
* "Inventories" -> "Add" -> "Inventory"
* Fill in:
* Name
* Description
* Organization
* Instance Groups
* Save
* Within the same inventory config, click the "Sources" button link
* Click the "Add Source" green button
* Fill in:
* Name
* Description
* Source: Sourced from a Project
* Project (select previously created Project)
* Inventory file (relative directory to project directory)
* Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
* Example configured: inventories/development/hosts
* Update Options
* Overwrite (Keep in sync with inventory source)
* Update on Project Change (Update inventory source when Project revision number is updated)
----
===== Create a Job Template =====
AWX requires you to create a job template in order to run Playbooks cloned from source control.
The templates define default run settings for the playbooks.
\\
To Create a Job Template
* Click "Templates" -> "Add" -> "Job Template"
* Required Fields:
* Name
* Job Type (Run/Check)
* Inventory
* Project
* Playbook (populated from Project)
* Credential
* Verbosity
* Any field: check "Prompt on launch" in order to prompt the user to select a field when launching the Job.
\\
==== Example Template Fields to Use ====
* **Name**: Playbook
* **Description**: Run the
* **Job Type**: Run
* **Inventory**: , check "**Prompt On Launch**"
* **Project**: System Admin Git Project
* **Playbook**:
* **Credential**: AWX Playbook Runner
* **Forks**: 10
* **Limit**: , check "**Prompt On Launch**"
* **Verbosity**: 0 (Normal), check "**Prompt On Launch**"
* **Job Tags**: , check "**Prompt On Launch**"
* **Skip Tags**: , check "**Prompt On Launch**"
* **Labels**:
* **Instance Groups**: tower
* **Show Changes**: Off
* **Options**
* Enable Privilege Escalation: **checked**
* Allow Provisioning Callbacks: not checked
* Enable Concurrent Jobs: not checked
* Use Fact Cache: **checked**
----
====== Operating AWX ======
AWX operations notes.
----
===== Service =====
**Enabled On Boot**
Check to see if the service is enabled on boot
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx
# AWX Services
systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
\\
**Service Status**
View the service status
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl status postgresql-9.6 memcached rabbitmq-server nginx
# AWX Services
systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
\\
**Service Start**
Start the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl start postgresql-9.6 memcached rabbitmq-server nginx
# AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
\\
**Service Stop**
Stop the services
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl stop postgresql-9.6 memcached rabbitmq-server nginx
# AWX Services
systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
----
===== Log Files =====
Log files are located:
* Database (Postgres): /var/log/messages
* Database caching (memcahed): /var/log/messages
* Message Broker (rabbitmq): /var/log/rabbitmq/
* Web Proxy (nginx): /var/log/nginx/
* AWX Web: /var/log/awx/web.log
----
===== Procedures =====
Common operational procedures.
==== Reboots ====
Reboot procedure and dependencies.
* Ensure no jobs are running
* Login to the web console: https://serverawx.mycorps.domain.org/#/login
* On the left menu, navigate to: Views -> Jobs
* If no job is currently running, proceed.
* Reboot system
----
==== Running Playbooks ====
To run a playbook via Ansible AWX:
* Login to the web portal: https://serverawx.mycorps.domain.org/#/login
* On the left side menu, navigate to: Views > Portal Mode
* Under the "Job Templates", find the desired template and click the rocket picture (Start a job using this template).
* Prompt window pop up
* **Inventory**
* Select which inventory (environment) to run against.
* **Other Prompts**
* Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems)
* Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues.
* Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run).
* Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags).
* **Preview**
* Verify all settings are good, then click "Launch" to start the job.
Jobs can be monitored a few ways
* Views > Portal Mode
* Right side
* Click "My Jobs" to view just jobs launched by you
* Click "All Jobs" to view all jobs
* Views > Jobs
----
==== Updating Playbook Runner LDAP Password ====
It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.
Examples
* Username: awx-runner
* Sudoers File: /etc/sudoers.d/ansible-awx
* Password required for elevated privileges.
When the LDAP password expires:
* Update the password in LDAP.
* Update the password in the Ansible AWX portal
* Login to the portal
* Navigate to: Resources > Credentials.
* Click the "AWX Playbook Runner" machine credential
* At the bottom, under "Privilege Escalation Password", click "Replace"
* Type the new password
----
===== Troubleshooting =====
Different troubleshooting scenarios and the fix.
----
==== General Playbook Errors ====
In general, if you run into errors while running a playbook job template:
* Increase the Verbosity and run it again.
* Views > Portal Mode
* Find Job Template to run, click the rocket (Start a job using this template)
* On the "Other Prompts" screen, click the "Verbosity" drop down box and increase it to 1 or higher.
----
==== Jobs Don't Start/Celery Workers Connection Errors ====
**Problem**: Jobs in the portal never start and the celery worker is showing connection errors in its service statussystemctl status awx-celery-worker
\\
**Cause**: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.
\\
**Fix**: Ensure that RabbitMQ is running
systemctl status rabbitmq-server
----
==== Jobs Don't Start/Celery Workers Unknown Tag Errors ====
**Problem**: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service statussystemctl status awx-celery-worker
\\
**Cause**: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.
\\
**Fix**: Stop all AWX services, reset RabbitMQ, start all AWX services# Stop all AWX services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
# Reset RabbitMQ
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
# Start all AWX services
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
----