General Information

Installation and operational notes for Ansible AWX (Tower).

“AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.”

Resources

• Github Project: https://github.com/ansible/awx
• Ansible AWX FAQ: https://www.ansible.com/products/awx-project/faq
• Copr RPM Repo: https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/
  • Github for Copr RPM repo: https://github.com/MrMEEE/awx-build

Start with a CentOS 7 minimal install.

Postgresql 9.6 required for AWX. Add the repo

yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Install Postgresql 9.6 Database and other required packages

yum install postgresql96-server rabbitmq-server wget memcached nginx

Add AWX Dev Repo

wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo

Install AWX

yum install awx

Initialize the database

/usr/pgsql-9.6/bin/postgresql96-setup initdb

Configure memcached to listen locally

vim /etc/sysconfig/memcached 
 
# Daemon
USER="memcached"
 
# Reserved Cache in MBs (Default: 64)
CACHESIZE="512"
 
# Memcached Options - Listen on localhost only
OPTIONS="-l 127.0.0.1"
 
# Networking
PORT="11211"
MAXCONN="1024"

Configure rabbitmq-server to listen locally

vim /etc/rabbitmq/rabbitmq.config
 
# Uncomment the following (and delete trailing comma in ipv6 line)
{tcp_listeners, [{"127.0.0.1", 5672},
                    {"::1",       5672}]}

Start and Enable some services.

Start/Enable Rabbit

systemctl start rabbitmq-server
systemctl enable rabbitmq-server

Start/Enable Memcached

systemctl start memcached
systemctl enable memcached

Start/Enable Postgresql

systemctl start postgresql-9.6
systemctl enable postgresql-9.6

Create Postgres user (awx) and database

su - postgres
createuser -S awx
createdb -O awx awx
 
exit

Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line

vim /etc/awx/settings.py
 
#CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)

• If you don't comment out the above line, migrations next will fail with an error.

Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines

vim /etc/awx/settings.py
 
#CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
#CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}

Migrate AWX App data into the database (fyi; this is a Django app)

sudo -u awx /opt/awx/bin/awx-manage migrate

Initialize the AWX Django App: Create admin user

echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell

Initialize the AWX Django App: Add tower instance (AWX Server)

sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)

Initialize the AWX Django App: Create some pre-loaded organization data

sudo -u awx /opt/awx/bin/awx-manage create_preload_data

Initialize the AWX Django App: Create a queue group

sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)

Nginx will act as the proxy to the AWX application.

Configure Nginx - Main Config (/etc/nginx/nginx.conf)

## NGINX - Main Configuration ##
 
# Context: Main - General Server Configuration
 
# User that worker processes run as
user  nginx;
 
# Number of worker processes (auto = set to number of CPUs)
worker_processes  auto;
 
# Error Log and PID of main process
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
 
# Context: Events - Connection Processing
events {
  # Max number of connections per worker process
  worker_connections  1024;
}
 
# Context: HTTP - HTTP Server Directives
http {
  # MIME - Include file and default type
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
 
  # Logging: Format and Main Access Log
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;
 
  # server_tokens off - Disable nginx version on error pages and response headers
  server_tokens off;
 
  ## Headers - Add additional headers ##
  # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin
  add_header X-Frame-Options SAMEORIGIN;
 
  # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks
  add_header X-Content-Type-Options nosniff;
 
  # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting
  #   1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected
  add_header X-XSS-Protection "1; mode=block" always;
 
  # Content-Security-Policy -> Prevent XSS, clickjacking, code injection
  add_header Content-Security-Policy "default-src 'self';" always;
 
  # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on
  # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send
  # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off,
  #               then tcp_nodelay forces the fast sending of the last data
 
  # Sendfile - Send files directly in kernel space
  # on -> keep on for locally stored files
  # off -> turn off for files served over network mounted storage
  sendfile        on;
 
  # tcp_nopush - Do not send data until packet reaches MSS
  # Dependency: sendfile MUST be on for this to work
  #tcp_nopush     on;
 
  # tcp_nodelay -  Send packets in buffer as soon as they are available
  #tcp_nodelay on;
 
  # Server side keepalive timeout in seconds (default: 75)
  keepalive_timeout  65;
 
  # Gzip - Compress responses using gzip
  #gzip  on;
 
  # AWX ADDED: Connection upgrade
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }
 
  # Include enabled configurations
  include /etc/nginx/conf.d/enabled/*.conf;
 
  # AWX ADDED: Upstream Apps
  upstream uwsgi {
        server 127.0.0.1:8050;
        }
 
    upstream daphne {
        server 127.0.0.1:8051;
    }
}

Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)

## Default Config - Catch All Matches ##
 
# HTTP (Port 80)
server {
    listen 80 default_server;
    server_name  _;
 
    # Redirect everything to HTTPS
    return 301 https://$http_host$request_uri;
}
 
# HTTPS (Port 443)
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;
 
    # HSTS (HTTPS Strict Transport Security)
    # 63072000 seconds = 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
 
    # SSL - Certificate Config
    ssl on;
    ssl_certificate /etc/pki/tls/current_cert.crt;
    ssl_certificate_key /etc/pki/tls/current_key.key;
    ssl_client_certificate /etc/pki/tls/current_ca.crt;
 
    # SSL - Session Config
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
 
    # SSL - Protocols and Ciphers
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM";
 
    # Locations for AWX
    location /static/ {
            alias /opt/awx/static/;
     }
 
    location /favicon.ico { alias /opt/awx/static/favicon.ico; }
 
    location /websocket {
      # Pass request to the upstream alias
      proxy_pass http://daphne;
      # Require http version 1.1 to allow for upgrade requests
      proxy_http_version 1.1;
      # We want proxy_buffering off for proxying to websockets.
      proxy_buffering off;
      # http://en.wikipedia.org/wiki/X-Forwarded-For
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # enable this if you use HTTPS:
      proxy_set_header X-Forwarded-Proto https;
      # pass the Host: header from the client for the sake of redirects
      proxy_set_header Host $http_host;
      # We've set the Host header, so we don't need Nginx to muddle
      # about with redirects
      proxy_redirect off;
      # Depending on the request value, set the Upgrade and
      # connection headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }
 
    # Location: Webserver root
    location / {
      # autoindex off - Disable directory listing output
      autoindex off;
      uwsgi_read_timeout 120s;
      uwsgi_pass uwsgi;
      include /etc/nginx/uwsgi_params;
    }
}

Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)

• /etc/pki/tls/current_cert.crt
• /etc/pki/tls/current_key.key
• /etc/pki/tls/current_ca.crt

Start/Enable Nginx

systemctl start nginx
systemctl enable nginx

Start/Enable AWX Services

systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

WARNING: Upgrading to the newest AWX is not guaranteed to work and might break your install. The project is fast moving and does not currently support upgrade paths.

To upgrade:

• Stop all services

  systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

• Upgrade AWX

  yum update awx

• Make Migrations

  sudo -u awx /opt/awx/bin/awx-manage makemigrations

• Migrate Database changes

  sudo -u awx /opt/awx/bin/awx-manage migrate

• Ensure other users still have read/execute permissions on awx directory

  chmod o+rx /var/lib/awx

• Clear RabbitMQ Queues

  rabbitmqctl stop_app
  rabbitmqctl reset
  rabbitmqctl start_app

• Start all services

  systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

To Upgrade:

• Install cli tool

  yum install ansible-tower-cli

• Export all data

  awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata

• Stop all AWX services

  systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

• Upgrade AWX

  yum update awx

• Drop and then re-create the database

  su - postgres -c "dropdb awx"
  su - postgres -c "createdb -O awx awx"

• Migrate app data back in

  sudo -u awx /opt/awx/bin/awx-manage migrate

• Initial app data setup in db

  echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell
  sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
  sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)

• Import saved data

  awx-cli send alldata

Other configuration steps.

Changes to ssh client settings (/etc/ssh/ssh_config)

# Disable ProxyCommand for Ansible AWX.
#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Get rid of the angry potato pictures.

Download AWX logos

wget https://github.com/ansible/awx-logos/archive/master.zip

Install unzip utility (if not installed)

yum install unzip

Unzip archive

unzip master.zip

Copy Logos to installed asset directory

cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/

• Note: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.

Example configuration for LDAP.

Tip: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.

Configure LDAP

• On the left navigation bar:
  • SETTINGS
  • SUB CATEGORY → LDAP

FreeIPA Example

• LDAP Server URI

  ldap://serverldap01.mycorps.domain.org

• LDAP BIND DN

  uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org

• LDAP BIND PASSWORD

  password here

• LDAP USER DN TEMPLATE

  uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org

• LDAP Group Type

  MemberDNGroupType

• LDAP Require Group

  cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org

• LDAP Deny Group

  #leave blank

• LDAP Start TLS: On

• LDAP User Search

  []

• LDAP Group Search

  [
  "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org",
  "SCOPE_SUBTREE",
  "(objectClass=groupofnames)"
  ]

• LDAP User Attribute Map

  {
  "first_name": "givenname",
  "last_name": "sn",
  "email": "mail"
  }

• LDAP Group Type Parameters

  {
   "member_attr": "member",
   "name_attr": "cn"
  }

• LDAP User Flags by group

  {
  "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org"
  }

• LDAP Organization Map

  {}

• LDAP Team Map

  {}

• Create an Organization
  • “Organizations” → “Add”
    • Fill in:
      • Name
      • Description
      • Instance Groups (what AWX instances the Organization will use)
    • Save

• Add credentials for source control
  • “Credentials”
    • Fill in:
      • Name
      • Description
      • Organization
      • Credential Type: Source Control
      • Username
      • Password (will be stored encrypted)
    • Save

• Add credentials for running jobs/playbooks on the remote hosts
  • “Credentials”
    • Fill in:
      • Name
      • Description
      • Organization
      • Credential Type: Machine
      • Username
      • Password (will be stored encrypted)
      • SSH Private Key: Copy/Paste from “—–BEGIN RSA PRIVATE KEY—–” to “—–END RSA PRIVATE KEY—–”
      • Privilege escalation method: sudo
    • Save

• Add a project
  • “Projects”
    • Fill in:
      • Name
      • Description
      • Organization
      • SCM Type (Git)
      • SCM URL
      • SCM Credential: select previously created
      • SCM Update Options
        • Clean
      • Save
        • Note: Initial sync begins immediate, watch progress on the “Jobs” page

• Schedule Regular Project Syncs
  • “Projects”
    • To the right of the target Project, under “Actions”, click the calendar (“Schedule SCM revision updates”)
    • Click the green “+ADD” button
      • Fill in
        • Name (unique)
        • Start Date
        • Start Time
        • Time zone
        • Repeat (hourly, etc)
        • Every X hours
        • End (never)

• Sync Inventory File from Project (Git source)
  • Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
  • “Inventories” → “Add” → “Inventory”
    • Fill in:
      • Name
      • Description
      • Organization
      • Instance Groups
    • Save
    • Within the same inventory config, click the “Sources” button link
    • Click the “Add Source” green button
    • Fill in:
      • Name
      • Description
      • Source: Sourced from a Project
      • Project (select previously created Project)
      • Inventory file (relative directory to project directory)
        • Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
        • Example configured: inventories/development/hosts
      • Update Options
        • Overwrite (Keep in sync with inventory source)
        • Update on Project Change (Update inventory source when Project revision number is updated)

AWX requires you to create a job template in order to run Playbooks cloned from source control.

The templates define default run settings for the playbooks.

To Create a Job Template

• Click “Templates” → “Add” → “Job Template”
  • Required Fields:
    • Name
    • Job Type (Run/Check)
    • Inventory
    • Project
    • Playbook (populated from Project)
    • Credential
    • Verbosity
  • Any field: check “Prompt on launch” in order to prompt the user to select a field when launching the Job.

• Name: <playbook name here> Playbook
• Description: Run the <playbook name here>
• Job Type: Run

• Inventory: <leave blank>, check “Prompt On Launch”
• Project: System Admin Git Project
• Playbook: <playbook file from drop down box>

• Credential: AWX Playbook Runner
• Forks: 10
• Limit: <leave blank>, check “Prompt On Launch”

• Verbosity: 0 (Normal), check “Prompt On Launch”
• Job Tags: <leave blank>, check “Prompt On Launch”
• Skip Tags: <leave blank>, check “Prompt On Launch”

• Labels: <leave blank>
• Instance Groups: tower
• Show Changes: Off

• Options
  • Enable Privilege Escalation: checked
  • Allow Provisioning Callbacks: not checked
  • Enable Concurrent Jobs: not checked
  • Use Fact Cache: checked

AWX operations notes.

Enabled On Boot

Check to see if the service is enabled on boot

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Service Status

View the service status

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl status postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Service Start

Start the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl start postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Service Stop

Stop the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl stop postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Log files are located:

• Database (Postgres): /var/log/messages
• Database caching (memcahed): /var/log/messages
• Message Broker (rabbitmq): /var/log/rabbitmq/
• Web Proxy (nginx): /var/log/nginx/
• AWX Web: /var/log/awx/web.log

Common operational procedures.

Reboot procedure and dependencies.

• Ensure no jobs are running
  • Login to the web console: https://serverawx.mycorps.domain.org/#/login
  • On the left menu, navigate to: Views → Jobs
    • If no job is currently running, proceed.
  • Reboot system

To run a playbook via Ansible AWX:

• Login to the web portal: https://serverawx.mycorps.domain.org/#/login
• On the left side menu, navigate to: Views > Portal Mode
• Under the “Job Templates”, find the desired template and click the rocket picture (Start a job using this template).
  • Prompt window pop up
    • Inventory
      • Select which inventory (environment) to run against.
    • Other Prompts
      • Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems)
      • Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues.
      • Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run).
      • Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags).
    • Preview
      • Verify all settings are good, then click “Launch” to start the job.

Jobs can be monitored a few ways

• Views > Portal Mode
  • Right side
    • Click “My Jobs” to view just jobs launched by you
    • Click “All Jobs” to view all jobs
• Views > Jobs

It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.

Examples

• Username: awx-runner
• Sudoers File: /etc/sudoers.d/ansible-awx
  • Password required for elevated privileges.

When the LDAP password expires:

• Update the password in LDAP.
• Update the password in the Ansible AWX portal
  • Login to the portal
  • Navigate to: Resources > Credentials.
  • Click the “AWX Playbook Runner” machine credential
    • At the bottom, under “Privilege Escalation Password”, click “Replace”
    • Type the new password

Different troubleshooting scenarios and the fix.

In general, if you run into errors while running a playbook job template:

• Increase the Verbosity and run it again.
  • Views > Portal Mode
    • Find Job Template to run, click the rocket (Start a job using this template)
    • On the “Other Prompts” screen, click the “Verbosity” drop down box and increase it to 1 or higher.

Problem: Jobs in the portal never start and the celery worker is showing connection errors in its service status

systemctl status awx-celery-worker

Cause: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.

Fix: Ensure that RabbitMQ is running

systemctl status rabbitmq-server

Problem: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status

systemctl status awx-celery-worker

Cause: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.

Fix: Stop all AWX services, reset RabbitMQ, start all AWX services

# Stop all AWX services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
 
# Reset RabbitMQ
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
 
# Start all AWX services
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web