linux_wiki:ansible_awx

Ansible AWX

General Information

Installation and operational notes for Ansible AWX (Tower).

“AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.”

Resources


Install

Start with a CentOS 7 minimal install.


Postgresql 9.6 required for AWX. Add the repo

yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm


Install Postgresql 9.6 Database and other required packages

yum install postgresql96-server rabbitmq-server wget memcached nginx


Add AWX Dev Repo

wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo


Install AWX

yum install awx

Initial Setup

Initialize the database

/usr/pgsql-9.6/bin/postgresql96-setup initdb


Configure memcached to listen locally

vim /etc/sysconfig/memcached 
 
# Daemon
USER="memcached"
 
# Reserved Cache in MBs (Default: 64)
CACHESIZE="512"
 
# Memcached Options - Listen on localhost only
OPTIONS="-l 127.0.0.1"
 
# Networking
PORT="11211"
MAXCONN="1024"


Configure rabbitmq-server to listen locally

vim /etc/rabbitmq/rabbitmq.config
 
# Uncomment the following (and delete trailing comma in ipv6 line)
{tcp_listeners, [{"127.0.0.1", 5672},
                    {"::1",       5672}]}

Start and Enable some services.


Start/Enable Rabbit

systemctl start rabbitmq-server
systemctl enable rabbitmq-server


Start/Enable Memcached

systemctl start memcached
systemctl enable memcached


Start/Enable Postgresql

systemctl start postgresql-9.6
systemctl enable postgresql-9.6

Create Postgres user (awx) and database

su - postgres
createuser -S awx
createdb -O awx awx
 
exit


Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line

vim /etc/awx/settings.py
 
#CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
  • If you don't comment out the above line, migrations next will fail with an error.


Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines

vim /etc/awx/settings.py
 
#CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
#CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}


Migrate AWX App data into the database (fyi; this is a Django app)

sudo -u awx /opt/awx/bin/awx-manage migrate


Initialize the AWX Django App: Create admin user

echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell


Initialize the AWX Django App: Add tower instance (AWX Server)

sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)


Initialize the AWX Django App: Create some pre-loaded organization data

sudo -u awx /opt/awx/bin/awx-manage create_preload_data


Initialize the AWX Django App: Create a queue group

sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)

Nginx will act as the proxy to the AWX application.


Configure Nginx - Main Config (/etc/nginx/nginx.conf)

## NGINX - Main Configuration ##
 
# Context: Main - General Server Configuration
 
# User that worker processes run as
user  nginx;
 
# Number of worker processes (auto = set to number of CPUs)
worker_processes  auto;
 
# Error Log and PID of main process
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
 
# Context: Events - Connection Processing
events {
  # Max number of connections per worker process
  worker_connections  1024;
}
 
# Context: HTTP - HTTP Server Directives
http {
  # MIME - Include file and default type
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
 
  # Logging: Format and Main Access Log
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;
 
  # server_tokens off - Disable nginx version on error pages and response headers
  server_tokens off;
 
  ## Headers - Add additional headers ##
  # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin
  add_header X-Frame-Options SAMEORIGIN;
 
  # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks
  add_header X-Content-Type-Options nosniff;
 
  # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting
  #   1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected
  add_header X-XSS-Protection "1; mode=block" always;
 
  # Content-Security-Policy -> Prevent XSS, clickjacking, code injection
  add_header Content-Security-Policy "default-src 'self';" always;
 
  # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on
  # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send
  # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off,
  #               then tcp_nodelay forces the fast sending of the last data
 
  # Sendfile - Send files directly in kernel space
  # on -> keep on for locally stored files
  # off -> turn off for files served over network mounted storage
  sendfile        on;
 
  # tcp_nopush - Do not send data until packet reaches MSS
  # Dependency: sendfile MUST be on for this to work
  #tcp_nopush     on;
 
  # tcp_nodelay -  Send packets in buffer as soon as they are available
  #tcp_nodelay on;
 
  # Server side keepalive timeout in seconds (default: 75)
  keepalive_timeout  65;
 
  # Gzip - Compress responses using gzip
  #gzip  on;
 
  # AWX ADDED: Connection upgrade
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }
 
  # Include enabled configurations
  include /etc/nginx/conf.d/enabled/*.conf;
 
  # AWX ADDED: Upstream Apps
  upstream uwsgi {
        server 127.0.0.1:8050;
        }
 
    upstream daphne {
        server 127.0.0.1:8051;
    }
}


Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)

## Default Config - Catch All Matches ##
 
# HTTP (Port 80)
server {
    listen 80 default_server;
    server_name  _;
 
    # Redirect everything to HTTPS
    return 301 https://$http_host$request_uri;
}
 
# HTTPS (Port 443)
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;
 
    # HSTS (HTTPS Strict Transport Security)
    # 63072000 seconds = 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
 
    # SSL - Certificate Config
    ssl on;
    ssl_certificate /etc/pki/tls/current_cert.crt;
    ssl_certificate_key /etc/pki/tls/current_key.key;
    ssl_client_certificate /etc/pki/tls/current_ca.crt;
 
    # SSL - Session Config
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
 
    # SSL - Protocols and Ciphers
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM";
 
    # Locations for AWX
    location /static/ {
            alias /opt/awx/static/;
     }
 
    location /favicon.ico { alias /opt/awx/static/favicon.ico; }
 
    location /websocket {
      # Pass request to the upstream alias
      proxy_pass http://daphne;
      # Require http version 1.1 to allow for upgrade requests
      proxy_http_version 1.1;
      # We want proxy_buffering off for proxying to websockets.
      proxy_buffering off;
      # http://en.wikipedia.org/wiki/X-Forwarded-For
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # enable this if you use HTTPS:
      proxy_set_header X-Forwarded-Proto https;
      # pass the Host: header from the client for the sake of redirects
      proxy_set_header Host $http_host;
      # We've set the Host header, so we don't need Nginx to muddle
      # about with redirects
      proxy_redirect off;
      # Depending on the request value, set the Upgrade and
      # connection headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }
 
    # Location: Webserver root
    location / {
      # autoindex off - Disable directory listing output
      autoindex off;
      uwsgi_read_timeout 120s;
      uwsgi_pass uwsgi;
      include /etc/nginx/uwsgi_params;
    }
}


Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)

  • /etc/pki/tls/current_cert.crt
  • /etc/pki/tls/current_key.key
  • /etc/pki/tls/current_ca.crt


Start/Enable Nginx

systemctl start nginx
systemctl enable nginx

Start/Enable AWX Services

systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Upgrade Steps

WARNING: Upgrading to the newest AWX is not guaranteed to work and might break your install. The project is fast moving and does not currently support upgrade paths.

To upgrade:

  • Stop all services
    systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
  • Upgrade AWX
    yum update awx
  • Make Migrations
    sudo -u awx /opt/awx/bin/awx-manage makemigrations
  • Migrate Database changes
    sudo -u awx /opt/awx/bin/awx-manage migrate
  • Ensure other users still have read/execute permissions on awx directory
    chmod o+rx /var/lib/awx
  • Clear RabbitMQ Queues
    rabbitmqctl stop_app
    rabbitmqctl reset
    rabbitmqctl start_app
  • Start all services
    systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

To Upgrade:

  • Install cli tool
    yum install ansible-tower-cli
  • Export all data
    awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata
  • Stop all AWX services
    systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
  • Upgrade AWX
    yum update awx
  • Drop and then re-create the database
    su - postgres -c "dropdb awx"
    su - postgres -c "createdb -O awx awx"
  • Migrate app data back in
    sudo -u awx /opt/awx/bin/awx-manage migrate
  • Initial app data setup in db
    echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell
    sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
    sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
  • Import saved data
    awx-cli send alldata

Configuration

Other configuration steps.


Changes to ssh client settings (/etc/ssh/ssh_config)

# Disable ProxyCommand for Ansible AWX.
#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Get rid of the angry potato pictures.


Download AWX logos

wget https://github.com/ansible/awx-logos/archive/master.zip


Install unzip utility (if not installed)

yum install unzip


Unzip archive

unzip master.zip


Copy Logos to installed asset directory

cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
  • Note: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.

Example configuration for LDAP.

Tip: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.


Configure LDAP

  • On the left navigation bar:
    • SETTINGS
    • SUB CATEGORY → LDAP

FreeIPA Example

  • LDAP Server URI
    ldap://serverldap01.mycorps.domain.org
  • LDAP BIND DN
    uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP BIND PASSWORD
    password here
  • LDAP USER DN TEMPLATE
    uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP Group Type
    MemberDNGroupType
  • LDAP Require Group
    cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP Deny Group
    #leave blank
  • LDAP Start TLS: On
  • LDAP User Search
    []
  • LDAP Group Search
    [
    "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org",
    "SCOPE_SUBTREE",
    "(objectClass=groupofnames)"
    ]
  • LDAP User Attribute Map
    {
    "first_name": "givenname",
    "last_name": "sn",
    "email": "mail"
    }
  • LDAP Group Type Parameters
    {
     "member_attr": "member",
     "name_attr": "cn"
    }
  • LDAP User Flags by group
    {
    "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org"
    }
  • LDAP Organization Map
    {}
  • LDAP Team Map
    {}

  • Create an Organization
    • “Organizations” → “Add”
      • Fill in:
        • Name
        • Description
        • Instance Groups (what AWX instances the Organization will use)
      • Save
  • Add credentials for source control
    • “Credentials”
      • Fill in:
        • Name
        • Description
        • Organization
        • Credential Type: Source Control
        • Username
        • Password (will be stored encrypted)
      • Save
  • Add credentials for running jobs/playbooks on the remote hosts
    • “Credentials”
      • Fill in:
        • Name
        • Description
        • Organization
        • Credential Type: Machine
        • Username
        • Password (will be stored encrypted)
        • SSH Private Key: Copy/Paste from “—–BEGIN RSA PRIVATE KEY—–” to “—–END RSA PRIVATE KEY—–”
        • Privilege escalation method: sudo
      • Save
  • Add a project
    • “Projects”
      • Fill in:
        • Name
        • Description
        • Organization
        • SCM Type (Git)
        • SCM URL
        • SCM Credential: select previously created
        • SCM Update Options
          • Clean
        • Save
          • Note: Initial sync begins immediate, watch progress on the “Jobs” page
  • Schedule Regular Project Syncs
    • “Projects”
      • To the right of the target Project, under “Actions”, click the calendar (“Schedule SCM revision updates”)
      • Click the green “+ADD” button
        • Fill in
          • Name (unique)
          • Start Date
          • Start Time
          • Time zone
          • Repeat (hourly, etc)
          • Every X hours
          • End (never)
  • Sync Inventory File from Project (Git source)
    • Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
    • “Inventories” → “Add” → “Inventory”
      • Fill in:
        • Name
        • Description
        • Organization
        • Instance Groups
      • Save
      • Within the same inventory config, click the “Sources” button link
      • Click the “Add Source” green button
      • Fill in:
        • Name
        • Description
        • Source: Sourced from a Project
        • Project (select previously created Project)
        • Inventory file (relative directory to project directory)
          • Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
          • Example configured: inventories/development/hosts
        • Update Options
          • Overwrite (Keep in sync with inventory source)
          • Update on Project Change (Update inventory source when Project revision number is updated)

AWX requires you to create a job template in order to run Playbooks cloned from source control.

The templates define default run settings for the playbooks.


To Create a Job Template

  • Click “Templates” → “Add” → “Job Template”
    • Required Fields:
      • Name
      • Job Type (Run/Check)
      • Inventory
      • Project
      • Playbook (populated from Project)
      • Credential
      • Verbosity
    • Any field: check “Prompt on launch” in order to prompt the user to select a field when launching the Job.


  • Name: <playbook name here> Playbook
  • Description: Run the <playbook name here>
  • Job Type: Run
  • Inventory: <leave blank>, check “Prompt On Launch
  • Project: System Admin Git Project
  • Playbook: <playbook file from drop down box>
  • Credential: AWX Playbook Runner
  • Forks: 10
  • Limit: <leave blank>, check “Prompt On Launch
  • Verbosity: 0 (Normal), check “Prompt On Launch
  • Job Tags: <leave blank>, check “Prompt On Launch
  • Skip Tags: <leave blank>, check “Prompt On Launch
  • Labels: <leave blank>
  • Instance Groups: tower
  • Show Changes: Off
  • Options
    • Enable Privilege Escalation: checked
    • Allow Provisioning Callbacks: not checked
    • Enable Concurrent Jobs: not checked
    • Use Fact Cache: checked

Operating AWX

AWX operations notes.


Enabled On Boot

Check to see if the service is enabled on boot

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Status

View the service status

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl status postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Start

Start the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl start postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Stop

Stop the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl stop postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Log files are located:

  • Database (Postgres): /var/log/messages
  • Database caching (memcahed): /var/log/messages
  • Message Broker (rabbitmq): /var/log/rabbitmq/
  • Web Proxy (nginx): /var/log/nginx/
  • AWX Web: /var/log/awx/web.log

Common operational procedures.

Reboot procedure and dependencies.


To run a playbook via Ansible AWX:

  • On the left side menu, navigate to: Views > Portal Mode
  • Under the “Job Templates”, find the desired template and click the rocket picture (Start a job using this template).
    • Prompt window pop up
      • Inventory
        • Select which inventory (environment) to run against.
      • Other Prompts
        • Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems)
        • Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues.
        • Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run).
        • Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags).
      • Preview
        • Verify all settings are good, then click “Launch” to start the job.

Jobs can be monitored a few ways

  • Views > Portal Mode
    • Right side
      • Click “My Jobs” to view just jobs launched by you
      • Click “All Jobs” to view all jobs
  • Views > Jobs

It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.

Examples

  • Username: awx-runner
  • Sudoers File: /etc/sudoers.d/ansible-awx
    • Password required for elevated privileges.

When the LDAP password expires:

  • Update the password in LDAP.
  • Update the password in the Ansible AWX portal
    • Login to the portal
    • Navigate to: Resources > Credentials.
    • Click the “AWX Playbook Runner” machine credential
      • At the bottom, under “Privilege Escalation Password”, click “Replace”
      • Type the new password

Different troubleshooting scenarios and the fix.


In general, if you run into errors while running a playbook job template:

  • Increase the Verbosity and run it again.
    • Views > Portal Mode
      • Find Job Template to run, click the rocket (Start a job using this template)
      • On the “Other Prompts” screen, click the “Verbosity” drop down box and increase it to 1 or higher.

Problem: Jobs in the portal never start and the celery worker is showing connection errors in its service status

systemctl status awx-celery-worker


Cause: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.


Fix: Ensure that RabbitMQ is running

systemctl status rabbitmq-server

Problem: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status

systemctl status awx-celery-worker


Cause: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.


Fix: Stop all AWX services, reset RabbitMQ, start all AWX services

# Stop all AWX services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
 
# Reset RabbitMQ
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
 
# Start all AWX services
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

  • linux_wiki/ansible_awx.txt
  • Last modified: 2019/05/25 23:50
  • (external edit)