linux_wiki:ansible_awx

This is an old revision of the document!


Ansible AWX

General Information

Installation and operational notes for Ansible AWX (Tower).

“AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.”

Resources


Install

Start with a CentOS 7 minimal install.


Postgresql 9.6 required for AWX. Add the repo

yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm


Install Postgresql 9.6 Database and other required packages

yum install postgresql96-server rabbitmq-server wget memcached nginx


Add AWX Dev Repo

wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo


Install AWX

yum install awx

Initial Setup

Initialize the database

/usr/pgsql-9.6/bin/postgresql96-setup initdb


Configure memcached to listen locally

vim /etc/sysconfig/memcached 
 
# Daemon
USER="memcached"
 
# Reserved Cache in MBs (Default: 64)
CACHESIZE="512"
 
# Memcached Options - Listen on localhost only
OPTIONS="-l 127.0.0.1"
 
# Networking
PORT="11211"
MAXCONN="1024"


Configure rabbitmq-server to listen locally

vim /etc/rabbitmq/rabbitmq.config
 
# Uncomment the following (and delete trailing comma in ipv6 line)
{tcp_listeners, [{"127.0.0.1", 5672},
                    {"::1",       5672}]}

Start and Enable some services.


Start/Enable Rabbit

systemctl start rabbitmq-server
systemctl enable rabbitmq-server


Start/Enable Memcached

systemctl start memcached
systemctl enable memcached


Start/Enable Postgresql

systemctl start postgresql-9.6
systemctl enable postgresql-9.6

Create Postgres user (awx) and database

su - postgres
createuser -S awx
createdb -O awx awx
 
exit


Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line

vim /etc/awx/settings.py
 
#CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
  • If you don't comment out the above line, migrations next will fail with an error.


Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines

vim /etc/awx/settings.py
 
#CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
#CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}


Migrate AWX App data into the database (fyi; this is a Django app)

sudo -u awx /opt/awx/bin/awx-manage migrate


Initialize the AWX Django App: Create admin user

echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell


Initialize the AWX Django App: Add tower instance (AWX Server)

sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)


Initialize the AWX Django App: Create some pre-loaded organization data

sudo -u awx /opt/awx/bin/awx-manage create_preload_data


Initialize the AWX Django App: Create a queue group

sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)

Nginx will act as the proxy to the AWX application.


Configure Nginx - Main Config (/etc/nginx/nginx.conf)

## NGINX - Main Configuration ##
 
# Context: Main - General Server Configuration
 
# User that worker processes run as
user  nginx;
 
# Number of worker processes (auto = set to number of CPUs)
worker_processes  auto;
 
# Error Log and PID of main process
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
 
 
# Context: Events - Connection Processing
events {
  # Max number of connections per worker process
  worker_connections  1024;
}
 
# Context: HTTP - HTTP Server Directives
http {
  # MIME - Include file and default type
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
 
  # Logging: Format and Main Access Log
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;
 
  # server_tokens off - Disable nginx version on error pages and response headers
  server_tokens off;
 
  ## Headers - Add additional headers ##
  # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin
  add_header X-Frame-Options SAMEORIGIN;
 
  # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks
  add_header X-Content-Type-Options nosniff;
 
  # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting
  #   1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected
  add_header X-XSS-Protection "1; mode=block" always;
 
  # Content-Security-Policy -> Prevent XSS, clickjacking, code injection
  add_header Content-Security-Policy "default-src 'self';" always;
 
  # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on
  # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send
  # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off,
  #               then tcp_nodelay forces the fast sending of the last data
 
  # Sendfile - Send files directly in kernel space
  # on -> keep on for locally stored files
  # off -> turn off for files served over network mounted storage
  sendfile        on;
 
  # tcp_nopush - Do not send data until packet reaches MSS
  # Dependency: sendfile MUST be on for this to work
  #tcp_nopush     on;
 
  # tcp_nodelay -  Send packets in buffer as soon as they are available
  #tcp_nodelay on;
 
  # Server side keepalive timeout in seconds (default: 75)
  keepalive_timeout  65;
 
  # Gzip - Compress responses using gzip
  #gzip  on;
 
  # AWX ADDED: Connection upgrade
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }
 
  # Include enabled configurations
  include /etc/nginx/conf.d/enabled/*.conf;
 
  # AWX ADDED: Upstream Apps
  upstream uwsgi {
        server 127.0.0.1:8050;
        }
 
    upstream daphne {
        server 127.0.0.1:8051;
    }
}


Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)

## Default Config - Catch All Matches ##
 
# HTTP (Port 80)
server {
    listen 80 default_server;
    server_name  _;
 
    # Redirect everything to HTTPS
    return 301 https://$http_host$request_uri;
}
 
# HTTPS (Port 443)
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;
 
    # HSTS (HTTPS Strict Transport Security)
    # 63072000 seconds = 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
 
    # SSL - Certificate Config
    ssl on;
    ssl_certificate /etc/pki/tls/current_cert.crt;
    ssl_certificate_key /etc/pki/tls/current_key.key;
    ssl_client_certificate /etc/pki/tls/current_ca.crt;
 
    # SSL - Session Config
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;
 
    # SSL - Protocols and Ciphers
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM";
 
    # Locations for AWX
    location /static/ {
            alias /opt/awx/static/;
     }
 
    location /favicon.ico { alias /opt/awx/static/favicon.ico; }
 
    location /websocket {
      # Pass request to the upstream alias
      proxy_pass http://daphne;
      # Require http version 1.1 to allow for upgrade requests
      proxy_http_version 1.1;
      # We want proxy_buffering off for proxying to websockets.
      proxy_buffering off;
      # http://en.wikipedia.org/wiki/X-Forwarded-For
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # enable this if you use HTTPS:
      proxy_set_header X-Forwarded-Proto https;
      # pass the Host: header from the client for the sake of redirects
      proxy_set_header Host $http_host;
      # We've set the Host header, so we don't need Nginx to muddle
      # about with redirects
      proxy_redirect off;
      # Depending on the request value, set the Upgrade and
      # connection headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }
 
    # Location: Webserver root
    location / {
      # autoindex off - Disable directory listing output
      autoindex off;
      uwsgi_read_timeout 120s;
      uwsgi_pass uwsgi;
      include /etc/nginx/uwsgi_params;
    }
}


Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)

  • /etc/pki/tls/current_cert.crt
  • /etc/pki/tls/current_key.key
  • /etc/pki/tls/current_ca.crt


Start/Enable Nginx

systemctl start nginx
systemctl enable nginx

Start/Enable AWX Services

systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Upgrade Steps

WARNING: Upgrading to the newest AWX is not guaranteed to work and might break your install. The project is fast moving and does not currently support upgrade paths.

To upgrade:

  • Stop all services
    systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
  • Upgrade AWX
    yum update awx
  • Make Migrations
    sudo -u awx /opt/awx/bin/awx-manage makemigrations
  • Migrate Database changes
    sudo -u awx /opt/awx/bin/awx-manage migrate
  • Ensure other users still have read/execute permissions on awx directory
    chmod o+rx /var/lib/awx
  • Clear RabbitMQ Queues
    rabbitmqctl stop_app
    rabbitmqctl reset
    rabbitmqctl start_app
  • Start all services
    systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

To Upgrade:

  • Install cli tool
    yum install ansible-tower-cli
  • Export all data
    awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata
  • Stop all AWX services
    systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web
  • Upgrade AWX
    yum update awx
  • Drop and then re-create the database
    su - postgres -c "dropdb awx"
    su - postgres -c "createdb -O awx awx"
  • Migrate app data back in
    sudo -u awx /opt/awx/bin/awx-manage migrate
  • Initial app data setup in db
    echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell
    sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
    sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
  • Import saved data
    awx-cli send alldata

Configuration

Other configuration steps.


Changes to ssh client settings (/etc/ssh/ssh_config)

# Disable ProxyCommand for Ansible AWX.
#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Get rid of the angry potato pictures.


Download AWX logos

wget https://github.com/ansible/awx-logos/archive/master.zip


Install unzip utility (if not installed)

yum install unzip


Unzip archive

unzip master.zip


Copy Logos to installed asset directory

cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
  • Note: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.

Example configuration for LDAP.

Tip: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.


Configure LDAP

  • On the left navigation bar:
    • SETTINGS
    • SUB CATEGORY → LDAP

FreeIPA Example

  • LDAP Server URI
    ldap://serverldap01.mycorps.domain.org
  • LDAP BIND DN
    uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP BIND PASSWORD
    password here
  • LDAP USER DN TEMPLATE
    uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP Group Type
    MemberDNGroupType
  • LDAP Require Group
    cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org
  • LDAP Deny Group
    #leave blank
  • LDAP Start TLS: On
  • LDAP User Search
    []
  • LDAP Group Search
    [
    "cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org",
    "SCOPE_SUBTREE",
    "(objectClass=groupofnames)"
    ]
  • LDAP User Attribute Map
    {
    "first_name": "givenname",
    "last_name": "sn",
    "email": "mail"
    }
  • LDAP Group Type Parameters
    {
     "member_attr": "member",
     "name_attr": "cn"
    }
  • LDAP User Flags by group
    {
    "is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org"
    }
  • LDAP Organization Map
    {}
  • LDAP Team Map
    {}

  • Create an Organization
    • “Organizations” → “Add”
      • Fill in:
        • Name
        • Description
        • Instance Groups (what AWX instances the Organization will use)
      • Save
  • Add credentials for source control
    • “Credentials”
      • Fill in:
        • Name
        • Description
        • Organization
        • Credential Type: Source Control
        • Username
        • Password (will be stored encrypted)
      • Save
  • Add credentials for running jobs/playbooks on the remote hosts
    • “Credentials”
      • Fill in:
        • Name
        • Description
        • Organization
        • Credential Type: Machine
        • Username
        • Password (will be stored encrypted)
        • SSH Private Key: Copy/Paste from “—–BEGIN RSA PRIVATE KEY—–” to “—–END RSA PRIVATE KEY—–”
        • Privilege escalation method: sudo
      • Save
  • Add a project
    • “Projects”
      • Fill in:
        • Name
        • Description
        • Organization
        • SCM Type (Git)
        • SCM URL
        • SCM Credential: select previously created
        • SCM Update Options
          • Clean
        • Save
          • Note: Initial sync begins immediate, watch progress on the “Jobs” page
  • Schedule Regular Project Syncs
    • “Projects”
      • To the right of the target Project, under “Actions”, click the calendar (“Schedule SCM revision updates”)
      • Click the green “+ADD” button
        • Fill in
          • Name (unique)
          • Start Date
          • Start Time
          • Time zone
          • Repeat (hourly, etc)
          • Every X hours
          • End (never)
  • Sync Inventory File from Project (Git source)
    • Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
    • “Inventories” → “Add” → “Inventory”
      • Fill in:
        • Name
        • Description
        • Organization
        • Instance Groups
      • Save
      • Within the same inventory config, click the “Sources” button link
      • Click the “Add Source” green button
      • Fill in:
        • Name
        • Description
        • Source: Sourced from a Project
        • Project (select previously created Project)
        • Inventory file (relative directory to project directory)
          • Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
          • Example configured: inventories/development/hosts
        • Update Options
          • Overwrite (Keep in sync with inventory source)
          • Update on Project Change (Update inventory source when Project revision number is updated)

AWX requires you to create a job template in order to run Playbooks cloned from source control.

The templates define default run settings for the playbooks.


To Create a Job Template

  • Click “Templates” → “Add” → “Job Template”
    • Required Fields:
      • Name
      • Job Type (Run/Check)
      • Inventory
      • Project
      • Playbook (populated from Project)
      • Credential
      • Verbosity
    • Any field: check “Prompt on launch” in order to prompt the user to select a field when launching the Job.


  • Name: <playbook name here> Playbook
  • Description: Run the <playbook name here>
  • Job Type: Run
  • Inventory: <leave blank>, check “Prompt On Launch
  • Project: System Admin Git Project
  • Playbook: <playbook file from drop down box>
  • Credential: AWX Playbook Runner
  • Forks: 10
  • Limit: <leave blank>, check “Prompt On Launch
  • Verbosity: 0 (Normal), check “Prompt On Launch
  • Job Tags: <leave blank>, check “Prompt On Launch
  • Skip Tags: <leave blank>, check “Prompt On Launch
  • Labels: <leave blank>
  • Instance Groups: tower
  • Show Changes: Off
  • Options
    • Enable Privilege Escalation: checked
    • Allow Provisioning Callbacks: not checked
    • Enable Concurrent Jobs: not checked
    • Use Fact Cache: checked

Operating AWX

AWX operations notes.


Enabled On Boot

Check to see if the service is enabled on boot

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Status

View the service status

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl status postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Start

Start the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl start postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web


Service Stop

Stop the services

# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl stop postgresql-9.6 memcached rabbitmq-server nginx
 
# AWX Services
systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web

Log files are located:

  • Database (Postgres): /var/log/messages
  • Database caching (memcahed): /var/log/messages
  • Message Broker (rabbitmq): /var/log/rabbitmq/
  • Web Proxy (nginx): /var/log/nginx/
  • AWX Web: /var/log/awx/web.log

  • linux_wiki/ansible_awx.1530936377.txt.gz
  • Last modified: 2019/05/25 23:50
  • (external edit)