====== Ansible AWX ======

**General Information**

Installation and operational notes for Ansible AWX (Tower).

"AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived." 

Resources
  * Github Project: https://github.com/ansible/awx
  * Ansible AWX FAQ: https://www.ansible.com/products/awx-project/faq
  * Copr RPM Repo: https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/
    * Github for Copr RPM repo: https://github.com/MrMEEE/awx-build

----

====== Install ======

Start with a CentOS 7 minimal install.

\\
Postgresql 9.6 required for AWX. Add the repo
<code bash>yum install -y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm</code>

\\
Install Postgresql 9.6 Database and other required packages
<code bash>
yum install postgresql96-server rabbitmq-server wget memcached nginx
</code>

\\
Add AWX Dev Repo
<code bash>
wget -O /etc/yum.repos.d/awx-rpm.repo https://copr.fedorainfracloud.org/coprs/mrmeee/awx-dev/repo/epel-7/mrmeee-awx-dev-epel-7.repo
</code>

\\
Install AWX
<code bash>
yum install awx
</code>

----

====== Initial Setup ======

Initialize the database
<code bash>
/usr/pgsql-9.6/bin/postgresql96-setup initdb
</code>

\\
Configure memcached to listen locally
<code bash>
vim /etc/sysconfig/memcached 

# Daemon
USER="memcached"
 
# Reserved Cache in MBs (Default: 64)
CACHESIZE="512"
 
# Memcached Options - Listen on localhost only
OPTIONS="-l 127.0.0.1"
 
# Networking
PORT="11211"
MAXCONN="1024"
</code>

\\
Configure rabbitmq-server to listen locally
<code bash>
vim /etc/rabbitmq/rabbitmq.config

# Uncomment the following (and delete trailing comma in ipv6 line)
{tcp_listeners, [{"127.0.0.1", 5672},
                    {"::1",       5672}]}
</code>

----

===== Services =====

Start and Enable some services.

\\
Start/Enable Rabbit
<code bash>
systemctl start rabbitmq-server
systemctl enable rabbitmq-server
</code>

\\
Start/Enable Memcached
<code bash>
systemctl start memcached
systemctl enable memcached
</code>

\\
Start/Enable Postgresql
<code bash>
systemctl start postgresql-9.6
systemctl enable postgresql-9.6
</code>

----

===== Database Setup =====

Create Postgres user (awx) and database
<code bash>
su - postgres
createuser -S awx
createdb -O awx awx

exit
</code>

\\
Workaround for 1.0.5.32 and up: Comment out CELERY_QUEUES line<code bash>vim /etc/awx/settings.py

#CELERY_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
</code>
  * If you don't comment out the above line, migrations next will fail with an error.

\\
Workaround for 1.6.8 and up: Comment out the CELERY_ROUTES lines<code bash>vim /etc/awx/settings.py

#CELERY_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
#CELERY_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}</code>

\\
Migrate AWX App data into the database (fyi; this is a Django app)
<code bash>
sudo -u awx /opt/awx/bin/awx-manage migrate
</code>

\\
Initialize the AWX Django App: Create admin user
<code bash>
echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | sudo -u awx /opt/awx/bin/awx-manage shell
</code>

\\
Initialize the AWX Django App: Add tower instance (AWX Server)
<code bash>
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
</code>

\\
Initialize the AWX Django App: Create some pre-loaded organization data
<code bash>
sudo -u awx /opt/awx/bin/awx-manage create_preload_data
</code>

\\
Initialize the AWX Django App: Create a queue group
<code bash>
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)
</code>

----

===== Proxy Setup =====

Nginx will act as the proxy to the AWX application.

\\
Configure Nginx - Main Config (/etc/nginx/nginx.conf)
<code bash>
## NGINX - Main Configuration ##

# Context: Main - General Server Configuration

# User that worker processes run as
user  nginx;

# Number of worker processes (auto = set to number of CPUs)
worker_processes  auto;

# Error Log and PID of main process
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


# Context: Events - Connection Processing
events {
  # Max number of connections per worker process
  worker_connections  1024;
}

# Context: HTTP - HTTP Server Directives
http {
  # MIME - Include file and default type
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  # Logging: Format and Main Access Log
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;

  # server_tokens off - Disable nginx version on error pages and response headers
  server_tokens off;

  ## Headers - Add additional headers ##
  # X-Frame-Options SAMEORIGIN -> Page can only be displayed in a frame on same origin
  add_header X-Frame-Options SAMEORIGIN;

  # X-Content-Type-Options nosniff -> Prevent MIME Type Attacks
  add_header X-Content-Type-Options nosniff;

  # X-XSS-Protection "1; mode=block" -> Prevent Some Cross Site Scripting
  #   1;mode=block -> XSS filter enabled, prevent rendering the page if attack detected
  add_header X-XSS-Protection "1; mode=block" always;

  # Content-Security-Policy -> Prevent XSS, clickjacking, code injection
  add_header Content-Security-Policy "default-src 'self';" always;

  # Combined directives: sendfile, tcp_nopush, tcp_nodelay all on
  # sendfile+tcp_nopush = use kernel dma to fill packets up to MSS, then send
  # tcp_nodelay = once the last packet is reached, tcp_nopush auto turned off,
  #               then tcp_nodelay forces the fast sending of the last data

  # Sendfile - Send files directly in kernel space
  # on -> keep on for locally stored files
  # off -> turn off for files served over network mounted storage
  sendfile        on;

  # tcp_nopush - Do not send data until packet reaches MSS
  # Dependency: sendfile MUST be on for this to work
  #tcp_nopush     on;

  # tcp_nodelay -  Send packets in buffer as soon as they are available
  #tcp_nodelay on;

  # Server side keepalive timeout in seconds (default: 75)
  keepalive_timeout  65;

  # Gzip - Compress responses using gzip
  #gzip  on;

  # AWX ADDED: Connection upgrade
  map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
  }

  # Include enabled configurations
  include /etc/nginx/conf.d/enabled/*.conf;

  # AWX ADDED: Upstream Apps
  upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }
}
</code>

\\
Configure Nginx - AWX Drop in Config (/etc/nginx/conf.d/available/awx.conf)
<code bash>
## Default Config - Catch All Matches ##

# HTTP (Port 80)
server {
    listen 80 default_server;
    server_name  _;

    # Redirect everything to HTTPS
    return 301 https://$http_host$request_uri;
}

# HTTPS (Port 443)
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;

    # HSTS (HTTPS Strict Transport Security)
    # 63072000 seconds = 2 years
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";

    # SSL - Certificate Config
    ssl on;
    ssl_certificate /etc/pki/tls/current_cert.crt;
    ssl_certificate_key /etc/pki/tls/current_key.key;
    ssl_client_certificate /etc/pki/tls/current_ca.crt;

    # SSL - Session Config
    ssl_session_timeout 5m;
    ssl_session_cache shared:SSL:50m;

    # SSL - Protocols and Ciphers
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "HIGH:!AECDH:!DHE:!EDH:!RC4:!ADH:!3DES:!MEDIUM";

    # Locations for AWX
    location /static/ {
            alias /opt/awx/static/;
     }

    location /favicon.ico { alias /opt/awx/static/favicon.ico; }

    location /websocket {
      # Pass request to the upstream alias
      proxy_pass http://daphne;
      # Require http version 1.1 to allow for upgrade requests
      proxy_http_version 1.1;
      # We want proxy_buffering off for proxying to websockets.
      proxy_buffering off;
      # http://en.wikipedia.org/wiki/X-Forwarded-For
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # enable this if you use HTTPS:
      proxy_set_header X-Forwarded-Proto https;
      # pass the Host: header from the client for the sake of redirects
      proxy_set_header Host $http_host;
      # We've set the Host header, so we don't need Nginx to muddle
      # about with redirects
      proxy_redirect off;
      # Depending on the request value, set the Upgrade and
      # connection headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    }

    # Location: Webserver root
    location / {
      # autoindex off - Disable directory listing output
      autoindex off;
      uwsgi_read_timeout 120s;
      uwsgi_pass uwsgi;
      include /etc/nginx/uwsgi_params;
    }
}
</code>

\\
Deploy your SSL certificates as (tip: use symlinks so you never have to update the nginx config file)
  * /etc/pki/tls/current_cert.crt
  * /etc/pki/tls/current_key.key
  * /etc/pki/tls/current_ca.crt

\\
Start/Enable Nginx
<code bash>
systemctl start nginx
systemctl enable nginx
</code>

----

===== AWX Services =====

Start/Enable AWX Services
<code bash>
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
systemctl enable awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
</code>

----

====== Upgrade Steps ======

**WARNING:** Upgrading to the newest AWX is **not guaranteed to work and might break your install**. The project is fast moving and does not currently support upgrade paths.

===== AWX RPM Method =====

To upgrade:

  * Stop all services<code bash>systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code>
  * Upgrade AWX<code bash>yum update awx</code>
  * Make Migrations<code bash>sudo -u awx /opt/awx/bin/awx-manage makemigrations</code>
  * Migrate Database changes<code bash>sudo -u awx /opt/awx/bin/awx-manage migrate</code>
  * Ensure other users still have read/execute permissions on awx directory<code bash>chmod o+rx /var/lib/awx</code>
  * Clear RabbitMQ Queues<code bash>rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app</code>
  * Start all services<code bash>systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code>

----

===== AWX Team Suggested Upgrade Steps =====

To Upgrade:

  * Install cli tool<code bash>yum install ansible-tower-cli</code>
  * Export all data<code bash>awx-cli receive --organization all --team all --credential_type all --credential all --notification_template all --user all --inventory_script all --inventory all --project all --job_template all --workflow all > alldata</code>
  * Stop all AWX services<code bash>systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code>
  * Upgrade AWX<code bash>yum update awx</code>
  * Drop and then re-create the database<code bash>su - postgres -c "dropdb awx"
su - postgres -c "createdb -O awx awx"</code>
  * Migrate app data back in<code bash>sudo -u awx /opt/awx/bin/awx-manage migrate</code>
  * Initial app data setup in db<code bash>echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'test')" | sudo -u awx /opt/awx/bin/awx-manage shell
sudo -u awx /opt/awx/bin/awx-manage provision_instance --hostname=$(hostname)
sudo -u awx /opt/awx/bin/awx-manage register_queue --queuename=tower --hostnames=$(hostname)</code>
  * Import saved data<code bash>awx-cli send alldata</code>

----

====== Configuration ======

Other configuration steps.

----

===== SSH Client Settings =====

Changes to ssh client settings (/etc/ssh/ssh_config)<code bash># Disable ProxyCommand for Ansible AWX.
#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h</code>

----

===== Logos =====

Get rid of the angry potato pictures.

\\
Download AWX logos
<code bash>wget https://github.com/ansible/awx-logos/archive/master.zip</code>

\\
Install unzip utility (if not installed)
<code bash>
yum install unzip
</code>

\\
Unzip archive
<code bash>unzip master.zip</code>

\\
Copy Logos to installed asset directory
<code bash>
cp -fv awx-logos-master/awx/ui/client/assets/logo-header.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/logo-login.svg /opt/awx/static/assets/
cp -fv awx-logos-master/awx/ui/client/assets/favicon.ico /opt/awx/static/assets/
</code>
  * **Note**: The logo-header and favicon.ico will be over written at first login; you will need to login first before replacing that image or re-copy it and refresh the browser to see the changes take effect. If you are having issues getting the replacement images to show up, clear browser cache/cookies/temp files.

----

===== LDAP Authentication =====

Example configuration for LDAP.

**Tip**: The fields on this page do zero error checking as of this writing. In order to save lots of re-typing, fill out one field at a time and click Save. Leave the page and come back to see if the change stayed there (if there is a problem with it, it will be reset to default). This helps track down the field that AWX doesn't like.

\\
Configure LDAP
  * On the left navigation bar:
    * SETTINGS
    * SUB CATEGORY -> LDAP

FreeIPA Example
  * LDAP Server URI<code bash>ldap://serverldap01.mycorps.domain.org</code>
  * LDAP BIND DN<code bash>uid=ldapbinduser,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org</code>
  * LDAP BIND PASSWORD<code bash>password here</code>
  * LDAP USER DN TEMPLATE<code bash>uid=%(user)s,cn=users,cn=accounts,dc=mycorps,dc=domain,dc=org</code>
  * LDAP Group Type<code bash>MemberDNGroupType</code>
  * LDAP Require Group<code bash>cn=awxusers,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org</code>
  * LDAP Deny Group<code bash>#leave blank</code>
  * LDAP Start TLS: **On**

  * LDAP User Search<code bash>[]</code>
  * LDAP Group Search<code bash>
[
"cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org",
"SCOPE_SUBTREE",
"(objectClass=groupofnames)"
]</code>
  * LDAP User Attribute Map<code bash>
{
"first_name": "givenname",
"last_name": "sn",
"email": "mail"
}</code>
  * LDAP Group Type Parameters<code bash>
{
 "member_attr": "member",
 "name_attr": "cn"
}</code>
  * LDAP User Flags by group<code bash>
{
"is_superuser": "cn=sysadmins,cn=groups,cn=accounts,dc=mycorps,dc=domain,dc=org"
}</code>
  * LDAP Organization Map<code bash>{}</code>
  * LDAP Team Map<code bash>{}</code>

----

===== Configure for Inventory =====

  * **Create an Organization**
    * "Organizations" -> "Add"
      * Fill in:
        * Name
        * Description
        * Instance Groups (what AWX instances the Organization will use)
      * Save

  * **Add credentials for source control**
    * "Credentials"
      * Fill in:
        * Name
        * Description
        * Organization
        * Credential Type: Source Control
        * Username
        * Password (will be stored encrypted)
      * Save

  * **Add credentials for running jobs/playbooks** on the remote hosts
    * "Credentials"
      * Fill in:
        * Name
        * Description
        * Organization
        * Credential Type: Machine
        * Username
        * Password (will be stored encrypted)
        * SSH Private Key: Copy/Paste from "-----BEGIN RSA PRIVATE KEY-----" to "-----END RSA PRIVATE KEY-----"
        * Privilege escalation method: sudo
      * Save

  * **Add a project**
    * "Projects"
      * Fill in:
        * Name
        * Description
        * Organization
        * SCM Type (Git)
        * SCM URL
        * SCM Credential: select previously created
        * SCM Update Options
          * Clean
        * Save
          * Note: Initial sync begins immediate, watch progress on the "Jobs" page

  * **Schedule Regular Project Syncs**
    * "Projects"
      * To the right of the target Project, under "Actions", click the calendar ("Schedule SCM revision updates")
      * Click the green "+ADD" button
        * Fill in
           * Name (unique)
           * Start Date
           * Start Time
           * Time zone
           * Repeat (hourly, etc)
           * Every X hours
           * End (never)

  * **Sync Inventory File from Project** (Git source)
    * Create a new inventory for EACH Ansible inventory file; ie dev, test, prod.
    * "Inventories" -> "Add" -> "Inventory"
      * Fill in:
        * Name
        * Description
        * Organization
        * Instance Groups
      * Save
      * Within the same inventory config, click the "Sources" button link
      * Click the "Add Source" green button
      * Fill in:
        * Name
        * Description
        * Source: Sourced from a Project
        * Project (select previously created Project)
        * Inventory file (relative directory to project directory)
          * Example on disk: /var/lib/awx/projects/_6__my_project/inventories/development/hosts
          * Example configured: inventories/development/hosts
        * Update Options
          * Overwrite (Keep in sync with inventory source)
          * Update on Project Change (Update inventory source when Project revision number is updated)

----

===== Create a Job Template =====

AWX requires you to create a job template in order to run Playbooks cloned from source control.

The templates define default run settings for the playbooks.

\\
To Create a Job Template
  * Click "Templates" -> "Add" -> "Job Template"
    * Required Fields:
      * Name
      * Job Type (Run/Check)
      * Inventory
      * Project
      * Playbook (populated from Project)
      * Credential
      * Verbosity
    * Any field: check "Prompt on launch" in order to prompt the user to select a field when launching the Job.

\\
==== Example Template Fields to Use ====

  * **Name**: <playbook name here> Playbook
  * **Description**: Run the <playbook name here>
  * **Job Type**: Run

  * **Inventory**: <leave blank>, check "**Prompt On Launch**"
  * **Project**: System Admin Git Project
  * **Playbook**: <playbook file from drop down box>

  * **Credential**: AWX Playbook Runner
  * **Forks**: 10
  * **Limit**: <leave blank>, check "**Prompt On Launch**"

  * **Verbosity**: 0 (Normal), check "**Prompt On Launch**"
  * **Job Tags**: <leave blank>, check "**Prompt On Launch**"
  * **Skip Tags**: <leave blank>, check "**Prompt On Launch**"

  * **Labels**: <leave blank>
  * **Instance Groups**: tower
  * **Show Changes**: Off

  * **Options**
    * Enable Privilege Escalation: **checked**
    * Allow Provisioning Callbacks: not checked
    * Enable Concurrent Jobs: not checked
    * Use Fact Cache: **checked**

----

====== Operating AWX ======

AWX operations notes.

----

===== Service =====

**Enabled On Boot**

Check to see if the service is enabled on boot
<code bash>
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl is-enabled postgresql-9.6 memcached rabbitmq-server nginx

# AWX Services
systemctl is-enabled awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
</code>

\\
**Service Status**

View the service status
<code bash>
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl status postgresql-9.6 memcached rabbitmq-server nginx

# AWX Services
systemctl status awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
</code>

\\
**Service Start**

Start the services
<code bash>
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl start postgresql-9.6 memcached rabbitmq-server nginx

# AWX Services
systemctl start awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
</code>

\\
**Service Stop**

Stop the services
<code bash>
# AWX Depedencies: Database (postgres), Database caching (memcached), Message broker (rabbitmq), Web proxy (nginx)
systemctl stop postgresql-9.6 memcached rabbitmq-server nginx

# AWX Services
systemctl stop awx-cbreceiver awx-celery-beat awx-celery-worker awx-channels-worker awx-daphne awx-web
</code>

----

===== Log Files =====

Log files are located:
  * Database (Postgres): /var/log/messages
  * Database caching (memcahed): /var/log/messages
  * Message Broker (rabbitmq): /var/log/rabbitmq/
  * Web Proxy (nginx): /var/log/nginx/
  * AWX Web: /var/log/awx/web.log

----

===== Procedures =====

Common operational procedures.

==== Reboots ====

Reboot procedure and dependencies.

  * Ensure no jobs are running
    * Login to the web console: https://serverawx.mycorps.domain.org/#/login
    * On the left menu, navigate to: Views -> Jobs
      * If no job is currently running, proceed.
    * Reboot system

----

==== Running Playbooks ====

To run a playbook via Ansible AWX:
  * Login to the web portal: https://serverawx.mycorps.domain.org/#/login
  * On the left side menu, navigate to: Views > Portal Mode
  * Under the "Job Templates", find the desired template and click the rocket picture (Start a job using this template).
    * Prompt window pop up
      * **Inventory**
        * Select which inventory (environment) to run against.
      * **Other Prompts**
        * Limit (Optional): Hostnames comma separated (if wanting to limit to specific systems)
        * Verbosity: Default of 0 (Normal) is fine. Increase if you need to debug issues.
        * Job Tags (Optional): Enter tags space separated (if wanting to limit what part of the playbook gets run).
        * Skip Tags (Optional): Enter tags space separated (if wanting to SKIP certain tags).
      * **Preview**
        * Verify all settings are good, then click "Launch" to start the job.

Jobs can be monitored a few ways
  * Views > Portal Mode
    * Right side
      * Click "My Jobs" to view just jobs launched by you
      * Click "All Jobs" to view all jobs
  * Views > Jobs

----

==== Updating Playbook Runner LDAP Password ====

It is recommended to use a LDAP user account to run the playbooks and a sudoers file that prompts for password.

Examples
  * Username: awx-runner
  * Sudoers File: /etc/sudoers.d/ansible-awx
    * Password required for elevated privileges.

When the LDAP password expires:
  * Update the password in LDAP.
  * Update the password in the Ansible AWX portal
    * Login to the portal
    * Navigate to: Resources > Credentials.
    * Click the "AWX Playbook Runner" machine credential
      * At the bottom, under "Privilege Escalation Password", click "Replace"
      * Type the new password

----

===== Troubleshooting =====

Different troubleshooting scenarios and the fix. 

----

==== General Playbook Errors ====

In general, if you run into errors while running a playbook job template:
  * Increase the Verbosity and run it again.
    * Views > Portal Mode
      * Find Job Template to run, click the rocket (Start a job using this template)
      * On the "Other Prompts" screen, click the "Verbosity" drop down box and increase it to 1 or higher.

----

==== Jobs Don't Start/Celery Workers Connection Errors ====

**Problem**: Jobs in the portal never start and the celery worker is showing connection errors in its service status<code bash>systemctl status awx-celery-worker</code>

\\
**Cause**: The queuing service (celery) is unable to contact the message broker to pick up new jobs. RabbitMQ is probably not running.

\\
**Fix**: Ensure that RabbitMQ is running
<code bash>systemctl status rabbitmq-server</code>

----

==== Jobs Don't Start/Celery Workers Unknown Tag Errors ====

**Problem**: Jobs in the portal never start and the celery worker is showing unknown tag errors in its service status<code bash>systemctl status awx-celery-worker</code>

\\
**Cause**: The queuing service (celery) is unable to pickup/create messages in RabbitMQ due to residual Rabbit configuration.

\\
**Fix**: Stop all AWX services, reset RabbitMQ, start all AWX services<code bash># Stop all AWX services
systemctl stop awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web

# Reset RabbitMQ
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

# Start all AWX services
systemctl start awx-celery-worker awx-cbreceiver awx-celery-beat awx-channels-worker awx-daphne awx-web</code>

----