GUnicorn - Uvicorn - Flask - Ansible Playbook fails

[gitsridhar]

Initial Checks

• I confirm this was discussed, and the maintainers suggest I open an issue.
• I'm aware that if I created this issue without a discussion, it may be closed without a response.

Discussion Link

Let me know how to initiate the discussion, but I provided the details in the description.

Description

We have a python application executing Ansible playbook running under Gunicorn -> Uvicorn -> Flask environment. This application responds to http/https requests and it fails while executing ansible playbook.
The error message is from ansible code, 'A worker was found in a dead state, rc=0'. This problem is found to be from /lib/python3.9/site-packages/ansible/executor/task_queue_manager.py (has_dead_workers method).

The _wait_on_pending_results() function will generate an AnsibleError exception ("A worker was found in a dead state") when has_dead_workers() returns True

• Ansible runs the playbook and execute the different tasks in order. We have only one thread running at the same time.  We see that the thread executing one of the playbook tasks exits with status = 1, which terminates the swagger server request in error. This can be on any task in the playbook, even for example "if pki exist.." which will just look if the file exists.
  3456567 0.000158 exit_group(1) = ?
  3456567 0.000008 +++ exited with 1 +++
  3456441 0.000044 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3456567, si_uid=0, si_status=1, si_utime=1, si_stime=2} ---
  3456441 0.000061 wait4(3456567, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 3456567

Ansible uses an unique thread for each task described in the playbook and one by one
When we run with uvicorn, each thread exits with si_status = 1

The test to know if the thread exits in error (and therefore we stop everything) is carried out in a loop but with other exit checks.
So it all depends on the timing of each check.

This explains why ansible exits with an error (A worker was found in a dead state), on one task or another. Is random

But the main question: why the si_status is equal to 1

When the ansible playbook is executed directly, there is no problem.

This problem is reproducible with the following command:
uvicorn --host 10.241.88.185 --port 8080 --factory swagger_server.wsgi:

the code in the app runs an ansible playbook.

When the run_playbook method of ansible is executed in the main thread of the uvicorn app, it works fine. The issue is only when the code is executed via swagger environment.

Example Code

No response

Python, Uvicorn & OS Version

# uvicorn --version
Running uvicorn 0.30.5 with CPython 3.9.19 on Linux

[gitsridhar]

If we use Python 3.8 instead of Python 3.9, this problem disappears.

[saanpritom]

I am also facing a similar type of issue but somewhat different. I am using FastAPI with Gunicorn and Unicorn worker types. My Ansible playbook executes using a FastAPI Post API and initiates the playbook using FastAPI Background Job.

My problem is Ansible playbook execution failed when the number of Gunicorn workers is more than 1. When the worker number is 1 Ansible succeeds.

In my case, the Ansible is not failing actually it's just executing a dependent hosts task before completing the parent host tasks which is causing the failures. What is happening here?