Latest Module Not Working With Ansible Execution Environment

[dk-rbrown]

I recently rebuilt a privately hosted EE to troubleshoot a different issue. When I did it also upgraded the HVAC library. My playbooks that utilize the lookup method for authentication are now failing. AAP claims that the HVAC library is not installed but as you can see in the code snippet below, it successfully installs the HVAC library.

I have deleted the container and rebuilt it multiple times with no luck.

Does anyone else have this issue or have any suggestions?

Successfully built ncclient pyVmomi ovirt-engine-sdk-python vSphere-Automation-SDK future pyhcl pycurl
Installing collected packages: boto, python-dateutil, jmespath, botocore, s3transfer, boto3, charset-normalizer, certifi, requests, awxkit, ansible-pylibssh, jxmlease, setuptools, cryptography, bcrypt, pynacl, paramiko, ncclient, netaddr, xmltodict, attrs, pyrsistent, jsonschema, future, textfsm, ttp, ipaddress, scp, pyhcl, hvac, typing-extensions, azure-core, portalocker, PyJWT, msal, msal-extensions, azure-identity, pyVmomi, cachetools, pyasn1, rsa, pyasn1-modules, google-auth, websocket-client, oauthlib, requests-oauthlib, kubernetes, jsonpointer, jsonpatch, pycurl, ovirt-engine-sdk-python, multidict, frozenlist, aiosignal, yarl, async-timeout, aiohttp, pyOpenSSL, vapi-runtime, vapi-client-bindings, vapi-common-client, vmc-client-bindings, nsx-python-sdk, nsx-policy-python-sdk, nsx-vmc-policy-python-sdk, nsx-vmc-aws-integration-python-sdk, vmc-draas-client-bindings, vSphere-Automation-SDK

[dk-rbrown]

Note: I had to place a hard requirement of version 0.11 to have it work again. Still wondering if there is a bug with AAP/EE and hvac latest release.

[briantist]

Unfortunately I don't think we'll be able to do much to help within this collection. It sounds like the issue is related to one or more of:

• the EE builder
• AAP
• ansible-runner (I think AAP uses this internally? maybe not? I'm not sure)
• hvac

I am also a maintainer for hvac and I don't know of any issues with the v1.x.y release packages. That said, if a cause is found within hvac after troubleshooting the other components, do let us know!

If you have support with AAP (I don't know much about it), I'd start there.

Next I would try to use your EE outside of AAP, like with ansible-navigator or something (also uses ansible-runner I think), to see if there's a difference in behavior or not.

You could also join the Ansible community chats on Matrix or IRC (they are bridged so it's the same either way):
https://docs.ansible.com/ansible/latest/community/communication.html#ansible-community-on-matrix

There's a room for AWX:
Matrix: #awx:ansible.com | IRC: #ansible-awx

---

If you do find an answer, I would appreciate you posting it here as well, just in case other people hit the issue and think to look here. Thanks!

[nwerker]

@dk-rbrown
Can you please provide your Execution Environment Files (execution-environment.yml) and your dependency files with the parts that would include this collection and the hvac library?

For debugging purposes you can also spin up an interactive container based on your execution environment, this often promises a faster result:
podman run --rm -it <IMAGE LABEL>:<IMAGE TAG> bash
for example: podman run --rm -it localhost/your-eelabel:latest bash
and then you could validate the installation with "pip freeze".

This is probably best to debug on the execution node where the EE should be used. One quick fix could be to check the Image Hashes / Versions on said execution node and if they match the image version in your container registry. Maybe you don't set the execution environment to "always pull" in your ansible controller, which means that he would not pull a new image if the execution node already has an image with the label and tag (ignoring the image hash) that could result in the image on the execution node being old not including your hvac dependency.

@briantist ansible-runner is an integral part within EEs and AAP, it's even the base image of the EE if you don't define a different base image, but all those provided by red hat are based on ansible-runner!