Intermittent failures reading from vault

[eshupps]

We have set up dozens of tests, most of which are scheduled to run every 15 minutes against our API's defined in APIC. Each test includes the X-IBM-Client-Id header required for APIC authorization. The value is stored in a vault variable named XIBMClientId. In the vast majority of test run instances this value is read properly from the vault; however, occasionally it returns a null value, causing the APIC call to result in a 401 unauthorized response. Here is an example of the same test at two different run times sending first a proper variable then a null value:

Good

Test Run: Fri 11 Sep 2020 12:50 PM UTC

POST https:///api/

POST /api/***** HTTP/1.1
host: .com
Content-Type: application/json
Accept: application/json
X-IBM-Client-Id: ********----**********
user-agent: Apache-HttpClient/4.3.2 (java 1.5)

{
"source": "VGhlIGJyb3duIGNvdw=="
}

Bad

Test run: Fri 11 Sep 2020 08:35 AM UTC

POST https:///api/

POST /api/***** HTTP/1.1
host: ******.com
Content-Type: application/json
Accept: application/json
X-IBM-Client-Id: null
user-agent: Apache-HttpClient/4.3.2 (java 1.5)

{
"source": "VGhlIGJyb3duIGNvdw=="
}

Note the null value for the X-IBM-Client-Id header in the second request. Same exact test, no changes have been made, several hours apart. This happens repeatedly, even when the retry option is enabled in the test settings.

[jameskhedley]

Thanks for the bug report!

We'll look into it but it could be quite a tricky one to replicate so I think the best way to deal with it is to use the retry option on the scheduled, it looks like: Try a second execution after 2m 30s if test fails

[eshupps]

Unfortunately, that doesn't mitigate the issue. I'm still getting null vault results even after adding the retry option to all of my tests.

Perhaps it's the number of concurrent read operations to the vault? I have 60 total tests, all of which occur every 15 minutes but are split into two groups - one group occurs at :05, :20, :35 and :50, the other at :00, :15, :30 and :45. So at any given time there shouldn't be more than 30 tests making read requests to the vault - perhaps that's still too many?

[jameskhedley]

The encrypted values in vault are stored in a proper database backend - it really shouldn't matter too much how many concurrent requests there are. I wonder if it's some kind of issue with the environment to be honest, we haven't been able to replicate it on our development systems.
Unfortunately it looks like this potentially would take a serious amount of time to dig into properly so it may not be resolved very quickly, given our other commitments.