Cannot signin : bot detected

[lasconic]

See https://community.finary.com/t/api-finasync-extension-open-source/10104/20

After a call to signin we get

{
    "errors": [
        {
            "message": "Bot traffic was detected. Please try again or contact support.",
            "code": "bot_detected"
        }
    ]
}

[lasconic]

My first thoughts:

• Could be as simple as a referer or user-agent check ? (I doubt but it's easy to try...)
• Could because there is no delay between the call with username and password and the call with the 2FA code... We could add a delay but it could fail because of the validity period of the 2FA code. Or we change our API, and we ask for the code interactively...

Any other ideas ?

[lasconic]

Actually, the error occurs right away on the first code, we don't even get the chance to submit the 2FA code.

[lasconic]

The following works, not sure why it doesn't work with requests yet.

curl 'https://clerk.finary.com/v1/client/sign_ins' \
  --http1.1 \
  -H 'User-Agent: finary_uapi 0.2.0' \
  -H 'Connection: keep-alive' \
  -H 'Origin: https://app.finary.com' \
  -H 'Referer: https://app.finary.com/' \
  -H 'Accept-Encoding: identity' \
  --data-raw 'password=PASSWORD&identifier=EMAIL' \
  --verbose

Translating this request exactly to requests causes the bot_detected error.
The same call using httpx works. There is something special about requests and clerk or finary detects it.

[lasconic]

@nmathey @MadeInPierre
A possible solution is to switch to httpx. There must be a subtle difference between requests and curl or httpx for requests to be detected as a bot with the configuration by default. I can't find this difference... Using httpx, most of the things seems to work for now... Is it a problem for your respective projects to use httpx instead of requests ?

[lasconic]

Note that even with httpx running from github actions doesn't work... clerk detects a bot.
Locally all the tests but 2 are passing.

[lasconic]

See https://github.com/lasconic/finary_uapi/tree/switch_httpx

[nmathey]

Moving away from requests shouldn't be a problem for me.
So from GitHub it still doesn't but you were able to get it working from your wrapper?

[lasconic]

Yes, the tests don't work from Github Actions server but they run fine from my home computer. You can try to signin with the main branch (and should get the bot detected error) and with this branch and you should get success.

[nmathey]

Got same error

INFO:httpx:HTTP Request: POST https://clerk.finary.com/v1/client/sign_ins "HTTP/1.1 403 Forbidden"
{
    "errors": [
        {
            "message": "Bot traffic was detected. Please try again or contact support.",
            "code": "bot_detected"
        }
    ]
}

Did your branch includes all headers additions you tested with curl?

--http1.1 \
 -H 'User-Agent: finary_uapi 0.2.0' \
 -H 'Connection: keep-alive' \
 -H 'Origin: https://app.finary.com' \
 -H 'Referer: https://app.finary.com/' \
 -H 'Accept-Encoding: identity' \

[lasconic]

See https://community.finary.com/t/api-finasync-extension-open-source/10104/24 for the current official status (by Finary CTO).

The behavior described in this bug is "expected". For me, the HTTPX branch works. I tried several headers but of course, I made sure to use exactly the same one while testing. I had the same header in the main branch (using requests), with curl and with HTTPX (including forcing HTTP 1.1 when possible). Clerk does smart things for their bot detection for sure.

For now, I would like to get more data. Does the main branch work for you ? Does the HTTPX branch work ?

[nmathey]

Sorry I'm bit confuse, and might be a dummy question, but can you confirm headers you successfully tested are hardcoded in your Httpx branch?
Or should I configured them myself somewhere?
Because when I tested your branch, I just did as is.

[lasconic]

@nmathey they aren't hardcoded but the HTTPX branch works for my laptop as it, without configuring anything.

[JulesGuesnon]

Hey!
I've been developing my own client with reqwest to automatically update balances of manual accounts, and I faced the same problem while developing:

• I tried to put the same headers than the browser but I was detected as a bot
• I tried CURL and it works

I was using this repo as a reference for implementation (thanks for the awesome work btw), and I naively tried to set my user-agent to: finary_uapi 0.2.0, and it worked. FYI, I'm also setting the following headers:

• Origin: https://app.finary.com
• Referer: https://app.finary.com
• Connection: keep-alive
• Accept: */*

I haven't tried anything else since then, but my guess is that Clerk detects http clients that try to behave like browsers as they're probably bots most of the times, and allows clients that are transparent and saying that they're an API.

So basically, I don't think it's an issue related to the http client, but it's probably linked to headers. Hope those information can help you!

[clemlesne]

I just pushed a PR to solve that issue: #210.

[lasconic]

Thank you for the new look. I checked every possible combinations 6 months ago but your headers seems to just work ! It can't hurt to merge the PR.

[lasconic]

@all-contributors please add @clemlesne for code

[allcontributors]

@lasconic

I've put up a pull request to add @clemlesne! 🎉

[lasconic]

@all-contributors please add @clemlesne for code

[allcontributors]

@lasconic

@clemlesne already contributed before to code

[nmathey]

Still getting same message with finary-uapi 0.2.1 b9a9a3f

{ "errors": [ { "message": "Bot traffic was detected. Please try again or contact support.", "code": "bot_detected" } ] }

Is that 100% time working on your side ?

[OxyFlax]

Don't you think Finary is actively monitoring our solutions here to update their bot detection with any found solution? I mean if they want to close their api it's not a big work to track this issue and ban any further solution proposed... :(

[lasconic]

@nmathey Locally it works for me, 100% of the time. I can run the tests without problems, and they do quite a lot of calls. You don't do any funny things ? like overriding the default headers ? Do you get this message on sign-in ? or after multiple calls ?

@OxyFlax I believe they have better things to do than monitoring our conversation... Also they did provide a demo account to test the code. I'm under they impression Finary just checked a box in Clerk admin UI for bot detection and Clerk is changing their bot detection algorithm continuously

[lasconic]

Let me add that the tests running on Github actions are not working and we still get a "bot traffic detected". See https://github.com/lasconic/finary_uapi/actions/runs/10584142604/job/29327710214#step:8:1424
Maybe Clerk uses an IP blacklist ? No idea.

[lasconic]

Yes, they provide testing tokens for development instances : https://clerk.com/docs/testing/overview#testing-tokens so they definitely do something against github actions runners.

[nmathey]

I'm using finary_uapi out of the box, and getting the error straight from on sign-in.
I'm using the same IP for both finary_uapi and browsing finary app/portal, so I don't think my IP is blacklisting (?)

[clemlesne]

I'm using finary_uapi out of the box, and getting the error straight from on sign-in.

It works for me, repeatably, for multiple days in a row.

Can you share reproduction steps?

I'm using the same IP for both finary_uapi and browsing finary app/portal, so I don't think my IP is blacklisting (?)

For sure that's not IP blacklist, at least for your residential IP. Those countermeasures are really rare with SaaS systems (imagine a VPN or a company proxy, you could disrupt hundred of legitimate users).

[nmathey]

I'm just trying the signin method (with 2FA or not):

abc@d3a3d0a14c9b:~/workspace/finary_uapi$ python -m finary_uapi signin
{
    "errors": [
        {
            "message": "Bot traffic was detected. Please try again or contact support.",
            "code": "bot_detected"
        }
    ]
}

[clemlesne]

This error should have raised an exception with a user-friendly format:

finary_uapi/finary_uapi/signin.py

Lines 61 to 63 in cf17b61

	if errors := xjson.get("errors", None):
	messages = ", ".join([error["long_message"] for error in errors])
	raise RuntimeError(f"Login failed: {messages}")

You're maybe not on the right code-base version, I guess.

[nmathey]

I'm not a git expert but my understanding is I'm using latest branch : am I correct ?

abc@d3a3d0a14c9b:~/workspace/finary_uapi$ cat .git/refs/heads/${branch-main}
cf17b614100875156e9052ac3c4864b78c4a5c1f