Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_post[http] fails in some systems #905

Open
sanvila opened this issue Jan 19, 2025 · 1 comment
Open

test_post[http] fails in some systems #905

sanvila opened this issue Jan 19, 2025 · 1 comment

Comments

@sanvila
Copy link

sanvila commented Jan 19, 2025 •
edited
Loading

Hello. As reported here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1087334

while building the Debian package for vcr.py version 7.0.0 on AWS virtual machines of type m7a.medium or r7a.medium, I get the following error:

E BrokenPipeError: [Errno 32] Broken pipe

Complete error message here:

vcr.py.txt

Incidentally, those AWS instances have 1 CPU, but we don't know if that's relevant or not (it may be the case that it fails because they are "too slow").

An error like this one started to happen simultaneously on packages vcr.py and pytest-httpbin, so we suspected of the common build-dependency python3-urllib3.

Now pytest-httpbin does not fail anymore but vcr.py still fails, so maybe there is indeed a problem in vcr.py.

Thanks.
@cjwatson
Copy link

I spent some time investigating this today. It's possible I'm missing something, but this seems to be a complex interaction of several different problems.

I'm seeing failures in tests/integration/test_urllib3.py::test_post (which has been around for a while) and tests/unit/test_stubs.py::TestVCRConnection::test_body_consumed_once_* (which were introduced in #851). In all cases, the error is essentially BrokenPipeError because the httpbin test server is closing the connection before the test has finished sending the request body. (This is likely to be somewhat racy, which is why we're only seeing this in certain circumstances.)

strace reveals that httpbin is closing the connection with a response along these lines (I've decoded it from the raw sendto calls for readability):

HTTP/1.1 501 NOT IMPLEMENTED
Date: Tue, 28 Jan 2025 14:08:45 GMT
Server: Pytest-HTTPBIN/0.1.0
Content-Type: text/html; charset=utf-8
Content-Length: 159
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: Close

<!doctype html>
<html lang=en>
<title>501 Not Implemented</title>
<h1>Not Implemented</h1>
<p>Chunked requests are not supported for server WSGIServer/0.2</p>

The relevant code is httpbin.core.before_request, which denies chunked requests unless it's running under gunicorn, to work around various bugs. It's thus hard to see how VCR.py can test chunked transfer requests using pytest-httpbin without some work elsewhere.

I tried hacking httpbin.core.before_request locally to run request.environ["wsgi.input_terminated"] = 1 instead of aborting, and the tests hung. I guess that's a relative of the flask problem mentioned in that comment in httpbin, but I wasn't able to track down the details.

While debugging this, I also found that the test_body_consumed_once_* unit tests are broken in a different way: because they don't send a Content-Type header, werkzeug doesn't bother to read the request body (I think). The simplest change there is probably to add headers={"Content-Type": "application/x-www-form-urlencoded"} to the request calls in _test_body_consumed_once and change the body iterators in the test methods from 1234567890 to a=1234567890 (and similar). I wasn't able to fully test this fix due to the problems above, though.

I don't know how to get much further here, because trying to test chunked requests using pytest-httpbin just seems kind of doomed at the moment. I also don't think this is really a new issue: kevin1024/pytest-httpbin#33 and (probably) kevin1024/pytest-httpbin#59 seem to be more or less the same thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cjwatson @sanvila