Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf · 2024-04-03T18:01:28Z

Should now correctly handle asynchronous httpx requests when called from synchronous functions. Previously, using asyncio.run() in _sync_vcr_send caused issues in environments with an existing event loop, like aiohttp servers.

Solution:

Implemented a run_async_from_sync function using ThreadPoolExecutor to execute async tasks from synchronous functions.
Replaced asyncio.run() in _sync_vcr_send with run_async_from_sync.

This change ensures httpx requests are processed correctly in vcr.py's synchronous operations.

I think this is related to #817

Here's a reproducer:

import asyncio

import httpx
import pytest
from aiohttp import web

import vcr

my_vcr = vcr.VCR(cassette_library_dir='cassettes', record_mode='once')

@my_vcr.use_cassette('test.yml')
async def handle_request(request):
    client = httpx.Client()
    try:
        response = client.get('http://httpbin.org/get')
    finally:
        client.close()
    return web.Response(text=response.text, status=response.status_code)

def create_app():
    app = web.Application()
    app.router.add_get('/', handle_request)
    return app

@pytest.mark.asyncio
async def test_server(aiohttp_client):
    app = create_app()
    client = await aiohttp_client(app)
    resp = await client.get('/')
    assert resp.status == 200

(there's some issue on my branch with an unclosed event loop but I couldn't figure it out, I think it's related to the asyncio fixture but I'm not sure)

Before:

============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/fabian/projects/github.com/kevin1024/vcrpy
configfile: pyproject.toml
plugins: anyio-3.7.1, recording-0.13.1, asyncio-0.23.6, aiohttp-1.0.5
asyncio: mode=Mode.STRICT
collected 1 item

repro.py FE                                                              [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_server _______________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x7fcb6d557c40>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: Callable[[], TResult],
        when: Literal["collect", "setup", "call", "teardown"],
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../../.local/lib/python3.12/site-packages/_pytest/runner.py:340: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../.local/lib/python3.12/site-packages/_pytest/runner.py:240: in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
../../../../.local/lib/python3.12/site-packages/pluggy/_hooks.py:501: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
../../../../.local/lib/python3.12/site-packages/pluggy/_manager.py:119: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../../../.local/lib/python3.12/site-packages/_pytest/threadexception.py:92: in pytest_runtest_teardown
    yield from thread_exception_runtest_hook()
../../../../.local/lib/python3.12/site-packages/_pytest/threadexception.py:63: in thread_exception_runtest_hook
    yield
../../../../.local/lib/python3.12/site-packages/_pytest/unraisableexception.py:95: in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            try:
                yield
            finally:
                if cm.unraisable:
                    if cm.unraisable.err_msg is not None:
                        err_msg = cm.unraisable.err_msg
                    else:
                        err_msg = "Exception ignored in"
                    msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                    msg += "".join(
                        traceback.format_exception(
                            cm.unraisable.exc_type,
                            cm.unraisable.exc_value,
                            cm.unraisable.exc_traceback,
                        )
                    )
>                   warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E                   pytest.PytestUnraisableExceptionWarning: Exception ignored in: <coroutine object _record_responses at 0x7fcb6d57bbc0>
E                   
E                   Traceback (most recent call last):
E                     File "/usr/lib64/python3.12/warnings.py", line 553, in _warn_unawaited_coroutine
E                       warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
E                   RuntimeWarning: coroutine '_record_responses' was never awaited

../../../../.local/lib/python3.12/site-packages/_pytest/unraisableexception.py:80: PytestUnraisableExceptionWarning
------------------------------ Captured log call -------------------------------
ERROR    aiohttp.server:web_protocol.py:421 Error handling request
Traceback (most recent call last):
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/_handle_coroutine.py", line 3, in handle_coroutine
    return await fn(cassette)
           ^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/repro.py", line 15, in handle_request
    response = client.get('http://httpbin.org/get')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 1055, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 828, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 184, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 177, in _sync_vcr_send
    asyncio.run(_record_responses(cassette, vcr_request, real_response, aread=False))
  File "/usr/lib64/python3.12/asyncio/runners.py", line 190, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
=================================== FAILURES ===================================
_________________________________ test_server __________________________________

aiohttp_client = <function aiohttp_client.<locals>.go at 0x7fcb6d556f20>

    @pytest.mark.asyncio
    async def test_server(aiohttp_client):
        app = create_app()
        client = await aiohttp_client(app)
        resp = await client.get('/')
>       assert resp.status == 200
E       AssertionError: assert 500 == 200
E        +  where 500 = <ClientResponse(http://127.0.0.1:33117/) [500 Internal Server Error]>\n<CIMultiDictProxy('Content-Type': 'text/plain; c...Length': '55', 'Date': 'Wed, 03 Apr 2024 19:02:16 GMT', 'Server': 'Python/3.12 aiohttp/3.9.3', 'Connection': 'close')>\n.status

repro.py:32: AssertionError
------------------------------ Captured log call -------------------------------
ERROR    aiohttp.server:web_protocol.py:421 Error handling request
Traceback (most recent call last):
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/_handle_coroutine.py", line 3, in handle_coroutine
    return await fn(cassette)
           ^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/repro.py", line 15, in handle_request
    response = client.get('http://httpbin.org/get')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 1055, in get
    return self.request(
           ^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 828, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 915, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 943, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/.local/lib/python3.12/site-packages/httpx/_client.py", line 980, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 184, in _inner_send
    return _sync_vcr_send(cassette, real_send, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fabian/projects/github.com/kevin1024/vcrpy/vcr/stubs/httpx_stubs.py", line 177, in _sync_vcr_send
    asyncio.run(_record_responses(cassette, vcr_request, real_response, aread=False))
  File "/usr/lib64/python3.12/asyncio/runners.py", line 190, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
=========================== short test summary info ============================
FAILED repro.py::test_server - AssertionError: assert 500 == 200
ERROR repro.py::test_server - pytest.PytestUnraisableExceptionWarning: Except...
========================== 1 failed, 1 error in 0.80s ==========================

After:

============================= test session starts ==============================
platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/fabian/projects/github.com/kevin1024/vcrpy
configfile: pyproject.toml
plugins: anyio-3.7.1, recording-0.13.1, asyncio-0.23.6, aiohttp-1.0.5
asyncio: mode=Mode.STRICT
collected 1 item

repro.py .E                                                              [100%]

==================================== ERRORS ====================================
_______________________ ERROR at teardown of test_server _______________________

    def _close_event_loop() -> None:
        policy = asyncio.get_event_loop_policy()
        try:
            loop = policy.get_event_loop()
        except RuntimeError:
            loop = None
        if loop is not None:
            if not loop.is_closed():
>               warnings.warn(
                    _UNCLOSED_EVENT_LOOP_WARNING % loop,
                    DeprecationWarning,
                )
E               DeprecationWarning: pytest-asyncio detected an unclosed event loop when tearing down the event_loop
E               fixture: <_UnixSelectorEventLoop running=False closed=False debug=False>
E               pytest-asyncio will close the event loop for you, but future versions of the
E               library will no longer do so. In order to ensure compatibility with future
E               versions, please make sure that:
E                   1. Any custom "event_loop" fixture properly closes the loop after yielding it
E                   2. The scopes of your custom "event_loop" fixtures do not overlap
E                   3. Your code does not modify the event loop in async fixtures or tests

../../../../.local/lib/python3.12/site-packages/pytest_asyncio/plugin.py:818: DeprecationWarning
=========================== short test summary info ============================
ERROR repro.py::test_server - DeprecationWarning: pytest-asyncio detected an ...
========================== 1 passed, 1 error in 0.65s ==========================

Resolved an issue in `vcr.py`'s handling of `httpx` calls, where asynchronous `_record_responses` could not be directly called within a synchronous `_sync_vcr_send` function due to `asyncio.run()` limitations in already running event loops. Introduced a `run_async_from_sync` utility function using `ThreadPoolExecutor` to bridge this gap, allowing async functions to run effectively in sync contexts without event loop conflicts. This fix enables seamless handling of async `httpx` requests in `vcr.py`'s synchronous operations. - Added `run_async_from_sync` for async execution in sync contexts. - Replaced `asyncio.run()` with `run_async_from_sync` in `_sync_vcr_send`. Signed-off-by: Fabian von Feilitzsch <[email protected]>

fabianvf · 2024-04-03T19:45:44Z

looks like this may be a dupe fix of #825

nilleb · 2024-04-27T19:40:06Z

looks like this may be a dupe fix of #825

Yes, maybe; I've tested both and yours just works (while the other don't).

Thanks, pal. 🙏🏻

tysonholub · 2024-10-08T19:40:57Z

Per #825, we upgraded a number of dependencies around langchain and began having an issue in one of our vcr recordings that uses the openai client synchronously (in a synchronous kafka consumer). I didn't dig into why exactly we only seem to have an issue around this specific synchronous playback.

However, I noticed this dupe and after testing can confirm it is working for us (after rebasing on master). I've closed #825 in favor of this.

Can we please get this rebased and merged for release?

fichitiu · 2024-11-22T08:03:06Z

Hello, can we get this released in the near future please?

jairhenrique · 2025-01-02T23:50:45Z

Fixed on #886

jwmatthews mentioned this pull request Jun 24, 2024

Consider moving to upstream vcrpy once the upstream async httpx issues are resolved konveyor/kai#207

Closed

tysonholub mentioned this pull request Oct 8, 2024

Fix httpx async / decoded responses #825

Closed

alekseik1 mentioned this pull request Dec 18, 2024

fix: correctly handle asyncio.run when loop exists #886

Merged

jairhenrique closed this Jan 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf commented Apr 3, 2024 •

edited

Loading

fabianvf commented Apr 3, 2024

nilleb commented Apr 27, 2024

tysonholub commented Oct 8, 2024

fichitiu commented Nov 22, 2024

jairhenrique commented Jan 2, 2025

Fixed Async httpx Request Handling in vcr.py Sync Contexts #833

Fixed Async httpx Request Handling in vcr.py Sync Contexts #833

Conversation

fabianvf commented Apr 3, 2024 • edited Loading

fabianvf commented Apr 3, 2024

nilleb commented Apr 27, 2024

tysonholub commented Oct 8, 2024

fichitiu commented Nov 22, 2024

jairhenrique commented Jan 2, 2025

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

Fixed Async `httpx` Request Handling in `vcr.py` Sync Contexts #833

fabianvf commented Apr 3, 2024 •

edited

Loading