Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSAN race in //tests:dynamic_api_test_cpu #26396

Open
hawkinsp opened this issue Feb 7, 2025 · 1 comment
Open

TSAN race in //tests:dynamic_api_test_cpu #26396

hawkinsp opened this issue Feb 7, 2025 · 1 comment
Assignees
@vfdev-5
Labels
bug Something isn't working free threading Issues found in free threading builds

Comments

@hawkinsp
Copy link
Collaborator

hawkinsp commented Feb 7, 2025

Description

In https://github.com/jax-ml/jax/actions/runs/13193583786/job/36830830038

I've seen this race show up a few times.

It smells like it's a CPython bug, but we'd need to distill it down to a reproducer to prove that.

WARNING: ThreadSanitizer: data race (pid=48937)
  Atomic write of size 8 at 0x7bb3aa782d18 by thread T7 (mutexes: read M0):
    #0 _Py_atomic_store_ssize_relaxed /__w/jax/jax/cpython/./Include/cpython/pyatomic_gcc.h:481:3 (python3.13+0x273284) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #1 split_keys_entry_added /__w/jax/jax/cpython/Objects/dictobject.c:225:5 (python3.13+0x273284)
    #2 insert_split_key /__w/jax/jax/cpython/Objects/dictobject.c:1743:9 (python3.13+0x273284)
    #3 insertdict /__w/jax/jax/cpython/Objects/dictobject.c:1796:25 (python3.13+0x263496) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #4 dict_dict_merge /__w/jax/jax/cpython/./Include/cpython/pyatomic_gcc.h (python3.13+0x268eae) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #5 dict_merge /__w/jax/jax/cpython/Objects/dictobject.c:3807:15 (python3.13+0x268eae)
    #6 PyDict_Merge /__w/jax/jax/cpython/Objects/dictobject.c:3897:12 (python3.13+0x276c5c) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #7 dict_update_arg /__w/jax/jax/cpython/Objects/dictobject.c (python3.13+0x276c5c)
    #8 dict_update_common /__w/jax/jax/cpython/Objects/dictobject.c:3555:18 (python3.13+0x276c5c)
    #9 dict_update /__w/jax/jax/cpython/Objects/dictobject.c:3573:9 (python3.13+0x27691c) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #10 method_vectorcall_VARARGS_KEYWORDS /__w/jax/jax/cpython/Objects/descrobject.c:358:14 (python3.13+0x2004ba) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #11 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ead4a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #12 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1ead4a)
    #13 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e264b) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #14 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #15 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #16 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #17 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eaf86) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #18 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eaf86)
    #19 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #20 bounded_lru_cache_wrapper /__w/jax/jax/cpython/./Modules/_functoolsmodule.c:1068:14 (python3.13+0x573e02) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #21 lru_cache_call /__w/jax/jax/cpython/./Modules/_functoolsmodule.c:1334:14 (python3.13+0x57266a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #22 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:361:18 (python3.13+0x1eafeb) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #23 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #24 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #25 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #26 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #27 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #28 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x572243) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #29 partial_vectorcall /__w/jax/jax/cpython/./Modules/_functoolsmodule.c:262:25 (python3.13+0x572243)
    #30 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eaf86) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #31 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eaf86)
    #32 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #33 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #34 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #35 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #36 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #37 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef2d5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #38 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:62:18 (python3.13+0x1ef2d5)
    #39 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:285:24 (python3.13+0x1eaf86) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #40 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eaf86)
    #41 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #42 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #43 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #44 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #45 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #46 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef440) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #47 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef440)
    #48 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #49 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #50 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #51 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #52 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #53 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #54 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #55 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1e9f3d) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #56 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1eba37) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #57 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9539:15 (python3.13+0x2f8858) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #58 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea1ac) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #59 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eae08) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #60 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eae08)
    #61 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e264b) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #62 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #63 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #64 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #65 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #66 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #67 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #68 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #69 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #70 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5649a2) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #71 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdca7) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)

  Previous read of size 8 at 0x7bb3aa782d18 by thread T4 (mutexes: read M0):
    #0 dict_dealloc /__w/jax/jax/cpython/Objects/dictobject.c:3170:42 (python3.13+0x26b02f) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #1 _Py_Dealloc /__w/jax/jax/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #2 _Py_MergeZeroLocalRefcount /__w/jax/jax/cpython/Objects/object.c (python3.13+0x2906a2)
    #3 Py_DECREF /__w/jax/jax/cpython/./Include/object.h:913:13 (python3.13+0x271621) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #4 Py_XDECREF /__w/jax/jax/cpython/./Include/object.h:1042:9 (python3.13+0x271621)
    #5 _PyObject_SetManagedDict /__w/jax/jax/cpython/Objects/dictobject.c:7166:9 (python3.13+0x271621)
    #6 PyObject_ClearManagedDict /__w/jax/jax/cpython/Objects/dictobject.c:7175:9 (python3.13+0x272317) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #7 PjitFunction_tp_dealloc /proc/self/cwd/external/xla/xla/python/pjit.cc:1072:3 (xla_extension.so+0x80e9c3) (BuildId: c328bac74b338f9b6eb64da5a20d1195bf36d5f5)
    #8 _Py_Dealloc /__w/jax/jax/cpython/Objects/object.c:2935:5 (python3.13+0x2906a2) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #9 _Py_MergeZeroLocalRefcount /__w/jax/jax/cpython/Objects/object.c (python3.13+0x2906a2)
    #10 Py_DECREF /__w/jax/jax/cpython/./Include/object.h:913:13 (python3.13+0x3ee064) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #11 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:3767:21 (python3.13+0x3ee064)
    #12 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #13 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #14 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #15 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef440) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #16 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef440)
    #17 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #18 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #19 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #20 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e4832) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #21 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #22 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #23 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #24 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1e9f3d) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #25 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1eba37) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #26 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9539:15 (python3.13+0x2f8858) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #27 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea1ac) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #28 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eae08) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #29 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eae08)
    #30 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e264b) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #31 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de77a) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #32 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1812:12 (python3.13+0x3de77a)
    #33 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb3bf) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #34 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef38f) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #35 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef38f)
    #36 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb033) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #37 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb033)
    #38 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb0b5) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #39 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x5649a2) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)
    #40 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bdca7) (BuildId: 4f0cd5ff36563504297698c79511892009bdb164)

System info (python version, jaxlib version, accelerator, etc.)

Python 3.13t from head
@hawkinsp hawkinsp added bug Something isn't working free threading Issues found in free threading builds labels Feb 7, 2025
@hawkinsp
Copy link
Collaborator Author

hawkinsp commented Feb 7, 2025

The same race looks like it happens in 3 different test cases in that build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vfdev-5 vfdev-5

Labels
bug Something isn't working free threading Issues found in free threading builds
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hawkinsp @vfdev-5