unhandled exception in select_reactor thread leads to process termination

[arvidn]

The select_reactor has a socket_select_interrupter object used to signal an interruption via. It has a long-lived socket that's added to the select() call to make it wake up on an interruption.

This long lived socket can sometimes be closed and has to be recreated. For example if the computer hibernates or if the network stack is restarted. To handle this, the interrupter sockets are recreated if reading from it fails. This is socket_select_interrupter::recreate().

Recreating this socket may fail with an exception. This exception will flow out of the call to run(), expected to be handled by the application.

The problem occurs in the select_reactor, which on windows spawns a thread. The thread calls run() and post_deferred_completions() in a loop until stop_thread_ is set. (socket_reactor::run_thread())

There is nothing catching exceptions at the top level of this thread. So when recreate() fails, the process is terminated.

An example stack trace from windows:

#  0 qbittorrent.exe      0x00007ff61e1499ec straceWin::getBacktrace()[ app\stacktrace_win.h : 220 ]
#  1 qbittorrent.exe      0x00007ff61e14ab6b sigAbnormalHandler(signum)[ app\main.cpp : 368 ]
#  2 qbittorrent.exe      0x00007ff61ef7a365 raise(signum)[ minkernel\crts\ucrt\src\appcrt\misc\signal.cpp : 541 ]
#  3 qbittorrent.exe      0x00007ff61ef86a94 abort()[ minkernel\crts\ucrt\src\appcrt\startup\abort.cpp : 64 ]
#  4 qbittorrent.exe      0x00007ff61ef7dd73 terminate()[ minkernel\crts\ucrt\src\appcrt\misc\terminate.cpp : 58 ]
#  5 qbittorrent.exe      0x00007ff61ef551b7 __scrt_unhandled_exception_filter(pointers)[ d:\agent\_work\1\s\src\vctools\crt\vcstartup\src\utility\utility_desktop.cpp : 91 ]
#  6 KERNELBASE.dll       0x00007ffb5f236391 UnhandledExceptionFilter()
#  7 ntdll.dll            0x00007ffb617aaeec memset()
#  8 ntdll.dll            0x00007ffb61793eb6 _C_specific_handler()
#  9 ntdll.dll            0x00007ffb617a8e4f _chkstk()
# 10 ntdll.dll            0x00007ffb61735e9a RtlRestoreContext()
# 11 ntdll.dll            0x00007ffb61733163 RtlRaiseException()
# 12 KERNELBASE.dll       0x00007ffb5f13474c RaiseException()
# 13 qbittorrent.exe      0x00007ff61ef73611 _CxxThrowException(pExceptionObject, pThrowInfo)[ d:\agent\_work\1\s\src\vctools\crt\vcruntime\src\eh\throw.cpp : 129 ]
# 14 qbittorrent.exe      0x00007ff61e14279b boost::throw_exception(e)[ g:\qbittorrent\boost_1_78_0\boost\throw_exception.hpp : 148 ]
# 15 qbittorrent.exe      0x00007ff61e13c8ed boost::asio::detail::do_throw_error(err, location)[ g:\qbittorrent\boost_1_78_0\boost\asio\detail\impl\throw_error.ipp : 38 ]
# 16 qbittorrent.exe      0x00007ff61e47929e boost::asio::detail::socket_select_interrupter::open_descriptors()[ g:\qbittorrent\boost_1_78_0\boost\asio\detail\impl\socket_select_interrupter.ipp : 66 ]
# 17 qbittorrent.exe      0x00007ff61e479e3e boost::asio::detail::select_reactor::run(ops, ops)[ g:\qbittorrent\boost_1_78_0\boost\asio\detail\impl\select_reactor.ipp : 266 ]
# 18 qbittorrent.exe      0x00007ff61e47a099 boost::asio::detail::select_reactor::run_thread()[ g:\qbittorrent\boost_1_78_0\boost\asio\detail\impl\select_reactor.ipp : 303 ]
# 19 qbittorrent.exe      0x00007ff61e4126cf boost::asio::detail::win_thread_function(arg)[ g:\qbittorrent\boost_1_78_0\boost\asio\detail\impl\win_thread.ipp : 119 ]
# 20 qbittorrent.exe      0x00007ff61ef864a0 thread_start(parameter, parameter)[ minkernel\crts\ucrt\src\appcrt\startup\thread.cpp : 115 ]
# 21 KERNEL32.DLL         0x00007ffb5f9154e0 BaseThreadInitThunk()
# 22 ntdll.dll            0x00007ffb6170485b RtlUserThreadStart()

related ticket: qbittorrent/qBittorrent#17082

[ghost]

@chriskohlhoff sorry for the ping but please take a look into this issue. A lot of qBittorrent users are suffering from crash on VPN/Network interface disconnect due to this bug.

[chriskohlhoff]

@arvidn I may have missed some, but as far as I can tell these are the only places that libtorrent uses reactor-style operations on sockets:

• https://github.com/arvidn/libtorrent/blob/6e03a8307dd3fcef97628c310fa57103ee2b6469/src/upnp.cpp#L193
• https://github.com/arvidn/libtorrent/blob/6e03a8307dd3fcef97628c310fa57103ee2b6469/src/upnp.cpp#L205
• https://github.com/arvidn/libtorrent/blob/6e03a8307dd3fcef97628c310fa57103ee2b6469/src/upnp.cpp#L510
• https://github.com/arvidn/libtorrent/blob/6e03a8307dd3fcef97628c310fa57103ee2b6469/src/lsd.cpp#L139
• https://github.com/arvidn/libtorrent/blob/6e03a8307dd3fcef97628c310fa57103ee2b6469/src/lsd.cpp#L222

Is there a reason they cannot be changed to receive directly into a buffer and endpoint as class data members? On modern Windows these would be the only operations that trigger creation of the select_reactor.

[chriskohlhoff]

Commit 4709210 moves the recreate() so that the exception can escape through io_context::run().

N.B. I still recommend changing those places to use async_receive_from, to avoid instantiating the select_reactor in the first place.

[xavier2k6]

@chriskohlhoff Has this made it in to upcoming Boost 1.80?

[chriskohlhoff]

It will.

[Chocobo1]

It seems there are still related crashes albeit the user says it occurs less than before:

This is using boost 1.80 with libtorrent 7bb4b410d704e13d73d0f02d85192d59dff1fb5e on RC_2_0.
Note that libtorrent already include arvidn/libtorrent@1f09a84 and arvidn/libtorrent@d6af615.

Taken from qbittorrent/qBittorrent#17082 (comment):

#  0 qbittorrent.exe      0x00007ff62fb59dbc straceWin::getBacktrace()[ app\stacktrace_win.h : 220 ]
#  1 qbittorrent.exe      0x00007ff62fb5af4b sigAbnormalHandler(signum)[ app\main.cpp : 367 ]
#  2 qbittorrent.exe      0x00007ff6309953f3 raise(signum)[ minkernel\crts\ucrt\src\appcrt\misc\signal.cpp : 541 ]
#  3 qbittorrent.exe      0x00007ff6309a2f58 abort()[ minkernel\crts\ucrt\src\appcrt\startup\abort.cpp : 64 ]
#  4 qbittorrent.exe      0x00007ff6309994a5 terminate()[ minkernel\crts\ucrt\src\appcrt\misc\terminate.cpp : 58 ]
#  5 qbittorrent.exe      0x00007ff6309c0a1c __C_specific_handler_noexcept(ExceptionRecord, EstablisherFrame, ContextRecord, DispatcherContext)[ d:\agent\_work\2\s\src\vctools\crt\vcruntime\src\eh\chandler_noexcept.cpp : 65 ]
#  6 ntdll.dll            0x00007ffee9a68fcf _chkstk()
#  7 ntdll.dll            0x00007ffee99f5e9a RtlRestoreContext()
#  8 ntdll.dll            0x00007ffee99f3163 RtlRaiseException()
#  9 KERNELBASE.dll       0x00007ffee746474c RaiseException()
# 10 qbittorrent.exe      0x00007ff630988ed1 _CxxThrowException(pExceptionObject, pThrowInfo)[ d:\agent\_work\2\s\src\vctools\crt\vcruntime\src\eh\throw.cpp : 129 ]
# 11 qbittorrent.exe      0x00007ff62fb52923 boost::throw_exception(e)[ g:\qbittorrent\boost_1_80_0\boost\throw_exception.hpp : 157 ]
# 12 qbittorrent.exe      0x00007ff62fb4c98d boost::asio::detail::do_throw_error(err, location)[ g:\qbittorrent\boost_1_80_0\boost\asio\detail\impl\throw_error.ipp : 42 ]
# 13 qbittorrent.exe      0x00007ff62fe88b1b boost::asio::detail::socket_select_interrupter::open_descriptors()[ g:\qbittorrent\boost_1_80_0\boost\asio\detail\impl\socket_select_interrupter.ipp : 66 ]
# 14 qbittorrent.exe      0x00007ff62fe89c06 boost::asio::detail::select_reactor::restart_reactor::do_complete(owner, base, __formal, __formal)[ g:\qbittorrent\boost_1_80_0\boost\asio\detail\impl\select_reactor.ipp : 334 ]
# 15 qbittorrent.exe      0x00007ff62fe23648 boost::asio::detail::win_iocp_io_context::do_one(this_thread, ec)[ g:\qbittorrent\boost_1_80_0\boost\asio\detail\impl\win_iocp_io_context.ipp : 473 ]
# 16 qbittorrent.exe      0x00007ff62fe22edf boost::asio::detail::win_iocp_io_context::run(ec)[ g:\qbittorrent\boost_1_80_0\boost\asio\detail\impl\win_iocp_io_context.ipp : 204 ]
# 17 qbittorrent.exe      0x00007ff62fe238bc boost::asio::io_context::run()[ g:\qbittorrent\boost_1_80_0\boost\asio\impl\io_context.ipp : 63 ]
# 18 qbittorrent.exe      0x00007ff62fe250e7 std::_LaunchPad >,std::default_delete > > > >::_Go()[ c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.16.27023\include\thr\xthread : 230 ]
# 19 qbittorrent.exe      0x00007ff62fe21f69 std::_Pad::_Call_func(_Data)[ c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.16.27023\include\thr\xthread : 208 ]
# 20 qbittorrent.exe      0x00007ff6309a266a thread_start(parameter, parameter)[ minkernel\crts\ucrt\src\appcrt\startup\thread.cpp : 97 ]
# 21 KERNEL32.DLL         0x00007ffee92454e0 BaseThreadInitThunk()
# 22 ntdll.dll            0x00007ffee99c485b RtlUserThreadStart()