-
Notifications
You must be signed in to change notification settings - Fork 195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: find and print backtraces #5424
Conversation
f38783a
to
a8b3a37
Compare
a9736ea
to
aa96e8b
Compare
Here you can find examples of how the backtrace files are displayed in the Azure log (by looking in the "Logs" step of a workflow, after "Test"): https://dev.azure.com/ECP-WarpX/WarpX/_build/results?buildId=19622&view=results. These crashes were induced artificially by manual injection of NaN in the code. It starts like this: Starting: Logs
==============================================================================
Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.246.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/4ae82368-ed46-4960-8efa-c629bc4b2273.sh
Backtrace
---------
build/bin/test_2d_langmuir_multi_psatd_vay_deposition_particle_shape_4/Backtrace.0.0
---------
Host Name: fv-az23-266
=== If no file names and line numbers are shown below, one can run
addr2line -Cpfie my_exefile my_line_address
to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
Or one can use amrex/Tools/Backtrace/parse_bt.py.
=== Please note that the line number reported by addr2line may not be accurate.
One can use
readelf -wl my_exefile | grep my_line_address'
to find out the offset for that line.
0: /home/vsts/work/1/s/build/lib/libamrex_2d.so(_ZN5amrex11BLBackTrace20print_backtrace_infoEP8_IO_FILE+0x4d) [0x7fbe6557805d]
?? ??:0
1: /home/vsts/work/1/s/build/lib/libamrex_2d.so(_ZN5amrex11BLBackTrace7handlerEi+0x3e1) [0x7fbe6557a8a1]
?? ??:0 Let me know if you think we should format the header differently. The current header is Backtrace
---------
build/bin/test_2d_langmuir_multi_psatd_vay_deposition_particle_shape_4/Backtrace.0.0
--------- followed by the backtrace. |
I think this should work to find and print backtraces after all CI tests have run.
This is for Azure only. Local backtraces still need be inspected manually.
After trying many solutions, the current strategy is:
cleanup
step of each test (we continue to remove all other files);The new Azure workflow step is labeled "Logs" and it comes right after the step labeled "Test".