Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: find and print backtraces #5424

Merged
merged 7 commits into from
Nov 4, 2024

Conversation

EZoni
Copy link
Member

@EZoni EZoni commented Oct 29, 2024

I think this should work to find and print backtraces after all CI tests have run.

This is for Azure only. Local backtraces still need be inspected manually.

After trying many solutions, the current strategy is:

  • avoid removing the backtrace files in the cleanup step of each test (we continue to remove all other files);
  • have a separate workflow step to find and print the backtrace files in the Azure job (this is executed always).

The new Azure workflow step is labeled "Logs" and it comes right after the step labeled "Test".

@EZoni EZoni added the component: tests Tests and CI label Oct 29, 2024
@EZoni EZoni mentioned this pull request Oct 29, 2024
3 tasks
@EZoni EZoni force-pushed the ci_print_backtraces branch 6 times, most recently from f38783a to a8b3a37 Compare October 30, 2024 03:51
@EZoni
Copy link
Member Author

EZoni commented Nov 2, 2024

@WeiqunZhang @RemiLehe

Here you can find examples of how the backtrace files are displayed in the Azure log (by looking in the "Logs" step of a workflow, after "Test"): https://dev.azure.com/ECP-WarpX/WarpX/_build/results?buildId=19622&view=results.

These crashes were induced artificially by manual injection of NaN in the code.

It starts like this:

Starting: Logs
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.246.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/4ae82368-ed46-4960-8efa-c629bc4b2273.sh

Backtrace
---------
build/bin/test_2d_langmuir_multi_psatd_vay_deposition_particle_shape_4/Backtrace.0.0
---------
Host Name: fv-az23-266
=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: /home/vsts/work/1/s/build/lib/libamrex_2d.so(_ZN5amrex11BLBackTrace20print_backtrace_infoEP8_IO_FILE+0x4d) [0x7fbe6557805d]
    ?? ??:0

 1: /home/vsts/work/1/s/build/lib/libamrex_2d.so(_ZN5amrex11BLBackTrace7handlerEi+0x3e1) [0x7fbe6557a8a1]
    ?? ??:0

Let me know if you think we should format the header differently. The current header is

Backtrace
---------
build/bin/test_2d_langmuir_multi_psatd_vay_deposition_particle_shape_4/Backtrace.0.0
---------

followed by the backtrace.

@EZoni EZoni changed the title [WIP] CI: find and print backtraces CI: find and print backtraces Nov 2, 2024
@RemiLehe RemiLehe self-assigned this Nov 4, 2024
@RemiLehe RemiLehe merged commit c8c78f4 into ECP-WarpX:development Nov 4, 2024
37 checks passed
@EZoni EZoni deleted the ci_print_backtraces branch November 5, 2024 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: tests Tests and CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants