gcc-UBSAN issue in gradient code

[trevorld]

• {ggpattern} was hit with a CRAN deadline because a vignette which saved an image with a gradient with {ragg} produced a gcc-UBSAN warning: gcc-UBSAN warning in a package {ggpattern} suggests trevorld/ggpattern#131 (I have CRAN workarounds in mind for the meantime such as using png(type="cairo") to save that vignette example)
• I can reproduce a similar gcc-UBSAN issue in the rocker/r-devel-san docker container with the RD binary:

UBSAN_OPTIONS="print_stacktrace=1" RD

with the following minimal, reproducible example:

library("grid")
ragg::agg_png()
grid.rect(gp=gpar(fill=linearGradient(colours=c("green", "#445566"))))
dev.off()

agg/include/agg_dda_line.h:39:30: runtime error: left shift of negative value -170
    #0 0x7c6ab6b08e78 in agg::dda_line_interpolator<14, 0>::dda_line_interpolator(int, int, unsigned int) agg/include/agg_dda_line.h:39
    #1 0x7c6ab6b08e78 in agg::color_interpolator<agg::rgba8T<agg::linear> >::color_interpolator(agg::rgba8T<agg::linear> const&, agg::rgba8T<agg::linear> const&, unsigned int) agg/include/agg_gradient_lut.h:70
    #2 0x7c6ab6b08e78 in agg::gradient_lut<agg::color_interpolator<agg::rgba8T<agg::linear> >, 512u>::build_lut() agg/include/agg_gradient_lut.h:222
    #3 0x7c6ab6d25bad in Pattern<agg::pixfmt_alpha_blend_rgba<agg::blender_rgba_pre<agg::rgba8T<agg::linear>, agg::order_rgba>, agg::row_accessor<unsigned char> >, agg::rgba8T<agg::linear> >::finish_gradient() /tmp/RtmpUdlm4l/R.INSTALL759d2d3154/ragg/src/pattern.h:103
    #4 0x7c6ab6d25bad in AggDevice<agg::pixfmt_alpha_blend_rgb<agg::blender_rgb_pre<agg::rgba8T<agg::linear>, agg::order_rgb>, agg::row_accessor<unsigned char>, 3u, 0u>, agg::rgba8T<agg::linear>, agg::pixfmt_alpha_blend_rgba<agg::blender_rgba_pre<agg::rgba8T<agg::linear>, agg::order_rgba>, agg::row_accessor<unsigned char> > >::createPattern(SEXPREC*) /tmp/RtmpUdlm4l/R.INSTALL759d2d3154/ragg/src/AggDevice.h:834
    #5 0x7c6ab6e85c4f in SEXPREC* agg_setPattern<AggDevicePng<agg::pixfmt_alpha_blend_rgb<agg::blender_rgb_pre<agg::rgba8T<agg::linear>, agg::order_rgb>, agg::row_accessor<unsigned char>, 3u, 0u> > >(SEXPREC*, _DevDesc*) /tmp/RtmpUdlm4l/R.INSTALL759d2d3154/ragg/src/init_device.h:205
    #6 0x7c6abf2fc805 in do_External /tmp/R-devel/src/main/dotcode.c:576
    #7 0x7c6abf4043d3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8122
    #8 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #9 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #10 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #11 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #12 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #13 0x7c6abf449f78 in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #14 0x7c6abf558a83 in dispatchMethod /tmp/R-devel/src/main/objects.c:473
    #15 0x7c6abf55a043 in Rf_usemethod /tmp/R-devel/src/main/objects.c:509
    #16 0x7c6abf55a82a in do_usemethod /tmp/R-devel/src/main/objects.c:579
    #17 0x7c6abf403ec3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8142
    #18 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #19 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #20 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #21 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #22 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #23 0x7c6abf449f78 in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #24 0x7c6abf558a83 in dispatchMethod /tmp/R-devel/src/main/objects.c:473
    #25 0x7c6abf55a126 in Rf_usemethod /tmp/R-devel/src/main/objects.c:513
    #26 0x7c6abf55a82a in do_usemethod /tmp/R-devel/src/main/objects.c:579
    #27 0x7c6abf403ec3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8142
    #28 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #29 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #30 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #31 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #32 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #33 0x7c6abf43ccdd in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #34 0x7c6abf43ccdd in Rf_eval /tmp/R-devel/src/main/eval.c:1280
    #35 0x7c6ab576c776 in resolveFill /tmp/R-devel/src/library/grid/src/gpar.c:288
    #36 0x7c6ab57705b3 in initGContext /tmp/R-devel/src/library/grid/src/gpar.c:551
    #37 0x7c6ab577bbb7 in gridRect /tmp/R-devel/src/library/grid/src/grid.c:3159
    #38 0x7c6ab57874d2 in L_rect /tmp/R-devel/src/library/grid/src/grid.c:3354
    #39 0x7c6abf310b9b in R_doDotCall /tmp/R-devel/src/main/dotcode.c:770
    #40 0x7c6abf3209a7 in do_dotcall /tmp/R-devel/src/main/dotcode.c:1437
    #41 0x7c6abf321cb8 in do_dotcallgr /tmp/R-devel/src/main/dotcode.c:1542
    #42 0x7c6abf40ad54 in bcEval_loop /tmp/R-devel/src/main/eval.c:8057
    #43 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #44 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #45 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #46 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #47 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #48 0x7c6abf449f78 in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #49 0x7c6abf558a83 in dispatchMethod /tmp/R-devel/src/main/objects.c:473
    #50 0x7c6abf55a126 in Rf_usemethod /tmp/R-devel/src/main/objects.c:513
    #51 0x7c6abf55a82a in do_usemethod /tmp/R-devel/src/main/objects.c:579
    #52 0x7c6abf403ec3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8142
    #53 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #54 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #55 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #56 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #57 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #58 0x7c6abf43ccdd in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #59 0x7c6abf43ccdd in Rf_eval /tmp/R-devel/src/main/eval.c:1280
    #60 0x7c6abf388d3d in do_recordGraphics /tmp/R-devel/src/main/engine.c:3259
    #61 0x7c6abf4043d3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8122
    #62 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #63 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #64 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #65 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #66 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #67 0x7c6abf449f78 in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #68 0x7c6abf558a83 in dispatchMethod /tmp/R-devel/src/main/objects.c:473
    #69 0x7c6abf55a043 in Rf_usemethod /tmp/R-devel/src/main/objects.c:509
    #70 0x7c6abf55a82a in do_usemethod /tmp/R-devel/src/main/objects.c:579
    #71 0x7c6abf403ec3 in bcEval_loop /tmp/R-devel/src/main/eval.c:8142
    #72 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7505
    #73 0x7c6abf43bbd3 in bcEval /tmp/R-devel/src/main/eval.c:7490
    #74 0x7c6abf43c79a in Rf_eval /tmp/R-devel/src/main/eval.c:1167
    #75 0x7c6abf443ea2 in R_execClosure /tmp/R-devel/src/main/eval.c:2393
    #76 0x7c6abf447379 in applyClosure_core /tmp/R-devel/src/main/eval.c:2306
    #77 0x7c6abf43ccdd in Rf_applyClosure /tmp/R-devel/src/main/eval.c:2328
    #78 0x7c6abf43ccdd in Rf_eval /tmp/R-devel/src/main/eval.c:1280
    #79 0x7c6abf4f4fe1 in Rf_ReplIteration /tmp/R-devel/src/main/main.c:265
    #80 0x7c6abf4f5890 in R_ReplConsole /tmp/R-devel/src/main/main.c:317
    #81 0x7c6abf4f5927 in run_Rmainloop /tmp/R-devel/src/main/main.c:1219
    #82 0x5d85fc2ea07a in main /tmp/R-devel/src/main/Rmain.c:29
    #83 0x7c6abe2b9d67 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #84 0x7c6abe2b9e24 in __libc_start_main_impl ../csu/libc-start.c:360
    #85 0x5d85fc2ea0a0 in _start (/usr/local/lib/R/bin/exec/R+0x10a0) (BuildId: 43097ec7028d3499d8f013f27c16272c0d0510f9)

[thomasp85]

Thanks for the report. Looks like a bug in AGG - I'll cook up a fix