Expensive-looking oddities in CUDA profile

[inducer]

Running this benchmark based on wave-op-mpi.py on 1c44c4b with the command

PYTHONHASHSEED=17 PYOPENCL_TEST=port:nvid setarch -R numactl -C 2 -m 0 nvprof -f -o yoink.nvvp python -O wave-op-mpi.py --dim=3 --order=4   

on dunkel gives me the following profile in Nvidia's visual profiler:

There are at least two things wrong here (both circled):

• There are a bunch of big gaps where nothing seems to be happening. Why?
• Every now and then, a cuLaunchKernel seems to take a very long time. Why?

Curiously, there seem to be periods that don't suffer from this:

If we could fix these two types of stalls, I suspect our performance story would look quite a bit different.

cc @matthiasdiener @lukeolson

Other versions in use, for reproducibility:

• inducer/meshmode@c7b7841
• inducer/arraycontext@77b03e5
• inducer/loopy@e232bb1
• inducer/pyopencl@a56281a

[inducer]

We could try using https://github.com/conda-forge/lttng-ust-feedstock for tracing. POCL already supports that tracing using LTTng: pocl/pocl@ef737d3

[kaushikcfd]

I faced a similar issue whenever I was using almost all of the device's global memory:

(Notice the two unexplainable voids in the profile)

On moving to a slightly coarser mesh, the phenomena wasn't seen.