Remove ProgressMeter.jl-based progress bar?

[tkf]

IIUC, PkgBenchmark.jl has its own _run function to support progress reports via ProgressMeter.jl. In JuliaCI/BenchmarkTools.jl#153 I suggested to use more flexible and open progress logging API on BenchmarkTools.jl side. If that PR is accepted, I suppose we can eventually use the plain run function?

My motivation is to run PkgBenchmark with verbose option passed to run. I think this is nice because:

• It helps package authors to locate time-consuming benchmarks.
• Some CI services don't like silent jobs. verbose option is a nice way to workaround it.

[KristofferC]

If BenchmarkTools gets its own progressl ogging system, then we can of course just use that. That's why I tired to upstream it here: JuliaCI/BenchmarkTools.jl#66

[tkf]

Thanks for the comment. JuliaCI/BenchmarkTools.jl#66 would be an improvement but I still think JuliaCI/BenchmarkTools.jl#153 is better. I commented my reasoning there: JuliaCI/BenchmarkTools.jl#66 (comment)

[KristofferC]

With JuliaCI/BenchmarkTools.jl#153 merged, what do we do here now?

[tkf]

JuliaCI/BenchmarkTools.jl#153 doesn't interfere with the existing ProgressMeter.jl-based approach in PkgBenchmark.jl so I don't think you'd need immediate actions.

But it'd be nice to have more natural support for logging-based progress bar in PkgBenchmark.jl. How do you want to handle the transition?

1. Default to TerminalLoggers.jl-based progressbar, remove progressoptions keyword argument from benchmarkpkg (so bump the version to 0.3).
2. Keep ProgressMeter.jl-based progress bar, just add support TerminalLoggers.jl-based approach

I think option 1 is better as it'd simplify the code. But an objection may be that TerminalLoggers.jl is still a young (0.1.0) package and not tested a lot in the wild yet. (cc @c42f)

The third option would be to somehow support merging log records of the subprocess to the main process so that it is automatically handled by user's default logger (e.g., Juno). But this requires logging infrastructure that doesn't exist yet.

[c42f]

I'm happy to stress test TerminalLoggers and fix problems which come from making it a dependency here 👍

Do we need to resolve JuliaLogging/TerminalLoggers.jl#22 before this is practical?

[tkf]

I don't think ProgressMeter.jl works well with subprocesses so I don't think we need to fix it to switch to the logging-based progress bars. Maybe the logging-based solution has a higher chance of fixing it.

[c42f]

Ok sounds fine then. I guess the scroll region based progress bars just end up writing over the top of each other anyway. Which is annoying but not fatal.