CSV file size is funbounded when append option is true

[jensck]

Browsing through the code, it looks like the CSV records file is "funbounded". (Funbounded, because what's more fun than a mystery file with an unbounded size silently filling up your disk? ;)

I see two issues with this:

1. As mentioned above, the file could start taking up a significant amount of disk space. On a Jenkins server in particular, which will run complex, complete builds, all day, every day, this could add up very quickly.
2. As the file grows, CSVSummaryReporter will get slower. If the file is big enough, this means that the build time plugin could, ironically, slow down the build by a meaningful amount (e.g.: more than a 1-2 seconds).

This isn't stopping me from using the plugin personally, but my team (and by extension, lots of other devs, I imagine) are uncomfortable adding a plugin which requires manual maintenance, since users will naturally forget to do it until the situation becomes unbearable, like running out of disk space, or build times going up by tens of seconds because it has to process a multi-gigabyte file.

[passy]

Thanks for opening the issue and the comprehensive explanation, @jensck! I'm also totally going to steal that term funbounded. :)

The same issue came up internally but we decided to defer working on a solution so far. To me, the second risk was actually more important than the first as the whole point of the plugin was to lead to faster builds - while this could lead to the exact opposite.

I'd like to avoid reinventing the wheel here and log truncation and rotation seems like it should be a pretty well researched topic. Do you - or anyone else - have any suggestions on what could be used here?

[jensck]

Glad you liked "funbounded" -- that's a gem from my former colleagues at Code42 which I thought deserved to enter the broader hacker lexicon. :)

As for the solution, some kind of log rollover approach is what came to mind for me. There might be more sophisticated approaches that are worth exploring if needed, but here's a simple first cut:

Add a config option specifying the number of days worth of events to keep. While this isn't a perfect solution for predicting log size, it's arguably the most useful, relatable measurement.

• It should have a default which seems reasonable for a moderately complex project on a reasonably recent laptop
• Allow for a value of -1 if people actually want funbounded logs
• when the limit is hit, just erase the events which are past the limit, instead of trying to roll them over into another file or something, but spit out a long message about what's happening

I don't have any specific thoughts about implementation, but the performance of the JVM and the availability of great IO libraries like Okio should make it fairly straightforward?