After some time, seeing: Query error: MySQL server has gone away

[pisexy-qnd]

Every few days, we will see a whole string of messages with the "queue size" increasing by 2 bytes each second.

We attempted the usrsigkill -

Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5902
Query error: MySQL server has gone away in flush peers with a qlength: 1982 queue size: 1000
Torrent flush queue size: 5902, next query length: 520
Peer flush queue size: 1000, next query length: 1982
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5904
Query error: MySQL server has gone away in flush peers with a qlength: 4824 queue size: 1000
Reloading from database
Loaded 12269 torrents
Loaded 0 tokens
Loaded 2898 users
Assuming no whitelist desired, disabling
Torrent flush queue size: 5904, next query length: 520
Peer flush queue size: 1000, next query length: 4824
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5906
Query error: MySQL server has gone away in flush peers with a qlength: 1741 queue size: 1000
Torrent flush queue size: 5906, next query length: 520
Peer flush queue size: 1000, next query length: 1741
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5908
Query error: MySQL server has gone away in flush peers with a qlength: 2751 queue size: 1000
Torrent flush queue size: 5908, next query length: 520
Peer flush queue size: 1000, next query length: 2751

But as you can see from the above, it didn't fix the problem.
Next is bouncing the tracker:

Peer flush queue size: 1000, next query length: 822
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5984
Query error: MySQL server has gone away in flush peers with a qlength: 570 queue size: 1000
Torrent flush queue size: 5984, next query length: 520
Peer flush queue size: 1000, next query length: 570
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5986
Query error: MySQL server has gone away in flush peers with a qlength: 573 queue size: 1000
Torrent flush queue size: 5986, next query length: 520
Peer flush queue size: 1000, next query length: 573
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5988
Query error: MySQL server has gone away in flush peers with a qlength: 577 queue size: 1000
Torrent flush queue size: 5988, next query length: 520
Peer flush queue size: 1000, next query length: 577
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5990
Query error: MySQL server has gone away in flush peers with a qlength: 580 queue size: 1000
Torrent flush queue size: 5990, next query length: 520
Peer flush queue size: 1000, next query length: 580
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5992
Query error: MySQL server has gone away in flush peers with a qlength: 701 queue size: 1000
^C
Session terminated, terminating shell...Caught SIGINT/SIGTERM
closing tracker... press Ctrl-C again to terminate
Torrent flush queue size: 5992, next query length: 520
Peer flush queue size: 1000, next query length: 701
Query error: MySQL server has gone away in flush torrents with a qlength: 520 queue size: 5994
Query error: MySQL server has gone away in flush peers with a qlength: 854 queue size: 1000
...killed.

And then we'll restart the process and everything will be fine until it hits a busy time with the database:

Clearing xbt_files_users and resetting peer counts...done
Loaded 2898 users
Loaded 12269 torrents
Loaded 0 tokens
Assuming no whitelist desired, disabling
Sockets up on port 34000, starting event loop!
0 open, 8 connections (2/s), 8 requests (2/s)
0 open, 277 connections (1/s), 277 requests (1/s)
0 open, 418 connections (1/s), 418 requests (1/s)
0 open, 534 connections (1/s), 534 requests (1/s)
Torrent flush queue size: 2, next query length: 374
Peer flush queue size: 1, next query length: 1076
Torrent flush queue size: 4, next query length: 374
Peer flush queue size: 2, next query length: 1076
Torrent flush queue size: 6, next query length: 374
Snatch flush queue size: 1, next query length: 97
Peer flush queue size: 3, next query length: 1076
MySQL error in flush_snatches: Too many connections
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
3 open, 708 connections (4/s), 705 requests (3/s)

Is there a way to have Ocelot reset its connections to the database?

I just attempted various mysql commands from the CLI, and inadvertently made the flush problem stop.

Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
Snatch flush queue size: 1, next query length: 97
3 open, 1971 connections (15/s), 1968 requests (16/s)
0 open, 2090 connections (2/s), 2090 requests (2/s)
0 open, 2239 connections (1/s), 2239 requests (1/s)
0 open, 2351 connections (2/s), 2351 requests (2/s)
0 open, 2546 connections (3/s), 2546 requests (3/s)
0 open, 2652 connections (1/s), 2652 requests (1/s)
Reloading from database
Loaded 12271 torrents
Loaded 0 tokens
Loaded 2898 users
Assuming no whitelist desired, disabling
1 open, 2756 connections (3/s), 2755 requests (3/s)
0 open, 2924 connections (2/s), 2924 requests (2/s)

(just above is an automatic USRSIG when new torrents have been added to the database. Might that be contributing to the problem? We made it not reload anytime sooner than 5 minutes even if 5 or 10 new torrents are added quickly.)

Thanks for any advice.

[pisexy-qnd]

Please close/disregard this - we saw performance issues across other things as well when full logging was turned on and other tuning steps were attempted. Seems it was related to MySQL 5.7. When rolled back to 5.6, the constant 20-50% CPU usage my MySQL dropped back to 0.1-5% CPU.

MySQL 5.7 was the problem.