Installed via docker-compose on unraid. 5X'ed my cpu util, EXTREMELY slow

[superdong69]

Describe the bug
Was routinely using 18% of my CPU on an R710 server, now using 80% util.
Collection is over 18TB. Created the library 5 days ago, took 5 days to finish. Is completely unusable.

Using postgres15 Server DB connection.

To Reproduce
Try to navigate to anything on the app. Just sits there thinking. Is there a way to fix this, or is this a limitation of the software?

[Floppy]

That's a big collection, so it's probably doing a lot of analysis work. Hard to tell though without knowing more... when you say it was using 18%, was that a previous version of Manyfold or is this the first time you've used it?

It's probably a good idea to set DEFAULT_WORKER_CONCURRENCY to 1, and you could also set PERFORMANCE_WORKER_CONCURRENCY to 0 while we get it under control. Could even set the default worker concurrency to zero as well to be sure.

There are some performance issues with large libraries, and this is the largest I've heard of so far, so we might be hitting that. It could potentially be due to a huge number of autogenerated tags, perhaps.

Once you've changed the background worker concurrency, the idle cpu use should drop, and it should become semi-usable, though if the tag problem is the root cause then it might still be very slow to respond.

If you can, take a look at the /admin area and see how many model, files and tags it's indexed, that will give us some clues.

[Floppy]

Thinking about this some more, I'm pretty sure this is the tags causing it, same as in #2084. You could try https://manyfold.app/sysadmin/advanced_tasks.html#remove-all-tags but that's a somewhat nuclear option as it will remove all your existing tags. If you can get a tag count for me, that'll definitely confirm one way or another. If even /admin won't load, then running ActsAsTaggableOn::Tag.count in the console should do it.

[superdong69]

So here's a bit of info.
I've never used this before. So my server without it running was running at a max of 18% CPU across all threads.
Where would I put these changes in you suggested? I'd really love to have this work normally, because I need it badly. LOL

Here's a brief example of tags, I couldn't find just how many tags I have, but on the library page, on the right, I have to scroll down for about 5 minutes to get to the bottom of the page, there are so many tags.
Here's a brief example.
[!new (81774)
[patreon (22796)]
[2021 (22566)]
[2022 (19270)]
[2023 (13633)]
[supported (9950)]
[2020 (9614)]
[kickstarter (6124)]
[unsupported (4402)]
[pre supported (3697]

[Floppy]

Ok, sounds like there are a lot, that's probably the issue. The concurrency stuff goes in environment variables, but I don't think that's your problem, so don't worry about it. Let me have a go at improving tag performance.

[Floppy]

Yep, pretty certain after some investigation that this is tag performance issues. I've got a handle on it and am making some improvements.

[superdong69]

Awesome! Can't wait!

[Floppy]

I've release v0.74.0 with a bunch of tag performance improvements; hopefully this will help! I'd also recommend changing your tag visibility threshold to at least 2 (the new default) in settings.

[superdong69]

Roger that! I'll try it and report back!

[superdong69]

Just updated. Light years better. Still overall load is still quite high. Too high to leave running for sure. Though it does occasionally drop to 30% it's more often just constantly cycling cpu into the 60% even though it's not actively doing anything. Page response has improved significantly. You're definitely on the way to something amazing.

Please let me know if there is anything I can provide you to help improve.

[Floppy]

OK, that's great, glad it improved things. Hopefully it's usable now! Does it go up to 60% even when you're not actively using the site?

[superdong69]

Correct. At idle, the overall cpu swings wildly, from 60% to 28%. Hovering most often north of 48%. Watching the individual cpus, they're getting thrashed.

[Floppy]

OK, thanks. Is there anything happening in the logs?

[Floppy]

If CPU is high, it's likely that the background workers are active; the app consumes a lot of memory when quiescent (see #1784 for when I get round to fixing that), but it shouldn't use a lot of CPU, and mine doesn't here. I suspect that if you visit /sidekiq you'll see a bunch of work being done, probably by the performance worker?

[superdong69]

I'm not sure how to answer that. This is what I see in Sidekiq.

Is there any other data I can get to you?

[superdong69]

As an aside, when I go into a model, and try to view the 3d rendering, it completely locks up my browser to crash.

Server stats when the docker is running.

[superdong69]

And here you can see when I end the docker container:

[Floppy]

Thanks! Doesn't look like background workers then, there's nothing running. Hmm. I will keep poking at it.

[Floppy]

@superdong69 can I check in on this, has performance improved since? There have been a bunch of memory and performance improvements; have they helped at all?