Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove new/assignment-approve statuses from the list of ACTIVE status #11246

Open
amaltaro opened this issue Aug 9, 2022 · 2 comments · May be fixed by #11263
Open

Remove new/assignment-approve statuses from the list of ACTIVE status #11246

amaltaro opened this issue Aug 9, 2022 · 2 comments · May be fixed by #11263

Comments

@amaltaro
Copy link
Contributor

amaltaro commented Aug 9, 2022

Impact of the new feature
ReqMgr2/WMStats and other services consuming this data

Is your feature request related to a problem? Please describe.
We have been discussing on how to alleviate the load on WMStats - and anything/anyone consuming ACTIVE data - and one of the ideas is to reduce the number of workflows that are constantly monitored by WMStats (and cached by DataCache WMStatsServer application).

That could be done by removing new and assignment-approved from the list of ACTIVE status defined here:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/ReqMgr/DataStructs/RequestStatus.py#L71

however, this will impact the data that is served by wmstatsserver app and the content provided, e.g. the protectedlfns and globallocks, used in the microservices for some dangerous and sensitive actions.

Describe the solution you'd like
Before we can shorten this list of ACTIVE status, we need to:

  • investigate what the impact is on the system
  • investigate if it could cause problems for those consuming the list of protected LFNs (output)
  • investigate if it could cause problems for those consuming the global locks (input + output)

if no negative side effects are expected, we should then stop monitoring workflows in status new or assignment-approvement in WMStats. This will also affect those querying ReqMgr2 with this query string status=ACTIVE.

Describe alternatives you've considered
At the current load (30k requests), the system is still behaving properly, but we could see a degradation at 40k or so. So we need to find ways to reduce the footprint on WMStats.

Additional context
Likely dependent on:
#11241
and
#11243

@amaltaro
Copy link
Contributor Author

amaltaro commented Aug 9, 2022

As discussed with PPD and P&R, the target is to deploy this in September.

@amaltaro
Copy link
Contributor Author

There is a candidate fix for this, provided in #11263

But I don't feel like that is the best (or the most cost-benefit) way to address this issue. I think we should instead work at the foundations of the WMStats data structure and caching mechanism to decrease the memory footprint and data volume (network bandwidth).

Given that I haven't put any effort on it for the last month or so, I am unassigning myself.

@amaltaro amaltaro removed their assignment Jan 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: ToDo
Development

Successfully merging a pull request may close this issue.

1 participant