-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider adding prometheus metrics for DMCI /insert, /update and maybe /validate endpoints #220
Comments
Also what my thought was for this issue, when we have a working prometheus exporter, is to see if we also can create some The worker returns a list of failed distributors, so we should have some metrics like:
From the failed array in https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/worker.py#L171. Maybe the code have to be rewritten a bit, since we dont get this array back into the /update and /insert endpoints, So the lines at https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/app.py#L74 need to get the failed array back. so the code
|
do I read correctly that the only place in practice where this 'failed' array would be not null is in this if block? https://github.com/metno/discovery-metadata-catalog-ingestor/blob/main/dmci/api/app.py#L159 |
Yes. You are right. you will have to return the failed array from the I think if |
ok this probably means creating custom metrics counters defined via prometheus_client and incrementing them in |
I am not sure myself how this is working.. but It should be possible somehow. I bet other projects using flask app also want to expose custom prometheus metrics for their business-logic and not just reponse codes from api endpoints |
anyway, for now still no default metrics exposure.. |
giving it a try https://gitlab.met.no/tjenester/s-enda/-/merge_requests/2928 |
Seems like it works now: But now I do not see those default
|
with this latest PR I try again with GunicornPrometheusMetrics + CollectorRegistry call + having all the metrics at port 9200.. again it works for me when I run the container locally |
maybe we can add to the mid-level instance the dashboard of the exporter customized with out extra metrics, I 'll look into it |
The https://github.com/rycus86/prometheus_flask_exporter, supports exporting flask metrics out of the box.
This could be beneficial to monitor HTTP error codes for the DMCI endpoints, if insert and update calls fails etc, and then we can get alerts if DMCI have a lot of failed inserts for example.
The text was updated successfully, but these errors were encountered: