Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support JSON response for /metrics endpoint, add client method for metrics #725

Merged
merged 9 commits into from
Jan 3, 2025

Conversation

arnavgarg1
Copy link
Contributor

@arnavgarg1 arnavgarg1 commented Jan 3, 2025

This PR adds a new metrics method to both the async and sync LoRAX client objects that returns a nearly equivalent payload to what the /metrics endpoint returns.

By default, the /metrics endpoint returns a response in plain/text format and in a format compatible with Prometheus. This PR modifies that endpoint to support a new optional query parameter called format that can optionally be set to JSON or just requires the application/json header.

curl -X GET "http://127.0.0.1:8080/metrics" -H "Accept: application/json"

On the Python client side, you can just call client.metrics(format="json") to get the relevant response back:

from lorax import Client

endpoint_url = "http://127.0.0.1:8080"
client = Client(endpoint_url)

metrics = client.metrics(format="json") # just call .metrics() if you want it in prometheus format

The structure follows this format:

{
        parent_metric_name: {
                "type": histogram | gauge | summary | counter
                "data": [
                     {"key": X, "value": Y},
                     {"key": X, "value": Y},
                     ...
                ]
        }
}

For metrics that are histograms, you will also find a sum and count key in the data objects.

Here's an example snippet when grabbing this output:

{'lorax_batch_current_max_tokens': {'data': [{'key': '', 'value': 0.0}],
                                    'type': 'gauge'},
 'lorax_batch_current_size': {'data': [{'key': '', 'value': 0.0}],
                              'type': 'gauge'},
 'lorax_batch_inference_count': {'data': [{'key': 'lorax_batch_inference_count{method="decode"}',
                                           'value': 508.0},
                                          {'key': 'lorax_batch_inference_count{method="prefill"}',
                                           'value': 4.0}],
                                 'type': 'counter'},
.....
 'lorax_request_skipped_tokens': {'data': [{'key': 'lorax_request_skipped_tokens{quantile="0"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="0.5"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="0.9"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="0.95"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="0.99"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="0.999"}',
                                            'value': 0.0},
                                           {'key': 'lorax_request_skipped_tokens{quantile="1"}',
                                            'value': 0.0},
                                           {'key': 'sum', 'value': 0.0},
                                           {'key': 'count', 'value': 512.0}],
                                                         'type': 'summary'}
....
}

@arnavgarg1 arnavgarg1 changed the title Client metrics Add new client method for getting metrics in json and prometheus format Jan 3, 2025
@arnavgarg1 arnavgarg1 changed the title Add new client method for getting metrics in json and prometheus format Support JSON response for /metrics endpoint, add client method for metrics Jan 3, 2025
@arnavgarg1 arnavgarg1 merged commit 7cd7d06 into main Jan 3, 2025
2 checks passed
@arnavgarg1 arnavgarg1 deleted the client_metrics branch January 3, 2025 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants