Kvrouter that will increase the kv-cache hits in case of multiple routing strategy

[Narsil]

What does this PR do?

Strategy is on purpose relatively stupid in order to account for many types of factor.

Currently the kv-cache hit rate (on 4 replicas) bumps for 62% to 93% with this router. This is done on standard tests, not actual production yet.

Code to test (thanks r1)

import requests

def get_cache_metrics(port):
    url = f"http://localhost:{port}/metrics"
    try:
        response = requests.get(url)
        response.raise_for_status()
        metrics = {}
        for line in response.text.split('\n'):
            if line.startswith('tgi_cache_hit'):
                metrics['hit'] = int(line.split()[1])
            elif line.startswith('tgi_cache_total'):
                metrics['total'] = int(line.split()[1])
        return metrics
    except requests.exceptions.RequestException as e:
        print(f"Error fetching metrics from port {port}: {e}")
        return None

def main():
    initial_metrics = {}
    updated_metrics = {}
    
    # First run
    print("=== FIRST RUN ===")
    for port in range(8000, 8004):
        metrics = get_cache_metrics(port)
        if metrics:
            server_id = f"Server {port}"
            initial_metrics[server_id] = {
                'hit': metrics.get('hit', 0),
                'total': metrics.get('total', 0)
            }
            print(f"{server_id}: Initial metrics recorded")

    # Wait for user action
    input("Press Enter when you're ready to start the second run...")

    # Second run
    print("\n=== SECOND RUN ===")
    for port in range(8000, 8004):
        metrics = get_cache_metrics(port)
        if metrics:
            server_id = f"Server {port}"
            updated_metrics[server_id] = {
                'hit': metrics.get('hit', 0),
                'total': metrics.get('total', 0)
            }
            print(f"{server_id}: Updated metrics recorded")

    # Calculate metrics change
    total_hit_change = 0
    total_request_change = 0
    for server_id in initial_metrics:
        initial = initial_metrics[server_id]
        updated = updated_metrics.get(server_id, {'hit': 0, 'total': 0})
        hit_change = updated['hit'] - initial['hit']
        request_change = updated['total'] - initial['total']
        if request_change < 0:
            request_change = 0

        total_hit_change += hit_change
        total_request_change += request_change

        print(f"{server_id}: Cache hits during runtime = {hit_change}")
        print(f"{server_id}: Cache requests during runtime = {request_change}\n")

    if total_request_change > 0:
        hit_rate = (total_hit_change / total_request_change) * 100
        print(f"Total Cache Hit Rate during runtime: {hit_rate:.2f}%\n")
    else:
        print("No new cache requests recorded across all servers during runtime")

if __name__ == "__main__":
    main()

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.