allow traffic split across model with user provided ratios

[kaushikmitr]

This pull request introduces significant updates to the benchmarking process in benchmark_serving.py and modifies the latency_throughput_curve.sh script to support traffic splitting across multiple models. The most important changes include the addition of traffic split functionality, refactoring of the benchmarking function, and updates to the script for handling new arguments.

Benchmarking improvements:

• Added a new function run_single_request to handle individual requests with model selection based on traffic split.
• Refactored the benchmark function to support model selection per request and save results separately for each model.
• Updated the main function to call the refactored benchmark function with traffic split arguments.

Script updates:

• Added --traffic-split argument to the argparse parser in benchmark_serving.py to specify traffic split proportions.
• Modified latency_throughput_curve.sh to include the --traffic-split argument in the Python command options. [1] [2]

removed save_aggregated_results:

This does not seem to be reliable as even with 1 model the overall metrics does not agree with individual model metrics with significant delta (> 10%). Did not remove flag as existing scripts might fail.

Minor fixes:

• Corrected a typo in the latency_throughput_curve.sh script from "Benchmaking" to "Benchmarking".

Tested with:

1. three model with traffic split: python3 benchmark_serving.py --save-json-results --host=35.240.228.178 --port=8000 --dataset=ShareGPT_V3_unfiltered_cleaned_split.json --tokenizer=meta-llama/Llama-2-7b-hf --request-rate=2 --backend=vllm --num-prompts=30 --max-input-length=1024 --max-output-length=1024 --file-prefix=lgp-ig-prod-v1-base --models=meta-llama/Llama-2-7b-hf,tweet-summary-0,tweet-summary-1 --output-bucket=kaushikmitra-llm-ig-benchmark --save-aggregated-result --output-bucket-filepath t --traffic-split=0.7,0.2,0.1
2. one model: python3 benchmark_serving.py --save-json-results --host=35.240.228.178 --port=8000 --dataset=ShareGPT_V3_unfiltered_cleaned_split.json --tokenizer=meta-llama/Llama-2-7b-hf --request-rate=2 --backend=vllm --num-prompts=10 --max-input-length=1024 --max-output-length=1024 --file-prefix=lgp-ig-
   prod-v1-base --models=meta-llama/Llama-2-7b-hf --output-bucket=kaushikmitra-llm-ig-benchmark --save-aggregated-result --output-bucket-filepath t --traffic-split=1
3. three model with traffic split does not add to 1: Errors out

[kaushikmitr]

list of changes in the PR description

[achandrasekar]

It is a little confusing to understand that this is req_latency - ttft / output_tokens - 1 since we are choosing different array indexes. Can we either change it to dict or add a comment above explaining what latency[2] and latency[1] are? We can also create separate variables for these which would make it easier to follow.

[kaushikmitr]

Yes, agreed. Done

[achandrasekar]

Have you verified by not passing in --traffic-split that it works the same as before?

[kaushikmitr]

Yes just added.

[achandrasekar]

Were we using this for anything specific before? Are we missing anything by removing it? cc @liu-cong

[kaushikmitr]

I was not using this, relying on individual model metrics

[achandrasekar]

Looks good. You might need to rebase.