WIP: Adding Logging

[mvanniasingheTT]

This WIP PR looks to add logging of raw stats. Currently tested with offline inference. I have the logger class in vllm-tt-metal-llama3-70b/src/logging_utils.py but I also have it under tests right now so you can see how it is used. Any feedback would be great thank you!

[tstescoTT]

If there is no decode in the generation, the logging will error out:

Traceback (most recent call last):                                                                                                                                                                                                               │····
  File "benchmark_vllm_offline_inference.py", line 207, in <module>                                                                                                                                                                              │····
    run_inference(                                                                                                                                                                                                                               │····
  File "benchmark_vllm_offline_inference.py", line 116, in run_inference                                                                                                                                                                         │····
    run_inference_perf(llm, prompt_token_ids, sampling_params)                                                                                                                                                                                   │····
  File "benchmark_vllm_offline_inference.py", line 139, in run_inference_perf                                                                                                                                                                    │····
    generate_tokens(                                                                                                                                                                                                                             │····
  File "benchmark_vllm_offline_inference.py", line 168, in generate_tokens                                                                                                                                                                       │····
    outputs = llm.generate(prompts, sampling_params, prompt_token_ids)                                                                                                                                                                           │····
  File "/home/user/vllm/vllm/utils.py", line 1073, in inner                                                                                                                                                                                      │····
    return fn(*args, **kwargs)                                                                                                                                                                                                                   │····
  File "/home/user/vllm/vllm/entrypoints/llm.py", line 353, in generate                                                                                                                                                                          │····
    outputs = self._run_engine(use_tqdm=use_tqdm)                                                                                                                                                                                                │····
  File "/home/user/vllm/vllm/entrypoints/llm.py", line 879, in _run_engine                                                                                                                                                                       │····
    step_outputs = self.llm_engine.step()                                                                                                                                                                                                        │····
  File "/home/user/vllm/vllm/engine/llm_engine.py", line 1454, in step                                                                                                                                                                           │····
    self.do_log_stats(scheduler_outputs, outputs)                                                                                                                                                                                                │····
  File "/home/user/vllm/vllm/engine/llm_engine.py", line 1562, in do_log_stats                                                                                                                                                                   │····
    logger.log(stats)                                                                                                                                                                                                                            │····
  File "/home/user/tests/mock_vllm_model.py", line 428, in log                                                                                                                                                                                   │····
    self._write_to_json(stats)                                                                                                                                                                                                                   │····
  File "/home/user/tests/mock_vllm_model.py", line 458, in _write_to_json                                                                                                                                                                        │····
    data["time to first token"][                                                                                                                                                                                                                 │····
KeyError: 'Inference num:1'  

Can you make the logging handle this case?

to reproduce,

python examples/offline_inference_tt.py --measure_perf --max_seqs_in_batch 32 --perf_prompt_len 128 --max_tokens 1 --greedy_sampling

[tstescoTT]

Could you remove test/data and add the .jsonl output to .gitignore?