Token / LLM call monitoring

[pieterjanvc]

Llamaindex has options to count tokens which might be interesting to implement in parts of the app to monitor how much data is sent / received from the LLM and help estimate costs / usage

[lee-t]

OpenAI-API uses TikToken library for tokenization of strings as part of their api call. The token_count has a set of functions that specifically uses the same library to count tokens in arbitrary strings in python.

for example: a function to count tokens in a chat prompt

from token_count import TokenCount

async def run_prompt(placeholder, meta, prompt, model):
    tc = TokenCount(model_name="gpt-3.5-turbo")
    start = time.time()
    stream = await client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": "You are a helpful assistant."},
                  {"role": "user", "content": prompt},],
        stream=True
    )
    streamed_text = ""
    async for chunk in stream:
        chunk_content = chunk.choices[0].delta.content
        if chunk_content is not None:
            streamed_text = streamed_text + chunk_content
            placeholder.write(streamed_text)
            end = time.time()
            time_taken = end-start
            tokens = tc.num_tokens_from_string(streamed_text)
            meta.info(f"""**Duration: :green[{time_taken:.2f} secs]**
            **Eval count: :green[{tokens} tokens]**
            **Eval rate: :green[{tokens / time_taken:.2f} tokens/s]**
            """)