You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am using haystack's OllamaGenerator which takes a streaming callback function for streaming message directly from llm, I am able to print this on server by below code but I am not sure how can I return that stream to caller instead of sync request.
@get("/query")
async def query_db(q: str) -> str:
result = query_documents(q, streaming_callback)
return result
def streaming_callback(chunk):
print(chunk.content, end="", flush=True)
I have tried to hack and store response in content and use a stream but no success, it still return me response at the end instead of streaming
content = []
# Global event to notify when content is updated
content_event = asyncio.Event()
async def my_generator_two() -> AsyncGenerator[bytes, None]:
while True:
# Wait until content is updated
await content_event.wait()
content_event.clear() # Reset event
# Stream the updated content to the client
yield encode_json({"content": content})
@get("/query", sync_to_thread=False)
async def query_db(q: str) -> Stream:
query_documents(q, streaming_callback))
return Stream(my_generator_two())
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I am using haystack's OllamaGenerator which takes a streaming callback function for streaming message directly from llm, I am able to print this on server by below code but I am not sure how can I return that stream to caller instead of sync request.
I have tried to hack and store response in
content
and use a stream but no success, it still return me response at the end instead of streaminghow can I make it happen?
Thanks
Beta Was this translation helpful? Give feedback.
All reactions