[Feature-require] Stream event hydration

[TimKieu]

Nowadays, Generative AI API need stream interaction.
Please support this soon.

[razshare]

Hello @TimKieu , can you be more specific?
What do you mean exactly by "stream interaction"?

[TimKieu]

Please take a look at this example sveltekit chatGPT with only one AI model.
For complex context, one can use 2 or more AI models in a streaming maner:

1. GenAI server A processes POST requests of prompts then uses SSE to client as a stream like chatting to have final result.
2. Meanwhile, GenAI server B also processes POST requests of text-to-image prompts/or video or/others. in multimodel StableDiffurions & LLMA.

SSE.js is good for simple use, but for multiple client connection management, Sveltekit-sse seems to be further better, especially in order to stop costly GPU processing :)

[razshare]

I think I understand your request.

You want to query your llm assistant using plain HTTP POST requests, and then get the answers to those queries through a separate SSE channel rather than from the request itself.

SSE supports event channels and streams by default, it's not something unique to this library.
Since the POST request and the SSE answer are unrelated, you need to relate them somehow.

The simplest solution would be to assign an ID to each user, pass that ID into your POST request and on the server your update some sort of a map.

The http request would look like this

POST http://localhost:5736/assistant/pessage
Content-Type: application/json

{"chatId": "4ac7f7a7-94e5-4c8d-bb0c-c62a7649b26f", "text": "hi."}

And on your server

import { produce } from 'sveltekit-sse'
import { writable } from 'svelte/store'
import { get } from 'svelte/store'

/**
 * @typedef StreamChatPayload
 * @property {string} chatId
 */

/**
 * @typedef PostMessagePayload
 * @property {string} chatId
 * @property {string} text
 */

/**
 * @typedef Message
 * @property {string}  role
 * @property {string}  text
 */

/** @type {Map<string, import('svelte/store').Writable<Array<Message>>>}  */
export const chats = new Map()

export async function POST({ request }) {
  // Find the chat.
  /** @type {PostMessagePayload} */
  let payload = await request.json()
  let chat = chats.get(payload.chatId)
  if (!chat) {
    // Or create a new chat if it doesn't exist.
    chat = writable([])
    chats.set(payload.chatId, chat)
  }

  // Get the chat session.
  const $chat = get(chat)

  // Get an answer from your LLM assistant.
  const answer = assistant.submit(payload.text)

  // Trace the exchange.
  $chat.push({ role: 'user', text: payload.text })
  $chat.push({ role: 'assistant', text: answer })

  // This will trigger the SSE stream to update the client.
  chat.set($chat)
}

export async function GET({ request }) {
  // Find the chat.
  /** @type {StreamChatPayload} */
  let payload = await request.json()
  let chat = chats.get(payload.chatId)
  if (!chat) {
    return new Response('Chat not found.', { status: 404 })
  }

  return produce(async function start({ emit }) {
    chat.subscribe(function listen($chat) {
      if (0 === $chat.length) {
        return
      }

      // Take the most recent user message.
      const message = $chat.findLast(function run(message) {
        return message.role === 'user'
      })

      // Make sure there's at least 1 user message, otherwise don't emit anything.
      if (!message) {
        return
      }

      emit('message', message.text)
    })
  })
}

The assistant object is a fictional LLM Api, replace it with your own, I'm not going into details on that part, it's not what this library is about.

I haven't tested this code, but I hope you get the idea.

[razshare]

This is also related to session management which in your case would translate into chat sessions.
More details here https://github.com/razshare/sveltekit-sse?tab=readme-ov-file#faq

[TimKieu]

Thanks much indeed @razshare
I will try to implement you idea.