-
Notifications
You must be signed in to change notification settings - Fork 368
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(dev): Add image upload (vision) support for internal testing #7279
base: main
Are you sure you want to change the base?
Conversation
feat(chat): Add image upload support This commit adds support for uploading images in the chat interface. Users can now upload images, which are then displayed as part of the chat messages. The changes include: - Adding a new `MediaUploadButton` component for handling image uploads. - Adding a new `ContextItemMedia` type to represent media context items. - Updating the `constructGeminiChatMessages` function to handle media content. - Updating the `renderContextItem` function to handle media content. - Updating the `ContextItemMentionNode` to render media context items. - Updating the `HumanMessageEditor` to include the `MediaUploadButton` and handle media uploads. - Adding `@google/generative-ai` as a dependency. - Updating the `Message` type to include `data` and `mimeType` for media. - Updating the `chat-question` telemetry event to include media context. - Updating the `googleChatClient` to use the `@google/generative-ai` library. - Removing the `GeminiChatMessage` interface.
Fixes an issue in the Kotlin code generator where property names with underscores were not properly converted to camelCase. For example, 'image_url' in TypeScript was incorrectly converted to 'image-url' in Kotlin when it should be'imageUrl'. The fix adds logic to detect and properly transform snake_case identifiers to camelCase while preserving the original string for serialization purposes. The problem was that when processing field names with underscores like image_url, the code wasn't converting them to camelCase format as is standard in Kotlin. Now, the updated code will: 1. Detect if a field name contains underscores 2. Replace each underscore followed by a character with the uppercase version of that character using a regex 3. For example, image_url will be converted to imageUrl rather than keeping the underscore ## Test Plan This fix ensures that when TypeScript property names with underscores are converted to Kotlin, they'll follow Kotlin naming conventions while still maintaining the original string literal value for serialization. Run `pnpm generate-agent-kotlin-bindings` to confirm all the exisiting behavior works currently, but for new property added with snake_case identifiers are transformed into camelCase correctly while preserving the original string for serialization purposes.
<Button | ||
variant="ghost" | ||
size="none" | ||
aria-label="Upload images (drag, select, or paste with Cmd+V)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be helpful to mention Ctrl+V for Windows/Linux users (maybe in a subsequent PR)
<div className="tw-text-center"> | ||
<div>Upload images (PNG, JPEG, WEBP, HEIC, HEIF)</div> | ||
<div className="tw-text-sm tw-opacity-75 tw-mt-1"> | ||
Drag & drop, paste Cmd+V, or click to select |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will be helpful to mention Ctrl+V for Windows/Linux users (maybe in a subsequent PR)
Building block for #7235
This commit adds support for multi-media content in messages, particularly for image handling with byo dev models from google, including support for uploading images in the chat interface. Users can now upload images when they are using the configured model , and the uploaded images are then displayed as part of the chat messages as at-mention chip.
vision.mov
Key changes:
Other changes:
MediaUploadButton
component for handling image uploads.ContextItemMedia
type to represent media context items.constructGeminiChatMessages
function to handle media content.renderContextItem
function to handle media content.ContextItemMentionNode
to render media context items.HumanMessageEditor
to include theMediaUploadButton
and handle media uploads.@google/generative-ai
as a dependency.Message
type to includedata
andmimeType
for media.chat-question
telemetry event to include media context.googleChatClient
to use the@google/generative-ai
library.Updated
Enterprise instances with Early Access feature flag enabled (e.g. sg02) could also enable vision models:
Test plan
Steps
For internal users: connected to sg02 instance
For BYOK
When sending to sg