Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add more possible file types in attachments #2624

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

MrHertal
Copy link
Contributor

@MrHertal MrHertal commented Aug 9, 2024

Currently you can only attach images to the prompt along with a text.

Some providers are handling more file types. This is the case for Bedrock Converse API which accept documents like pdf, docx etc.

In this PR, I created an additional part type called file which is more generic than the current image part type.

I didn't want to break anything so I duplicated the existing mechanism of image without replacing it.
You can pass the file blob or a URL to the file, just like with the images.

For now I only updated Bedrock provider to handle this new part type. I wanted some feedbacks before working on the other providers.

@lgrammel
Copy link
Collaborator

Which provider did you use to try this out? Anthropic? are you sure that the contents of the PDF have been used?

@MrHertal
Copy link
Contributor Author

Which provider did you use to try this out? Anthropic? are you sure that the contents of the PDF have been used?

Yes I tried with Bedrock and Claude 3. If you run the example:

$ pnpm tsx src/stream-text/amazon-bedrock-multimodal.ts
...
Overall, this document appears to be a standard Lorem Ipsum placeholder text, commonly used as a sample or filler text in design and publishing projects when the final content is not yet available.

it reads the pdf which is a Lorem ipsum

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants