Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX] Pasting Images in Chat with Clipboard or CTRL+V #259

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6a0370e
Merge remote-tracking branch 'upstream/main'
Oct 28, 2024
6a66cdd
Merge remote-tracking branch 'upstream/main'
Oct 28, 2024
134340a
Merge remote-tracking branch 'upstream/main'
charlwillia6 Oct 29, 2024
3f07de8
Merge remote-tracking branch 'upstream/main'
charlwillia6 Oct 30, 2024
8f7f236
Merge remote-tracking branch 'upstream/main'
charlwillia6 Oct 30, 2024
447600c
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 1, 2024
02b8de3
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 2, 2024
99f66ae
Merge remote-tracking branch 'upstream/main'
Nov 4, 2024
527e486
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 7, 2024
a0e9d50
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 8, 2024
b907905
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 10, 2024
0b5244d
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 11, 2024
7e47d71
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 11, 2024
0df9372
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 12, 2024
ab14a70
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 14, 2024
a4eed44
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 15, 2024
7936495
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 19, 2024
64ad339
Merge remote-tracking branch 'upstream/main'
charlwillia6 Nov 20, 2024
ebe21bb
Merge remote-tracking branch 'upstream/main'
charlwillia6 Dec 3, 2024
a960793
Merge remote-tracking branch 'upstream/main'
charlwillia6 Dec 6, 2024
28c7c01
Merge remote-tracking branch 'upstream/main'
charlwillia6 Dec 12, 2024
bc98cf8
Merge remote-tracking branch 'upstream/main'
charlwillia6 Dec 16, 2024
f0dfa6c
Merge remote-tracking branch 'upstream/main'
charlwillia6 Jan 24, 2025
2ffd589
Merge remote-tracking branch 'upstream/main'
charlwillia6 Jan 31, 2025
b168592
Merge remote-tracking branch 'upstream/main' into charlwillia6/fix-im…
charlwillia6 Feb 4, 2025
3b26ec1
Fixed an issue with the TopGuiDiv resizer not working
charlwillia6 Feb 4, 2025
774a889
Fixed a spacing issue
charlwillia6 Feb 4, 2025
1085ddc
Fixed an issue with pasting images in the chat input box
charlwillia6 Feb 4, 2025
5b71846
Adjusted the height of the status bar.
charlwillia6 Feb 4, 2025
1597561
Fixed a validateDOMNesting issue
charlwillia6 Feb 4, 2025
6e40b0f
Fixes memory leak in image handling
charlwillia6 Feb 4, 2025
0649ab8
Fixes unnecessary rerenders
charlwillia6 Feb 4, 2025
acf1b0a
Improves image pasting reliability
charlwillia6 Feb 4, 2025
86dbaa0
Improves image upload handling
charlwillia6 Feb 4, 2025
75d6e43
Fixes DOM text reinterpreted as HTML
charlwillia6 Feb 4, 2025
8eba6c1
Fixed plain text pasting.
charlwillia6 Feb 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions gui/src/components/StatusBar.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ const StatusBar = () => {
});

return (
<div className="items-center flex justify-between">
<div className="items-center flex justify-between pb-2">
<div className="flex items-center gap-2">
{/* Indexing Progress Bar */}
<IndexingProgressBar indexingState={indexingState} />
Expand All @@ -50,4 +50,4 @@ const StatusBar = () => {
);
};

export default StatusBar;
export default StatusBar;
2 changes: 1 addition & 1 deletion gui/src/components/gui/StepContainer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ function StepContainer({

{!active && (
<div
className="px-2 flex gap-1 justify-between -mt-[6px]"
className="px-2 flex gap-1 justify-between mt-2"
style={{
color: lightGray,
fontSize: getFontSize() - 3,
Expand Down
137 changes: 89 additions & 48 deletions gui/src/components/mainInput/TipTapEditor.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@
import { isAiderMode, isPerplexityMode } from "../../util/bareChatMode";
import { TipTapContextMenu } from './TipTapContextMenu';


const InputBoxDiv = styled.div`
position: relative;
resize: none;
Expand Down Expand Up @@ -125,7 +124,6 @@
pointer-events: none;
`;


const getPlaceholder = (historyLength: number, location: any) => {
if (location?.pathname === "/aiderMode" || location?.pathname === "/inventory/aiderMode") {
return historyLength === 0
Expand Down Expand Up @@ -163,6 +161,59 @@
onChange?: (newState: JSONContent) => void;
}

export const handleImageFile = async (
charlwillia6 marked this conversation as resolved.
Show resolved Hide resolved
file: File,
onError?: (message: string) => void
): Promise<[HTMLImageElement, string] | undefined> => {
const filesize = file.size / 1024 / 1024; // filesize in MB

if (
[
"image/jpeg",
"image/jpg",
"image/png",
"image/gif",
"image/svg",
"image/webp",
].includes(file.type) &&
filesize < 10
) {
// check dimensions
const img = new window.Image();
img.src = URL.createObjectURL(file);
Fixed Show fixed Hide fixed
return await new Promise((resolve) => {
const safeRevokeURL = () => {
try {
URL.revokeObjectURL(img.src);
} catch (error) {
console.error('Error revoking URL:', error);
}
};

img.onload = function () {
const dataUrl = getDataUrlForFile(file, img);
const image = new window.Image();

image.src = dataUrl;
image.onload = function () {
resolve([image, dataUrl]);
safeRevokeURL();
};
image.onerror = function() {
safeRevokeURL();
resolve(undefined);
};
};
img.onerror = function() {
safeRevokeURL();
resolve(undefined);
};
});
} else if (onError) {
onError("Images need to be in jpg or png format and less than 10MB in size.");
}
};

export const handleCopy = (editor: Editor) => {
const selection = editor.state.selection;
const text = editor.state.doc.textBetween(selection.from, selection.to, '\n');
Expand All @@ -177,7 +228,32 @@
};

export const handlePaste = async (editor: Editor) => {
try {
const items = await navigator.clipboard.read();

for (const item of items) {
// Handle images
if (item.types.includes('image/png') || item.types.includes('image/jpeg')) {
try {
const imageBlob = await item.getType(item.types.find(type => type.startsWith('image/')) || 'image/png');
const file = new File([imageBlob], 'pasted-image.png', { type: 'image/png' });
const result = await handleImageFile(file);

if (result) {
const [, dataUrl] = result;

editor.commands.setImage({ src: dataUrl });
return true;
}
} catch (err) {
console.error('Failed to paste image:', err);
}
}
}

// Fall back to text handling if no image
const clipboardText = await navigator.clipboard.readText();

if (clipboardText) {
const lines = clipboardText.split(/\r?\n/);
const { tr } = editor.state;
Expand All @@ -198,6 +274,10 @@
editor.view.dispatch(tr);
return true;
}
} catch (error) {
console.error('Error handling paste:', error);
}

return false;
};

Expand Down Expand Up @@ -275,6 +355,7 @@
(store: RootState) => store.state.contextItems,
);
const defaultModel = useSelector(defaultModelSelector);
const defaultModelRef = useUpdatingRef(defaultModel);
const getSubmenuContextItemsRef = useUpdatingRef(getSubmenuContextItems);
const availableContextProvidersRef = useUpdatingRef(availableContextProviders)

Expand All @@ -296,47 +377,6 @@

const activeRef = useUpdatingRef(active);

async function handleImageFile(
file: File,
): Promise<[HTMLImageElement, string] | undefined> {
const filesize = file.size / 1024 / 1024; // filesize in MB
// check image type and size
if (
[
"image/jpeg",
"image/jpg",
"image/png",
"image/gif",
"image/svg",
"image/webp",
].includes(file.type) &&
filesize < 10
) {
// check dimensions
const _URL = window.URL || window.webkitURL;
const img = new window.Image();
img.src = _URL.createObjectURL(file);

return await new Promise((resolve) => {
img.onload = function () {
const dataUrl = getDataUrlForFile(file, img);

const image = new window.Image();
image.src = dataUrl;
image.onload = function () {
resolve([image, dataUrl]);
};
};
});
} else {
ideMessenger.post("errorPopup", {
message:
"Images need to be in jpg or png format and less than 10MB in size.",
});
}
return undefined;
}

const mainEditorContent = useSelector(
(store: RootState) => store.state.mainEditorContent,
);
Expand Down Expand Up @@ -391,16 +431,17 @@
props: {
handleDOMEvents: {
paste(view, event) {
console.log("Pasting image");
const items = event.clipboardData.items;
for (const item of items) {
const file = item.getAsFile();
const model = defaultModelRef.current;

file &&
modelSupportsImages(
defaultModel.provider,
defaultModel.model,
defaultModel.title,
defaultModel.capabilities,
model.provider,
model.model,
model.title,
model.capabilities,
) &&
handleImageFile(file).then((resp) => {
if (!resp) {
Expand Down
4 changes: 2 additions & 2 deletions gui/src/pages/gui.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ function GUI() {
clearTimeout(timeoutId);
window.removeEventListener("scroll", handleScroll);
};
}, [topGuiDivRef.current]);
}, [topGuiDivRef]);

useEffect(() => {
const resizeObserver = new ResizeObserver(() => {
Expand All @@ -243,7 +243,7 @@ function GUI() {
}

return () => resizeObserver.disconnect();
}, [isNewSession]);
}, [isNewSession, inputContainerRef, topGuiDivRef]);

useEffect(() => {
const listener = (e: any) => {
Expand Down
8 changes: 4 additions & 4 deletions gui/src/pages/history.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -415,24 +415,24 @@ export function History({
<Fragment key={index}>
{from === 'continue' && index === 0 && date > yesterday && (
<SectionHeader style={{ top: `${headerHeight - 1}px` }}>
Today
<td>Today</td>
</SectionHeader>
)}
{from === 'continue' && date < yesterday &&
date > lastWeek &&
prevDate > yesterday && (
<SectionHeader style={{ top: `${headerHeight - 1}px` }}>
This Week
<td>This Week</td>
</SectionHeader>
)}
{from === 'continue' && date < lastWeek &&
date > lastMonth &&
prevDate > lastWeek && (
<SectionHeader style={{ top: `${headerHeight - 1}px` }}>
This Month
<td>This Month</td>
</SectionHeader>
)}
<Tr key={index}>
<Tr key={`row-${index}`}>
<TableRow
session={session}
date={date}
Expand Down