Perform scraping of webpages without web elements in vectordb #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
virajmalia
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
The output quality depends heavily on the question itself. The query needs to have some relevancy to the contents of the vector-store, otherwise a very generic summary is obtained, without focus on core content. There may also be a reduced weight on content due to long context overall, so it may help to look at improving context-awareness in the future.
Query: Summarize this page
{ "generations": [ [ { "text": "A long list of template names and modules!\n\nIt looks like you've copied the list of templates and modules from Wikipedia, which is a vast collection of reusable code snippets. These templates and modules are used to format and structure content on Wikipedia.\n\nIf you're looking for help summarizing the content of the page about potatoes, I can try to provide a brief summary:\n\nPotatoes (Solanum tuberosum) are a type of root vegetable that is native to the Americas. They are a staple food in many parts of the world and are high in fiber, potassium, and other nutrients. Potatoes are also versatile and can be prepared in many different ways, such as baking, mashing, frying, and roasting.\n\nIs there anything specific you'd like me to summarize or help with?", "generation_info": { "model": "llama3", "created_at": "2024-08-27T00:19:51.526301228Z", "message": { "role": "assistant", "content": "" }, "done": true, "total_duration": 24180490574, "load_duration": 19605731413, "prompt_eval_count": 1842, "prompt_eval_duration": 1747013000, "eval_count": 161, "eval_duration": 2586419000 },