Ingest larger documents (over 800MB)

[bru-singh]

I'm running ingest.py for ingesting a txt file containing question and answer pairs, it is over 800MB (I know it's a lot). I am using the instruct-xl as the embedding model to ingest. Any approximate idea for how long will it take to complete the ingest process.

Loads all documents from the source documents directory, ignoring specified files
Loading new documents: 100%|██████████████████████| 1/1 [00:11<00:00, 11.11s/it]
Loaded 1 new documents from //SOURCE_DOCUMENTS LOCATION
Split into 981498 chunks of text (max. 1000 tokens each)
Creating embeddings. May take some minutes...
Using embedded DuckDB with persistence: data will be stored in: DB

[PromtEngineer]

I am curious what system you are running this on 😃 These are a lot of chunks. Curious if this is finished.

[bru-singh]

Im currently using a system with 16GB RAM, and 3 nvidia TESLA GPUs (16GBs each) but the localGPT only took 4-5 GB of GPU space.
it took some time to finish the ingesting (I kept it running for a day). But the inferences from the model takes some time. Any way to reduce that?