Using FAISS in production!

[anubhav562]

Hello FAISS team!

Thanks for building and maintaining the FAISS project!

I have a use case and some follow up questions related to it:

Use Case:

We want to build a vector similarity engine of a scale between 500M - 1B vectors. The number of vectors grow each day non uniformly. We have budget constraints for resources so we are thinking of sticking to the [Flat Quantizer + IVFPQ] or [HNSW Quantizer + IVFPQ] composite indices. Regarding this use case, mentioned below are the follow up questions:

Questions:

1. Which Quantizer should we prefer? We have memory constraints!

2. Is it advisible to train FAISS index on n = 500M vectors? Or should we train on a small subset and add rest of them to the index!

3. Are there any benchmarking values that show how fast the GPU training could be?

Your answers would be really helpful to us!

Thanks and Best Regards,
Anubhav Chhabra