Faiss v1.10.0 Release

[mnorris11]

Faiss v1.10.0 Release

The latest open-source release of Faiss has been published to Conda with 34 contributors since 1.9.0 from several companies, including AMD, Intel, Nvidia, Zilliz, and more.

• https://anaconda.org/pytorch/faiss-cpu
• https://anaconda.org/pytorch/faiss-gpu
• https://anaconda.org/pytorch/faiss-gpu-cuvs
• https://anaconda.org/pytorch/libfaiss

Thank you to all contributors!

Changelog

Added

• Add desc_name to dataset descriptor (Add desc_name to dataset descriptor #3935)
• implement ST_norm_from_LUT for the ResidualQuantizer (implement ST_norm_from_LUT for the ResidualQuantizer #3917)
• Add example of how to build, link, and test an external SWIG module (Add example of how to build, link, and test an external SWIG module #3922)
• add copyright header (add copyright header #3948)
• Add some SVE implementations (Add some SVE implementations #3933)
• Enable linting: lint config changes plus arc lint command (Enable linting: lint config changes plus arc lint command #3966)
• Re-add example of how to build, link, and test an external SWIG module (Re-add example of how to build, link, and test an external SWIG module"" #3981)
• demo: IndexPQ: separate codes from codebook (demo: IndexPQ: separate codes from codebook #3987)
• add all wrapped indexes to the index_read (add all wrapped indexes to the index_read #3988)
• add validity check AlignedTableTightAlloc clear method (add validity check AlignedTableTightAlloc clear method #3997)
• Add index binary to telemetry (Add index binary to telemetry #4001)
• Add VectorTransform read from filename to the C API (Add VectorTransform read from filename to the C API #3970)
• Added IndexLSH to the demo (Added IndexLSH to the demo #4009)
• write distributed_kmeans centroids and assignments to hive tables (write distributed_kmeans centroids and assignments to hive tables #4017)
• introduce data splits in dataset descriptor (introduce data splits in dataset descriptor #4012)
• Faiss GPU: bfloat16 brute-force kNN support (Faiss GPU: bfloat16 brute-force kNN support (#4014) #4018)
• ROCm support for bfloat16 (ROCm support for bfloat16 #4039)
• Unit tests for distances_simd.cpp (Unit tests for distances_simd.cpp #4058)
• add cuda-toolkit for GPU ([2/n] add cuda-toolkit for GPU #4057)
• Add more unit testing for IndexHNSW [1/n] (Add more unit testing for IndexHNSW [1/n] #4054)
• Add more unit testing for IndexHNSW [2/n] (Add more unit testing for IndexHNSW [2/n] #4056)
• Add more unit testing for HNSW [3/n] (Add more unit testing for HNSW [3\n] #4059)
• Add more unit testing for HNSW [4/n] (Add more unit testing for HNSW [4/n] #4061)
• Add more unit tests for index_read and index_write (Add more unit tests for index_read and index_write #4068)
• Add testing for utils/hamming.cpp (Add testing for utils/hamming.cpp #4079)
• Test sa_decode methd on IndexIVFFlat (Test sa_decode methd on IndexIVFFlat #4098)
• Conditionally compile extras like benchmarks and demos (Conditionally compile extras like benchmarks and demos #4094)
• Add a new architecture mode: 'avx512_spr'. (Add a new architecture mode: 'avx512_spr'. #4025)
• Use _mm512_popcnt_epi64 to speedup hamming distance evaluation. (Use _mm512_popcnt_epi64 to speedup hamming distance evaluation. #4020)
• PQ with pytorch (PQ with pytorch #4116)
• add range_search() to IndexRefine (add range_search() to IndexRefine #4022)
• Expose accumulate_to_mem from faiss interface (Expose accumulate_to_mem from faiss interface #4099)
• Windows Arm64 support (Windows Arm64 support #4087)
• add test to cover GPU (add test to cover GPU #4130)
• Added support for building without MKL (Added support for building without MKL #4147)

Changed

• Move train, build and search to their respective operators (Move train, build and search to their respective operators #3934)
• PQFS into Index trainer (PQFS into Index trainer #3941)
• Place a useful cmake function 'link_to_faiss_lib' into a separate file (Place a useful cmake function 'link_to_faiss_lib' into a separate file #3939)
• Cache device major version value to avoid multiple calls of getCudaDeviceProperties (Cache device major version value to avoid multiple calls of getCudaDeviceProperties #3950)
• Consolidate set_target_properties() calls in faiss/CMakeLists.txt (Consolidate set_target_properties() calls in faiss/CMakeLists.txt #3973)
• Removing Manual Hipify Build Step (Removing Manual Hipify Build Step #3962)
• Allow to replace graph structure for NSG graphs (Allow to replace graph structure for NSG graphs #3975)
• Adjust nightly build (Adjust nightly build #3978)
• Update RAFT CI with pytorch 2.4.1 (Update RAFT CI with pytorch 2.4.1 #3980)
• Moved add_sa_codes, sa_code_size to Index, IndexBinary base classes (Moved add_sa_codes, sa_code_size to Index, IndexBinary base classes #3989)
• Update autoclose.yml (Update autoclose.yml #4000)
• Migrate from RAFT to CUVS (Migrate from RAFT to CUVS #3549)
• Pin to numpy<2 (Pin to numpy<2 #4033)
• (1/n) - Preload datasets in manifold so that subsequent stages of training, indexing and search can use those instead of each trainer or indexer downloading data. (Add a class variable to benchmark CodecDescriptor,IndexDescriptor and KnnDescriptor for the filename prefix  #4034)
• Constrain conda version for Windows build (Constrain conda version for Windows build #4040)
• Updates to faiss-gpu-cuvs nightly pkg (Updates to faiss-gpu-cuvs nightly pkg #4032)
• pin the dependecies version for x86_64 ([1/n] pin the dependecies version for x86_64 #4046)
• pin arm64 dependency ([3/n] pin arm64 dependency #4060)
• Pin conda build ([4/n] Pin conda build  #4062)
• Improve naming due to codemod (Improve naming due to codemod #4063)
• Improve naming due to codemod (Improve naming due to codemod #4064)
• Improve naming due to codemod (Improve naming due to codemod #4065)
• separare the github build into two conditions ([5/n] separare the github build into two conditions #4066)
• Improve naming due to codemod (Improve naming due to codemod #4070)
• improve naming due to codemod (improve naming due to codemod #4067)
• improve naming due to codemod (improve naming due to codemod #4071)
• improve naming due to codemod (improve naming due to codemod #4072)
• fix nightily build (fix nightily build #4080)
• Change github action workflows name (Change github action workflows name #4083)
• Resolve Packaging Issues (Resolve Packaging Issues #4044)
• Update init.py (Update __init__.py #4086)
• Exhaustive IVF probing in scalar quantizer tests (Exhaustive IVF probing in scalar quantizer tests #4075)
• Pin Nightlies with testing on PR (Pin Nightlies with testing on PR #4088)
• Update benchmarking library code to work for IdMap index as well (Update benchmarking library code to work for IdMap index as well #4093)
• Update action.yml (Update action.yml #4100)
• Upgrade CUVS to 24.12 (Upgrade CUVS to 24.12 #4021)
• Link cuVS Docs (Link cuVS Docs #4084)
• Set KnnDescriptor.desc_name in the Benchmarking core framework in FAISS like other descriptors (Set KnnDescriptor.desc_name in the Benchmarking core framework in FAISS like other descriptors #4109)
• enable quiet mode for conda install (enable quiet mode for conda install #4112)
• Disable retry build (Disable retry build #4124)
• Add ngpu default argument to knn_ground_truth (Add ngpu default argument to knn_ground_truth #4123)
• Update code comment to reflect the range of IF from [1, k] (Update code comment to reflect the range of IF from [1, k] #4139)
• Reenable auto retry workflow (Reenable auto retry workflow #4140)
• Migration off defaults to conda-forge channel (Start migration off defaults to conda-forge channel #4126)
• Benchmarking Scripts for cuVS Index, more docs updates (Benchmarking Scripts for cuVS Index, more docs updates #4117)

Fixed

• Fix total_rows (Fix total_rows #3942)
• Fix INSTALL.md due to failure of conflict resolving (Fix INSTALL.md due to failure of conflict resolving #3915)
• Back out "Add example of how to build, link, and test an external SWIG module" (Back out "Add example of how to build, link, and test an external SWIG module" #3954)
• Fix shadowed variable in faiss/IndexPQ.cpp (Fix shadowed variable in faiss/IndexPQ.cpp #3959)
• Fix shadowed variable in faiss/IndexIVFAdditiveQuantizer.cpp (Fix shadowed variable in faiss/IndexIVFAdditiveQuantizer.cpp #3958)
• Fix shadowed variable in faiss/impl/HNSW.cpp (Fix shadowed variable in faiss/impl/HNSW.cpp #3961)
• Fix shadowed variable in faiss/impl/simd_result_handlers.h (Fix shadowed variable in faiss/impl/simd_result_handlers.h #3960)
• Fix shadowed variable in faiss/utils/NeuralNet.cpp (Fix shadowed variable in faiss/utils/NeuralNet.cpp #3952)
• Resolve "incorrect-portions-license" errors: add no license lint to top of GPU files with both licenses (add no license lint to top of GPU files with both licenses #3965)
• Resolve "duplicate-license-header": Find and replace duplicate license headers (Resolve "duplicate-license-header": Find and replace duplicate license headers #3967)
• fix some more nvidia licenses that get erased (fix some more nvidia licenses that get erased #3977)
• fix merge_flat_ondisk stress run failures (fix merge_flat_ondisk stress run failures #3999)
• Fix reverse_index_factory formatting of ScalarQuantizers (Fix reverse_index_factory formatting of ScalarQuantizers #4003)
• Fix shadowed variable in faiss/IndexAdditiveQuantizer.cpp (Fix shadowed variable in faiss/IndexAdditiveQuantizer.cpp #4011)
• facebook-unused-include-check in fbcode/faiss (facebook-unused-include-check in fbcode/faiss #4029)
• fix linter (fix linter #4035)
• Some chore fixes (Some chore fixes #4010)
• Fix unused variable compilation error (Fix unused variable compilation error #4041)
• stop dealloc of coarse quantizer when it is deleted (stop dealloc of coarse quantizer when it is deleted #4045)
• Fix SCD Table test flakiness (Fix SCD Table test flakiness #4069)
• Fix IndexIVFFastScan reconstruct_from_offset method (Fix IndexIVFFastScan reconstruct_from_offset method #4095)
• more fast-scan reconstruction (more fast-scan reconstruction #4128)
• Fix nightly cuVS 11.8.0 failure (Fix nightly cuVS 11.8.0 failure #4149)
• Correct capitalization of FAISS to Faiss (Correct capitalization of FAISS to Faiss #4155)
• Fix cuVS 12.4.0 nightly failure (Fix cuVS 12.4.0 nightly failure #4153)
• Fix build-release workflow (Fix build-release workflow #4162)

Deprecated

• Remove unused-variable in dumbo/backup/dumbo/service/tests/ChainReplicatorTests.cpp (Remove unused-variable in dumbo/backup/dumbo/service/tests/ChainReplicatorTests.cpp +3 #4024)
• remove inconsistent oom exception test (remove inconsistent oom exception test #4052)
• Remove unused(and wrong) io macro (Remove unused(and wrong) io macro #4122)