Assertion Error while index.search

[ravijoe]

Hi , so I have trained a faiss index and when I search for similar query I get an assertion error.

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(data)
# print(vectorizer.get_feature_names())
# ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
print(X.shape)
svdT = TruncatedSVD(n_components=50)
svdTFit = svdT.fit_transform(X)

svdTFit shape=100,50

-----------------> now creating index and using search
import numpy as np
import torch
import os
import pandas as pd
import faiss

dimension = 50   # dimensions of each vector                     
n = 100    # number of vectors              
np.random.seed(1)            

nlist = 5  # number of clusters
quantiser = faiss.IndexFlatL2(dimension) 
index = faiss.IndexIVFFlat(quantiser, dimension, nlist,   faiss.METRIC_L2)
print(index.is_trained)   # False
svdTFit=np.array(svdTFit,dtype=np.float32)
index.train(svdTFit)  # train on the database vectors
print(index.ntotal)   # 0
index.add(svdTFit)   # add the vectors and update the index
print(index.is_trained)  # Trueprint(index.ntotal)   # 200



def search(query):
   t=time.time()
   query_vector = vectorizer.transform([query])
   k = 2
   top_k = index.search(query_vector, k)
   print('totaltime: {}'.format(time.time()-t))
   return [data[_id] for _id in top_k[1].tolist()[0]]
query=str(input())
results=search(query)
print('results :')
for result in results:
   print('\t',result) 

and i get error :- """---------------------------------------------------------------------------

AssertionError Traceback (most recent call last)

in ()
34 return [data[_id] for _id in top_k[1].tolist()[0]]
35 query=str(input())
---> 36 results=search(query)
37 print('results :')
38 for result in results:

1 frames

/usr/local/lib/python3.6/dist-packages/faiss/init.py in replacement_search(self, x, k, D, I)
130 def replacement_search(self, x, k, D=None, I=None):
131 n, d = x.shape
--> 132 assert d == self.d
133
134 if D is None:

"""

[beauby]

Could you print query_vector.shape from within your search() method? It looks like it's not the right shape.

[ravijoe]

Oh ok. I'll do the experiment from scratch today n update you

…

On Tue, Feb 16, 2021, 1:51 AM Lucas Hosseini ***@***.***> wrote: Could you print query_vector.shape from within your search() method? It looks like it's not the right shape. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAIDMA2T5T4NJV7NNYXGC3S7F64PANCNFSM4XUSPQ3Q> .

[ravijoe]

Could you print query_vector.shape from within your search() method? It looks like it's not the right shape.

Ok so i think the shapes were not in the right format now its working. Can you tell me what is the difference between the method I used in the post and this ->

# index = faiss.IndexIDMap(faiss.IndexFlatIP(471))
# # index.add_with_ids(svdTFit, np.array(range(0, df.shape[0])))
# index.add_with_ids(svdTFit, np.array(range(0, len(data))))
# faiss.write_index(index, 'emp')

# index = faiss.read_index('emp')

in above method we don't need to define clusters right? i think this way is better but above mentioned code is not working