Minimum rows for dataset

[sroy-forr]

Thank you for building this awesome project.
I am testing models using a local dataset and I wanted to ask if there's a minimum number of rows that's expected in the corpus for this to work. My evaluations are leading to all 0's.

[KennethEnevoldsen]

We have fairly small datasets ~200 samples (see e.g. twitterhjerne), but 0 is naturally not possible.

[sroy-forr]

I am using 30 to test but I must have done something wrong with the setup. Does this look okay to you?

# dataset format: Id, topic, passage
self.corpus, self.queries, self.relevant_docs = {}, {}, {}

for idx, split in enumerate(self.metadata.eval_splits):
    dataset = self.dataset[idx]  # HF defaulted a list object (index 0) without train/test
    answers = dataset["passage"]
    self.corpus[split] = {
        str(idx): {"_id": str(idx), "title": "", "text": answer}
        for idx, answer in enumerate(answers)
    }

    questions = dataset["topic"]
    self.queries[split] = {
        str(idx): question for idx, question in enumerate(questions)
    }
    self.relevant_docs[split] = {
        str(idx): {str(idx): 1} for idx in range(len(answers))
    }

[KennethEnevoldsen]

Seems like idx here is over/mispecified.

dataset = self.dataset[idx] # should be splits surprised that it does not break

{
        str(corpus_id): {"title": "", "text": answer}
        for corpus_id, answer in enumerate(answers)   # changed to corpus id
    }

self.queries[split] = {
        str(query_id): question for query_id, question in enumerate(questions)
    }

 self.relevant_docs[split] # this is then a list of positive pairs between query id and corpus id.
 

if you do it on a small sample you can just print it out as show how it looks now (around 5 examples should do)

[sroy-forr]

I am not splitting anything. When I pass split='train' to HF load_dataset it just gives me 1 set like this

[Dataset({
     features: ['id', 'topic', 'passage'],
     num_rows: 30
 })]

https://huggingface.co/docs/datasets/v2.18.0/en/package_reference/loading_methods#datasets.load_dataset.split

Then I store the value at index 0 (idx=0) to self.dataset. I changed all idx to resolve any issues with misspecification but didn't change the 0 outcome. Printing the corpus, queries, and relevant docs also seems to be okay.
Not sure what's going wrong. I will sleep over it.

Thank you for assisting!