Implement Dataset Transformation Scripts for ColPali and Msmarco (training+corpus)

[Samantha-Zhan]

The Design

A unified training data format

{
	"query_id": str
	"query_text": str
	"query_image": PIL.Image
	"positive_document_ids": List[str]
	"negative_document_ids": List[str]
	"source": 'msmarco'
        "answer": str
}

A unified corpus data format

{
	"docid": str,
	"image": PIL.Image,
	"text": str,
	"source": str,
}

Execution

Simply navigate to /tevatron/scripts/dataset_transform_scripts/, and call python ./<script_name>

Results

We have successfully created 4 new Huggingface datasets by executing the scripts, transforming original datasets according to the new schema

Msmarco

• Tevatron/msmarco-passage-corpus-new
• Tevatron/msmarco-passage-new

ColPali

• Tevatron/colpali
• Tevatron/colpali-corpus

[MXueguang]

let's follow the snake style for variable names and function names.
each word is separated by an underscore character

e.g. loadDatasets -> load_datasets