Improve reindexing

[khoroshevskyi]

1. Reindexing is pretty slow.
2. Reindexing doesn't have log, that will track what files were reindexed (in case of failure)

[khoroshevskyi]

In the latest release, we added batch updates in Qdrant, which slightly improved indexing speed. However, indexing 20K files still takes 25 hours, which is quite slow. Help wanted to optimize indexing

[nsheff]

Clarification: Here "reindexing" means

• empty qdrant database
• re-compute BED embedding for each file,
• re-load that into qdrant database

At this point, the time bottleneck is the time it takes to compute the embedding.

It's probably the tokenization.

[khoroshevskyi]

https://github.com/databio/geniml/blob/94a5f743db63ecef6c82a89da89c5beb7fc0b50d/geniml/region2vec/main.py#L341-L355

[ClaudeHu]

https://github.com/databio/geniml/blob/94a5f743db63ecef6c82a89da89c5beb7fc0b50d/geniml/region2vec/main.py#L341-L355

That for loop is inefficient. Padding the 2D list of token ids and directly converting it to torch.tensor as input of Region2Vec.projection (which is a torch.nn.Embedding) can reduce the total time cost by around 50%.