Auto batching

[thijndehaas]

I know auto batching is still in beta but I have some feedback/questions about it.

I was wondering when auto batching will start indexing. It seems like as long as I'm actively writing data to the API the queued data will start processing very slow. When i'm not adding new data, the remaining queue will start indexing fast.

There are already a lot of processing tasks but the server has plenty of resources left to do more:

F.Y.I. This is currently not running in docker anymore but I still use the same directory.

For now this won't be a problem but I'm wondering what will happen when we deploy this with for example 10.000.000 products instead of 600.000.

[Kerollmops]

Hey @thijndehaas,

I would like to know the parameters you use with your Meilisearch, as you specified --enable-auto-batching maybe you set more settings?

Thank you and have a good day!

[thijndehaas]

Hello @Kerollmops ,

Apologies for the late reply. However this still seems to be the case in the latest version where te auto-batching is enabled by default. Somehow tasks even seem to stay in queue even if the queue is inactive for a while. The hanging tasks get processed when I add new tasks to the queue.

This is the startup command (I added the \ symbols and new lines for visibility here:

/home/meilisearch/meilisearch \
--http-addr=127.0.0.1:7700 \
--env=production \
--db-path=/home/meilisearch/data.ms \
--master-key=HIDDEN \
--debounce-duration-sec=3 \
--max-indexing-threads=20

[thijndehaas]

This is currently an amount of 3419 tasks with JSON data. I have made a back-up of the data.ms directory and I'm trying to reproduce this in a clean set-up to find out when this starts happening.

[thijndehaas]

Currently I have indexed 100k/600k products. Is it normal that the tasks always way for 20 enqueued tasks? Before processing the next one?
The first result is a minute after starting my script:

18:39
[Statuses] Enqueued: 20
[Queue tasks] Add: 8 | Delete: 12
[Queue amount] Add: 16.575 | Delete: 365.075

18:40
[Statuses] Enqueued: 20
[Queue tasks] Add: 9 | Delete: 11
[Queue amount] Add: 10.230 | Delete: 329.452

18:41
[Statuses] Enqueued: 20
[Queue tasks] Add: 9 | Delete: 11
[Queue amount] Add: 8.155 | Delete: 400.132

18:43
[Statuses] Enqueued: 20
[Queue tasks] Add: 10 | Delete: 10
[Queue amount] Add: 18.567 | Delete: 43.664

18:44
[Statuses] Enqueued: 20
[Queue tasks] Add: 10 | Delete: 10
[Queue amount] Add: 29.364 | Delete: 24.398

18:48
[Statuses] Enqueued: 20
[Queue tasks] Add: 10 | Delete: 10
[Queue amount] Add: 6.341 | Delete: 136.558

[thijndehaas]

My sync script is now completed and it just seems to be waiting for the 21st task before processing the last task:

[Statuses] Enqueued: 20
[Queue tasks] Add: 10 | Delete: 10
[Queue amount] Add: 14.601 | Delete: 37.430

[thijndehaas]

I have found something I missed when receiving the tasks in PHP:

$taskQuery = new TasksQuery(); // Added now
$taskQuery->setLimit(10000); // Added now
$tasks = $index->getTasks($taskQuery)->getResults(); // Modified
echo count($tasks);

New response (with all tasks statuses):

[Statuses] Enqueued: 570 | Succeeded: 573
[Queue tasks] Add: 281 | Delete: 289
[Queued task documents] Add: 217.787 | Delete: 2.177.413
[Total tasks] Add: 545 | Delete: 576
[Total task documents] Add: 572.799 | Delete: 6.339.452

I had changed the code to receive tasks after updating to the latest version. So this explains why I had just 20 tasks staying enqueued but not why they are not completed faster with all CPU/RAM/DISK stlil available.

[irevoire]

So if I understand correctly, meilisearch is not really stuck but simply super slow?

[thijndehaas]

Since my problem with the 20 tasks might be solved. Processing jobs still seems slow and (I guess not in bulk mode since only one tasks is processing at a time):
[19:21] Queued: 570
[20:04] Queued: 333

Can the processing prevent bulk mode because I add a lot of delete/addition tasks after eachother?

[irevoire]

Yes, the auto-batching only works with documents addition of the same type on the same index currently 😖

You should see a significant performance improvement when deleting documents in the next update, though!

[thijndehaas]

I have resolved the problem with my code preventing bulk indexing.

I also submitted my bulk queue for removing documents when submitting my bulk queue for adding documents. Because most of our products are actually not on stock we remove more products than we add when doing a full sync.

Here is a very simplified example class:

class Example
{
    private $additionsQueue;
    private $removeQueue;

    public function add($item)
    {
        $this->additionsQueue = $item;
        $this->syncItems();
    }

    public function remove($itemId)
    {
        $this->removeQueue = $itemId;
        $this->syncItems();
    }

    public function syncItems()
    {
        if (count($this->additionsQueue) > 5_000) { // Now changed to 10_000
            $this->index->addDocuments($this->additionsQueue);
            $this->additionsQueue = [];

            $this->index->removeDocuments($this->removeQueue); // The problem
            $this->removeQueue = []; // The problem
        }

        if (count($this->removeQueue) > 50_000) { // Now changed to 100_000
            $this->index->removeDocuments($this->removeQueue);
            $this->removeQueue = [];
        }
    }
}

A task queue example:

[batch 1] documentAdditionOrUpdate
[batch 2] documentDeletion 
[batch 3] documentAdditionOrUpdate // because the task from batch 2 was added it is not in batch 1
[batch 4] documentDeletion  // because the task from batch 3 was added it is not in batch 2
[batch 5] documentAdditionOrUpdate
[batch 6] documentDeletion
[batch 7] documentAdditionOrUpdate
[batch 8] documentDeletion
[batch 8] documentDeletion
[batch 8] documentDeletion
[batch 8] documentDeletion
[batch 8] documentDeletion
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 9] documentAdditionOrUpdate
[batch 10] documentDeletion
[batch 11] documentAdditionOrUpdate // because the task from batch 10 was added it is not in batch 9

[thijndehaas]

Isn't it possible to combine document addition and delete tasks in the following way?

3 tasks that could be in a single batch:

existing documents: [1, 5, 7]

[task_id] task action below

// addition queue: [], delete queue: []

[1] add document: 15
[1] add document: 23
[1] add document: 18

// addition queue: [15, 23, 18], delete queue: []

[2] delete document: 7
[2] delete document: 15 // extra: cancel addition of document
[2] delete document: 23 // extra: cancel addition of document

// addition queue: [18], delete queue: [7, 15, 23]

[3] add document: 15 // extra: cancel delete document

// addition queue: [18, 15], delete queue: [7, 23]

final batch action:

addition: [15, 18]
delete: [7, 23]

This would also prevent document 23 from ever being indexed when everything is handled in a single batch because it was never needed.

[irevoire]

Yes, it's possible, but it's not trivial; we would need to add a lot of context to the add_documents routine.
Currently, we only focused on the batching of the add/update documents routes because that's what most people were doing.

I don't think your use case is common enough to prioritize it right now. We're preparing ourselves for v1.0, and there are a lot of little things to do before working again on this kind of improvement, but I'll talk to the team about it Monday 👍

Thanks for thinking about it; you rocks 🪨

Also, if you could share your workflow/use-case with us and why you need to do it this way, maybe we could find a workaround or understand better why we should prioritize it.

[thijndehaas]

We have a lot of duplicate products that we merge for the MeiliSearch index
For example if product A, B, C and D are duplicate we decide the primary product, this has not to be the first product so this can be product C. So we index product C and add a remove action for A, B and D just for the cases where product C is new or was not the primary product.

I have resolved this now by creating even bigger internal queues (500.000 for remove and 50.000 for add). When this limit is reached I split them by (100.000 for remove and 10.000 for add). So I get 5 tasks for 100.000 remove documents that can be parsed together with the auto bulk system.

[thijndehaas]

Just as status update, I have updated to Meilisearch v0.29.0rc2 and I'm running my script with 3x more instances and still can't create a large tasks queue because they are processed really fast. It seems like something changed in a good way 😁

[gmourier]

Thanks, @thijndehaas

This is excellent news! Let us know if you run into any problems at any time.

It's possible that, even when you've tried auto-batching in the past, you've gotten confused about the various auto-batching options leading you to a not optimal auto-batching behavior. We found that they were difficult to explain and use.

We removed them to keep auto-batching super basic; if extra options are needed in the future, we will catch them with user feedback asking for them.

[thijndehaas]

I removed the debounce duration:
--debounce-duration-sec=3

[gmourier]

Thanks, @thijndehaas.

We have also added performance optimizations that should help a lot for a heavy throughput document deletion/update.

If a document to be updated is the exact same as the one found in the DB, there is no indexing to speak of. Thus speeding the whole indexing process. meilisearch/meilisearch#2600

If documents are given to be deleted, they are internally marked as soft-deleted by the system (they are discarded from the search while being marked as such and will be physically hard deleted later) meilisearch/meilisearch#2583

There are also some performance gains due to other noticeable changes (such as the memory allocator replaced by mimalloc or linking time opts)