[WORKFLOW SDK BUG] Workflow Execution Scheduling Timeout

[relusion]

Expected Behavior

Workflows are expected to be scheduled and executed promptly when there is sufficient compute or queue capacity. They should retry automatically in the case of recoverable errors, and eventually complete as long as resources are available.

Actual Behavior

Workflows are being scheduled but are not executed. The Dapr sidecar continuously generates the following error messages:

"scheduling workflow execution with durabletask engine" app_id=consumer-app instance=daec2dca0950 scope=dapr.wfengine.backend.actors type=log ver=1.13.6" "Workflow actor '5db209e4-f5b9-405e-b5f2-25fc1a340bae': execution failed with a recoverable error and will be retried later: 'timed-out trying to schedule a workflow execution - this can happen if there are too many in-flight workflows or if the workflow engine isn't running: context deadline exceeded'" app_id=consumer-app instance=daec2dca0950 scope=dapr.wfengine.backend.actors type=log ver=1.13.6"

Despite allowing the system to wait and retry, the scheduled workflows are never executed, and this issue persists indefinitely. You may need to run the producer application multiple times before the issue surfaces, as sometimes the workflows execute without errors.

Reproduction Steps:

1. Clone the repository: https://github.com/relusion/dapr_pub_sub_workflow_case
2. Start the environment: Run docker-compose up --build to start RabbitMQ, Redis, and the Consumer application.
3. Generate messages:
   • Navigate to the producer directory.
   • Run python producer.py, which generates 100 messages and sends them to RabbitMQ.
4. Observe:
   • The Consumer application should schedule workflows for the incoming messages, but they will not be executed.
   • Check the Dapr logs for repeated errors similar to the one listed above.