fleet_state not getting received after some time

[cwrx777]

Hi,

I'm checking with the community if they have experienced not being able to echo /fleet_states after some time. /door_states and my custom topic that's published by the fleet adapter periodically are still able to echo.
OS: Ubuntu 22.04
rmf branch: 22.09

To troubleshoot this issue, i added some debug logs in FleetUpdateHandle.cpp in the following places as follows:
method: publish_fleet_state_topic

method: update_fleet_state

When /fleet_states is not received in ros2 topic echo, I noticed that the log message in publish_fleet_state_topic and in update_fleet_state method were not printed. Does it seem that the wall timer here and here failed?

However, a custom topic that I published from within this code block can be echoed.

[cwrx777]

Additional information:
I run the following using iron_20231229 release.
When /fleet_states are not published, the purple icon in rviz will not be shown. See the video below from 0:18 to 0:48.
The terminal beside rviz shows ros2 topic hz /fleet_states and ros2 topic echo /fleet_states | grep nanosec.

missing_fleet_states_2024-05-28_12-42-33.mp4

[mxgrey]

I don't think the wall timer is at fault. Based on the video I suspect something is blocking up the main event loop for longer than it should.

Are you doing anything in particular around the same time that this is happening, for example submitting new task requests? In iron and older releases, task planning gets done inside the main event loop, so if there are many tasks queued up then submitting a new task can clog up the main event loop for an inordinate amount of time. This is fixed in the latest main branch.

Other than that, the only thing I can think of that tends to block up a thread is traffic planning and negotiation, but I wouldn't suspect those of being problems because:

1. Those have been decoupled from the main event loop for a long time now, well before the humble release.
2. There is some risk that planning or negotiation can occupy the thread that gets used by the main event loop, but the probability is low unless you are using a computer with a very small number of threads.
3. This is a very small nav graph so motion planning should be almost instantaneous.
4. There are only two robots so negotiation should be almost instantaneous.

Is there anything unusual about the computer you are running this on? Like having very few CPU cores?

my custom topic that's published by the fleet adapter periodically are still able to echo

Can you link us to where this topic is being published and how it gets triggered? When I try to go to the provided link I get a 404 error.

If the publishing is triggered by a wall timer that was with the fleet adapter node but is still being published while the fleet states are not then that would disprove my hypothesis. All the wall timers created by the adapter node will only be triggered within the thread of the main event loop.

[mxgrey]

My custom topic is published from within this link

Based on the code snippet it looks like you're using rclpy and an event loop within Python to publish the custom message. In that case it will be able to publish even if the C++ main event loop in rmf_fleet_adapter is blocked. So I think there is indeed something blocking the main event loop.

how and when follow_new_path in the python fleet adapter get called?

This function will be called in the following situations:

• A new plan was generated for the robot because
  • the robot's task needs it to go to a new place, or
  • a negotiation was finished
• The robot is done waiting for something to happen such as
  • a door to open
  • a lift to arrive
  • another robot to pass

The follow_new_path function does block the C++ main event loop which is why all the examples we provide will quickly offload the execution of following the path into a separate Python thread. If you're using rmf_demos_fleet_adapter without significant modifications then this should be okay.

the fleet adapter Executor is using MultiThreadedExecutor?

We can't use a multi-threaded executor in the C++ implementation without adding an enormous number of mutexes throughout the codebase and using them very meticulously. This would give us a very high risk of data races (which can lead to undefined behavior and program crashes) and generally make the implementation even harder to work with than it already is.

This is a big part of why we're migrating the next generation effort to Bevy which will be able to do high performance multi-threading without having these risks.

video: from 00:18

In this example I can confidently say that the system got jammed up by the negotiation. This particular type of negotiation, where two robots are suddenly head-to-head and one needs to back off so that the other can get out of the way, is a particularly challenging situation for the current system to work out. I'm not very surprised that all the threads would be eaten up by this, especially when it's limited to 8 threads. (Another benefit of Bevy is that we'll be able to limit how many threads are used by the planners and negotiations, so the main event loop doesn't get pushed down by them.)

What I find most strange about the video is that tinyRobot1 seemed to teleport at t=0:10 to suddenly be blocking tinyRobot2. The negotiation system works best when it can anticipate conflicts in advance and prevent them from ever occurring. I'm not sure how that teleportation was possible in the first place.

[cwrx777]

What I find most strange about the video is that tinyRobot1 seemed to teleport at t=0:10 to suddenly be blocking tinyRobot2. The negotiation system works best when it can anticipate conflicts in advance and prevent them from ever occurring. I'm not sure how that teleportation was possible in the first place

The 'teleport'at 00:10 was because I've shortened the video to the time of interest. I should've mentioned this earlier.

FYI, The demo captured in the video was done using the rmf_demos_fleet_adapter in the iron 20231329 release.

[mxgrey]

In this case the time period that was cut out might have been very informative. It looks like tinyRobot2 was supposed to have moved out of the charger but somehow remained there until tinyRobot1 was blocking it. I can't tell if the simulated robot was unresponsive or if the traffic negotiation was misbehaving to get it stuck like that.

But I think the main issue here is that the robots are being commanded to go to the same waypoints at the same time, and those waypoints don't have good traffic flow to let the robots get around each other.

We intend to fix this with the reservation system which is in a testing phase right now, but in the meantime this kind of issue can be mitigated with two techniques:

1. Have uni-directional entry and exit lanes for the target waypoints that allow the robots to pass each other.
2. In addition to (1), strategically use mutex groups on the lanes and waypoints.

In this case (1) should be enough, but adding (2) will help keep the flow extra safe.

[cwrx777]

Hi @mxgrey

Here is the more complete video:

missing_fleet_states_.8vcpu.2024-05-29_10-17-52_v1.1.mp4

The main issue that we are facing is the frequent pauses, long delay with the robot movement, the follow_new_path comand doesn't get called soon after the earlier one is completed. i have created another discussion (#470) to understand how the follow_new_path gets called.

[mxgrey]

These cases tend to be situational and can typically be managed with with explicit traffic flows, i.e. separate uni-directional enter and exit lanes.

The worst cases usually arise from multiple robots trying to approach the same destination at the same time. This is something we're actively working to fix with the reservation system, and we're attempting to get a basic prototype into the current generation of RMF, while a more sophisticated implementation will be available in the next generation.