0

Rosanswers logo

Hi,

It's hard to believe, but according to my research I am the only one who would like to accelerate simulation of a ROS robot... and it seems to be an almost impossible task.

It looks like when I create a node that generates faster than real time clock topic, it totally messes up the simulation. Not even bags can be replayed faster, than real time.

The reason, as I found is that a huge part of the ROS system relies on WallTime and WallDurations, and not just by using these classes, but by doing waits and sleeps in wall times relevant in the simulation.

I tried to create an accelerated version of ROS:

  • Removed the definitions of WallTime and SteadyTime, replaced them with typedefs to Time
  • Removed the definition of WallDuration and replaced it with typedef to Duration
  • Scaled all timed waits with my desired acceleration factor
  • Instead of a clock topic I share the time from a timer process in shared memory, so all calls to Time::now() are in sync

But now I get stuck in various timing issues related to transformations, more specifically I get errors, like:

Lookup would require extrapolation into the future.  Requested time 1270387817.717999935 but the latest data is at time 1270387817.482000113

(Note that the timestamp is intentionally in the past, as it is an accelerated time, not the real time.)

Now in the stack trace, as I see, this is happening in waitForTransform() call, which is called with Time(0) by the simulation (stdr_simulator). And this is weird, semantically, as I would think (by the name of it) that waitForTransform() is intended to wait, until the transform is available, not throw an exception that it is not (yet).

Could please someone help here?

UPDATE I think the problem is somehow related to BufferCore::getLatestCommonTime(). Somehow the cached times does not match the cached transforms.

Originally posted by shardator on ROS Answers with karma: 43 on 2017-12-20

Post score: 1

Original comments

Comment by gvdhoorn on 2017-12-20:
Time(0) has special semantics in TF, it basically means "latest transform".

But in general if use_sim_time is set and there is a publisher on /clock, I would expect things to 'just work'. I've not done it myself, but what you're doing seems rather convoluted.

Comment by shardator on 2017-12-20:
No, it does not, because timer delays are expected to be real time. So if you set a timer to 10Hz, it will fire 10Hz real frequency, and not accelerated frequency. This is because WallTime is used, and pthread conditional waits are called with the nominal duration.

Comment by shardator on 2017-12-20:
Also, I know about the Time(0) semantics, it is just after a while, inside waitForTransform() is replaced by some common time of the two frames (which we are trying to obtain the transformation between). Now this common time is somehow incorrect, as no actual transformation exists with that.

Comment by gvdhoorn on 2017-12-20:
Using simulated time is done with Gazebo and some other sims as well, so your statements confuse me.

I'm not saying you're wrong, but I would just make sure what you're doing is absolutely necessary, as it would be unfortunate otherwise.

Comment by gvdhoorn on 2017-12-20:
If what you're doing is inside stdr, then that could be true. It may be that the stdr authors didn't (want to) consider FTR / STR cases and just use WallTime everywhere.

Comment by shardator on 2017-12-20:
No, I found that ROS core things fire at wrong times. So wrong, that my robot teleports off the map. It has nothing to do with the simulator, I have already excluded that option via debugging. I have recompiled the whole ROS infrastructure with my changes, as I found many places which depend on the

Comment by shardator on 2017-12-20:
speed of the clock...

Comment by gvdhoorn on 2017-12-20:
Then I would urge you to report that on the proper issue trackers.

If with "ROS core" you actually mean roscpp and the other pkgs in ros_comm, then the tracker would be ros/ros_comm/issues.

Comment by shardator on 2017-12-20:
I will, but I doubt that this will be fixed, as for real time use, this is a non-issue. I need a working workaround, as unfortunately this is not a hobby project... :(

Comment by gvdhoorn on 2017-12-20:
Then I think posting an initial issue on the ros_comm tracker would be a good idea, as either:

  1. this is really wrong and the ppl there should be able to help
  2. it's already supported but you're not seeing it

In either case it could save you a lot of work.

Comment by shardator on 2017-12-20:
Thanks. I've opened it as https://github.com/ros/ros_comm/issues/1272

Comment by gvdhoorn on 2017-12-20:
Thanks for opening that, but your issue is rather vague. You give the impression you have many examples of where WallTime should not be used, why not link to those? You also refer to a 'stacktrace' you see, why not include that?

Comment by shardator on 2017-12-20:
For the record, I updated the ticket, I hope that helps a bit.

2 Answers2

0

Rosanswers logo

It is generally recommended to use the standard ROS Time by default. There are places which interact with hardware etc that WallTime is necessary since hardware timeouts etc cannot reflect accelerated or declerated time.

I know of many systems that are using simulated time both faster and slower than realtime and have used it successfully myself in both cases. If there are modules that are using WallTime incorrectly please ticket them directly.

And if you're having issues with transforms please provide a reproducible example and open a ticket here: https://github.com/ros/geometry2/issues

Reading your post again it sounds like you're publishing /clock from a node other than the simulator? "when I create a node that generates faster than real time clock topic, it totally messes up the simulation." That suggests that you will have two sources of a /clock and that will wreak havoc on your system. If you're using a simulator and want to speed up time you should tell the simulator to accelerate time not just publish it yourself from another node. With two conflicting /clock sources nodes will be getting whiplash jumping between the two timelines. That will cause immediate timeouts and I would not expect any time dependent algorithm to function correctly.

Originally posted by tfoote with karma: 58457 on 2017-12-20

This answer was ACCEPTED on the original site

Post score: 1

Original comments

Comment by shardator on 2017-12-20:
Hi, The simulator does not provide a clock, I checked that. There is no clock publisher in that. Also, there is no way not to use WallTime/SteadyTime as some parts of ROS utilize that. TimerManager in particular. And we definitely have timers.

Comment by tfoote on 2017-12-20:
I don't know about stdr's implementation but both stage and gazebo ros packages implement the /clock API and can run faster than realtime.

Comment by tfoote on 2017-12-20:
Can you please provide an example how roscpp::TimerManager cannot work with accelerated realtime? Looking at the code it does use WallTime/SteadyTIme but only for collecting statistics, but otherwise it's templated to use the time source of the timer object that you passed into it.

Comment by shardator on 2017-12-20:
Pardon me my ignorance, but I fail to see how. If I set a timer in my robot's own code, it will fire with physical, real frequency I've given. All timeouts, every mutex waits happen in real time, unaccelerated in the code of ROS core libs. How is it possible to run an equivalent sim then accelerated

Comment by tfoote on 2017-12-20:
If you use the ROS Time abstraction and enable simulated time. The roscpp Timers will also follow the abstraction. Please try some small test nodes to see how things work

Comment by shardator on 2017-12-20:
I did that. It simply does not work correctly for anything complex. Only for very simple scenarios. Do you happen to know, how timers work in simulated time conditions? I know for fact that SteadyTime is being used, which is not correct for acceleration.

Comment by shardator on 2017-12-20:
We even get incorrect timings in unaccelerated simulated time, which does not actually contain the current real time. It works, but with tiny, but unacceptable glitches, which ruin our sensitive mapping and routing algorithm. When clock contains real time values, it works.

Comment by tfoote on 2017-12-20:
They work like this: https://github.com/ros/ros_comm/blob/lunar-devel/clients/roscpp/include/ros/timer_manager.h#L552-L584

Please provide a reproducible example of your "unacceptable glitches" otherwise I don't think there's much more I can help you with.

Comment by shardator on 2017-12-21:
OK, so if your timers are 20Hz, and the simulation is accelerated to 25x speed, you cannot have more than 2 timers on queue, otherwise they will not fire in time (nearly.) We are using a HP G8 for simulation, so accel factor of 25 is not that big.

Comment by shardator on 2017-12-21:
This is because of the 1ms hard-coded delay. 20Hz on 25x acceleration is 500Hz, which is 2ms per timer. If you have 3 timers, especially if they take some time to execute, the hard-coded 1ms delay will be too long (see line 580 in timer_manager.h)

Comment by tfoote on 2017-12-21:
@shardator I'm happy to help you make this work, but would appreciate if you would change your tone to be a little more constructive. It feels like you have a conclusion that you're trying to find evidence to support instead of trying to understand what's there.

Comment by tfoote on 2017-12-21:
There is a maximum frequency at which it will poll for new timer deadlines of 1000Hz. Multiple callbacks can be called per cycle so the number does not matter. Also this thread does not execute the callbacks so the length of time of the callbacks does not matter since they don't block execution.

Comment by tfoote on 2017-12-21:
With respect to the speed ratio 25x is actually a pretty big deal even on a large machine. If you want to run at 25x speed you can only use a maximum of 4% when running in realtime.

Comment by tfoote on 2017-12-21:
I don't know what your application is but if you're doing any vision processing or planning I suspect that keeping it under 4% will be a challenge. (For reference on a 16 core system you need to be able to run your whole demo on 2/3rds of one core before you speed it up.)

Comment by tfoote on 2017-12-21:
The 1kHz sampling rate was picked as a compromise between having high overhead from polling and performance. If you application really needs higher resolution you can tune that. Your use case seems to be different than the average.

Comment by tfoote on 2017-12-21:
If you increase the frequency of how often you poll for a new clock message, make sure that you increase the rate at which you publish the /clock topic.

Comment by tfoote on 2017-12-21:
Though if you'd like to improve the high speed playback performance I'd suggest having the /clock subscription callback notifiy the condition varialble on which it's waiting and then the maximum resolution will only be limited by your publishing.

Comment by shardator on 2017-12-22:
Sorry, I did not mean to be rude. I'm struggling with this for some time now, and am a bit frustrated. Yes, I think the clock somehow needs to synchronize with the rest of the system, otherwise it won't work correctly.

Comment by shardator on 2017-12-22:
I'm also trying to create a simple example, but I'm stuck with some stupid issue with nanosleep() in my /clock generator. Namely, if I init ROS, nanosleep() will return early without an error. And I can't figure out, why. (So nanosleep(60000ns) sleeps for appx 3-4000ns, and errno=0.)

Tully
  • 24,992
  • 1
  • 17
  • 35
0

Rosanswers logo

When using Gazebo for simulation, there is a simple way to accelerate time: In the right panel, there is a tab World. In this tab go to Physics and search for the value "real time update rate".

Simply increase this value to accelerate the simulation time (e.g. double it to accelerate by factor 2).

Originally posted by Felix Widmaier with karma: 382 on 2017-12-22

This answer was NOT ACCEPTED on the original site

Post score: 2

Original comments

Comment by shardator on 2017-12-22:
Hi! Thanks. We need to do batch simulations (with perturbed parameters) for optimization, so we need like 20x accelerated, scripted simulation. Is it possible to do that with Gazebo? Is not Gazebo using too much resources (we just do 2D simulations, we don't need 3D...)

Comment by Felix Widmaier on 2017-12-22:
You are right, Gazebo takes a lot of resources. I have successfully accelerated by factor 2 or 3 in the past but 20 is probably too much (at least for a normal computer).

Comment by shardator on 2017-12-22:
:( We've explored simple options already. This looks like a difficult task. I think it would be nice to have a 10x accelerated simulation, but I can't even do that consistently.