4

My team has a ROS2 humble docker image we use for development. It contains our dependencies along with a source installation of Humble targeting Ubuntu arm64 architecture on a Nvidia Jetson device. Building our source code takes a very long time (up to an hour for a clean build). We are currently using the following commands in a fresh terminal inside the docker container for a clean build. source /opt/ros2/humble/install/setup.bash && colcon build --merge-install

If the code was already built then running above still results in a 1min 9s build time which is still quite long for a no-change build. I feel this is unreasonably long as our ROS1 noetic builds using catkin took a small fraction of this time.

I have seen discussions online about ros2 subscribers eating up some build time, but even so this still seems excessive. And, that still doesn't explain the 1m9s build time for a no-change build. So I feel this may be colcon or maybe ament specific rather than just slow c++ build times. It's worth noting the python packages take about 15-20s each as well.

Are there any best practices for reducing build times or recommendations for profiling colcon to identify the bottleneck?

user1433734
  • 173
  • 4
  • Something seems odd for your setup - perhaps Jetson related? Python packages are virtually instantaneous for me since its not compiling anything and running over pre-compiled C++ packages take < 5 seconds each. This on a 8th gen i7/8GB RAM (currently upgrading to something in the last 5 years this week). – Steve Macenski Nov 30 '23 at 21:00
  • Well it's good to know I'm not crazy thinking it should be faster. I imagine your device isn't so much faster than the jetson. It could be a jetson or docker issue though it's unfortunately a bit difficult to extract the build from that without also changing loads of dependencies. – user1433734 Nov 30 '23 at 22:04
  • @user1433734 can you share the packages? I would like to look into it. Someone else told me the same issue which I find interesting but their code is private. – Craig Dec 01 '23 at 12:23
  • Our stuff is mostly private as well, but I'm happy to pull a specific open source package and try building that in the image as a reference if you have a suggestion. – user1433734 Dec 01 '23 at 15:42

1 Answers1

2

It may happen that there are cases where your files are not setup correctly or there are some dependencies issues. In my case I just had to change the Nvidia Graphics driver. Below I have provided some of the things which may work , else you may try playing with the drivers .

  1. Parallel Builds:

    • As mentioned earlier, use the -j option with colcon build to enable parallel builds. This allows multiple packages to be built simultaneously, which can significantly reduce overall build times. Adjust the number of parallel jobs based on the available CPU cores.
    source /opt/ros2/humble/install/setup.bash
colcon build --merge-install --cmake-args -j4

  2. Incremental Builds:

    • Ensure that you are taking advantage of incremental builds by specifying the --packages-up-to option when building. This will only build packages that have changes, saving time during development.
    source /opt/ros2/humble/install/setup.bash
colcon build --merge-install --packages-up-to package_name

  3. Cache Usage:

    • Make sure caching is enabled. This can speed up subsequent builds by reusing previously built artifacts.
    source /opt/ros2/humble/install/setup.bash
colcon build --merge-install --cmake-args -DCMAKE_UNITY_BUILD=ON

  4. Check for Dependency Issues:

    • Verify that your dependencies are correctly specified and installed. Incorrect or missing dependencies can lead to longer build times.

  5. Hardware Acceleration:

    • Ensure that your Nvidia Jetson device is configured correctly for maximum performance. You might need to optimize GPU settings for build operations that can benefit from hardware acceleration.

  6. Build Isolation:

    • Experiment with build isolation settings. While colcon's default behavior is to isolate builds for better dependency management, you can try turning off isolation to see if it improves build times in your specific scenario.
    source /opt/ros2/humble/install/setup.bash
colcon build --merge-install --cmake-args -DCOLCON_ISOLATE_PREFIX=OFF

  7. Profiler:

    • Use profiling tools to identify bottlenecks in the build process. Tools like time, cachegrind, or specific ROS2 profiling tools can help you pinpoint where most of the build time is spent.

  8. ROS2 Configuration:

    • Check your ROS2 configuration files, especially any ROS2-specific settings in ament and colcon. Ensure that they are optimized for your development environment.

By combining these strategies and tailoring them to your specific use case, you should be able to identify and address the factors contributing to the long build times in your ROS2 development environment.

Avisheet Srivastava
  • 21
  • 3