Skip to content

ROS

JUnit Output support in Autotest Framework

DALL·E 2023-11-23 00.05.54 - A humorous cartoon image of a drone wearing a graduation cap and glasses, sitting at a computer and looking at a screen displaying 'JUnit Test Results|500x500

Hello ArduPilot Enthusiasts,

We've just merged a new feature for our autotest framework: JUnit output! Here's a brief summary of what this entails. Seasoned developers will probably know all this, but it should be new knowledge for non-developers.

The autotest framework.

This is the software suite we use each time somebody makes modifications to the ArduPilot codebase. It has the huge task of running tests automatically. ArduPilot’s AutoTest suite allows for the creation of repeatable tests which help prevent regressions in ArduPilot’s behavior. It is based on ArduPilot’s SITL architecture - a fully-software-based solution.

What kind of tests ?

We do a lot of tests! Some check code syntax and common writing errors. But we also do unit tests and functional tests.

Unit tests focus on the smallest parts of code—individual functions or methods. They are designed to ensure that each unit of the software performs as designed. These cover most of our math functions.

Functional tests are concerned with the overall operation of the system. They test larger sections of the application, often entire features. They are performed in an environment that closely mirrors production, so with SITL in our case ! There are close to 1000 of these, trying to test most of ArduPilot's features.

|500x500

How much do we test ?

We've got code coverage support. It's a metric used to gauge the effectiveness of tests by determining which areas of the codebase have been tested and which have not.

On 22/11/2023, here are our coverage statistics (https://coveralls.io/builds/64044563):

Global : 57.646% Lines

  • AntennaTracker : 68.03%
  • ArduCopter : 66.01%
  • ArduPlane: 71.2%
  • ArduSub: 46.38%
  • Rover: 61.08
  • AP_Periph: 54.55%
  • libraries: 56.33%

We can compare with one years ago (https://coveralls.io/builds/54387841):

Global : 56.114% Lines

  • AntennaTracker :67.05%
  • ArduCopter : 64.63%
  • ArduPlane: 67.16%
  • ArduSub: 33.96%
  • Rover: 61.97
  • AP_Periph: 58.59%
  • libraries: 55.05%

We can see that we are slowly raising our coverage rates. Ideally, we should aim for 100% coverage, but this is challenging, especially in a FOSS project like ArduPilot. Writing tests is quite lengthy and sometimes difficult, which can delay contributions to the project.

How to use Autotest ?

Well, we've got some documentation about this, so let’s read the wiki! It isn’t hard: https://ardupilot.org/dev/docs/the-ardupilot-autotest-framework.html !

Don’t forget, if you implement a feature and have a test for it, or better yet, you write a test for it, you can run tests individually: running-a-specific-sub-test

That saves a lot of time by testing only what you need.

Plus, our shell completion system makes test execution a breeze – get ready to make the most of your Tab key!

Junit ?

The JUnit format is a widely-used standard for reporting the results of software tests. It's not a formal standard, but rather a convention that has become a de facto standard due to the widespread adoption of JUnit and similar testing frameworks in the Java ecosystem. Test results in JUnit are typically outputted in an XML format. This XML format is structured and easy to parse, making it compatible with a variety of tools and systems for further processing, like continuous integration servers.

Previously, running the test would provide a small summary at the end in your terminal with information like this:

>>>> PASSED STEP: test.CopterTests1a at Thu Nov 23 00:01:26 2023

With the JUnit output, you get an XML file in addition that looks like this:

<?xml version="1.0" encoding="utf-8"?>

<testsuites>

<testsuite name="Autotest ArduCopter test.CopterTests1a" timestamp="2023-11-16T14:43:23" tests="27" errors="0" failures="0" skipped="0" time="233.136">

<testcase name="PIDTuning" classname="ArduCopter" time="5.101137638092041"/>

<testcase name="ArmFeatures" classname="ArduCopter" time="20.81020736694336"/>

<testcase name="SetHome" classname="ArduCopter" time="11.95091986656189"/>

<testcase name="ConfigErrorLoop" classname="ArduCopter" time="2.7126312255859375"/>

<testcase name="CPUFailsafe" classname="ArduCopter" time="5.354845762252808"/>

<testcase name="Parameters" classname="ArduCopter" time="9.803567171096802"/>

<testcase name="LoggerDocumentation" classname="ArduCopter" time="3.151670217514038"/>

<testcase name="Logging" classname="ArduCopter" time="13.548582315444946"/>

<testcase name="GetCapabilities" classname="ArduCopter" time="0.07680463790893555"/>

<testcase name="InitialMode" classname="ArduCopter" time="2.385150909423828"/>

<testcase name="NavDelayTakeoffAbsTime" classname="ArduCopter" time="7.005534887313843"/>

<testcase name="NavDelayAbsTime" classname="ArduCopter" time="5.902580738067627"/>

<testcase name="NavDelay" classname="ArduCopter" time="5.835424184799194"/>

<testcase name="GuidedSubModeChange" classname="ArduCopter" time="7.143741607666016"/>

<testcase name="MAV_CMD_CONDITION_YAW" classname="ArduCopter" time="14.092246770858765"/>

<testcase name="LoiterToAlt" classname="ArduCopter" time="7.683399677276611"/>

<testcase name="PayloadPlaceMission" classname="ArduCopter" time="9.601360082626343"/>

<testcase name="PrecisionLoiterCompanion" classname="ArduCopter" time="7.640118598937988"/>

<testcase name="Landing" classname="ArduCopter" time="6.856537818908691"/>

<testcase name="PrecisionLanding" classname="ArduCopter" time="13.348423719406128"/>

<testcase name="SetModesViaModeSwitch" classname="ArduCopter" time="0.9609272480010986"/>

<testcase name="SetModesViaAuxSwitch" classname="ArduCopter" time="0.33498406410217285"/>

<testcase name="AuxSwitchOptions" classname="ArduCopter" time="0.5513660907745361"/>

<testcase name="AuxFunctionsInMission" classname="ArduCopter" time="3.2609589099884033"/>

<testcase name="AutoTune" classname="ArduCopter" time="23.587228536605835"/>

<testcase name="AutoTuneYawD" classname="ArduCopter" time="40.96126747131348"/>

<testcase name="NoRCOnBootPreArmFailure" classname="ArduCopter" time="3.4743480682373047"/>

<properties>

<property name="Firmware Version Major" value="4"/>

<property name="Firmware Version Minor" value="5"/>

<property name="Firmware Version Rev" value="0"/>

<property name="Firmware hash" value="d3d0f91c"/>

<property name="Git hash" value="d3d0f91c92a8d76a42d6fd7aa5ecb398bb464d6a"/>

<property name="Mavproxy Version Major" value="1"/>

<property name="Mavproxy Version Minor" value="8"/>

<property name="Mavproxy Version Rev" value="67"/>

</properties>

</testsuite>

</testsuites>

Another example with an output converted to HTML (smaller to fit in a picture): image|512x500

We can now integrate this into many CI systems to make useful statistics on our tests and have a better understanding of what is failing and when. This is an important feature for professionals using ArduPilot in their fleets.

Ha, yes ! To output JUnit xml, just add --junit on your autotest invocation, like this : `Tools/autotest/autotest.py build.Copter test.CopterTests2a --junit

In Conclusion

The introduction of JUnit output marks a significant step forward in enhancing our testing capabilities. By continuously refining our testing practices, we're committed to maintaining the high reliability and performance standards that ArduPilot users expect.

Stay tuned for more updates, and as always, happy testing (and flying) !

Plotjuggler Dataflash plotting!

https://youtu.be/ocyd0ikqQfo

Hello guys,

You may be used to MAVExplorer.py or MissionPlanner for Dataflash analysis. I offer you another alternative : Plotjuggler

This is a tool that people using ROS may know already, but it is extensible with plugins support. So I made an ArduPilot Dataflash plugin : https://github.com/khancyr/plotjuggler-apbin-plugins

What are the advantages of Plotjuggler : - it is fast - it supports livegraphing - it got 2D graph support - it got a LUA script math engine - it got totally unprofessional Splash screen

There is one important issue with the plugin currently. Plotjuggler is expecting lower case file extension for logs, when ArduPilot is using capital BIN type file. So you need to rename your logs to lower case bin.

Have fun with it !

Speeding up compilation time

compiling|413x360 https://xkcd.com/303/

Hello friends,

Some time ago, I left my job for new adventures. Doing so, I lost my company recent laptop and then my development workflow get really downgraded. I am now using an old 2011 laptop with i5-2410M (2 cores at 2.3Ghz) and 4Go RAM DDR3. That is still decent but when you taste the speed you cannot get back ...

Well, my issue is the compile time for reviewing PR on ArduPilot.

Standard setup

With the default installation a full build with waf takes 7m14s.

Yep, you should use just waf or waf copter for example instead of waf -j4. The -j stand for --jobs and is the number of compilation jobs you want to use. The more you use the more CPU cores and computation power for compilation you will use. Generally, you scale the number of jobs with the number of threads your computer support. On my laptop, I got a 2 cores CPU with 4 threads. That means that I could do 4 compilation jobs in parallel ! On contrary of make that need the number of jobs explicitly passed, waf is already taking care of maximizing the number of jobs on your machine.

Fortunately, like make and other builds system, waf is smart enough to not do full rebuild each time we made some change. But this generally won't work when we switch the branch on git or, obviously, do a waf clean.

Best standard setup

If you followed our installation instructions correctly, you should have seen that when you are using waf configure the output looks like :

Checking for 'g++' (C++ compiler) : /usr/lib/ccache/g++
Checking for 'gcc' (C compiler) : /usr/lib/ccache/gcc

instead of

Checking for 'g++' (C++ compiler) : /usr/bin/g++
Checking for 'gcc' (C compiler) : /usr/bin/gcc

What does it mean ? On the second case, waf is detecting GCC and G++ as the C and C++ compiler, that is the intended setup. On the first and rightful case, waf is detecting ccache as the compiler. Ccache is a compiler cache. It will put in cache previous compilation file to reuse them !

You can use ccache -s to get a summary of you cache usage. In my case :

cache directory /home/khancyr/.ccache
primary config /home/khancyr/.ccache/ccache.conf
secondary config  (readonly) /etc/ccache.conf
stats updated Wed Feb 19 13:31:20 2020
stats zeroed Mon Nov 11 19:20:57 2019
cache hit (direct) 3513
cache hit (preprocessed) 117
cache miss 12804
cache hit rate 22.09 %
called for link 483
called for preprocessing 78
compile failed 33
cleanups performed 208
files in cache 117
cache size 1.4 MB
max cache size 5.0 GB

It reuses 20% of the cache instead of compiling, and that is pretty interesting to speed up your builds !

After a small change on ArduPilot file, using waf but this time with ccache, I get a build time of 17.9s. Well, mostly everything is in cache, so I don't have to recompile everything !

Sadly, that won't work in all case, and plenty of time in need the full and long build. Then, how to speed up compilation ?

Using another computer to speed up compilation

I got a gaming desktop computer with a 4 core i5 and 16Go DDR3 RAM. That is a also an old computer but that is a beast comparing to the laptop ! As I got Windows, games and most of my important file (picture, papers, etc.) on it, I don't want to risk myself to dual boot it to have a Linux on it. The simpler way was to instead a virtual machine on it. I used VirtualBox and setup a Ubuntu VM.

On Linux, it exists a utility called distcc that allow to distribute the compilation tasks across multiple computer ! That what I am going to use.

Setup

On Ubuntu the installation is simple : sudo apt install distcc gcc g++ ccache Obviously, you need a compiler to make it works, and I also install ccache as it will serve. You can use systemctl enable distccd to auto start it on your machine.

Now you get distcc on your VM waiting for compilation order. You need then to show waf how to use it. I have create a new file called distcc_config with content :

export CCACHE_PREFIX="distcc"
export CC="ccache gcc"
export CXX="ccache g++"
export DISTCC_HOSTS='localhost/4 10.42.0.79/5,lzo'
export DISTCC_JOBS=$(distcc -j)
echo "Building with $DISTCC_JOBS parallel jobs on following servers:"
for server in `distcc --show-hosts`; do
server=$(echo $server | sed 's/:.*//')
echo -e "\t$server"
done

Here is what it does :

  • CCACHE_PREFIX allow us to use distcc on our computer in combination with ccache. I would be a same to not have it.
  • export CC and export CXX explicitly set the compilers for distcc.
  • DISTCC_HOSTS need, unfortunately to be set manually. It said to distcc what computer use and the number of jobs they can handle. In my case, localhost/4 is for my laptop : 4 jobs. 10.42.0.79/8,lzo for my desktop computer : 8 jobs and lzo to compress file to send.

Now you can invoke waf -j $(distcc -j) to ask waf to compile with distcc max number of jobs, in my case 12 jobs. distcc2|634x500

The result is a full compilation in 4m10s with the drawback to use a lot my network, but as I am on a gigabyte Ethernet network with nobody watching 4k video, that isn't an issue for me !

My solution was to use VirtualBox, but if you are more used to Docker, you could just use a lightly machine with distcc only for the same purpose.

Edit: Funny things, I give another try on WSL and success make it work. It was even better that with VirtualBox as it achieved compilation in 3m36.658s

Limits

Limits to distcc usage : - you need the same version of the compiler on each computer. - it will use a lot your network to transfers file to compile and compilation results. - using your dusted RPIs won't bring much help against using a decent CPU.

Conclusion

I hope you learn something on how to speed up compilation on ArduPilot, unless you really need it, I won't recommend you to use distcc as the setup and maintenance can be tricky. The default ArduPilot environment with Waf and Ccache should be enough in most case to get the maximum performance out of your machine.

You can still help us on the project to make consecutive compilation faster by helping us to clean the inclusion dependency on ArduPilot. With clean inclusion, waf will be able, on its own, to recompile only the part that need to instead of compiling back everything.

Multi systems patrol simulations

https://www.youtube.com/watch?v=YLDb-ti9bjU

Hello everyone,

I am often asked about what can be done with ArduPilot or with ROS, and my answers are always : you can do just about anything you’d like to do! ArduPilot is not just an autopilot for drones, it is also a well-grown robotic platform ! It is not the same as ROS, in the sense that ROS concentrates more on the high level tasks, whereas ArduPilot focuses on low-level vehicles control ! Both have strengths and drawbacks. And by using both, you will be able to create incredible robots !

During my PhD, I tried to focus my work on multi-robots systems, which are also known as swarms (from an academic perspective, swarms are just a special case of multi-robots systems). ROS by itself can excel for single robot design, but is difficult to use on multi-robots systems and flying robots like multi-rotors. On the other hand, ArduPilot allows to use multi-rotors more simply, and with safe behaviors.

In order to demonstrate the usefulness of multi-robots systems and be able to evaluate the global performance of the system, I set up some identical simulations with ROS and ArduPilot. The goal was to demonstrate the performance of a multi-robot system with various numbers of quadcopters, and rovers on patrolling tasks. I then simulated systems with 1 to 20 quadcopters, and 1 to 20 rovers and system with the same number of quadcopters and rovers. Those were simulated with SITL from ArduPilot, and I will detail at the end the few modifications that I made and that I am pushing into ArduPilot. The patrolling task was done with ROS. Each vehicle got a Mavros instance (that is, a ROS program that makes the bridge between ROS language and MAVLink), a simple python script that handles the patrolling behavior, and a master python script that controls the group. But that was too simple ! So I implemented 3 patrolling behavior, that are quite simple :

  • Random patrol (RANDOM). It is a decentralized algorithm. Each robot will go at a random patrolling point

  • Reactive patrol (HIGHEST). It is a centralized algorithm. When a robot arrived at his destination, it notifies the master script (like a base) and gets the next patrol point.

  • Cyclique patrol (CYCLE). It is a decentralized algorithms. At start, each robot create a patrol cycle.

For the full detail of the simulation models and hypothesis, please refer to my PhD manuscript and/or directly the code !

image|690x193

The results were those graphs for 1 hour of patrolling on this field, the number represent the patrol points : image|402x500

For copter only:

image|499x500

For rover only:

image|507x499

For rover and copter :

image|690x419 image|690x330

All on same graphs : image|494x500 image|523x500 image|524x500

You can see a video here of the patrolling : https://www.youtube.com/watch?v=YLDb-ti9bjU

I won't discuss the results here, but you should be worried by SKYNET ^^ as the group behavior outperform largely the single robot patrolling ! But what it was really interesting, it is that I could make all those simulation on my own laptop : just a i5 + 8go RAM and with simulation accelerated 10 times ! Right ! (If I find the budget to build 20 quads and 20 rovers, I would be able to confirm the simulation with the real platform)

ArduPilot allows to simulate entire fleets of robots with the same code that will be embedded and at a small cost ! That is right, for those kind of behavior, you don't need to use big simulator like Gazebo on big 32 cores servers. Simulate simply with SITL !

I don't say that you should use Gazebo or other well known robotic simulator, but when it comes to simulate only behavior without worrying about collision, SITL with excel !

All these were permitted thanks to OPEN SOURCE and numerous contributions ! PR welcome !

F.A.Q

Where is your code :

Github ardupilot https://github.com/ArduPilot/ardupilot and branch PhD on my fork https://github.com/khancyr/ardupilot/tree/PHD : I am currently getting back those changes into ardupilot master.

Modifications done : - allow to set sysid at the start of SITL (PR in master)

  • disable mavlink routing on a channel. Finally I didn’t use this, this was intend to prevent mavlink spam on a broadcast channel for the monitor. I still believe that we need a way to stop mavlink routing explicitly on some channel, as on swarm configuration, there is to much spam.

  • clear battery failsafe : I didn’t use this, I got another code that I will push to allow to refill the battery and clear the failsafe. On my code, the ROS part is monitoring the battery and decide when to go on base to change the battery.

  • disable MSG_POSITION_TARGET_GLOBAL_INT sending, as there is a bug in MAVROS. It has been reported and I need to make a fix on MAVROS.

  • Disable some stream and unused features. This won’t be push back on master.

  • Prevent switching in guided is EKF is not ready for Rover. (Need a PR)

  • disarm rover at the end of RTL : I am not sure we want this into master, maybe with a parameter.

  • correct battery estimation for rover : PR done.

ROS :

https://github.com/khancyr/patrol

I want to do the same with Gazebo :

If you need only a 3d visualization see https://github.com/khancyr/gazebo_ardu . This can be adapted to any 3D framework you want.

To use the gazebo plugin : https://github.com/khancyr/ardupilot_gazebo . Beware, the simulation of rover isn't simple yet as the ground contacts are a pain and the friction gives a hard time to make a skid steering rover.

Why use ArduPilot over PX4:

PX4 don't support rover and surface vehile like boat and sub. Moreover, the governance of ArduPilot is open whereas PX4 is money dronecode driven

I don't understand anything but the colors were nice :

Thanks you.