DevOps for Embedded C/C++

How to spot and fix the red flags in your workflow



Part of the problem is that hardware vendors provide very basic BSPs and tools. Outdated best practices and restrictive project budgets usually leave us limited to use these “vendor supported” set of tools. The other part of the problem is the general lack of awareness of what tools and trick are actually at our disposal. I suppose the Stockholm syndrome of doing things the hard way is also a factor with experienced C++ dev. I can’t help with the masochism issues, but the said lack of awareness is what I am attempting to address in this article.

I’ll be listing a bunch of common pain points with there corresponding solutions and best practices. These solutions in general will strive to improve the “quality of life” by reducing the amount of busy work and by abstracting away the more complex parts of the programming language and tool set.

One caveat is that some of what I am about to list may not be viable to your particular hardware platform, project size or budget. My condolences to you.


Red Flags : Individual workflow

(A) Build, copy to device, run to confirm functionality, repeat

  • Consider using an unit test framework like GoolgeTest to write unit test to confirm the functionality natively on x86. This will speed things up and also force you to write more testable code.
  • If native x86 compiles are not an option and the target processor is fast enough. Consider a remote development workflow over SSH as described here
  • If all fails, at least make sure that the deployment to device is automated as part of your build step.

(B) Printing debug messages to track down issues

  • Use an IDE with GDB support to step through code, print logs and inspect memory.
  • Don’t be afraid to leverage more advance tricks like adding conditional break points and expressions evaluation support. VS Code, Clion and Eclipse IDEs work well with this regard.
  • Basically, you should not be using printf and std::cout as a debugging tool in 2021!

(C) Wasting time figuring out the root cause of a segfault

  • Again, what you shouldn’t be doing is adding more log messages and reproducing the issue multiple times.

(D) Editor is not able to show compilation error preview

  • Speed up your development with a properly configured and functioning IDE. In my experience Jetbrain’s Clion IDE is the clear winner here.

Red Flags : Project structure

(A) Relying only on system tests

  • It is still necessary to run the tests on the actual hardware; but that can be deferred to periodic scheduled tests like nightly and weekly builds. Again, our focus here is to unblock developers and allow them to churn out code faster.
  • Integration tests are a bit tricky to run as they usually involve some form of hardware interaction. This is hard to dependency inject and abstract out, but still possible. There are projects that use elaborate testing frameworks to simulate the hardware interfaces; usually written in Python. This has tremendous benefits, but due to the effort that is required, it is only viable for larger projects that needs to be maintained over a long period.
  • JFYI, there are some commercial solution available to manage board farms for on device testing requirements:

(B) It takes hours to setup a fresh development environment

  • VM images are discouraged due to their higher overhead, but is still viable.

(C) Compiling takes a long time

  • Package mangers can also help cut down the repeated builds.
  • In addition. there are two ways to use a second more powerful machine to speed up build times.
    #1 Remotely develop on a shared server
    #2 Use a distributed build system: e.g. distcc, Incredibuild
  • Both options are not very viable with the new WFH norm though
  • Don’t rely on your CI pipeline as the primary method to do builds. The only exception is for a multi hour rootfs build.
  • In general it pays to invest and provide adequately powerful computers to each developer. This would allow them to quickly build and test locally. The time saving is worth it in the long run.
  • The general rule of thumb is to prefer desktops over laptops and thicker and heavier laptops (workstation replacement types) over more portable, thin and light laptops.

(D) Having to mess with compiler flags and linker flags

  • You should be using a build generator like CMake to define your modules and then express the dependencies between them without having to step down to the compiler specific flags level. Search for “Modern CMake” for more info about related best practices.

(E) Keeping third party lib sources in your source code

  • Consider using Conan and it’s tight integration with CMake as an alternative to keeping foreign source code in your repo.

(F) Using an ancient toolchain with limited C++ feature support

  • These open source solutions have up-to-date toolchain support and are generally well tested for ARM devices.
  • If you must have commercial support to fallback on, there are vendors like Timesys that will provide such services.

(G) Application developers need to setup the entire rootfile build system


Red Flags : CI related

(A) Catching spacing/indentation issues and naming convention violations at code review time

  • IDEs like VS Code and Clion automatically pickup ClangFormat config files and auto indent and generate code accordingly.

(B) Reliance on senior engineers to enforce best practices through code reviews

  • Cppcheck static analyzer is a very popular option for CI stages as it has a low false positive rate.
  • In addition to static analysis, relatively cheap runtime analysis is now viable using compiler sanitizers. It is much like Valgrind, but much faster in terms of execution speed because it is injecting additional machine code to catch things like out of bound memory address, free after use and invoking undefined behavior. I’d recommend to enable Sanitizers during everyday development. On the CI front, definitely have you test suite run with Sanitizers On; at least as a separate stage.

(C) Release builds are done on a “golden” machine

  • Do release build through your CI tool

(D) No central location to get the build binaries from

  • Some popular examples include Maven and Artifactory. The former works better with CMake and Conan.

Other pointers:

(A) Link your issue tracker and git repo

  • This allows for features like branches and PRs automatically getting linked to issues, etc.

(B) Compiler optimizations

  • There is a faster “Thin-LTO” option supported by Clang that brings in most of the performance benefits of regular “Full LTO” without the compile time hit for incremental builds.

(C) Clang for x86 builds

  • Error messages are also generally better with Clang.
  • Try it for your native unit test builds during everyday development.

(D) Use Ninja instead of make

Embedded Systems & Linux Techie