DevOps for Embedded C/C++

How to spot and fix the red flags in your workflow

source

Intro

DevOps is a significant part of a web related tech developer’s life. A lot of smart people have put a lot of effort to develop tools and establish processes to optimize throughput. In contrast for the more embedded focused folk like myself; it feels like the world has left us behind to wallow around with our legacy tools and ponder the purpose of our existence.

Part of the problem is that hardware vendors provide very basic BSPs and tools. Outdated best practices and restrictive project budgets usually leave us limited to use these “vendor supported” set of tools. The other part of the problem is the general lack of awareness of what tools and trick are actually at our disposal. I suppose the Stockholm syndrome of doing things the hard way is also a factor with experienced C++ dev. I can’t help with the masochism issues, but the said lack of awareness is what I am attempting to address in this article.

I’ll be listing a bunch of common pain points with there corresponding solutions and best practices. These solutions in general will strive to improve the “quality of life” by reducing the amount of busy work and by abstracting away the more complex parts of the programming language and tool set.

One caveat is that some of what I am about to list may not be viable to your particular hardware platform, project size or budget. My condolences to you.

source

Red Flags : Individual workflow

(A) Build, copy to device, run to confirm functionality, repeat

• Does this describe your workflow? We can do better.
• Consider using an unit test framework like GoolgeTest to write unit test to confirm the functionality natively on x86. This will speed things up and also force you to write more testable code.
• If native x86 compiles are not an option and the target processor is fast enough. Consider a remote development workflow over SSH as described here https://code.visualstudio.com/docs/remote/remote-overview
• If all fails, at least make sure that the deployment to device is automated as part of your build step.

(B) Printing debug messages to track down issues

• Get with the program and setup GDB before you start working on a project
• Use an IDE with GDB support to step through code, print logs and inspect memory.
• Don’t be afraid to leverage more advance tricks like adding conditional break points and expressions evaluation support. VS Code, Clion and Eclipse IDEs work well with this regard.
• Basically, you should not be using printf and std::cout as a debugging tool in 2021!

(C) Wasting time figuring out the root cause of a segfault

• Use runtime analysis tools like compiler sanitizers and Valgrind to get at the root cause, fast! They will dump a lot of information including offending line numbers. I’d advise leaving sanitizers ON while developing.
• Again, what you shouldn’t be doing is adding more log messages and reproducing the issue multiple times.

(D) Editor is not able to show compilation error preview

• Don’t settle for an IDE or Text editor that is not able to highlight things like syntax errors.
• Speed up your development with a properly configured and functioning IDE. In my experience Jetbrain’s Clion IDE is the clear winner here.

source

Red Flags : Project structure

(A) Relying only on system tests

• Running anything on remote hardware is expensive. It takes time to transfer, it’s harder to debug and takes longer to execute. So it goes without saying that the ability to run all the tests on x86 is a boon to speed up everyday development.
• It is still necessary to run the tests on the actual hardware; but that can be deferred to periodic scheduled tests like nightly and weekly builds. Again, our focus here is to unblock developers and allow them to churn out code faster.
• Integration tests are a bit tricky to run as they usually involve some form of hardware interaction. This is hard to dependency inject and abstract out, but still possible. There are projects that use elaborate testing frameworks to simulate the hardware interfaces; usually written in Python. This has tremendous benefits, but due to the effort that is required, it is only viable for larger projects that needs to be maintained over a long period.
• JFYI, there are some commercial solution available to manage board farms for on device testing requirements: https://www.timesys.com/open-source-embedded/board-farm/

(B) It takes hours to setup a fresh development environment

• Use scripts, Docker images or WSL images to quickly and easily share a pre-configured build environment.
• VM images are discouraged due to their higher overhead, but is still viable.

(C) Compiling takes a long time

• Enable compiler caching. This is specially useful if you are switching branches a lot or if you need to compiler the same third party libraries from source over and over.
• Package mangers can also help cut down the repeated builds.
• In addition. there are two ways to use a second more powerful machine to speed up build times.
  #1 Remotely develop on a shared server
  #2 Use a distributed build system: e.g. distcc, Incredibuild
• Both options are not very viable with the new WFH norm though
• Don’t rely on your CI pipeline as the primary method to do builds. The only exception is for a multi hour rootfs build.
• In general it pays to invest and provide adequately powerful computers to each developer. This would allow them to quickly build and test locally. The time saving is worth it in the long run.
• The general rule of thumb is to prefer desktops over laptops and thicker and heavier laptops (workstation replacement types) over more portable, thin and light laptops.

(D) Having to mess with compiler flags and linker flags

• In this day and age, adding a new target or dependency to a project should not involve having to fight with build config files, compiler flags and linker flags.
• You should be using a build generator like CMake to define your modules and then express the dependencies between them without having to step down to the compiler specific flags level. Search for “Modern CMake” for more info about related best practices.

(E) Keeping third party lib sources in your source code

• Package management has been notoriously difficult with C++. As such, in the past it has been a common practice to include code from third party dependencies in your repo. But now there are new package management tools available that address this need more elegantly. e.g Conan & VPKG
• Consider using Conan and it’s tight integration with CMake as an alternative to keeping foreign source code in your repo.

(F) Using an ancient toolchain with limited C++ feature support

• You don’t have to only use the toolchain that your SoC vendor officially supports. Especially with ARM based designs; you are better off spinning up your own BSP with something like Buildroot or Yocto.
• These open source solutions have up-to-date toolchain support and are generally well tested for ARM devices.
• If you must have commercial support to fallback on, there are vendors like Timesys that will provide such services.

(G) Application developers need to setup the entire rootfile build system

• All modern build systems like Yocto support the ability to export a SDK (toolchain and sysroot) that application developers can use to cross-compile apps for your target hardware.

source

Red Flags : CI related

(A) Catching spacing/indentation issues and naming convention violations at code review time

• Use a code formatter like ClangFormat to catch and fix such issues via local git commit hooks and enforce them automatically via a CI pipeline stage.
• IDEs like VS Code and Clion automatically pickup ClangFormat config files and auto indent and generate code accordingly.

(B) Reliance on senior engineers to enforce best practices through code reviews

• This is what static analyzers are for. Clang-Tidy is my favorite option. I find it to be a great learning tool too since it catches most violations of the C++ Core Guidelines.
• Cppcheck static analyzer is a very popular option for CI stages as it has a low false positive rate.
• In addition to static analysis, relatively cheap runtime analysis is now viable using compiler sanitizers. It is much like Valgrind, but much faster in terms of execution speed because it is injecting additional machine code to catch things like out of bound memory address, free after use and invoking undefined behavior. I’d recommend to enable Sanitizers during everyday development. On the CI front, definitely have you test suite run with Sanitizers On; at least as a separate stage.

(C) Release builds are done on a “golden” machine

• Please use Docker and not Clonezilla to “save” your build environment
• Do release build through your CI tool

(D) No central location to get the build binaries from

• Do leverage dedicated binary repositories to host your build artifacts. One cool feature this enables is the ability to programmatically download a particular version of a binary/library from your CI build scripts.
• Some popular examples include Maven and Artifactory. The former works better with CMake and Conan.

source

Other pointers:

(A) Link your issue tracker and git repo

• Github, GitLab and Atlassian support this with their product offerings.
• This allows for features like branches and PRs automatically getting linked to issues, etc.

(B) Compiler optimizations

• Don’t leave performance on the table. Remember to enable compiler optimizations and link time optimizations (LTO). As long as you are not invoking undefined behavior, enabling higher levels of optimizations shouldn’t break anything.
• There is a faster “Thin-LTO” option supported by Clang that brings in most of the performance benefits of regular “Full LTO” without the compile time hit for incremental builds.

(C) Clang for x86 builds

• Clang and the lld linker can compile a lot faster for x86 (vs GCC and ld).
• Error messages are also generally better with Clang.
• Try it for your native unit test builds during everyday development.

(D) Use Ninja instead of make

That’s it folks!