diff options
Diffstat (limited to 'Documentation')
| -rw-r--r-- | Documentation/technotes/2020-03-unit-testing-coreboot.md | 319 | ||||
| -rw-r--r-- | Documentation/technotes/index.md | 1 |
2 files changed, 320 insertions, 0 deletions
diff --git a/Documentation/technotes/2020-03-unit-testing-coreboot.md b/Documentation/technotes/2020-03-unit-testing-coreboot.md new file mode 100644 index 0000000000..0d1d8ece49 --- /dev/null +++ b/Documentation/technotes/2020-03-unit-testing-coreboot.md @@ -0,0 +1,319 @@ +# Unit testing coreboot + +## Preface +First part of this document, Introduction, comprises disambiguation for what +unit testing is and what is not. This definition will be a basis for the whole +paper. + +Next, Rationale, explains why to use unit testing and how coreboot specifically +may benefit from it. + +This is followed by evaluation of different available free C unit test +frameworks. Firstly, collection of requirements is provided. Secondly, there is +a description of a few selected candidates. Finally, requirements are applied to +candidates to see if they might be a good fit. + +Fourth part is a summary of evaluation, with proposal of unit test framework +for coreboot to be used. + +Finally, Implementation proposal paragraph touches how build system and coreboot +codebase in general should be organized, in order to support unit testing. This +comprises couple of design considerations which need to be addressed. + +## Introduction +A unit test is supposed to test a single unit of code in isolation. In C +language (in contrary to OOP) unit usually means a function. One may also +consider unit under test to be a single compilation unit which exposes some +API (set of functions). A function, talking to some external component can be +tested if this component can be mocked out. + +In other words (looking from C compilation angle), there should be no extra +dependencies (executables) required beside unit under test and test harness in +order to compile unit test binary. Test harness, beside code examining a +routines, may comprise test framework implementation. + +It is hard to apply this strict definition of unit test to firmware code in +practice, mostly due to constraints on speed of execution and size of final +executable. coreboot codebase often cannot be adjusted to be testable. Because +of this, coreboot unit testing subsystem should allow to include some additional +source object files beside unit under test. That being said, the default and +goal wherever possible, should be to isolate unit under test from other parts. + +Unit testing is not an integration testing and it doesn't replace it. First of +all, integration tests cover larger set of components and interactions between +them. Positive integration test result gives more confidence than a positive +unit test does. Furthermore, unit tests are running on the build machine, while +integration tests usually are executed on the target (or simulator). + +## Rationale +Considering above, what is the benefit of unit testing, especially keeping in +mind that coreboot is low-level firmware? Unit tests should be quick, thus may +be executed frequently during development process. It is much easier to build +and run a unit test on a build machine, than any integration test. This in turn +may be used by dev to gather extra confidence early during code development +process. Actually developer may even write unit tests earlier than the code - +see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept. + +That being said, unit testing embedded C code is a difficult task, due to +significant amount of dependencies on underlying hardware. Mocking can handle +some hardware dependencies. However, complex mocks make the unit test +susceptible to failing and can require significant development effort. + +Writing unit tests for a code (both new and currently existing) may be favorable +for the code quality. It is not only about finding bugs, but in general - easily +testable code is a good code. + +coreboot benefits the most from testing common libraries (lib/, commonlib/, +payloads/libpayload) and coreboot infrastructure (console/, device/, security/). + +## Evaluation of unit testing frameworks + +### Requirements +Requirements for unit testing frameworks: + +* Easy to use +* Few dependencies + + Standard C library is all we should need + +* Isolation between tests +* Support for mocking +* Support for some machine parsable output +* Compiler similarity + + Compiler for the host _must_ support the same language standards as the target + compiler. Ideally the same toolchain should be used for building firmware + executables and test binaries, however the host complier will be used to build + unit tests, whereas the coreboot toolchain will be used for building the + firmware executables. For some targets, the host compiler and the target + compiler could be the same, but this is not a requirement. + +* Same language for tests and code + + Unit tests will be written in C, because coreboot code is also written in C + +### Desirables + +* Easy to integrate with build system/build tools + + Ideally JUnit-like XML output format for Jenkins + +* Popularity is a plus + + We want a larger community for a couple of reasons. Firstly, easier access to + people with knowledge and tutorials. Secondly, bug fixes for the top of tree + are more frequent and known issues are usually shorter in the pending state. + Last but not least, larger reviewer pool means better and easier upstream + improvements that we would like to submit. + +* Extra features may be a plus +* Compatible license + + This should not be a blocker, since test binaries are not distributed. + However ideally compatible with GPL. + +* IDE integration + +### Candidates +There is a lot of frameworks which allow unit testing C code +([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from +Wikipedia). While not all of them were evaluated, because that would take an +excessive amount of time, couple of them were selected based on the good +opinions among C devs, popularity and fitting above criteria. + +* [SputUnit](https://www.use-strict.de/sput-unit-testing/) +* [GoogleTest](https://github.com/google/googletest) +* [Cmocka](https://cmocka.org/) +* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) + +We looked at several other test frameworks, but decided not to do a full evaluation +for various reasons such as functionality, size of the developer community, or +compatibility. + +### Evaluation +* [SputUnit](https://www.use-strict.de/sput-unit-testing/) + * Pros + * No dependencies, one header file to include - that’s all + * Pure C + * Very easy to use + * BSD license + * Cons + * Main repo doesn’t have support for generating JUnit XML reports for + Jenkins to consume - this feature is available only on the fork from + SputUnit called “Sput_report”. It makes it niche in a niche, so there are + some reservations whether support for this will be satisfactory + * No support for mocks + * Not too popular + * No automatic test registration +* [GoogleTest](https://github.com/google/googletest) + * Pros + * Automatic test registration + * Support for different output formats (including XML for Jenkins) + * Good support, widely used, the biggest and the most active community out + of all frameworks that were investigated + * Available as a package in the most common distributions + * Test fixtures easily available + * Well documented + * Easy to integrate with an IDE + * BSD license + * Cons + * Requires C++11 compiler + * To make most out of it (use GMock) C++ knowledge is required +* [Cmocka](https://cmocka.org/) + * Pros + * Self-contained, autonomous framework + * Pure C + * API is well documented + * Multiple output formats (including XML for Jenkins) + * Available as a package in the most common distributions + * Used in some popular open source projects (libssh, OpenVPN, Samba) + * Test fixtures available + * Support for exception handling + * Cons + * No automatic test registration + * It will require some effort to make it work from within an IDE + * Apache 2.0 license (not compatible with GPLv2) +* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling) + * Pros + * Pure C (Unity testing framework itself, not test runner) + * Support for different output formats (including XML for Jenkins) + * There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse) + * MIT license + * Cons + * Test runner (Ceedling) is not written in C - uses Ruby + * Mocking/Exception handling functionalities are actually separate tools + * No automatic test registration + * Not too popular + +### Summary & framework proposal +After research, we propose using the Cmocka unit test framework. Cmocka fulfills +all stated evaluation criteria. It is rather easy to use, doesn’t have extra +dependencies, written fully in C, allows for tests fixtures and some popular +open source projects already are using it. Cmocka also includes support for +mocks. + +Cmocka's limitations, such as the lack of automatic test registration, are +considered minor issues that will require only minimal additional work from a +developer. At the same time, it may be worth to propose improvement to Cmocka +community or simply apply some extra wrapper with demanded functionality. + +## Implementation + +### Framework as a submodule or external package +Unit test frameworks may be either compiled from source (from a git submodule +under 3rdparty/) or pre-compiled as a package. The second option seems to be +easier to maintain, while at the same time may bring some unwanted consequences +(different version across distributions, frequent changes in API). It makes sense +to initially experiment with packages and check how it works. If this will +cause any issues, then it is always possible to switch to submodule approach. + +### Integration with build system +To get the most out of unit testing framework, it should be integrated with +Jenkins automation server. Verification of all unit tests for new changes may +improve code reliability to some extent. + +### Build configuration (Kconfig) +While building unit under test object file, it is necessary to apply some +configuration (config) just like when building usual firmware. For simplicity, +there will be one default tests .config `qemu_x86_i440fx` for all unit tests. At +the same time, some tests may require running with different values of particular +config. This should be handled by adding extra header, included after config.h. +This header will comprise #undef of old CONFIG values and #define of the +required value. When unit testing will be integrated with Jenkins, it may be +preferred to use every available config for periodic builds. + +### Directory structure +Tests should be kept separate from the code, while at the same time it must be +easy to match code with test harness. + +We create new directory for test files ($(toplevel)/tests/) and mimic the +structure of src/ directory. + +Test object files (test harness, unit under tests and any additional executables +are stored under build/tests/<test_name> directory. + +Below example shows how directory structure is organized for the two test cases: +tests/lib/string-test and tests/device/i2c-test: + +```bash +├── src +│ ├── lib +│ │ ├── string.c <- unit under test +│ │ +│ ├── device +│ ├── i2c.c +│ +├── tests +│ ├── include +│ │ ├── mocks <- mock headers, which replace original headers +│ │ +│ ├── Makefile.inc <- top Makefile for unit tests subsystem +│ ├── lib +│ │ ├── Makefile.inc +│ │ ├── string-test.c <- test code for src/lib/string.c +│ │ │ +│ ├── device +│ │ ├── Makefile.inc +│ ├── i2c-test.c +│ +├── build +│ ├── tests <-all test-related executables + ├── config.h <- default config used for tests builds + ├── lib + │ ├── string-test <- all string-test executables + │ │ ├── run <- final test binary + │ │ ├── tests <- all test harness executables + │ │ ├── lib + │ │ ├── string-test.o <-test harness executable + │ │ ├── src <- unit under test and other src executables + │ │ ├── lib + │ │ ├── string.o <- unit under test executable + ├── device + ├── i2c-test + ├── run + ├── tests + │ ├── device + │ ├── i2c-test.o + ├── src + ├── device + ├── i2c.o +``` + +### Adding new tests +For purpose of this description, let's assume that we want to add a new unit test +for src/device/i2c.c module. Since this module is rather simple, it will be enough +to have only one test module. + +Firstly (assuming there is no tests/device/Makefile.inc file) we need to create +Makefile.inc in main unit test module directory. Inside this Makefile.inc, one +need to register new test and can specify multiple different attributes for it. + +```bash +# Register new test, by adding its name to tests variable +tests-y += i2c-test + +# All attributes are defined by <test_name>-<attribute> variables +# <test_name>-srcs is used to register all input files (test harness, unit under +# test and others) for this particular test. Remember to add relative paths. +i2c-test-srcs += tests/device/i2c-test.c +i2c-test-srcs += src/device/i2c.c + +# We can define extra cflags for this particular test +i2c-test-cflags += -DSOME_DEFINE=1 + +# For mocking out external dependencies (functions which cannot be resolved by +# linker), it is possible to register a mock function. To register new mock, it +# is enough to add function-to-be-mocked name to <test_name>-mocks variable. +i2c-test-mocks += platform_i2c_transfer + +# Similar to coreboot concept, unit tests also runs in the context of stages. +# By default all unit tests are compiled to be ramstage executables. If one want +# to overwrite this setting, there is <test_name>-stage variable available. +i2c-test-stage:= bootblock +``` + +### Writing new tests +Full description of how to write unit tests and Cmocka API description is out of +the scope of this document. There are other documents related to this +[Cmocka API](https://api.cmocka.org/) and +[Mocks](https://lwn.net/Articles/558106/). diff --git a/Documentation/technotes/index.md b/Documentation/technotes/index.md index 7c231fc672..5367e69aa2 100644 --- a/Documentation/technotes/index.md +++ b/Documentation/technotes/index.md @@ -2,3 +2,4 @@ * [Dealing with Untrusted Input in SMM](2017-02-dealing-with-untrusted-input-in-smm.md) * [Rebuilding coreboot image generation](2015-11-rebuilding-coreboot-image-generation.md) +* [Unit testing coreboot](2020-03-unit-testing-coreboot.md) |