summaryrefslogtreecommitdiff
path: root/Documentation/tutorial/part3.md
blob: 7ccee877546e9cfa21bf7448b144df7c72968214 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
# Writing unit tests for coreboot

## Introduction
General thoughts about unit testing coreboot can be found in
[Unit testing coreboot](../technotes/2020-03-unit-testing-coreboot.md).

This document aims to guide developers through the process of adding and writing
unit tests for coreboot modules.

As an example of unit under test, `src/device/i2c.c` (referred hereafter as UUT
"Unit Under Test") will be used. This is simple module, thus it should be easy
for the reader to focus solely on the testing logic, without the need to spend
too much time on digging deeply into the source code details and flow of
operations. That being said, a good understanding of what the unit under test is
doing is crucial for writing unit tests.

This tutorial should also be helpful for developers who want to follow
[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even though TDD
has a different work flow of building tests first, followed by the code that
satisfies them, the process of writing tests and adding them to the tree is the
same.

## Analysis of unit under test
First of all, it is necessary to precisely establish what we want to test in a
particular module. Usually this will be an externally exposed API, which can be
used by other modules.

```eval_rst
.. admonition:: i2c-test example

   In case of our UUT, API consist of two methods:

   .. code-block:: c

     int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
                 uint8_t mask, uint8_t shift)
     int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t data,
                 uint8_t mask, uint8_t shift)

   For sake of simplicity, let's focus on `i2c_read_field` in this document.
```

Once the API is defined, the next question is __what__ this API is doing (or
what it will be doing in case of TDD). In other words, what outputs we are
expecting from particular functions, when providing particular input parameters.

```eval_rst
.. admonition:: i2c-test example

   .. code-block:: c

     int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data,
                 uint8_t mask, uint8_t shift)

   This is a method which means to read content of register `reg` from i2c device
   on i2c `bus` and slave address `chip`, applying bit `mask` and offset `shift`
   to it. Returned data should be placed in `data`.
```

The next step is to determine all external dependencies of UUT in order to mock
them out. Usually we want to isolate the UUT as much as possible, so that the
test result depends __only__ on the behavior of UUT and not on the other
modules. While some software dependencies may be hard to be mock (for example
due to complicated dependencies) and thus should be simply linked into the test
binaries, all hardware dependencies need to be mocked out, since in the
user-space host environment, targets hardware is not available.

```eval_rst
.. admonition:: i2c-test example

   `i2c_read_field` is calling `i2c_readb`, which eventually invokes
   `i2c_transfer`. This method simply calls `platform_i2c_transfer`. The last
   function in the chain is a hardware-touching one, and defined separately for
   different SOCs. It is responsible for issuing transactions on the i2c bus.
   For the purpose of writing unit test, we should mock this function.
```

## Adding new tests
In order to keep the tree clean, the `tests/` directory should mimic the `src/`
directory, so that test harness code is placed in a location corresponding to
UUT. Furthermore, the naming convention is to add the suffix `-test` to the UUT
name when creating a new test harness file.

```eval_rst
.. admonition:: i2c-test example

   Considering that UUT is `src/device/i2c.c`, test file should be named
   `tests/device/i2c-test.c`. When adding a new test file, it needs to be
   registered with the coreboot unit testing infrastructure.
```

Every directory under `tests/` should contain a Makefile.inc, similar to what
can be seen under the `src/`. Register a new test in Makefile.inc, by
__appending__ test name to the `tests-y` variable.

```eval_rst
.. admonition:: i2c-test example

   .. code-block:: c

     tests-y += i2c-test
```

Next step is to list all source files, which should be linked together in order
to create test binary. Usually a tests requires only two files - UUT and test
harness code, but sometimes more is needed to provide the test environment.
Source files are registered in `<test_name>-srcs` variable.

```eval_rst
.. admonition:: i2c-test example

   .. code-block:: c

     i2c-test-srcs += tests/device/i2c-test.c
     i2c-test-srcs += src/device/i2c.c
```

Above minimal configuration is a basis for further work. One can try to build
and run test binary either by invoking `make tests/<test_dir>/<test_name>` or by
running all unit tests (whole suite) for coreboot `make unit-tests`.

```eval_rst
.. admonition:: i2c-test example

   .. code-block:: c

     make tests/device/i2c-test

   or

   .. code-block:: c

     make unit-tests
```

When trying to build test binary, one can often see linker complains about
`undefined reference` to couple of symbols. This is one of solutions to
determine all external dependencies of UUT - iteratively build test and resolve
errors one by one. At this step, developer should decide either it's better to
add an extra module to provide necessary definitions or rather mock such
dependency. Quick guide through adding mocks is provided later in this doc.

## Writing new tests
In coreboot, [Cmocka](https://cmocka.org/) is used as unit test framework. The
project has exhaustive [API documentation](https://api.cmocka.org/). Let's see
how we may incorporate it when writing tests.

### Assertions
Testing the UUT consists of calling the functions in the UUT and comparing the
returned values to the expected values. Cmocka implements
[a set of assert macros](https://api.cmocka.org/group__cmocka__asserts.html) to
compare a value with an expected value. If the two values do not match, the test
fails with an error message.

```eval_rst
.. admonition:: i2c-test example

   In our example, the simplest test is to call UUT for reading our fake devices
   registers and do all calculation in the test harness itself. At the end, let's
   compare integers with `assert_int_equal`.

   .. code-block:: c

     #define MASK        0x3
     #define SHIFT        0x1

     static void i2c_read_field_test(void **state)
     {
             int bus, slave, reg;
             int i, j;
             uint8_t buf;

             mock_expect_params_platform_i2c_transfer();

             /* Read particular bits in all registers in all devices, then compare
                with expected value. */
             for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++)
                     for (j = 0; j < ARRAY_SIZE(i2c_ex_devs[0].regs); j++) {
                             i2c_read_field(i2c_ex_devs[i].bus,
                                     i2c_ex_devs[i].slave,
                                     i2c_ex_devs[i].regs[j].reg,
                                     &buf, MASK, SHIFT);
                             assert_int_equal((i2c_ex_devs[i].regs[j].data &
                                     (MASK << SHIFT)) >> SHIFT, buf);
                     };
     }
```

### Mocks

#### Overview
Many coreboot modules are low level software that touch hardware directly.
Because of this, one of the most important and challenging part of
writing tests is to design and implement mocks. A mock is a software component
which implements the API of another component so that the test can verify that
certain functions are called (or not called), verify the parameters passed to
those functions, and specify the return values from those functions. Mocks are
especially useful when the API to be implemented is one that accesses hardware
components.

When writing a mock, the developer implements the same API as the module being
mocked. Such a mock may, for example, register a set of driver methods. Behind
this API, there is usually a simulation of real hardware.

```eval_rst
.. admonition:: i2c-test example

   For purpose of our i2c test, we may introduce two i2c devices with set of
   registers, which simply are structs in memory.

   .. code-block:: c

     /* Simulate two i2c devices, both on bus 0, each with three uint8_t regs
        implemented. */
     typedef struct {
             uint8_t reg;
             uint8_t data;
     } i2c_ex_regs_t;

     typedef struct {
             unsigned int bus;
             uint8_t slave;
             i2c_ex_regs_t regs[3];
     } i2c_ex_devs_t;

     i2c_ex_devs_t i2c_ex_devs[] = {
             {.bus = 0, .slave = 0xA, .regs = {
                     {.reg = 0x0, .data = 0xB},
                     {.reg = 0x1, .data = 0x6},
                     {.reg = 0x2, .data = 0xF},
             } },
             {.bus = 0, .slave = 0x3, .regs = {
                     {.reg = 0x0, .data = 0xDE},
                     {.reg = 0x1, .data = 0xAD},
                     {.reg = 0x2, .data = 0xBE},
             } },
     };

   These fake devices will be accessed instead of hardware ones:

   .. code-block:: c

             reg = tmp->buf[0];

             /* Find object for requested device */
             for (i = 0; i < ARRAY_SIZE(i2c_ex_devs); i++, i2c_dev++)
                     if (i2c_ex_devs[i].slave == tmp->slave) {
                             i2c_dev = &i2c_ex_devs[i];
                             break;
                     }

             if (i2c_dev == NULL)
                     return -1;

             /* Write commands */
             if (tmp->len > 1) {
                     i2c_dev->regs[reg].data = tmp->buf[1];
             };

             /* Read commands */
             for (i = 0; i < count; i++, tmp++)
                     if (tmp->flags & I2C_M_RD) {
                             *(tmp->buf) = i2c_dev->regs[reg].data;
                     };
```

Cmocka uses a feature that gcc provides for breaking dependencies at the link
time. It is possible to override implementation of some function, with the
method from test harness. This allows test harness to take control of execution
from binary (during the execution of test), and stimulate UUT as required
without changing the source code.

coreboot unit test infrastructure supports overriding of functions at link time.
This is as simple as adding a `name_of_function` to be mocked into
<test_name>-mocks variable in Makefile.inc. The result is that every time the
function is called, `wrap_name_of_function` will be called instead.

```eval_rst
.. admonition:: i2c-test example

   .. code-block:: c

     i2c-test-mocks += platform_i2c_transfer

   Now, dev can write own implementation of `platform_i2c_transfer` and define it
   as `wrap_platform_i2c_transfer`. This implementation instead of accessing real
   i2c bus, will write/read from fake structs.

   .. code-block:: c

     int __wrap_platform_i2c_transfer(unsigned int bus, struct i2c_msg *segments,
             int count)
     {
     }
```

#### Checking mock's arguments
A test can verify the parameters provided by the UUT to the mock function. The
developer may also verify that number of calls to mock is correct and the order
of calls to particular mocks is as expected (See
[this](https://api.cmocka.org/group__cmocka__call__order.html)). The Cmocka
macros for checking parameters are described
[here](https://api.cmocka.org/group__cmocka__param.html). In general, in mock
function, one makes a call to `check_expected(<param_name>)` and in the
corresponding test function, `expect*()` macro, with description which parameter
in which mock should have particular value, or be inside a described range.

```eval_rst
.. admonition:: i2c-test example

   In our example, we may want to check that `platform_i2c_transfer` is fed with
   number of segments bigger than 0, each segment has flags which are in
   supported range and each segment has buf which is non-NULL. We are expecting
   such values for _every_ call, thus the last parameter in `expect*` macros is
   -1.

   .. code-block:: c

     static void mock_expect_params_platform_i2c_transfer(void)
     {
             unsigned long int expected_flags[] = {0, I2C_M_RD, I2C_M_TEN,
                     I2C_M_RECV_LEN, I2C_M_NOSTART};

             /* Flags should always be only within supported range */
             expect_in_set_count(__wrap_platform_i2c_transfer, segments->flags,
                     expected_flags, -1);

             expect_not_value_count(__wrap_platform_i2c_transfer, segments->buf,
                     NULL, -1);

             expect_in_range_count(__wrap_platform_i2c_transfer, count, 1, INT_MAX,
                     -1);
     }

   And the checks below should be added to our mock

   .. code-block:: c

             check_expected(count);

             for (i = 0; i < count; i++, segments++) {
                     check_expected_ptr(segments->buf);
                     check_expected(segments->flags);
             }
```

#### Instrument mocks
It is possible for the test function to instrument what the mock will return to
the UUT. This can be done by using the `will_return*()` and `mock()` macros.
These are described in
[the Mock Object section](https://api.cmocka.org/group__cmocka__mock.html) of
the Cmocka API documentation.

```eval_rst
.. admonition:: Example

   There is an non-coreboot example for using Cmocka available
   `here <https://lwn.net/Articles/558106/>`_.
```

### Test runner
Finally, the developer needs to implement the test `main()` function. All tests
should be registered there and cmocka test runner invoked. All methods for
invoking Cmocka test are described
[here](https://api.cmocka.org/group__cmocka__exec.html).

```eval_rst
.. admonition:: i2c-test example

   We don't need any extra setup and teardown functions for i2c-test, so let's
   simply register test for `i2c_read_field` and return from main value which is
   output of Cmocka's runner (it returns number of tests that failed).

   .. code-block:: c

     int main(void)
     {
             const struct CMUnitTest tests[] = {
                     cmocka_unit_test(i2c_read_field_test),
             };

             return cmocka_run_group_tests(tests, NULL, NULL);
     }
```