# FreeBSD Catch2 test report for cppTango 10.3.3

## Introduction

This report records Catch2 verification on a FreeBSD host using the
`tango-10.3.3` source distribution. It includes the test-harness patch,
Release configuration and build commands, the full CTest output, and the
results of rerunning failed tests.

## Summary

**Result: the Catch2 target builds on FreeBSD, and 335 of 338 CTest entries
passed in the valid Release run.** Three tests failed: a test-server diagnostic
check and asynchronous read/write timeout cases. On rerun, the diagnostic
check passed, while both asynchronous timeout tests failed again. Setup and
cleanup passed on the rerun.

The patch changes only the Catch2 test harness. The cppTango library
implementation was not modified.

## Patch contents

The patch changes three paths under `lib/cpp/tests/catch2`:

1. `utils/platform/impl_unix.cpp` includes `<signal.h>` on Unix platforms.
   The prior conditional included it only on macOS, leaving FreeBSD without the
   declarations for `sigaction`, `sigprocmask`, signal constants, and `kill`.
2. `tango_catch2_tests.cmake` selects a FreeBSD-specific file watcher when
   `CMAKE_SYSTEM_NAME` is `FreeBSD`. Linux uses `inotify` and `prctl` headers
   that FreeBSD does not provide.
3. `utils/platform/unix/impl_freebsd.cpp` implements the Catch2 harness watcher
   with FreeBSD `kqueue`/`EVFILT_VNODE` write notifications and uses `pselect`
   to wait for file events or child signals. As in the existing macOS harness,
   parent-death notification is a no-op because this implementation does not
   use Linux's `PR_SET_PDEATHSIG`.

The patch was prepared from the corresponding 10.3.0 files, then checked for
applicability against the modified 10.3.3 source tree. All build and test
results in this report are from 10.3.3. Apply the patch from the root of a clean
10.3.3 source distribution with:

```sh
patch -p1 < /path/to/freebsd/patches/0001-catch2-freebsd-support.patch
```

## Test environment

- Operating system: FreeBSD 64-bit
- Compiler: Clang 22.1.7 (`/usr/local/llvm22/bin/clang++`)
- CMake: 3.31.12
- Generator: Ninja
- Catch2: 3.15.2
- Source: `/home/giacomo/devel/tango-10.3.3/lib/cpp`
- Build directory: `/home/giacomo/devel/tango-10.3.3/lib/cpp/build/release`
- Build type: Release
- `BUILD_TESTING=ON`
- `TANGO_SKIP_OLD_TESTS=ON` (only Catch2 tests were requested)
- `TANGO_USE_TELEMETRY=OFF`
- `CMAKE_INSTALL_PREFIX=/usr/local/tango-10.3.3` (configured; installation was
  not needed to run Catch2 and was not performed)

The build and tests ran directly from the 10.3.3 source tree. The test command
sets `LD_LIBRARY_PATH` to the Release build directory so FreeBSD loads the
10.3.3 library under test.

## Commands run

The suggested `cmake --workflow ci-debug-linux` command was an example of the
compile-and-test workflow. This run used explicit CMake configure, build, and
CTest commands for the FreeBSD host, in Release mode, with the old non-Catch2
tests disabled. The failed-test retry used CTest's `--rerun-failed` option;
there were no source changes between the initial run and that retry, so no
rebuild was needed.

Configure:

```sh
cmake -S . -B build/release -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTING=ON \
  -DCMAKE_INSTALL_PREFIX=/usr/local/tango-10.3.3 \
  -DTANGO_SKIP_OLD_TESTS=ON \
  -DTANGO_USE_TELEMETRY=OFF
```

Build the Catch2 executable, its test-server link, and its log directories:

```sh
cmake --build build/release \
  --target Catch2Tests TestServer Catch2ServerLogs \
  --parallel 4
```

Run only registered Catch2 tests:

```sh
env LD_LIBRARY_PATH=/home/giacomo/devel/tango-10.3.3/lib/cpp/build/release \
  ctest --test-dir build/release \
  -R '^catch2::' --output-on-failure
```

Rerun the failures recorded by CTest:

```sh
env LD_LIBRARY_PATH=/home/giacomo/devel/tango-10.3.3/lib/cpp/build/release \
  ctest --test-dir build/release \
  --output-on-failure --rerun-failed
```

## Results

### Build

Configuration succeeded on FreeBSD and found Catch2 3.15.2. The first build
without the patch failed because the shared Unix test source lacked FreeBSD
signal declarations and CMake selected `impl_linux.cpp`, which includes
`sys/prctl.h`. After applying the patch, the Release Catch2 target and its
supporting targets built successfully. The build compiled the FreeBSD watcher
and shared Unix source, linked `Catch2Tests`, and created the `TestServer`
symlink.

### Full Catch2 run

- Total CTest entries: 338 (including Catch2 setup and cleanup fixtures)
- Passed: 335
- Failed: 3
- Elapsed time: 599.36 seconds

Failures:

1. `catch2::Scenario: test server crashes, exceptions and timeouts are
   reported` — expected one captured diagnostic log and received none.
2. `catch2::Scenario: Device timeout with read_attribute_asynch` — failed with
   `API_AsynReplyNotArrived` while waiting for asynchronous callback replies.
3. `catch2::Scenario: Device timeout with write_attribute_asynch` — failed
   with `API_AsynReplyNotArrived`; the expected `attr_asyn_to` callback entry
   was also absent.

### Rerun of failed tests

CTest reran five entries: setup, the three failed tests, and cleanup. Setup
passed. The test-server diagnostic check passed on retry. The asynchronous
read and write timeout tests failed again with `API_AsynReplyNotArrived`, and
the write test again lacked the expected `attr_asyn_to` entry. Cleanup passed.
Thus one of the three initial failures was intermittent; the two async timeout
failures reproduced.

## Follow-up

The FreeBSD harness now compiles and the majority of Catch2 tests pass. The
remaining investigation is in test/runtime behavior rather than the original
FreeBSD compile blockers:

- Investigate why async callback replies are not received for the two timeout
  scenarios on this FreeBSD setup.
- The test-server diagnostic check failed once and passed on retry, so it is
  intermittent and may warrant another run if this suite is repeated.

The captured CTest output is stored in this `freebsd` directory:
[`catch2-ctest-output.log`](catch2-ctest-output.log) is the full run, and
[`catch2-ctest-rerun-output.log`](catch2-ctest-rerun-output.log) is the retry.
Each contains the corresponding CTest `LastTest.log` output. The concise failed
test lists are in [`catch2-full-run-failed-tests.log`](catch2-full-run-failed-tests.log)
and [`catch2-rerun-failed-tests.log`](catch2-rerun-failed-tests.log).