Sample real-time anomaly detection evaluation report

This section contains a sample report, with real-time anomaly detection evaluation metrics, generated during CI.

The CI is set up as follows:

  • Environment: Github Actions

  • Task: Anomaly Detection on minispot dataset

  • ML training framework: PyTorch, ran with Kenning’s generic CNN model wrapper

  • ML execution framework: TFLite Micro, running under kenning-inference-lib from Kenning Zephyr Runtime,

  • Platform: stm32f746g_disco board with two I2C accelerometers

Real-time model evaluation, with time-based metrics

Commands used

Note

This section was generated using:

kenning test \
    --cfg \
        scenario.yml \
    --measurements \
        results.json

kenning report \
    --report-name \
        Real-time model evaluation, with time-based metrics \
    --report-path \
        /home/runner/work/zephyr-sensor-anomalies/zephyr-sensor-anomalies/docs/source/generated/sample-real-time-evaluation-report/report.md \
    --root-dir \
        /home/runner/work/zephyr-sensor-anomalies/zephyr-sensor-anomalies/docs/source \
    --img-dir \
        /home/runner/work/zephyr-sensor-anomalies/zephyr-sensor-anomalies/docs/source/generated/sample-real-time-evaluation-report/img \
    --measurements \
        results.json \
    --smaller-header

General information for results.json

Model framework:

  • torch ver. 2.8.0+cu128

Input JSON:

{
    "dataset": {
        "type": "kenning.datasets.tabular_dataset.TabularDataset",
        "parameters": {
            "dataset_path": "https://dl.antmicro.com/kenning/datasets/anomaly_detection/minispot.csv",
            "cols_x": [
                "roll",
                "pitch",
                "gz",
                "ax",
                "ay",
                "az"
            ],
            "col_y": "anomaly",
            "window_size": 8,
            "shuffle_data": false,
            "expand_classes": true,
            "dataset_root": "build/data",
            "inference_batch_size": 1,
            "download_dataset": true,
            "force_download_dataset": false,
            "external_calibration_dataset": null,
            "split_fraction_test": 0.2,
            "split_fraction_val": null,
            "split_seed": 1234,
            "reduce_dataset": 1.0
        }
    },
    "dataconverter": {
        "type": "kenning.dataconverters.modelwrapper_dataconverter.ModelWrapperDataConverter",
        "parameters": {}
    },
    "optimizers": [
        {
            "type": "kenning.optimizers.tflite.TFLiteCompiler",
            "parameters": {
                "model_framework": "any",
                "target": "default",
                "inference_input_type": "float32",
                "inference_output_type": "float32",
                "dataset_percentage": 0.25,
                "quantization_aware_training": false,
                "use_tf_select_ops": false,
                "resolver_template_path": null,
                "resolver_output_path": null,
                "epochs": 3,
                "batch_size": 32,
                "optimizer": "adam",
                "disable_from_logits": false,
                "save_to_zip": false,
                "compiled_model_path": "build/fp32.1.tflite",
                "location": "host"
            }
        }
    ],
    "platform": {
        "type": "kenning.platforms.zephyr.ZephyrPlatform",
        "parameters": {
            "zephyr_build_path": "../../build",
            "llext_binary_path": null,
            "sensors": [
                "i2c1.lis2ds12_1",
                "i2c1.lis2ds12_2"
            ],
            "sensors_frequency": 99.5,
            "enable_zephelin_gdb": false,
            "enable_zephelin": false,
            "zephyr_base": null,
            "uart_port": "/tmp/renode_uart_mlwydezj/uart",
            "uart_baudrate": 115200,
            "uart_log_port": "/tmp/renode_uart_mlwydezj/uart_log",
            "uart_log_baudrate": 115200,
            "auto_flash": false,
            "openocd_path": "openocd",
            "sensor": null,
            "number_of_batches": 16,
            "simulated": true,
            "runtime_binary_path": null,
            "platform_resc_path": "gh://antmicro:kenning-zephyr-runtime/renode/scripts/stm32f746g_disco.resc;branch=main",
            "resc_dependencies": [],
            "post_start_commands": [],
            "disable_opcode_counters": false,
            "disable_profiler": true,
            "profiler_dump_path": null,
            "profiler_interval_step": 10.0,
            "runtime_init_log_msg": "*** Booting Zephyr OS build",
            "runtime_init_timeout": 30,
            "gdb_port": 3333,
            "name": "stm32f746g_disco",
            "platforms_definitions": [
                "kenning:///platforms/platforms.yml",
                "/home/runner/work/zephyr-sensor-anomalies/zephyr-sensor-anomalies/.venv/lib/python3.11/site-packages/kenning/resources/platforms/platforms.yml"
            ]
        }
    },
    "protocol": {
        "type": "kenning.protocols.uart.UARTProtocol",
        "parameters": {
            "port": "/tmp/renode_uart_mlwydezj/uart",
            "baudrate": 115200,
            "error_recovery": true,
            "timeout": 30
        }
    },
    "model_wrapper": {
        "type": "kenning.modelwrappers.classification.pytorch_generic.PyTorchGenericClassification",
        "parameters": {
            "training_batch_size": 512,
            "model_source": "./model.py",
            "learning_rate": 0.01,
            "num_epochs": 10,
            "export_dict": true,
            "model_path": "./net.pt",
            "model_name": null
        }
    },
    "runtime": {
        "type": "kenning.runtimes.tflite.TFLiteRuntime",
        "parameters": {
            "save_model_path": "build/fp32.1.tflite",
            "delegates_list": null,
            "num_threads": 4,
            "llext_binary_path": null,
            "batch_size": 1,
            "disable_performance_measurements": false
        }
    },
    "inference_loop": {
        "type": "kenning.inferenceloops.anomaly_realtime.AnomalyDetectionInferenceLoop",
        "parameters": {
            "smoothing_window_size": 10,
            "inference_limit": 512
        }
    }
}

Inference quality metrics for results.json

Confusion matrix

Fig. 1 Confusion matrix

Table 1 Inference quality metrics

Statistic

Value

Accuracy

0.943249

Mean precision

0.768386

Mean sensitivity

0.833025

G-mean

0.823265

Detection quality metrics for results.json

Anomaly Fault Detection Rate and False Alarm Rate plot

Fig. 2 Anomaly Fault Detection Rate and False Alarm Rate plot

Anomaly Average Detection Delay plot

Fig. 3 Anomaly Average Detection Delay plot

Last update: 2026-05-21