Sample AutoML report¶

This section contains a sample AutoML report generated during CI.

The CI is set up as follows:

Environment: Github Actions
Task: Anomaly Detection
AutoML framework: Auto-PyTorch
Compiler framework: TFLite,
Platform: MAX32690 Evaluation Kit

AutoML statistics¶

Optimized metric: f1
The number of generated models: 40
The number of trained and evaluated models: 24
The number of successful training processes: 31
The number of models that caused a crash: 0
The number of models that failed due to the timeout: 0
The number of models that failed due to the too large size: 9
The number of models that failed due to incompatibility: 0

Training overview¶

Bokeh Plot

Figure 12 Loss value during AutoML training process¶

Bokeh Plot

Figure 13 Comparison of loss value across models¶

Summary of generated models¶

Bokeh Plot

Figure 14 Metrics of models trained by AutoML flow¶

Table 5 Summary of generated models’ parameters¶
Model ID	Number of layers	Optimized model size [KB]	Total parameters	Trainable parameters
3	7	17.8515625	2815	2814
4	10	56.0625	11623	11622
5	17	36.93359375	7498	7497
6	21	37.2578125	7613	7612
7	27	64.43359375	14094	14093
8	14	57.20703125	11841	11840
9	21	40.7578125	7834	7833
10	17	41.953125	8732	8731
11	23	57.3125	11691	11690
12	21	37.65234375	7656	7655
13	17	37.7734375	7799	7798
14	12	41.64453125	8227	8226
15	10	45.86328125	10264	10263
16	13	48.98828125	10455	10454
17	8	22.78515625	4364	4363
18	27	50.55078125	10720	10719
19	13	22.328125	4046	4045
20	19	49.92578125	10707	10706
21	19	44.93359375	9318	9317
22	19	38.32421875	7967	7966
23	21	24.71484375	4251	4250
24	19	57.14453125	12877	12876
25	25	74.7265625	16820	16819
26	23	49.66015625	10292	10291
27	23	37.51953125	7311	7310
28	23	38.56640625	7591	7590
29	13	30.2109375	5802	5801
30	17	24.7109375	4206	4205
31	9	27.79296875	5493	5492
32	15	69.015625	14880	14879
33	11	30.69140625	6079	6078
34	14	64.64453125	13745	13744
35	8	59.04296875	13616	13615

Classification comparison¶

Comparison of inference time, F1 score and model size¶

Bokeh Plot

Figure 15 Model size, speed and quality summary. The F1 score of the model is presented on Y axis. The inference time of the model is presented on X axis. The size of the model is represented by the size of its point.¶

Table 6 Comparison of model inference time, accuracy and size¶
Model name	Mean Inference time [s]	Size [MB]	F1 score
workspace.automl-results.1234_27_5.0.measurements.json	0.001214	0.038	0.375000
workspace.automl-results.1234_30_1.6666666666666665.measurements.json	0.000962	0.026	0.333333
workspace.automl-results.1234_30_5.0.measurements.json	0.000959	0.026	0.333333
workspace.automl-results.1234_3_5.0.measurements.json	0.000560	0.018	0.250000
workspace.automl-results.1234_12_5.0.measurements.json	0.001248	0.040	0.250000

Detailed metrics comparison¶

Bokeh Plot

Figure 16 Radar chart representing the accuracy, precision and recall for models¶

Table 7 Summary of classification metrics for models¶
Model name	Accuracy	Mean precision	Mean sensitivity	G-mean	ROC AUC	F1 score
workspace.automl-results.1234_27_5.0.measurements.json	0.960000	0.856707	0.622899	0.498948	0.622899	0.375000
workspace.automl-results.1234_30_1.6666666666666665.measurements.json	0.952000	0.731557	0.618697	0.496839	0.618697	0.333333
workspace.automl-results.1234_30_5.0.measurements.json	0.952000	0.731557	0.618697	0.496839	0.618697	0.333333
workspace.automl-results.1234_3_5.0.measurements.json	0.952000	0.729675	0.579132	0.406529	0.579132	0.250000
workspace.automl-results.1234_12_5.0.measurements.json	0.952000	0.729675	0.579132	0.406529	0.579132	0.250000

Inference comparison¶

Performance metrics¶

Bokeh Application

Figure 17 Plot represents changes of inference time over time for all models.¶

Table 8 Summary of inference time metrics for models¶
Model name	Median [s]	Minimum [s]	Mean [s]	Maximum [s]	Standard deviation [s]
workspace.automl-results.1234_27_5.0.measurements.json	0.001204	0.001155	0.001214	0.001295	0.000030
workspace.automl-results.1234_30_1.6666666666666665.measurements.json	0.000951	0.000920	0.000962	0.001057	0.000027
workspace.automl-results.1234_30_5.0.measurements.json	0.000951	0.000916	0.000959	0.001025	0.000025
workspace.automl-results.1234_3_5.0.measurements.json	0.000550	0.000518	0.000560	0.000652	0.000026
workspace.automl-results.1234_12_5.0.measurements.json	0.001241	0.001203	0.001248	0.001333	0.000025

Mean comparison plots¶

Bokeh Plot

Figure 18 Violin chart representing distribution of values for performance metrics for models¶

Table 9 Performance metric for models¶
Model name	Inference time [s]
workspace.automl-results.1234_27_5.0.measurements.json	0.001214
workspace.automl-results.1234_30_1.6666666666666665.measurements.json	0.000962
workspace.automl-results.1234_30_5.0.measurements.json	0.000959
workspace.automl-results.1234_3_5.0.measurements.json	0.000560
workspace.automl-results.1234_12_5.0.measurements.json	0.001248

Renode performance measurements¶

Executed instructions counters¶

Bokeh Application

Figure 19 Count of executed instructions per second for cpu0 during benchmark¶

Bokeh Plot

Figure 20 Cumulative count of executed instructions for cpu0 during benchmark¶

Memory access counters¶

Bokeh Application

Figure 21 Count of memory reads per second during benchmark¶

Bokeh Plot

Figure 22 Cumulative count of memory reads during benchmark¶

Bokeh Application

Figure 23 Count of memory writes per second during benchmark¶

Bokeh Plot

Figure 24 Cumulative count of memory writes during benchmark¶

Peripheral access counters¶

Bokeh Application

Figure 25 Count of nvic0 reads per second during benchmark¶

Bokeh Plot

Figure 26 Cumulative count of nvic0 reads during benchmark¶

Bokeh Application

Figure 27 Count of nvic0 writes per second during benchmark¶

Bokeh Plot

Figure 28 Cumulative count of nvic0 writes during benchmark¶

Bokeh Application

Figure 29 Count of uart0 reads per second during benchmark¶

Bokeh Plot

Figure 30 Cumulative count of uart0 reads during benchmark¶

Bokeh Application

Figure 31 Count of uart0 writes per second during benchmark¶

Bokeh Plot

Figure 32 Cumulative count of uart0 writes during benchmark¶

Bokeh Application

Figure 33 Count of uart2 reads per second during benchmark¶

Bokeh Plot

Figure 34 Cumulative count of uart2 reads during benchmark¶

Bokeh Application

Figure 35 Count of uart2 writes per second during benchmark¶

Bokeh Plot

Figure 36 Cumulative count of uart2 writes during benchmark¶

Exceptions counters¶

Bokeh Application

Figure 37 Count of raised exceptions per second during benchmark¶

Bokeh Plot

Figure 38 Cumulative count of raised exceptions during benchmark¶

Last update: 2025-12-01