Sample AutoML report¶

This section contains a sample AutoML report generated during CI.

The CI is set up as follows:

Environment: Github Actions
Task: Anomaly Detection
AutoML framework: Auto-PyTorch
Compiler framework: TFLite,
Platform: MAX32690 Evaluation Kit

AutoML statistics¶

Optimized metric: f1
The number of generated models: 43
The number of trained and evaluated models: 30
The number of successful training processes: 37
The number of models that caused a crash: 2
The number of models that failed due to the timeout: 1
The number of models that failed due to the too large size: 3
The number of models that failed due to incompatibility: 0

Training overview¶

Bokeh Plot

Figure 12 Loss value during AutoML training process¶

Bokeh Plot

Figure 13 Comparison of loss value across models¶

Summary of generated models¶

Bokeh Plot

Figure 14 Metrics of models trained by AutoML flow¶

Table 5 Summary of generated models’ parameters¶
Model ID	Number of layers	Optimized model size [KB]	Total parameters	Trainable parameters
3	7	15.5546875	2815	2814
4	10	49.73046875	11623	11622
5	17	33.08203125	7498	7497
6	21	33.4921875	7613	7612
7	27	61.02734375	14094	14093
8	14	50.16015625	11841	11840
9	21	35.60546875	7834	7833
10	17	37.4296875	8732	8731
11	23	54.3359375	11691	11690
12	21	30.37890625	7656	7655
13	17	33.13671875	7799	7798
14	12	37.5625	8227	8226
15	10	42.421875	10264	10263
16	13	45.5546875	10455	10454
17	8	19.8671875	4364	4363
18	27	47.5078125	10720	10719
19	25	49.37109375	13209	13208
20	19	45.875	10707	10706
21	12	44.40234375	10171	10170
22	14	58.640625	14009	14008
23	9	27.69921875	6301	6300
24	10	24.9453125	5151	5150
25	9	50.67578125	12130	12129
26	25	51.265625	11652	11651
27	11	30.75390625	8095	8094
28	10	36.19140625	8007	8006
29	21	24.62890625	4788	4787
30	23	30.2890625	6668	6667
31	13	26.79296875	6005	6004
32	19	41.4453125	8872	8871
33	19	42.16015625	9754	9753
34	15	31.8359375	6497	6496
35	11	42.99609375	9219	9218
36	13	20.0625	4112	4111
37	15	31.16796875	6868	6867
38	17		9512	9511

Classification comparison¶

Comparison of inference time, F1 score and model size¶

Bokeh Plot

Figure 15 Model size, speed and quality summary. The F1 score of the model is presented on Y axis. The inference time of the model is presented on X axis. The size of the model is represented by the size of its point.¶

Table 6 Comparison of model inference time, accuracy and size¶
Model name	Mean Inference time [s]	Size [MB]	F1 score
workspace.automl-results.1234_3_5.0.measurements.json	0.000426	0.016	0.250000
workspace.automl-results.1234_12_5.0.measurements.json	0.000853	0.032	0.250000
workspace.automl-results.1234_19_1.6666666666666665.measurements.json	0.004035	0.052	0.352941
workspace.automl-results.1234_19_5.0.measurements.json	0.004034	0.052	0.352941
workspace.automl-results.1234_31_5.0.measurements.json	0.000748	0.028	0.250000

Detailed metrics comparison¶

Bokeh Plot

Figure 16 Radar chart representing the accuracy, precision and recall for models¶

Table 7 Summary of classification metrics for models¶
Model name	Accuracy	Mean precision	Mean sensitivity	G-mean	ROC AUC	F1 score
workspace.automl-results.1234_3_5.0.measurements.json	0.952000	0.729675	0.579132	0.406529	0.579132	0.250000
workspace.automl-results.1234_12_5.0.measurements.json	0.952000	0.729675	0.579132	0.406529	0.579132	0.250000
workspace.automl-results.1234_19_1.6666666666666665.measurements.json	0.956000	0.781633	0.620798	0.497895	0.620798	0.352941
workspace.automl-results.1234_19_5.0.measurements.json	0.956000	0.781633	0.620798	0.497895	0.620798	0.352941
workspace.automl-results.1234_31_5.0.measurements.json	0.952000	0.729675	0.579132	0.406529	0.579132	0.250000

Inference comparison¶

Performance metrics¶

Bokeh Application

Figure 17 Plot represents changes of inference time over time for all models.¶

Table 8 Summary of inference time metrics for models¶
Model name	Standard deviation [s]	Minimum [s]	Maximum [s]	Median [s]	Mean [s]
workspace.automl-results.1234_3_5.0.measurements.json	0.000029	0.000381	0.000516	0.000413	0.000426
workspace.automl-results.1234_12_5.0.measurements.json	0.000030	0.000811	0.000929	0.000840	0.000853
workspace.automl-results.1234_19_1.6666666666666665.measurements.json	0.000180	0.003648	0.004534	0.004027	0.004035
workspace.automl-results.1234_19_5.0.measurements.json	0.000178	0.003640	0.004549	0.004036	0.004034
workspace.automl-results.1234_31_5.0.measurements.json	0.000030	0.000689	0.000846	0.000736	0.000748

Mean comparison plots¶

Bokeh Plot

Figure 18 Violin chart representing distribution of values for performance metrics for models¶

Table 9 Performance metric for models¶
Model name	Inference time [s]
workspace.automl-results.1234_3_5.0.measurements.json	0.000426
workspace.automl-results.1234_12_5.0.measurements.json	0.000853
workspace.automl-results.1234_19_1.6666666666666665.measurements.json	0.004035
workspace.automl-results.1234_19_5.0.measurements.json	0.004034
workspace.automl-results.1234_31_5.0.measurements.json	0.000748

Last update: 2026-01-30