Sample autogenerated report

This section contains a CI sample report for running inference on a compiled model.

The CI is set up as follows:

  • Environment: Github Actions

  • Task: dog and cat breeds classification based on the Oxford-IIIT Pet Dataset,

  • Training framework: TensorFlow,

  • Compiler framework: TVM,

  • Target: CPU (using LLVM target on TVM).

Pet Dataset classification using TVM-compiled TensorFlow model

Commands used

Note

This section was generated using:

python -m kenning.__main__ \
    optimize \
    test \
    --modelwrapper-cls \
        kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2 \
    --dataset-cls \
        kenning.datasets.pet_dataset.PetDataset \
    --measurements \
        ./build/local-cpu-tvm-tensorflow-classification.json \
    --compiler-cls \
        kenning.optimizers.tvm.TVMCompiler \
    --runtime-cls \
        kenning.runtimes.tvm.TVMRuntime \
    --model-path \
        kenning:///models/classification/tensorflow_pet_dataset_mobilenetv2.h5 \
    --model-framework \
        keras \
    --target \
        llvm \
    --compiled-model-path \
        ./build/compiled-model.tar \
    --opt-level \
        3 \
    --save-model-path \
        ./build/compiled-model.tar \
    --target-device-context \
        cpu \
    --dataset-root \
        ./build/PetDataset/ \
    --inference-batch-size \
        1 \
    --verbosity \
        INFO

python -m kenning.__main__ \
    report \
    --report-path \
        docs/source/generated/local-cpu-tvm-tensorflow-classification.md \
    --report-name \
        Pet Dataset classification using TVM-compiled TensorFlow model \
    --root-dir \
        docs/source/ \
    --img-dir \
        docs/source/generated/img \
    --report-types \
        performance \
        classification \
    --measurements \
        build/local-cpu-tvm-tensorflow-classification.json \
    --smaller-header

General information for build.local-cpu-tvm-tensorflow-classification.json

Model framework:

  • tensorflow ver. 2.11.1

Input JSON:

{
    "dataset": {
        "type": "kenning.datasets.pet_dataset.PetDataset",
        "parameters": {
            "classify_by": "breeds",
            "image_memory_layout": "NHWC",
            "dataset_root": "build/PetDataset",
            "inference_batch_size": 1,
            "download_dataset": true,
            "force_download_dataset": false,
            "external_calibration_dataset": null,
            "split_fraction_test": 0.2,
            "split_fraction_val": null,
            "split_seed": 1234
        }
    },
    "model_wrapper": {
        "type": "kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2",
        "parameters": {
            "model_path": "/home/runner/.kenning/models/classification/tensorflow_pet_dataset_mobilenetv2.h5",
            "model_name": null
        }
    },
    "runtime": {
        "type": "kenning.runtimes.tvm.TVMRuntime",
        "parameters": {
            "save_model_path": "build/compiled-model.tar",
            "target_device_context": "cpu",
            "target_device_context_id": 0,
            "runtime_use_vm": false,
            "llext_binary_path": null,
            "disable_performance_measurements": false
        }
    },
    "data_converter": {
        "type": "kenning.dataconverters.modelwrapper_dataconverter.ModelWrapperDataConverter",
        "parameters": {}
    },
    "optimizers": [
        {
            "type": "kenning.optimizers.tvm.TVMCompiler",
            "parameters": {
                "model_framework": "keras",
                "target": "llvm",
                "target_microtvm_board": null,
                "target_host": null,
                "zephyr_header_template": null,
                "zephyr_llext_source_template": null,
                "opt_level": 3,
                "libdarknet_path": "/usr/local/lib/libdarknet.so",
                "compile_use_vm": false,
                "output_conversion_function": "default",
                "conv2d_data_layout": "",
                "conv2d_kernel_layout": "",
                "use_fp16_precision": false,
                "use_int8_precision": false,
                "use_tensorrt": false,
                "dataset_percentage": 0.25,
                "compiled_model_path": "build/compiled-model.tar",
                "location": "host"
            }
        }
    ]
}

Inference performance metrics for build.local-cpu-tvm-tensorflow-classification.json

Inference time

Bokeh Application

Figure 6 Inference time

  • First inference duration (usually including allocation time): 0.07668648100002429,

  • Mean: 0.07305015110544145 s,

  • Standard deviation: 0.00037517915849804396 s,

  • Median: 0.07301363600004152 s.

Average CPU usage

Bokeh Application

Figure 7 Average CPU usage during benchmark

  • Mean: 53.76792372881356 %,

  • Standard deviation: 4.238942461168583 %,

  • Median: 52.5 %.

Memory usage

Bokeh Application

Figure 8 Memory usage during benchmark

  • Mean: 9.793389830508476 %,

  • Standard deviation: 0.06126104658577814 %,

  • Median: 9.8 %.

Inference quality metrics for build.local-cpu-tvm-tensorflow-classification.json

Bokeh Plot

Figure 9 Confusion matrix

  • Accuracy: 0.9578231292517007

  • Top-5 accuracy: 0.9959183673469387

  • Mean precision: 0.9591048246905213

  • Mean sensitivity: 0.9572500515056033

  • G-mean: 0.9563343851604948

Last update: 2024-12-06