Sample autogenerated report

This section contains the CI sample report for running inference on compiled model.

The CI setup is as follows:

  • Environment: Github Actions

  • Task: dog and cat breeds classification based on the Oxford-IIIT Pet Dataset,

  • Training framework: TensorFlow,

  • Compiler framework: TVM,

  • Target: CPU (using LLVM target on TVM).

Pet Dataset classification using TVM-compiled TensorFlow model

Commands used

Note

This section was generated using:

python -m kenning.scenarios.inference_tester \
    kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2 \
    kenning.datasets.pet_dataset.PetDataset \
    ./build/local-cpu-tvm-tensorflow-classification.json \
    --compiler-cls \
        kenning.compilers.tvm.TVMCompiler \
    --runtime-cls \
        kenning.runtimes.tvm.TVMRuntime \
    --model-path \
        ./kenning/resources/models/classification/tensorflow_pet_dataset_mobilenetv2.h5 \
    --model-framework \
        keras \
    --target \
        llvm \
    --compiled-model-path \
        ./build/compiled-model.tar \
    --opt-level \
        3 \
    --save-model-path \
        ./build/compiled-model.tar \
    --target-device-context \
        cpu \
    --dataset-root \
        ./build/pet-dataset/ \
    --inference-batch-size \
        1 \
    --download-dataset \
    --verbosity \
        INFO

python -m kenning.scenarios.render_report \
    Pet Dataset classification using TVM-compiled TensorFlow model \
    docs/source/generated/local-cpu-tvm-tensorflow-classification.rst \
    --root-dir \
        docs/source/ \
    --img-dir \
        docs/source/generated/img \
    --report-types \
        performance \
        classification \
    --measurements \
        inferencetest/local-cpu-tvm-tensorflow-classification.json

General information for TensorFlowPetDatasetMobileNetV2-TVMCompiler-TVMRuntime

Model framework:

  • tensorflow ver. 2.5.0

Compiler framework:

  • tvm ver. 0.8.dev0

Input JSON:

Note

{
    "dataset": {
        "type": "kenning.datasets.pet_dataset.PetDataset",
        "parameters": {
            "classify_by": "breeds",
            "image_memory_layout": "NHWC",
            "dataset_root": "build/pet-dataset",
            "inference_batch_size": 1,
            "download_dataset": true,
            "external_calibration_dataset": null
        }
    },
    "model_wrapper": {
        "type": "kenning.modelwrappers.classification.tensorflow_pet_dataset.TensorFlowPetDatasetMobileNetV2",
        "parameters": {
            "model_path": "kenning/resources/models/classification/tensorflow_pet_dataset_mobilenetv2.h5"
        }
    },
    "runtime": {
        "type": "kenning.runtimes.tvm.TVMRuntime",
        "parameters": {
            "save_model_path": "build/compiled-model.tar",
            "target_device_context": "cpu",
            "target_device_context_id": 0,
            "runtime_use_vm": false,
            "disable_performance_measurements": true
        }
    },
    "optimizers": [
        {
            "type": "kenning.compilers.tvm.TVMCompiler",
            "parameters": {
                "model_framework": "keras",
                "target": "llvm",
                "target_host": null,
                "opt_level": 3,
                "libdarknet_path": "/usr/local/lib/libdarknet.so",
                "compile_use_vm": false,
                "output_conversion_function": "default",
                "conv2d_data_layout": "",
                "conv2d_kernel_layout": "",
                "use_fp16_precision": false,
                "use_int8_precision": false,
                "dataset_percentage": 0.25,
                "compiled_model_path": "build/compiled-model.tar"
            }
        }
    ]
}

Inference performance metrics for TensorFlowPetDatasetMobileNetV2-TVMCompiler-TVMRuntime

Inference time

Inference time

Fig. 1 Inference time

  • First inference duration (usually including allocation time): 0.14481551399990167,

  • Mean: 0.14180965715280916 s,

  • Standard deviation: 0.0010850960440062978 s,

  • Median: 0.14170252099984282 s.

Mean CPU usage

Mean CPU usage

Fig. 2 Mean CPU usage during benchmark

  • Mean: 50.504113407669294 %,

  • Standard deviation: 2.1429701172036686 %,

  • Median: 50.0 %.

Memory usage

Memory usage

Fig. 3 Memory usage during benchmark

  • Mean: 20.705031275496328 %,

  • Standard deviation: 0.19437767854233368 %,

  • Median: 20.8 %.

Inference quality metrics for TensorFlowPetDatasetMobileNetV2-TVMCompiler-TVMRuntime

Confusion matrix

Fig. 4 Confusion matrix

  • Accuracy: 0.957273098380732

  • Top-5 accuracy: 0.9961899578173902

  • Mean precision: 0.9571990916404879

  • Mean sensitivity: 0.9574413909758527

  • G-mean: 0.9569616085080422