Large Language Models with Safetensors¶

This example demonstrates how to optimize large language models using the safetensors format. This is useful when deploying quantized LLMs by quickly switching between different quantization methods, such as GPTQ and AWQ. For the purpose of demonstration, SmolLM2 is used for quick optimization and report generation.

Setup¶

To get started, clone the Kenning repository:

git clone https://github.com/antmicro/kenning.git
cd kenning/

and install it with

uv pip install ".[llm,sparsity-aware-kernel,reports]"

Info

This example requires a CUDA-enabled GPU.

Depending on the example, you may also need:

CUDA Toolkit (13.0 recommended), including NVCC and NVRTC, to compile CUDA code.
cuDNN (9 recommended) for operations that use cuDNN.
NVML to generate GPU reports.

Check the example to see which of these components are needed.

AWQ Optimization¶

The first example shows how to use AWQ optimization to protect important weights in order to maintain model accuracy. The configuration specifies the model_framework as safetensors to ensure compatible loading.

Use the following configuration:

Listing 6 llm-smol-awq.yaml¶

dataset:
  type: kenning.datasets.cnn_dailymail.CNNDailymailDataset
  parameters:
    dataset_root: ./build/CNNDailymailDataset
    inference_batch_size: 128
    gather_predictions: true
    split_fraction_test: 0.001
model_wrapper:
  type: kenning.modelwrappers.llm.smol.SmolLM2
  parameters:
    model_path: hf://HuggingFaceTB/SmolLM2-135M-Instruct
optimizers:
  - type: kenning.optimizers.awq.AWQOptimizer
    parameters:
      compiled_model_path: ./build/smollm2_awq_4bit
      target_precision: 4
      group_size: 64
runtime:
  type: kenning.runtimes.vllm.VLLMRuntime
  parameters:
    save_model_path: ./build/smollm2_awq_4bit
    max_tokens: 8192

Run the optimization flow:

kenning optimize --cfg llm-smol-awq.yaml

SparseGPTQ Optimization¶

Kenning supports SparseGPTQ which combines quantization with structured pruning. It is useful when reduction in both the memory footprint and computational cost is required. This approach allows the LLM to run on hardware very limited RAM and processing power.

Use the following configuration:

Listing 7 llm-smol-sparsegptq.yaml¶

dataset:
  type: kenning.datasets.cnn_dailymail.CNNDailymailDataset
  parameters:
    dataset_root: ./build/CNNDailymailDataset
    inference_batch_size: 128
    gather_predictions: true
    split_fraction_test: 0.001
model_wrapper:
  type: kenning.modelwrappers.llm.smol.SmolLM2
  parameters:
    model_path: hf://HuggingFaceTB/SmolLM2-135M-Instruct
optimizers:
  - type: kenning.optimizers.awq.GPTQSparseGPTOptimizer
    parameters:
      compiled_model_path: ./build/smollm2_sparsegptq_4bit
      group_size: 64
runtime:
  type: kenning.runtimes.vllm.VLLMRuntime
  parameters:
    save_model_path: ./build/smollm2_sparsegptq_4bit
    max_tokens: 8192
    sparse_gptq_kernel: true
    max_model_len: 1024

Run the optimization flow:

kenning optimize --cfg llm-smol-sparsegptq.yaml

vLLM Runtime¶

The resulting safetensors models can be deployed using the vLLM runtime. vLLM is required to perform the tests and evaluation.

Results¶

First, perform the evaluation for each model:

kenning test \
  --cfg llm-smol-awq.yaml \
  --measurements build/awq.json

and if you want to evaluate sparsegpt tests:

kenning test \
  --cfg llm-smol-sparsegptq.yaml \
  --measurements build/sparsegptq.json

Then, generate the comparison report:

kenning report \
  --measurements \
    build/awq.json \
    build/sparsegptq.json \
  --report-path build/llm-comparison.md \
  --to-html

Last update: 2026-10-29