Quickstart

Prerequisites

The Python SDK installed — see Install.
Familiarity with runtime choice — qairt for Qualcomm AI Hub Models, llama_cpp for any GGUF.

The SDK follows the same design as Hugging Face transformers — load with AutoModelForCausalLM.from_pretrained(), then call .generate().

LLM inference (GGUF)

Any GGUF model from Hugging Face runs via llama_cpp. Model weights are downloaded on first use.

from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B-GGUF",     # HF repo id of a GGUF model, or a local .gguf path
    device_map="auto",          # "auto" | "cpu" | "gpu" | "npu" | "hybrid"
                                # | "<runtime>" | "<runtime>:<compute-unit>"
                                # auto -> npu for both llama_cpp and qairt
)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
      f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")

# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

LLM inference (QAIRT)

Pre-compiled bundles from Qualcomm AI Hub run entirely on the Hexagon NPU via the qairt runtime. Use device_map="qairt" (or "npu"). Model weights are downloaded on first use.

from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "ai-hub-models/Qwen3-4B",   # Qualcomm AI Hub model id
    device_map="qairt",         # NPU-only
)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
      f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")

# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

VLM inference (QAIRT)

Download a sample image first:

curl -o demo.jpg https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/demo.jpg

Then run inference:

import os
from geniex import AutoModelForCausalLM

image_path = os.path.abspath("demo.jpg")

model = AutoModelForCausalLM.from_pretrained(
    "ai-hub-models/Qwen2.5-VL-7B-Instruct",  # Qualcomm AI Hub VLM bundle
    device_map="qairt",
)
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image_path},
        {"type": "text", "text": "Describe the image."},
    ],
}]
prompt = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)

streamer = model.generate(prompt, images=[image_path], max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

Jupyter notebook walkthrough

For laptop users, follow the step-by-step Jupyter notebook at examples/python/windows.ipynb — it covers environment setup and inference end-to-end.

Next steps

API reference

All classes, methods, and parameters for the Python SDK.

Models

Supported models, GGUF on Hugging Face, and self-converted Qualcomm AI Engine Direct bundles.

Was this page helpful?

Yes

Get Started

Run

Tutorials

Resources

Prerequisites

LLM inference (GGUF)

LLM inference (QAIRT)

VLM inference (QAIRT)

Jupyter notebook walkthrough

Next steps

API reference

Models

​Prerequisites

​LLM inference (GGUF)

​LLM inference (QAIRT)

​VLM inference (QAIRT)

​Jupyter notebook walkthrough

​Next steps

API reference

Models

Prerequisites

LLM inference (GGUF)

LLM inference (QAIRT)

VLM inference (QAIRT)

Jupyter notebook walkthrough

Next steps