An inference engine leveraging Arc graphics.
中文

A high-performance LLM inference engine for Intel Arc GPUs, built with SYCL + oneDNN. Features hand-optimized kernels for bitsandbytes 4-bit (NF4) quantized models, fused dequant+matmul, and GPU-accelerated DeltaNet recurrence for Qwen3.5 hybrid architectures.

• ⚡ Bitsandbytes 4-bit (NF4) inference — run quantized models directly on Intel Arc with fused dequant+matmul kernels, hand-written GEMV decode, and fused gate+up+SiLU projection
• 🔢 Bfloat16 inference — dense (unquantized) models via oneDNN matmul primitives
• 🏗️ Qwen3.5 Hybrid architecture — full GPU acceleration for the dual attention (GQA + DeltaNet linear attention) architecture
• 📐 Qwen3 Dense architecture — standard Transformer with GQA, QK-norm, and SwiGLU FFN
• 👁️ Vision (Qwen3.5) — image understanding with CPU preprocessing and GPU vision transformer
• 🎙️ Audio (Qwen3-ASR) — speech-to-text transcription with audio preprocessing and GPU-accelerated audio encoder. Supports both offline wav transcription and real-time streaming input ASR
• 🎯 Forced Alignment — word-level timestamp alignment from audio + text, with CJK character-level tokenization and LIS-based timestamp correction
• 🔊 Audio (Qwen3-TTS) — text-to-speech synthesis with Mimi Vocoder and native zero-shot voice cloning. Supports direct raw audio WAV generation, voice cloning via reference audio, and offline Mimi decoding
• 🎤 CustomVoice / VoiceDesign — named speaker presets (vivian, ryan, etc.) for instant voice selection and text-described voice styles via VoiceDesign for creative TTS control
• 🔉 Streaming TTS — real-time raw 24kHz mono f32 PCM audio output to stdout for low-latency streaming speech synthesis
• ⚡ Native bf16 GEMV kernel — hand-optimized SG=16 vec8+FMA bf16 GEMV for TTS decode, delivering 9x faster TTS (RTF 8.08 to 0.89 on 0.6B)
• 🔄 Streaming output — token-level streaming callback with abort support
• 💬 Interactive CLI — multi-turn conversation with runtime commands (/clear, /greedy, /sample, etc.)
• 📊 Benchmark mode — measure prefill and decode throughput separately
• 🔌 C API — stable C FFI interface (Python, C#, Rust, Go, Java) — see docs/C_API.md
• 💭 Chat formatting — llama.cpp-style Jinja rendering, fixed Qwen3.5 template, structured reasoning/tool-call parsing, and JSONL chat stream events

Other Qwen3 / Qwen3.5 model sizes may work if they match the supported architecture pattern.

• Intel Arc A770 (16 GB) — primary development and test target
• Other Intel Arc discrete GPUs (A750, A580, A380, B580) with ≥8 GB VRAM should work
• Integrated GPUs (Xe-LP, Xe-LPG) may work for small models but are not tested

• Windows 10 22H2 or later / Windows 11

• Intel Arc Graphics Driver
• The Aila-vX.Y.Z-win64.zip release bundle includes all required runtime DLLs

1. Install the latest Intel Arc Graphics Driver.
2. Download Aila-vX.Y.Z-win64.zip from the Releases page.
3. Extract to a directory of your choice.
4. Place your model files in a directory (e.g. ./models/qwen3.5-0.8B-bnb-nf4-offline/).

The Windows release uses a split runtime layout:

integration_root/
|-- AilaShared.dll
`-- aila_runtime/
    |-- AilaWorker.exe
    |-- Aila.exe
    `-- <oneAPI and other runtime DLLs>

AilaShared.dll is a C ABI proxy without oneAPI inference imports. Each initialized engine runs inference in its own AilaWorker.exe worker process. For C API embedding, set AILA_RUNTIME_DLL_DIR=aila_runtime before loading the proxy. Relative values resolve relative to AilaShared.dll; absolute paths are also accepted and are normalized internally. If the variable is unset or empty, the proxy looks beside itself, preserving the legacy flat layout. That fallback is for old deployment compatibility, not DLL-search isolation: exposing a flat directory exposes both the proxy and private runtime. Python, ComfyUI, and other embedding hosts must use the split layout and explicitly set AILA_RUNTIME_DLL_DIR. Non-Windows builds continue to use in-process inference.

Python example:

import ctypes
import os

os.environ["AILA_RUNTIME_DLL_DIR"] = "aila_runtime"
lib = ctypes.CDLL(r".\AilaShared.dll")
lib.aila_engine_create.restype = ctypes.c_void_p
engine = lib.aila_engine_create()

Only add the proxy directory to the host's DLL search path. Never call os.add_dll_directory("aila_runtime") or add aila_runtime to the host PATH; that would expose Aila's private oneAPI DLLs to Python and defeat isolation. The existing generation, ASR, alignment, and Aila-owned memory-release entry points remain source-compatible. The TTS streaming API has changed in 0.1.7: it now returns an AilaTTSStream*, and wait/destroy operate on that stream handle; see docs/C_API.md for the migration.

The worker uses the runtime directory as its working directory and receives an isolated child PATH. Missing AilaWorker.exe, proxy/worker build ID mismatch, startup timeout, and worker exit are returned by the existing C API error functions. Failed operations are not automatically retried. Keep AilaShared.dll and AilaWorker.exe from the same release; cross-version worker compatibility is not guaranteed. CLI examples below assume Aila.exe is run from aila_runtime/ or is otherwise addressed by its full path.

ASR and TTS stream handles retain shared ownership of their engine. Destroy ASR streams, and wait for or cancel then destroy TTS streams, before destroying AilaEngine; otherwise, worker shutdown can be delayed until the final stream handle is destroyed.

Benchmark on Intel Arc A770 16 GB, Qwen3.5-4B, pp=2048 tg=1024:

Aila delivers the highest prefill throughput and competitive decode performance against Vulkan while using a more accurate NF4 4-bit quantization that retains vision capabilities.

TTS benchmark on Intel Arc A770 16 GB, Qwen3-TTS-12Hz-0.6B-Base:

Real-time factor (RTF) < 1 means the engine synthesizes speech faster than real-time. The native bf16 GEMV kernel delivers a 9x TTS speedup.

# Interactive conversation
Aila.exe -m ./models/Qwen3.5-4B-BNB-NF4-with-vision

# Offline audio transcription (ASR)
Aila.exe -m ./models/Qwen3-ASR-1.7B --transcribe input.wav

# Forced alignment (word-level timestamps)
Aila.exe -m ./models/Qwen3-ForcedAligner-0.6B-BNB-NF4 --align-text "你好世界" --align-audio input.wav --align-lang Chinese

# Text-to-speech synthesis (TTS)
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-Base --synthesize "Hello world!" --output-wav output.wav

# TTS with voice cloning (zero-shot)
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-Base --synthesize "今天天气真好。" --ref ./reference_speaker.wav --output-wav cloned.wav

# Single prompt from JSON file
Aila.exe -m ./models/Qwen3.5-4B-BNB-NF4-with-vision --messages-json prompt.json

# Single prompt from stdin
echo '{"messages":[{"role":"user","content":"hello"}]}' | Aila.exe -m ./models/qwen3.5-0.8B-bnb-nf4-offline --messages-json -

# Benchmark (greedy)
Aila.exe -m ./models/Qwen3.5-4B-BNB-NF4-with-vision --bench --bench-pp 512 --bench-tg 128

# Benchmark (sampling)
Aila.exe -m ./models/Qwen3.5-4B-BNB-NF4-with-vision --bench --sample

--messages-json accepts OpenAI-style chat requests with messages, tools, tool_choice, chat_template_kwargs, and generation parameters. Qwen3.5 Hybrid models use Aila's built-in fixed Qwen3.5 Jinja template unless a request or environment override is provided.

By default --messages-json prints raw assistant text. Add --chat-output-json to print structured assistant JSON with content, reasoning_content, tool_calls, raw_text, finish_reason, warnings, and metadata. Use --chat-stream-jsonl instead to print structured stream events as newline-delimited JSON.

Aila formats and parses tool calls but does not execute tools. Callers should execute returned tool_calls externally and send tool results back as tool messages. tool_policy can be "warn" or "strict"; strict policy marks violations with finish_reason: "tool_policy". Reasoning can be bounded with reasoning_budget / --thinking-budget. C API callers can use aila_generate_chat_json_ex and aila_generate_chat_json_stream_ex with AilaGenConfigV2 for ABI-safe chat options. Streaming callers can use the final JSONL/C API event with finish_reason: "tool_calls" as the handoff point for executing tools and starting the next request.

Aila supports zero-shot voice cloning for Qwen3-TTS models. The speaker embedding is extracted via a native C++ ECAPA-TDNN encoder (no Python dependency).

Reference audio requirements:

# Clone a voice from reference audio
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-Base `
    --synthesize "你好世界" `
    --ref ./reference_speaker.wav `
    --output-wav cloned_output.wav

# CustomVoice — use a named speaker preset
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-CustomVoice `
    --speaker vivian `
    --synthesize "Hello world!" `
    --output-wav vivian_output.wav

# VoiceDesign — describe the voice style in natural language
Aila.exe -m ./models/Qwen3-TTS-12Hz-1.7B-VoiceDesign `
    --instruct "A deep, warm voice with a slow pace" `
    --synthesize "Hello world!" `
    --output-wav styled_output.wav

# CustomVoice + language + instruct override
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-CustomVoice `
    --speaker ryan `
    --language english `
    --instruct "whispering softly" `
    --synthesize "Hello" `
    --output-wav ryan_english.wav

# Streaming TTS — real-time PCM audio output
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-Base `
    --stream-tts --synthesize "Hello!" 2>/dev/null | pcm_play

# Streaming TTS with custom batch size
Aila.exe -m ./models/Qwen3-TTS-12Hz-0.6B-Base `
    --stream-tts --stream-batch 8 --synthesize "Hello!" 2>/dev/null | pcm_play

The --rep-penalty flag controls repetition penalty (auto-set to 1.1 for TTS). Increase it (e.g. --rep-penalty 1.3) if the output has repetitive artifacts, or decrease it (e.g. --rep-penalty 1.0) for more variation.

Speaker embeddings are automatically cached: in-memory for the session lifetime, and persisted to disk (as <audio_path>.ref.bin) to avoid re-extraction on subsequent runs. Use --ref-cache-dir <dir> or AILA_REF_CACHE_DIR to store cache files in a centralized directory.

Supports standard OpenAI compatible JSON objects with "messages" as the top-level key, which also allows passing sampling/generation parameters directly (e.g. temperature, max_tokens/max_new_tokens, seed/do_sample):

{
  "messages": [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user",   "content": [{"type": "text", "text": "Introduce yourself"}]}
  ],
  "temperature": 0.7,
  "max_tokens": 128
}

It also backward-compatibly supports a raw JSON array as the top-level key: [{"role": "user", "content": "..."}].

Supports text, image, audio, and video content types. Image parts accept image, image_url (with base64 Data URI or file path), or {"image_url":{"url":"..."}}. Audio parts accept audio, audio_url (with base64 Data URI or file path), or {"input_audio": {"data": "base64", "format": "wav"}}.

See docs/C_API.md for the full C API reference (Python ctypes, C# P/Invoke, Rust FFI, etc.).

See docs/Environment_Variables.md for the complete list of configuration variables.

Use export_bnb_nf4.py to quantize a Hugging Face model to BNB NF4 format:

# Text-only model
python export_bnb_nf4.py \
    --source Qwen/Qwen3.5-0.8B \
    --output ./models/Qwen3.5-0.8B-BNB-NF4

# Model with vision
python export_bnb_nf4.py \
    --source Qwen/Qwen3.5-4B \
    --output ./models/Qwen3.5-4B-BNB-NF4-with-vision \
    --vision

# ASR model
python export_bnb_nf4.py \
    --source Qwen/Qwen3-ASR-1.7B \
    --output ./models/Qwen3-ASR-1.7B-BNB-NF4

# ForceAligner model (auto-detected, classify_head kept dense)
python export_bnb_nf4.py \
    --source Qwen/Qwen3-ForcedAligner-0.6B \
    --output ./models/Qwen3-ForcedAligner-0.6B-BNB-NF4

# From local directory, overwriting existing export
python export_bnb_nf4.py \
    --source ./Qwen3-0.6B \
    --output ./models/Qwen3-0.6B-BNB-NF4 \
    --overwrite

Requirements: torch, transformers, bitsandbytes with Intel XPU backend.

# Requires: Intel oneAPI Base Toolkit 2026.1+, CMake 3.24+, Ninja
.\build.ps1

# Clean build
.\build.ps1 -Clean

# Debug build
.\build.ps1 -Config Debug

Outputs:

On Windows, cmake --build build --target release stages the integration layout under build/Release/bin: only the proxy is at the root, and the CLI, worker, and runtime DLLs are under aila_runtime/. Build provenance remains in build/build_info.json and is not part of the release bundle.

Aila/
├── include/
│   ├── aila_api.h              # Public C API header
│   └── engine/Engine.hpp       # InferenceEngine class
├── src/
│   ├── main.cpp                 # CLI entry point
│   ├── api/aila_api.cpp         # C API implementation
│   ├── cli/                     # CLI argument parsing & interactive loop
│   ├── core/                    # SYCL context & tensor management
│   ├── memory/                  # KV cache
│   ├── models/                  # Model backends (Qwen3, Qwen3.5, BNB4)
│   ├── ops/                     # SYCL kernels (attention, RMSNorm, Bnb4BitLinear, etc.)
│   ├── profile/                 # Logging, profiling & device info
│   ├── audio/                   # Audio preprocessing (ASR) + speaker encoder (TTS voice cloning)
│   ├── utils/                   # Tokenizer, SafeTensors, memory-mapped I/O
│   └── vision/                  # Vision encoder (Qwen3.5)
├── docs/
│   ├── C_API.md                 # C API documentation
│   └── Environment_Variables.md # Environment variable reference
├── third_party/simdjson/        # JSON parsing
├── build.ps1                    # Build script
├── bench.ps1                    # Benchmark script
├── smoke.ps1                    # Smoke test script
└── CMakeLists.txt

• oneDNN — Intel's deep neural network library providing the matmul primitives used for bf16 inference
• bitsandbytes — NF4 quantization format and dequantization reference
• simdjson — Fast JSON parser used for model config and tokenizer metadata
• dr_libs — Single-header audio decoding libraries (dr_wav, dr_mp3, dr_flac) used for ASR audio preprocessing
• llama.cpp jinja module — jinja chat template parser module, used for parsing and building chat template

See LICENSE for details.