Kazeia
/
kazeia
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
kazeia/executorch-patches
History
Kazeia Team f32b5ddfdd LLM no-root: validate end-to-end pipeline, fix kv_io_bit_width detection 
End-to-end validation on OnePlus Pad 3 with stream_llm intent:
  Prompt:   'Bonjour, comment vas-tu ?'
  Response: 'Bonjour ! Je suis là pour t'écouter. Comment vas-tu aujourd'hui ?'
  TTS:      Talker(PTE) 37ms/step, CP(PTE) 73ms/step, audio synthesized.
  No su, no Magisk prompts.

Two fixes since the previous commit:
1. ExecuTorchLlmEngine: pass echo=false to LlmModule.generate() — by default
   the runner echoes the prompt tokens back via the callback, which fed the
   ChatML wrap (<|im_start|>user …) into the SentenceStreamer and TTS.
2. jni_layer_llama.cpp: pick Runner<uint8_t> vs Runner<uint16_t> based on the
   model's get_kv_io_bit_width metadata, mirroring qnn_llama_runner.cpp main().
   The hard-coded uint16_t was wrong for our Qwen3-4B export (which uses 8-bit
   KV I/O) and produced fluent-looking but completely random tokens
   ("blocked罩ug darkestSOLEQuotes作者本人 …") — same symptom whether greedy or
   sampled, the smoking gun for a width-mismatched KV cache reinterpretation.

Other tweaks:
- temperature=0.0 in the QNN_LLAMA branch of jni_layer_llama.cpp (greedy,
  matches the working qnn_llama_runner --temperature 0 invocation)
- shared_buffer=true (same as binary defaults)
- Kotlin chat template mirrors qnn_llama_runner.cpp's get_formatted_prompt for
  Qwen3 (user-first, then optional system, then "<|im_start|>assistant" with
  no trailing newline — that quirky ordering is what the .pte was trained on)

TFTT is ~4 s for a 77-token prompt on kv-only mode (sequential prefill, one
forward per token). To get a sub-second TTFT we'd need to re-export the model
in --model_mode hybrid which adds a parallel prefill_forward graph; not
required for the conversational use case.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 		2026-04-14 11:11:23 +02:00
..
README.md LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
llm_in_process_jni.patch LLM no-root: validate end-to-end pipeline, fix kv_io_bit_width detection 2026-04-14 11:11:23 +02:00
qwen3_4b_decoder.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
torchtune_quantization.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00

README.md

Executorch patches for Kazeia

Local modifications to /opt/Kazeia/executorch (upstream pytorch/executorch @ v1.2.0) required to export Qwen3-4B to QNN for OnePlus Pad 3 (Snapdragon 8 Elite, Hexagon V79).

Not upstreamable as-is (phi_4_mini torchtune guard is a local dependency workaround; Qwen3_4B class matches upstream style but hasn't been submitted).

qwen3_4b_decoder.patch

Applied to: /opt/Kazeia/executorch/

cd /opt/Kazeia/executorch && git apply ../executorch-patches/qwen3_4b_decoder.patch

Adds:

  • examples/qualcomm/oss_scripts/llama/__init__.py:
    • try/except around convert_phi_4_mini_weights import (phi_4_mini pulls torchtune which conflicts with our torchao 0.17 pin).
    • New Qwen3_4B class registered as qwen3-4b, num_sharding=2 (4B at num_sharding=1 OOMed during QNN compile even with 48 GB free RAM; sharding=2 is the minimum that lets the compile partitioner split the HTP context).
  • examples/qualcomm/oss_scripts/llama/decoder_constants.py:
    • Adds "qwen3-4b": "qwen3" to DECODER_MODEL_VERSION.

torchtune_quantization.patch

Applied to: /opt/Kazeia/et_venv/lib64/python3.10/site-packages/torchtune/training/quantization.py

torchao 0.17+ removed int4_weight_only and int8_dynamic_activation_int4_weight. torchtune 0.6.1 still imports them. Since our Qwen3 QNN export path doesn't use either, wrap the import in try/except and set them to None on ImportError.

Host env reminders (not in patches)

  • symlink libc++.so.1 and libc++abi.so.1 in backends/qualcomm/sdk/libcxx-14.0.0/
  • copy build-x86/backends/qualcomm/PyQnn*.so to backends/qualcomm/python/
  • QNN_SDK_ROOT=/opt/Kazeia/executorch/backends/qualcomm/sdk/qnn
  • LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang:.../sdk/libcxx-14.0.0
  • PATH+=build-x86/third-party/flatc_ep/bin
  • PYTHONPATH=/opt/Kazeia

RAM/swap for 4B export

Peak RAM during prepare_pt2e + QNN compile: 46 GB anon-rss. On a 62 GB + 8 GB zram box this OOMs. Fix: add a swapfile:

sudo dd if=/dev/zero of=/swapfile bs=1M count=49152
sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Compile then uses ~59 GB RAM + 24 GB swap, completes in ~30 min wall. Put --artifact on /home not /tmp (the 25 GB decode_qdq.pt2 overflows tmpfs).