Kazeia
/
kazeia
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
kazeia/executorch-patches
History
Kazeia Team 67de8d4767 LLM: enable hybrid-mode export via num_sharding=1 — TTFT 2.9s → 113ms 
Re-exported Qwen3-4B in hybrid mode (prefill_forward + kv_forward) with
num_sharding=1 after discovering that sharding=2 produces a multi-context
.pte that the LlmModule loader cannot restore (error 5010 "Context group 1
does not exist"). Single-context hybrid .pte loads cleanly through the JNI
runner and the auto-detected eval_mode=1 path.

The peak RAM during export hit 49 GB, which is why sharding=2 was used
originally — the /swapfile (192 GB) now absorbs it. Compile wall time with
sharding=1 + hybrid is ~73 min (two graphs) vs ~30 min for sharding=2 +
kv-only (one graph).

End-to-end on tablet, same 'Bonjour, comment vas-tu ?' prompt:
  Before (kv-only, short prompt):  TTFT 2865 ms, total 4034 ms
  After  (hybrid, short prompt):   TTFT  113 ms, total 1471 ms
  Gain: -2752 ms TTFT (96% reduction, 25× faster)
  Response: "Bonjour ! Je vais bien, merci de me demander. Comment vas-tu ?"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 		2026-04-14 15:08:31 +02:00
..
README.md LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
llm_in_process_jni.patch LLM JNI: auto-detect eval_mode from .pte methods (kv-only vs hybrid) 2026-04-14 12:45:10 +02:00
qwen3_4b_decoder.patch LLM: enable hybrid-mode export via num_sharding=1 — TTFT 2.9s → 113ms 2026-04-14 15:08:31 +02:00
torchtune_quantization.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00

README.md

Executorch patches for Kazeia

Local modifications to /opt/Kazeia/executorch (upstream pytorch/executorch @ v1.2.0) required to export Qwen3-4B to QNN for OnePlus Pad 3 (Snapdragon 8 Elite, Hexagon V79).

Not upstreamable as-is (phi_4_mini torchtune guard is a local dependency workaround; Qwen3_4B class matches upstream style but hasn't been submitted).

qwen3_4b_decoder.patch

Applied to: /opt/Kazeia/executorch/

cd /opt/Kazeia/executorch && git apply ../executorch-patches/qwen3_4b_decoder.patch

Adds:

  • examples/qualcomm/oss_scripts/llama/__init__.py:
    • try/except around convert_phi_4_mini_weights import (phi_4_mini pulls torchtune which conflicts with our torchao 0.17 pin).
    • New Qwen3_4B class registered as qwen3-4b, num_sharding=2 (4B at num_sharding=1 OOMed during QNN compile even with 48 GB free RAM; sharding=2 is the minimum that lets the compile partitioner split the HTP context).
  • examples/qualcomm/oss_scripts/llama/decoder_constants.py:
    • Adds "qwen3-4b": "qwen3" to DECODER_MODEL_VERSION.

torchtune_quantization.patch

Applied to: /opt/Kazeia/et_venv/lib64/python3.10/site-packages/torchtune/training/quantization.py

torchao 0.17+ removed int4_weight_only and int8_dynamic_activation_int4_weight. torchtune 0.6.1 still imports them. Since our Qwen3 QNN export path doesn't use either, wrap the import in try/except and set them to None on ImportError.

Host env reminders (not in patches)

  • symlink libc++.so.1 and libc++abi.so.1 in backends/qualcomm/sdk/libcxx-14.0.0/
  • copy build-x86/backends/qualcomm/PyQnn*.so to backends/qualcomm/python/
  • QNN_SDK_ROOT=/opt/Kazeia/executorch/backends/qualcomm/sdk/qnn
  • LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang:.../sdk/libcxx-14.0.0
  • PATH+=build-x86/third-party/flatc_ep/bin
  • PYTHONPATH=/opt/Kazeia

RAM/swap for 4B export

Peak RAM during prepare_pt2e + QNN compile: 46 GB anon-rss. On a 62 GB + 8 GB zram box this OOMs. Fix: add a swapfile:

sudo dd if=/dev/zero of=/swapfile bs=1M count=49152
sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Compile then uses ~59 GB RAM + 24 GB swap, completes in ~30 min wall. Put --artifact on /home not /tmp (the 25 GB decode_qdq.pt2 overflows tmpfs).