Kazeia
/
kazeia
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
kazeia/executorch-patches
History
Kazeia Team f4b15a72a7 LLM JNI: auto-detect eval_mode from .pte methods (kv-only vs hybrid) 
Replace the hardcoded eval_mode=0 in the QNN_LLAMA branch with a runtime
check on the loaded module's method names: if the .pte exposes a
prefill_forward graph, switch to EvalMode::kHybrid (1) — the runner can
then batch the entire prompt through prefill_forward in one parallel pass
instead of running 52 ms/token sequentially through kv_forward. Falls
back to kKVCached (0) when only kv_forward exists, matching the current
.pte behaviour exactly so this is a safe in-place upgrade ahead of the
hybrid re-export.

Sanity-tested with the kv-only Qwen3-4B .pte already on the tablet:
  Prompt 'Bonjour, ça va ?' → "Bonjour ! Ça va, merci de me demander ça.
  Tu as une question ?", TTFT 2728 ms, total 4158 ms — no change vs the
  hardcoded eval_mode=0 build.

Once the hybrid Qwen3-4B export finishes (~50 min compile, both
prefill_forward + kv_forward graphs), the same JNI binary will pick up
the new .pte and TTFT should drop to <1 s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 		2026-04-14 12:45:10 +02:00
..
README.md LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
llm_in_process_jni.patch LLM JNI: auto-detect eval_mode from .pte methods (kv-only vs hybrid) 2026-04-14 12:45:10 +02:00
qwen3_4b_decoder.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
torchtune_quantization.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00

README.md

Executorch patches for Kazeia

Local modifications to /opt/Kazeia/executorch (upstream pytorch/executorch @ v1.2.0) required to export Qwen3-4B to QNN for OnePlus Pad 3 (Snapdragon 8 Elite, Hexagon V79).

Not upstreamable as-is (phi_4_mini torchtune guard is a local dependency workaround; Qwen3_4B class matches upstream style but hasn't been submitted).

qwen3_4b_decoder.patch

Applied to: /opt/Kazeia/executorch/

cd /opt/Kazeia/executorch && git apply ../executorch-patches/qwen3_4b_decoder.patch

Adds:

  • examples/qualcomm/oss_scripts/llama/__init__.py:
    • try/except around convert_phi_4_mini_weights import (phi_4_mini pulls torchtune which conflicts with our torchao 0.17 pin).
    • New Qwen3_4B class registered as qwen3-4b, num_sharding=2 (4B at num_sharding=1 OOMed during QNN compile even with 48 GB free RAM; sharding=2 is the minimum that lets the compile partitioner split the HTP context).
  • examples/qualcomm/oss_scripts/llama/decoder_constants.py:
    • Adds "qwen3-4b": "qwen3" to DECODER_MODEL_VERSION.

torchtune_quantization.patch

Applied to: /opt/Kazeia/et_venv/lib64/python3.10/site-packages/torchtune/training/quantization.py

torchao 0.17+ removed int4_weight_only and int8_dynamic_activation_int4_weight. torchtune 0.6.1 still imports them. Since our Qwen3 QNN export path doesn't use either, wrap the import in try/except and set them to None on ImportError.

Host env reminders (not in patches)

  • symlink libc++.so.1 and libc++abi.so.1 in backends/qualcomm/sdk/libcxx-14.0.0/
  • copy build-x86/backends/qualcomm/PyQnn*.so to backends/qualcomm/python/
  • QNN_SDK_ROOT=/opt/Kazeia/executorch/backends/qualcomm/sdk/qnn
  • LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang:.../sdk/libcxx-14.0.0
  • PATH+=build-x86/third-party/flatc_ep/bin
  • PYTHONPATH=/opt/Kazeia

RAM/swap for 4B export

Peak RAM during prepare_pt2e + QNN compile: 46 GB anon-rss. On a 62 GB + 8 GB zram box this OOMs. Fix: add a swapfile:

sudo dd if=/dev/zero of=/swapfile bs=1M count=49152
sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Compile then uses ~59 GB RAM + 24 GB swap, completes in ~30 min wall. Put --artifact on /home not /tmp (the 25 GB decode_qdq.pt2 overflows tmpfs).