Kazeia
/
kazeia
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
kazeia/executorch-patches
History
Kazeia Team 809a6d4fed LLM no-root: migrate to in-process LlmModule (JNI) — zero su calls 
The root cause of the previous su-c requirement was that Qualcomm's FastRPC
kernel driver rejects processes spawned via ProcessBuilder fork+exec because
they lose supplementary GIDs on exec. Zygote-forked app processes retain the
proper init-configured credentials and are accepted by the adsprpcd service,
which is why ORT-QNN (Whisper, in-process) worked while the subprocess
qnn_llama_runner did not. Running the LLM in-process via ExecuTorch's
LlmModule bypasses the fork+exec path entirely.

What this commit does:
- ExecuTorchLlmEngine now uses org.pytorch.executorch.extension.llm.LlmModule
  with MODEL_TYPE_QNN_LLAMA=4 (routes to example::Runner in jni_layer_llama.cpp,
  the same C++ runner that qnn_llama_runner embeds).
- All su, ProcessBuilder, file-based prompt/response plumbing, and run_llm.sh
  gone. ChatML template is built in Kotlin; tokens stream in via LlmCallback.

Supporting changes under executorch-patches/llm_in_process_jni.patch:
1. backends/qualcomm/CMakeLists.txt — gate PyQnnManagerAdaptor on NOT ANDROID.
   The original guard (CMAKE_SYSTEM_PROCESSOR MATCHES x86_64) misfires in a
   nested scope during Android cross-compile and tried to build the host
   Python bindings.
2. extension/android/jni/jni_layer_llama.cpp — hardcode decoder_model="qwen3"
   (was "llama3") and pass eval_mode=0 (EvalMode::kKVCached) + shared_buffer=true
   to match our hybrid_llama_qnn.pte which only contains kv_forward, not
   prefill_forward.

Build: scripts/build_android_library.sh arm64-v8a with QNN_SDK_ROOT pointing
to /opt/Kazeia/qnn_sdk_242/qairt/2.42.0.251225 and EXECUTORCH_BUILD_QNN=ON.
Produces libexecutorch_jni.so (192 MB) with QNN v2.42 backend + the llama
runner code, plus libqnn_executorch_backend.so. Both staged in jniLibs.

Validated on OnePlus Pad 3: LlmModule.load() completes in 4.2 s, no su
prompts, Pipeline ready with STT(WhisperHybridEngine) → [VoiceCommands →
LLM] → TTS(Qwen3TtsEngine). TTS .pte still loads with the upgraded v2.42
runtime — no regression.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 		2026-04-14 10:39:50 +02:00
..
README.md LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
llm_in_process_jni.patch LLM no-root: migrate to in-process LlmModule (JNI) — zero su calls 2026-04-14 10:39:50 +02:00
qwen3_4b_decoder.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00
torchtune_quantization.patch LLM NPU: Qwen3-4B QNN export patches + deployment notes 2026-04-13 22:56:42 +02:00

README.md

Executorch patches for Kazeia

Local modifications to /opt/Kazeia/executorch (upstream pytorch/executorch @ v1.2.0) required to export Qwen3-4B to QNN for OnePlus Pad 3 (Snapdragon 8 Elite, Hexagon V79).

Not upstreamable as-is (phi_4_mini torchtune guard is a local dependency workaround; Qwen3_4B class matches upstream style but hasn't been submitted).

qwen3_4b_decoder.patch

Applied to: /opt/Kazeia/executorch/

cd /opt/Kazeia/executorch && git apply ../executorch-patches/qwen3_4b_decoder.patch

Adds:

  • examples/qualcomm/oss_scripts/llama/__init__.py:
    • try/except around convert_phi_4_mini_weights import (phi_4_mini pulls torchtune which conflicts with our torchao 0.17 pin).
    • New Qwen3_4B class registered as qwen3-4b, num_sharding=2 (4B at num_sharding=1 OOMed during QNN compile even with 48 GB free RAM; sharding=2 is the minimum that lets the compile partitioner split the HTP context).
  • examples/qualcomm/oss_scripts/llama/decoder_constants.py:
    • Adds "qwen3-4b": "qwen3" to DECODER_MODEL_VERSION.

torchtune_quantization.patch

Applied to: /opt/Kazeia/et_venv/lib64/python3.10/site-packages/torchtune/training/quantization.py

torchao 0.17+ removed int4_weight_only and int8_dynamic_activation_int4_weight. torchtune 0.6.1 still imports them. Since our Qwen3 QNN export path doesn't use either, wrap the import in try/except and set them to None on ImportError.

Host env reminders (not in patches)

  • symlink libc++.so.1 and libc++abi.so.1 in backends/qualcomm/sdk/libcxx-14.0.0/
  • copy build-x86/backends/qualcomm/PyQnn*.so to backends/qualcomm/python/
  • QNN_SDK_ROOT=/opt/Kazeia/executorch/backends/qualcomm/sdk/qnn
  • LD_LIBRARY_PATH=$QNN_SDK_ROOT/lib/x86_64-linux-clang:.../sdk/libcxx-14.0.0
  • PATH+=build-x86/third-party/flatc_ep/bin
  • PYTHONPATH=/opt/Kazeia

RAM/swap for 4B export

Peak RAM during prepare_pt2e + QNN compile: 46 GB anon-rss. On a 62 GB + 8 GB zram box this OOMs. Fix: add a swapfile:

sudo dd if=/dev/zero of=/swapfile bs=1M count=49152
sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

Compile then uses ~59 GB RAM + 24 GB swap, completes in ~30 min wall. Put --artifact on /home not /tmp (the 25 GB decode_qdq.pt2 overflows tmpfs).