« Reste optionnel » : tenté le streaming LLM→TTS pour CosyVoice (synthèse de la
1ʳᵉ phrase pendant que le LLM décode, comme le chemin Qwen TTS-B).
Mesuré device : CONTRE-PRODUCTIF. LLM decode (6 threads CPU) et synthèse
CosyVoice (8 threads CPU) saturent tous les cœurs ; les chevaucher écroule le
LLM (17 → 0,55 tok/s, tour 1,1s → 41s, 1er son à 43s). Reverté.
Le streaming LLM→TTS n'a de sens que si les 2 étages utilisent des compute
distincts (Qwen3 NPU + TTS CPU). Pour CosyVoice (tout CPU) : laisser le LLM
finir vite PUIS synthesizeAndPlay, qui streame déjà phrase-par-phrase EN INTERNE
(synthèse N+1 pendant lecture N, sans contention). Net diff = un commentaire
garde-fou dans KazeiaService pour ne pas re-tenter ; code de streaming retiré.
Re-validé batch : LLM 17,1 tok/s (1,9s), TTS 1er son ~8s (RTF CosyVoice inhérent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bug : sessionReset()+sessionAsk() renvoyait VIDE. Cause = q35-lmq4 est l'archi
hybride Qwen3.5 (DeltaNet/SSM) ; llama_memory_seq_rm d'un suffixe « Returns false
if a partial sequence cannot be removed » (l'état récurrent ne se rembobine pas
partiellement). Le code ignorait ce retour → KV incohérent → tour suivant vide.
Fix natif (kazeia-engine dist/jni/kazeia_engine_jni.cpp, .so rebuildé CPU-only,
drop-in) : sessionReset teste le retour de seq_rm ; s'il échoue (récurrent/
hybride) → reset COMPLET (llama_memory_clear + re-prefill du system mémorisé) +
llama_sampler_reset. sessionStart mémorise désormais les tokens system.
App : EngineLlmAdapter.resetSession() rebascule sur le session.reset() natif
(au lieu du contournement newSession). Validé device : après reset, le tour
n'est plus vide ET la mémoire est effacée (ne connaît plus le prénom),
prefill ~285ms, decode ~17 tok/s, tours suivants OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Intègre les 3 APIs livrées par l'engine (libkazeia_engine.so drop-in, façade
EngineLlmEngine.kt : generateStream / newSession / LlmSession / GenStats).
EngineLlmAdapter : crée une LlmSession au load (system prefillé 1×), et chaque
generate() fait session.ask() en STREAMING (onToken câblé au pipeline existant).
Le system n'est plus re-prefillé à chaque tour.
Mesuré device (q35-lmq4, t=6, v0.1.7) :
- tour : 2991→~1100 ms (prefill 2700→~300 ms, system caché). Decode réel
17 tok/s (le « 3,2 » était l'artefact prefill-inclus ; tps vient maintenant
de getLastStats, débit décode).
- streaming callback OK sous vraie JVM (le seul point que le dev n'avait pas
pu tester) — aucun crash.
- mémoire conversationnelle (bonus) : rappelle « Richard » sur 3 tours.
Confidentialité : la mémoire est vidée aux bornes de conversation —
EngineLlmAdapter.resetSession() (recrée la session) appelé sur CLEAR_CHAT et
sur changement de profil actif (onProfileStoreChanged). Validé : après switch
de profil, le LLM ne connaît plus le prénom.
⚠ Bug natif à corriger côté engine : sessionReset()+sessionAsk() renvoie vide
(n_past non restauré à n_sys après seq_rm). Contourné en recréant la session
(newSession). Le mode mémoire (ask→ask sans reset) est, lui, OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Directive 2026-06-15 : si on passe par kazeia-engine, on n'utilise plus de .pte.
- ConfigStore.llmEngine défaut "prod"(.pte) → "lib" (EngineLlmAdapter +
libkazeia_engine + GGUF). Le .pte n'est plus chargé au runtime ; il reste
câblé en repli d'urgence seulement (et sur cette plateforme son runner QNN
échoue de toute façon : « Failed to load llm runner [1] »).
- ModelRegistry.defaultSpeaker() "qwen3-4b-seq1024"(.pte) → "qwen3.5-4b" (GGUF,
q35-lmq4) : config cohérente, le mode lib charge sp.ggufPath() directement
sans repli, zéro référence .pte.
Validé device (release v0.1.6, SM8750) : libkazeia_engine charge q35-lmq4.gguf
en 1.8 s (backend='lib'), 2 générations FR thérapeutiques cohérentes
(« C'est compréhensible, je suis là avec toi… » / « C'est vraiment épuisant,
je te comprends. »). Règle aussi le mode perroquet observé avec le .pte.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
La voix n'est plus choisie côté patient ; elle est définie PAR PROFIL dans l'app
admin (le champ Profile.voiceId + le dropdown admin existaient déjà ; seul le
runtime ne l'appliquait pas).
App patient :
- ChatActivity : suppression de setupVoiceSelector() + des listes voiceFiles/
voiceNames/voiceColors. activity_chat.xml : suppression de la voiceBar (spinner),
l'orbe se contraint désormais sous tvStatus (couleur = défaut service).
- KazeiaService : nouvelle source unique = le profil actif.
- applyActiveProfileVoice() lit ProfileStore.activeProfile().voiceId (défaut
DEFAULT_VOICE_ID="damien" en invité), appelée à l'init ET via un listener
ProfileStore → propagation LIVE quand l'admin édite la voix / change de profil.
- setVoiceId(id) résout selon le moteur actif : CosyVoice → cosyvoice/<id>.cvps ;
Qwen3/lib → ../voix/<id>.wav. setVoice(pathOrId) conservé (intents test) délègue.
Validé device (release v0.1.5, SM8750, moteur cosyvoice) :
- plus de barre de voix dans l'UI patient (confirmé).
- profil voice_id=elodie → service charge 'elodie' (≠ défaut) ; changement live
via provider voice_id=damien → ré-appliqué sans redémarrage.
- moteur 'lib' charge toujours (pas de régression du stack gelé).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Câblage Kotlin/APK du moteur TTS clonage vocal CosyVoice3 distillé livré par
kazeia-engine (dist/cosyvoice). Mirror du pattern sttEngine/ttsEngine="lib".
- CosyVoiceTtsEngine.kt : CosyVoiceJni (5 symboles) + adapter implémentant
com.kazeia.core.TtsEngine. Cache de voix (.cvps), synthèse phrase-par-phrase,
streaming AudioTrack MODE_STREAM en ENCODING_PCM_FLOAT @24 kHz (1er son joué
pendant que la phrase suivante se génère ; RTF~1.4 masqué par le streaming),
suspension jusqu'au drain réel (marker) pour garder le micro coupé (anti-écho).
- KazeiaService : branche when(ttsEngine) { "cosyvoice" -> ... } à l'init.
- ConfigStore : doc du nouveau backend.
⚠ NON validé device : BLOQUEUR côté libs engine. Les .so livrés embarquent leurs
propres libggml*.so (SONAME identiques au stack LLM/TTS gelé, 535 symboles ggml_*
en collision) → interposition ELF dans le namespace JNI (RTLD_GLOBAL) : le premier
libggml chargé (LLM, au boot) gagnerait, libcosyvoice se lierait au mauvais ggml →
crash. Ni écraser ni omettre les libggml app ne marche (consommés par libllama/
libkazeia_engine/libkazeia_tts gelés). Fix attendu : ggml statique + visibilité
hidden dans libcosyvoice.so côté engine. Code prêt à valider dès libs corrigées.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deux bugs de prod trouvés en validant le cycle OTA corpus r2 :
1. ttsEngine default "prod" → toute install fraîche (release patient) chargeait
Qwen3TtsEngine legacy → SIGSEGV déterministe dans llama_context ctor
(libtts_talker_cpu compilé contre llama-upstream, libllama.so embarqué =
fork ql depuis 2026-06-02, dérive ABI llama_context_params). La tablette
release crash-loopait au démarrage du service depuis la migration ; la
tablette dev ne le voyait pas (config tts_engine=lib persistée).
→ default "lib" (seul backend compatible avec les jniLibs livrés).
2. Le corpus RAG était résolu via MODELS_DIR/../rag_corpus : sur tablette dev,
MODELS_DIR = legacy /data/local/tmp (modèles adb) → syncDir lisait l'ancien
corpus alors que l'OTA dépose le composant rag_corpus dans le stockage
externe. → KazeiaPaths.ragCorpusDir avec priorité INVERSE des modèles :
l'externe (installeur/OTA) prime dès qu'il est non vide, fallback legacy.
versionCode 4 / 0.1.3, publié catalog v6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
L'auto-ingestion ne jouait qu'à base vide (1er démarrage) : une MAJ OTA du
composant rag_corpus déposait les nouveaux fichiers mais le RAG continuait de
servir l'ancien corpus, sans aucun déclencheur (sauf intent manuel rag_ingest).
Rag.syncDir(dir) : ingère tout fichier .txt/.md absent de la base ou dont le
texte diffère du brut stocké (comparaison exacte). Idempotent, n'embed que le
delta → appelé à CHAQUE buildRag, remplace le cas spécial « base vide ».
Ne supprime jamais (docs admin et docs retirés du dossier restent — suppression
via l'admin).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code-review (2e passe) :
- Faux négatif CrisisGuard : « je serais/suis mieux mort », « mieux mort que
vivant » ne déclenchaient aucun motif. Ajout du motif « mieux mort » + 3 cas
de test. (Sécurité vitale — sensibilité d'abord.)
- normalize() recompilait 3 Regex à chaque appel (1×/message patient) :
hoistées en constantes (MARKS/APOS/SPACES).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rag.ingest : embed D'ABORD, n'écrit la base qu'ensuite, via la nouvelle
RagDb.replaceSource (delete+insert dans UNE transaction). Un échec transitoire
d'embedding ne détruit plus les chunks existants (auparavant deleteSource était
appelé avant la boucle d'embed → corpus tronqué/vidé en silence, non rattrapé
par reingestMissing qui ne reprend que les docs à zéro chunk).
KazeiaService.buildRag/teardownRag : synchronized(ragLock) + buildRag idempotent
(retourne l'instance existante si déjà active). onCreate et un toggle config RAG
peuvent se chevaucher (deux coroutines serviceScope) → sans verrou, deux
EngineEmbedder natifs étaient créés et le perdant fuyait (jamais close()).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CrisisGuard (sécurité vitale) :
- normalize() : suppression accents (NFD) + apostrophes→espace + compactage,
pour matcher la sortie STT FR (souvent sans accents) et la frappe libre.
- PATTERNS étendus : formulations directes manquantes (« je veux mourir »,
« idées suicidaires/noires », « disparaître », « me pendre », « passer à
l'acte », « plus la force de vivre »…). « me pendre » autonome (aucun usage
bénin) ; « me noyer » reste gardé (idiome « se noyer dans le travail »).
- Tests étendus : 12 positifs + 4 négatifs (idiomes « mourir de rire/faim »,
« idées plein la tête », « passe à la pharmacie »). 11/11 verts.
Rag.retrieve() : un chunk plus long que le budget ne fait plus retomber tout
le bloc à null — on garantit ≥1 hit (tronqué si besoin) puis on empile tant
qu'il reste du budget. Évite la perte de contexte pertinent.
Rag.close() : @Synchronized — sérialise avec retrieve()/searchScored() pour
qu'un teardown (toggle RAG off) ne libère pas le contexte natif de l'embedder
pendant une requête provider en cours (évite le SIGSEGV).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migration tablette dev debug→release + cycle OTA complet exercé de bout en bout
pour la première fois :
- uninstall debug → install release v1 (signé clé Kazeia, cf RELEASE_SIGNING.md)
- bump versionCode 2 / versionName 0.1.1, assembleRelease (verifyJniLibs OK)
- publication Nextcloud : APK v2 + composants RAG (e5 + corpus) + catalog v3
- sur device : Onboarding détecte « Mise à jour disponible v0.1.1 », télécharge
(106 Mo, SHA-256 vérifié), PackageInstaller installe in-place → versionCode=2.
PREUVE CLÉ (la raison d'être de la signature release) : un marqueur de config
posé en v1 (rag_enabled=1, rag_threshold=0.85) a SURVÉCU à la mise à jour v2 →
signature cohérente entre versions → une MAJ ne détruit pas les données patient.
Note : catalog.spec.json/dist non suivis (miroir data). versionCode=2 est la
seule trace git ; la prochaine release repart de là.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Les 23 .so de app/src/main/jniLibs/ (474 Mo, 5 chaînes de build : kazeia-engine,
ExecuTorch, QNN SDK 2.42, Genie, TTS pipeline) sont des artefacts gitignorés
posés à la main — « ça ne buildait que sur cette machine ». On ne les versionne
pas (l'historique gonflerait de ~500 Mo par update engine) ; on versionne leur
DÉFINITION :
- scripts/jnilibs.MANIFEST.sha256 : état attendu (sha256 + provenances).
- scripts/jnilibs.sh : verify (CI/Gradle) | sync-engine (rafraîchit le
sous-ensemble kazeia-engine depuis /opt/Kazeia-engine/dist) | update-manifest
(re-fige, à committer) | export/import (tarball pour autre machine).
- Garde Gradle : preReleaseBuild dépend de verifyJniLibs → un build RELEASE avec
des libs manquantes/modifiées/non déclarées ÉCHOUE. Debug reste libre.
Validé : release passe avec libs conformes ; échoue sur lib altérée (MODIFIÉ
libomp.so) ; verify OK après restauration. Ménage au passage : 3 .bak morts
(~293 Mo) sortis de jniLibs vers _jnilibs_backup_baks/.
Nouvelle machine : git clone + ./scripts/jnilibs.sh import <tarball> (export
déposé sur box.kazeia.com privé) → build garanti conforme.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Premier harnais de tests du projet. testImplementation junit:4.13.2, src/test/.
CrisisGuardTest — NON-RÉGRESSION DE SÉCURITÉ : 26 formulations d'idéation qui
DOIVENT déclencher (toutes les familles de motifs, casse, variantes STT collées
validées device), 10 messages de détresse ordinaire qui ne doivent PAS déclencher
(tristesse, insomnie, colère, deuil d'autrui, « envie de dormir »…), 2 limites
ASSUMÉES documentées (sensibilité d'abord : « en finir avec ces insomnies » et
mention indirecte du suicide déclenchent), invariants de la réponse (112,
honnêteté « programme », anti-isolement). Toute retouche des PATTERNS doit garder
ces tests verts ; le clinicien ÉTEND les cas, ne supprime pas de positif.
VectorIndexTest : classement cosinus, top-k, seuil, index vide/dim incohérente.
ChunkerTest : tag source, découpe+overlap sous la limite ubatch, texte vide.
./gradlew :app:testDebugUnitTest → 11 tests, 0 échec, <5 s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keystore release créé : /opt/Kazeia/keystore/kazeia-release.jks (RSA 4096,
alias kazeia, valide 2056), credentials dans keystore/credentials.properties
(chmod 600, hors git — la liste blanche du .gitignore racine l'exclut d'office).
Les deux modules lisent credentials.properties : présent → assembleRelease signe
en release ; absent (autre machine/CI) → repli debug + warning, le build ne casse
pas. MÊME clé pour com.kazeia et com.kazeia.admin → permettra la
signature-permission sur le ContentProvider (plan admin).
Validé : assembleRelease des deux apps OK, apksigner confirme le certificat
CN=Kazeia (SHA-256 4b408d9d…) sur les deux APK.
docs/RELEASE_SIGNING.md : gestion complète — câblage, procédure de release via
catalogue (bump versionCode obligatoire), SAUVEGARDE du keystore (le point qui
ne pardonne pas), scénarios de panne (perte clé/mdp/machine/compromission),
migration one-shot des tablettes debug→release, dev quotidien reste en debug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fusionne les garde-fous du prompt de Damien dans la voix chaleureuse de Kazeia,
et ajoute un filet de sécurité crise indépendant du LLM.
Bornes "soft" (niveau prompt, DEFAULT_SPEAKER_PROMPT + DEFAULT_SYSTEM_PROMPT +
copie admin, synchronisés) : pas de diagnostic / pas de pathologie nommée ;
médicament/dose/traitement -> décision médicale, renvoi médecin/psychiatre ;
refus des jeux de rôle/simulations ; réponse cadrée sur le stockage local.
Tutoiement + ton chaleureux préservés (≠ style clinique/vouvoiement de Damien).
Borne "hard" (CrisisGuard, DÉTERMINISTE) : on ne laisse jamais un 4B sous
contrainte de brièveté improviser sur une idéation suicidaire. Détection par
motifs FR (tunables clinicien) -> court-circuite Thinker+Speaker -> réponse fixe,
validée, honnête (Kazeia = programme), oriente vers l'humain + 112 (urgence EU,
valable FR et Luxembourg). Placé en tête de processLlmResponse.
Validé device : "envie d'en finir / me faire du mal / vaudrait mieux d'en finir"
-> [CRISIS] -> réponse de sécurité synthétisée (TTS frames=192) + jouée.
"arrêter mes médicaments" -> redirection médecin (prompt, pas crise). "triste et
fatigué" -> réponse normale, AUCUN faux positif crise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Active le RAG de bout en bout dans la vraie conversation :
- Rag.ingestDir(dir) : ingère tous les .txt/.md d'un dossier (source = nom fichier).
- KazeiaService : auto-ingestion du corpus au 1er démarrage si base vide
(.../kazeia/rag_corpus/), + intent `rag_ingest` (clear+reingest, brique admin future).
- Injection live : seuil relevé à 0.82 (calé e5 : base ~0.80 / pertinent ~0.88), k=3,
pour ne PAS injecter de hors-sujet.
- Corpus seed FR de psychoéducation (6 fichiers) dans docs/rag_corpus_seed/.
Validé device (ragEnabled ON, Speaker lib qwen3.5-4b) : 6 chunks ingérés, tour
patient « je n'arrive pas à dormir… » -> retrieve 1 hit (top 0.84, le chunk sommeil ;
les 5 autres filtrés par le seuil) -> bloc CONTEXTE injecté (577 c) -> réponse ancrée.
Aucun crash, coexistence RAM OK. ragEnabled reste OFF par défaut (opt-in clinicien).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Le dev kazeia-engine a livré le bridge CPU-only avec embedText (commit engine
655d65a) + un GGUF multilingual-e5-small fonctionnel (converter patché pour le
tokenizer XLM-R/unigram). Bridge swappé dans jniLibs (ABI cohérente : libllama/
libggml déjà = build CPU-only du 2 juin).
- Rag.ingest : les passages portent désormais le préfixe "passage: " (e5), en
symétrie du "query: " de retrieve (EngineEmbedder.embedPassage).
- Intent `rag_real_test` : valide le VRAI EngineEmbedder (e5) in-app.
Validé device (CPU-only, untrusted_app) : embedder ready=true dim=384, AUCUN
crash SELinux/fastrpc (la .so ne tire plus hexagon), retrieval FR correct —
requête sommeil → doc insomnie en tête (top=0.88). embedText fonctionne in-app
end-to-end. Le .so (gitignored) est un artefact build, hors git.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Module com.kazeia.rag, conçu d'après le RAG PC de Damien mais porté on-device :
- Embedder : interface + FakeEmbedder (déterministe, test hors engine) +
EngineEmbedder (réel, branché sur EngineJni.loadEmbedder/embedText — inerte
via UnsatisfiedLinkError tant que la lib n'expose pas ces symboles).
- VectorIndex : recherche exhaustive cosinus (produit scalaire, vecteurs L2),
top-k + seuil. Pas d'ANN : injustifié à l'échelle (10^2–10^3 chunks).
- Chunker : découpage par phrases + overlap, tag [Source: x] (mieux que le
découpage au caractère du PC).
- RagDb : SQLite (vecteurs en BLOB float32 LE), garde-fou modèle+dim.
- Rag : façade ingest/loadIndex/retrieve -> bloc CONTEXTE budgété en tokens.
Bindings JNI embedder ajoutés à EngineJni (loadEmbedder/embedText/freeEmbedder).
Flag ragEnabled (ConfigStore + provider rag_enabled), DEFAULT OFF -> production
inchangée. Câblage live dans KazeiaService : si activé + embedder prêt + contexte
au-dessus du seuil, préfixe le bloc CONTEXTE au prompt patient (défensif, le RAG
ne porte jamais la gestion de crise).
Validé device via l'intent `rag_test` (FakeEmbedder) : ingest 2 docs -> SQLite ->
index -> retrieve classe correctement (requête sommeil -> doc insomnie en tête).
Démarrage prod : "[RAG] désactivé (config)", rag_enabled=0. Aucune régression.
Reste (dépend de l'engine) : embedText livré + modèle e5/bge FR + ingestion d'un
vrai corpus côté admin + entrée catalogue. Cf docs/RAG_EMBEDDINGS_ENGINE_SPEC.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture l'état courant de kazeia-android (migration vers le moteur lib unifié :
suppression des anciens AudioCaptureManager/VadEngine/SileroVad, WhisperJni/
WhisperLiteRt/WhisperNpu/AndroidStt, KazeiaLlmJni/LlamaCppLlmEngine,
AndroidTts/ChatterboxTts ; édits app/JNI/gradle associés). L'app construit et
tourne sur device dans cet état. Commit de préservation pour ne rien perdre
avant de restreindre le périmètre de suivi du dépôt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ModelRegistry devient backend-aware (PTE | GGUF) : enum Backend, ggufFile +
ggufPath() sous MODELS_DIR, fileExists() par backend, et un type de template
ChatTemplate (QWEN35_THINKOFF | PLAIN_CHATML). Ajout de deux entrées Speaker
GGUF servies par le moteur lib : Qwen3.5-4B (q35-lmq4, le défaut lib jusqu'ici
codé en dur, désormais visible) et Qwen2.5-7B (Q4_K_M, arch qwen2 dense).
KazeiaService : le path lib route le GGUF du Speaker sélectionné (sp.ggufPath())
au lieu de q35-lmq4 en dur, et passe plainChatml selon le template. Pour les
modèles sans thinking (Qwen2.5), EngineLlmAdapter construit un ChatML standard
sans bloc <think> via generateRaw (le natif l'injecterait sinon).
Affordances de test/maintenance (pilotage adb sans UI) :
- llm_text / llm_max : génération LLM pure loggée (tok/s, chars), hors pipeline TTS.
- cfg_set : persiste engine+model+prompt via am --es (contourne content --bind
qui casse sur ':'); cfg_sys=DEFAULT résout le prompt thérapeutique en interne.
Provider /models : expose le chemin GGUF pour les entrées GGUF.
Validé device : les deux GGUF chargent, FR cohérent, templates propres.
Bench : Qwen3.5-4B ~13.5 tok/s, Qwen2.5-7B ~6.8 tok/s (CPU-only) ; qualité FR
thérapeutique nettement en faveur du Qwen3.5-4B.
Note : KazeiaService.kt et les 3 fichiers nouveaux portent aussi du travail
antérieur non commité (migration moteur lib + app admin) ; ce commit en capture
l'état courant en plus de la fonctionnalité Speaker GGUF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three tightly-coupled UX fixes the user flagged during live testing.
**Voice hot-swap (Qwen3TtsEngine.setVoice)**: previously a no-op
stub — the spinner callback updated the orb color but the actual
audio kept using Damien's cached prefix/suffix embeddings. Now we
derive the voice id from the WAV basename (elodie.wav → 'elodie'),
look up `<id>_voice_prefix.bin` + `<id>_voice_suffix.bin` in the
model dir, parse their headers, and atomically replace the embedding
arrays so the NEXT synthesized segment uses the new voice. If the
files aren't present we log a clear warning pointing at
prepare_tts_native.py — the hot-swap is wired, but per-voice prefix/
suffix still need to be generated offline and adb-pushed.
KazeiaService.setVoice now forwards to Qwen3TtsEngine in addition to
the Chatterbox branch.
**Emoji stripping**: the model loves closing on "😊" and it was
reaching TTS as a standalone segment that synthesized a fraction of
a second of junk. KazeiaPipeline.speakText now runs each sentence
through stripNonSpeakable before enqueueing — drops Unicode emoji /
dingbat / pictograph / flag blocks plus variation selectors and
zero-width joiners, then trims. Empty-after-strip sentences are
skipped entirely. The chat bubble still shows the original text
(with emojis) — only the audio path drops them.
**Typing dots indicator**: while LLM is done but TTS synthesis is
still running (~3–5 s for the first segment), the Kazeia bubble now
shows an animated ". / .. / ..." cycle at 400 ms cadence instead of
sitting empty. The moment the first segment actually starts playing,
the cycle cancels, the bubble resets to empty, and the existing
word-by-word reveal takes over. A defensive finally block also
cancels the job when no segment ever fires (e.g. all-emoji reply).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dropped the internal waveform lines — not what we wanted visually —
and replaced them with a spectrum-driven deformation of the sphere
outline itself. Each of the 12 log-spaced bands drives one Fourier
mode of the perimeter (band b → mode b + 2, so modes 0/1 stay
circular and higher bands produce tighter ripples). Low bands pull
the shape into wide asymmetric bumps that feel like formants; high
bands add quick sibilant-like tremors. Phase advances faster for
higher modes so tight ripples visually match high-frequency content.
Overall displacement is gated by the RMS envelope so silence is
quiet and loud syllables distort strongly. Fill + highlight are
clipped to the deformed path so the gradient follows the shape and
it reads as a single living object rather than a circle with stuff
bolted on.
Removed drawSpectrumBars and drawWaveformLine.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
**Regression fix**: when synthesis of segment N+1 ran longer than the
playback of segment N (e.g. 5 s synth for a 1.5 s "Bonjour !"), the
previous MediaPlayer was already in the Completed state by the time
we queued the next one. setNextMediaPlayer() on a Completed player is
a documented silent no-op — so the second sentence never started and
the user only heard the first part of the reply.
Rewrote playChainedMediaPlayers with per-player CompletableDeferred
tracking: before calling setNext we check whether current's done has
fired; after awaiting completion we verify next really auto-started
(checking isPlaying / currentPosition) and call start() explicitly if
the chain missed. Belt-and-suspenders against the race either way.
Removed the now-unused waitForPlaybackCompletion helper.
**Visual change**: in-sphere spectrum bars replaced with three
superimposed Bézier "deforming lines", mirrored above/below a central
baseline, with a soft cosine taper so the curves decay to zero at the
sphere's left/right edges (matches the circular mask). Each line has
its own slow-moving phase + gain + thickness + alpha so the three
overlap to give depth — closer to an oscilloscope trace than an EQ.
Low-level sin jitter keeps the lines alive during quiet passages,
amplitude-gated so true silence is a flat line.
User-facing change: no bars anymore. The sphere now "breathes" with
flowing waveforms matching its voice.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete redesign of AudioVisualizerView based on feedback: the orb
is now the app's visual face, takes the top ~60% of the chat area,
and has clearly distinct behaviour in each state.
- **Idle**: slow 5 s breathing (scale 0.88 → 1.00 via cos easing),
pure round shape, soft halo in phase. No high-frequency motion.
- **Listening**: organic blob outline built from 8 Fourier modes
whose amplitude scales with live mic RMS; a thin shimmering arc
rotates around the orb while mic energy is present; continuous
micro-ripples pulse outward. Looks clearly 'alive and attentive'
vs Idle's static breathing.
- **Speaking**: the orb becomes a contained spectrometer. A pre-
computed log-spaced spectrogram (12 bands, 120 Hz–4 kHz,
Hann-windowed FFT, one column per 50 ms of audio) is rendered as
vertical rounded-rectangle bars CLIPPED to the sphere outline so
they really look like the sphere itself speaking. Bar heights
interpolate between spectrogram frames and exponentially smooth
toward the target for fluid 60 fps motion. Outer halo pulses with
the RMS envelope; ripples release on envelope peaks.
- **Per-voice color**. Eight-entry palette (Damien lavender,
Elodie rose, Jerome aqua, Richard amber, Amir emerald, Didier
indigo, Sid peach, Zelda periwinkle). Halo, accent, bars, ring,
and ripples are all derived from a single voiceColor so switching
the voice spinner tweens the entire scene to the new identity
over a few frames. Color stored on both KazeiaService (for
persistence across process/view rebinds) and pushed directly to
the view for instant feedback at selection time.
Sidecar pipeline changes:
- Qwen3TtsEngine now computes per-segment spectrogram alongside the
RMS envelope (new computeSpectrogram + an in-place radix-2 FFT).
FFT_SIZE = 1024, hop = 50 ms, 12 log-spaced bands. SegmentReady
carries both arrays; onSegmentPlaying is (sentence, durationMs,
rmsEnvelope, spectrogram).
- KazeiaPipeline.speakText forwards the new callback shape.
- KazeiaService.VisualizerSignal.Speaking now carries the
spectrogram and the new voiceColor StateFlow.
- ChatActivity passes both to the view and collects voiceColor.
Layout: vertical chain between audioViz (weight 3) and rvMessages
(weight 2) so the orb owns ~60% of the chat panel and the chat list
takes the remainder. Removed the fixed 140 dp constraint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a breathing lavender orb centred above the chat list that tracks
the actual audio state of the app:
- **Idle**: slow respiratory pulsation (~4 s cycle) at 20 fps. The
chatbot is visually "awake" without animating loudly.
- **Listening**: halo swells with live mic RMS from the VAD loop, so
the user sees Kazeia hearing them even before Whisper has produced
any transcription. Mic RMS is normalised with the same sqrt
squashing the TTS envelope uses so quiet speech still reads visibly.
- **Speaking**: amplitude + halo driven by a pre-computed RMS envelope
(50 ms windows, sqrt-normalised) produced at synthesis time. Ripples
fire on local peaks above 0.35 — matches speech rhythm without
overwhelming. Timer is internal to the view, synced to the segment's
durationMs; no MediaPlayer position polling.
Architecture:
- Sidecar RMS envelope. Computed in Qwen3TtsEngine.generateSegmentAudioVC
right after PCM is available, packed into SegmentReady, and handed to
onSegmentPlaying(sentence, durationMs, rmsEnvelope) when each MediaPlayer
starts. Zero extra IO — runs on the same PCM we already write to WAV.
- KazeiaService exposes VisualizerSignal (Idle | Listening(rms) |
Speaking(env, dur)) as a StateFlow. The VAD loop pushes Listening,
processLlmResponse pushes Speaking from the per-segment TTS callback,
and finally clears to Idle when no mic is open.
- AudioVisualizerView renders via Choreographer.FrameCallback, self-
throttled to 20 fps at Idle and full refresh during Listening/
Speaking. Hardware layer. Pure Kotlin + Canvas, no deps. ~280 LOC.
Layout: 140 dp strip between voiceBar and rvMessages in activity_chat.xml.
No 3D engine, no Unity, no splash extension. The avatar design work
remains on disk for a later phase when the TTS+streaming pipeline
stabilises enough to spend time on DECA/FLAME integration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Matches the 'conversation' feel the user asked for. Previously the
full LLM response appeared in the chat as soon as generation
finished, then audio played 5–10 s later — text and sound felt
decoupled. Now:
- The KAZEIA bubble is created empty and only starts filling when
the first TTS segment actually starts playing through the speaker
(we already split the response by sentence for the chained-
MediaPlayer pipeline; that split drives the reveal too).
- Inside each sentence, words are appended one by one at a cadence
of (audio duration / word count) — slower sentences reveal slower,
matching speech pacing. The first word of each sentence appears
immediately so audio and text stay aligned at the start.
Implementation:
- Qwen3TtsEngine: added `onSegmentPlaying(sentence, durationMs)`
listener, invoked from the chained-MediaPlayer worker the moment
each segment's MediaPlayer.start() lands. Sentence + duration are
carried end-to-end via a new SegmentReady data class.
- KazeiaPipeline.speakText: forwards an optional listener down to
the TTS engine, same signature.
- KazeiaService: new updateMessageText(id, text) helper. In
processLlmResponse, the bubble is added empty before speakText and
grown by a reveal coroutine per sentence; after speakText returns
we snap to the full text as a safety net.
No change to the stream_llm debug intent path — it still uses the
old enqueueSentence flow directly and doesn't need the reveal (no
UI bubble there).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The kill-background approach worked at startup (−1.6 GB) but respawns
pulled most of that back within 1–3 min, and the periodic sweep only
kept the first sweep's reclaim stable rather than going lower. Net
practical benefit over "just let Android manage it" is small, not
worth a custom optimizer + normal-perm + file maintenance.
Removes:
- MemoryOptimizer.kt
- KazeiaService.onCreate calls to freeRamForModels / startPeriodicOptimizer
- KILL_BACKGROUND_PROCESSES permission from the manifest
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First sweep reclaimed 1.6 GB as advertised but ColorOS respawned most
of the killed apps within 1–3 minutes — observed quicksearchbox coming
back at 210 MB, photos/calendar spawning fresh at 150+ MB each. Two
changes:
1. Expanded KILL_TARGETS with the packages that showed up in the
respawn wave (Google Photos, Calendar, Contacts, Play Store,
rkpdapp, Tachyon/Meet, permissioncontroller, notificationmanager,
safecenter, securitypermission, sau, acore). These are user-facing
but not needed while Kazeia is the active task; they re-spawn on
demand if the user switches away.
2. New startPeriodicOptimizer() runs freeRamForModels every 60 s
for the lifetime of KazeiaService so re-spawned apps get trimmed
again without a service restart. Tied to serviceScope so it stops
cleanly on destroy.
Net effect observed: avail RAM stays ~1.2–1.5 GB higher than without
the sweep. Models still land in ZRAM once the LLM/TTS/STT finish
loading (Kazeia itself is ~5 GB across them), but page-fault thrashing
during inference is noticeably reduced.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three unrelated fixes rolled into one so testing on the tablet stayed
coherent. All were driven by what the user was observing during live
audio tests, not by pre-planned refactors.
1. **Audio playback actually audible.** ColorOS's AudioFlinger
silently muted our AudioTrack ~600 ms after play() every time
(dumpsys audio showed `event:muted updated source:clientVolume`
and playbackHeadPosition stuck at 0), regardless of USAGE_MEDIA /
USAGE_ASSISTANT / USAGE_VOICE_COMMUNICATION, regardless of audio
focus grant, regardless of FGS type including mediaPlayback. A
MediaPlayer path using the SAME usage attributes works because it
routes through a different AudioFlinger thread that isn't under
the same background-hardening policy. `USE_MEDIAPLAYER_FALLBACK`
in Qwen3TtsEngine.kt flips playback to a WAV-per-segment pipeline.
Two MediaPlayer instances are chained via `setNextMediaPlayer()`
so segments transition without re-arming the DAC (that re-arm was
audible as "beg beg" pops between sentences). Synth of seg N+1
runs in parallel with playback of seg N via a capacity-2 Channel,
hiding synthesis latency behind playback for all but the first seg.
2. **Mic no longer loops TTS back into STT.** The continuous-
listening VAD in KazeiaService already had a guard to drop frames
while `pipelineState is Speaking`, but that state was never set by
any caller — so the mic kept recording during playback and fed our
own speaker output back to Whisper, creating the infinite
"Kazeia talks to Kazeia" loop the user observed. Both the
stream_llm intent path and the main `processLlmResponse` TTS path
now wrap the TTS call with `Speaking → Idle/Listening`.
3. **Free 1.6 GB of RAM at service start.** The OnePlus Pad 3 with
ColorOS keeps ~7 GB of Google + OPLUS background services
resident at idle. With Qwen3-4B (3.2 GB) + Qwen3-TTS (1 GB) +
Whisper (0.5 GB) on top, most of our model weights were going to
ZRAM swap — "the NPU is stuck" reports were actually page faults
paging 3 GB of LLM weights back in before each inference. New
`MemoryOptimizer` kills 30-ish non-essential background packages
(Google optional: YouTube, Wallet, Chromecast, Messaging, AICore,
Quicksearchbox; OPLUS optional: smartsidebar, cosa, pantanal,
nhs, midas, …) via `ActivityManager.killBackgroundProcesses`.
Measured reclaim on first run: **avail RAM 8468 MB → 10112 MB,
+1644 MB**. Uses KILL_BACKGROUND_PROCESSES (normal perm, no user
prompt); system-critical packages and the launcher/systemui are
explicitly excluded from the target list.
Collateral changes:
- Added FOREGROUND_SERVICE_MEDIA_PLAYBACK permission + fgsType flag
(didn't fix the mute on its own, but it's correct per Android 14
policy and leaving it without would be a latent compliance risk).
- Kept `USE_STREAMING_DECODE` + CP↔BigVGAN overlap code intact
behind the MediaPlayer-fallback branch so reverting to the
AudioTrack streaming path is a single-const flip if ColorOS ever
lifts the hardening (or we move to a device without it).
- New AudioTrack path has a keep-alive silence watchdog and a
playback-head drain wait on stop. Both were attempts to fix the
mute that didn't pan out on their own; leaving them in so the
streaming path stays usable on non-hardened devices.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ResourceMonitor.init ran `su -c id` at every ChatActivity launch to see
if root was available, then used root to read /sys/class/kgsl/... and
/sys/bus/platform/devices/soc:qcom,msm-cdsp-rm/... for GPU/NPU usage %.
That probe was the only thing still triggering the Magisk auth dialog
on each app start after the no-root LLM migration.
Remove the root probe and the execRoot helper. GPU/NPU reads now return
-1 (UI already renders "—" for negative values). The non-root
/sys/class/kgsl/kgsl-3d0/gpubusy path is kept as a best-effort — it's
world-readable on some devices, silently fails otherwise. CPU and RAM
readouts are unaffected (never needed root).
Dead-code `su -c ...` calls remain in Qwen3TtsEngine (hexStartRunner,
hexStartCpRunner, hexStopRunner, etc.) and WhisperNpuSttEngine, but all
are gated behind fallback paths that don't execute under the current
PTE-only config (talkerPteModule != null && cpPteModule != null short-
circuits before any su call). Left in place to avoid churning the TTS
Hexagon fallback; can be purged in a later cleanup pass if needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Streaming variant of the per-segment decode pipeline. As soon as SEQ_LEN
codes are accumulated from the talker/CP loop, BigVGAN is dispatched on
a background coroutine while the producer keeps generating the rest of
the segment. The BigVGAN consumer feeds a streaming crossfader that
emits stable audio as it arrives and holds back overlapSamples for the
next chunk's blend.
Mirrors decodeChunked's semantics exactly so final audio is bit-identical
modulo the fadeOut application location (now applied to the final
emission tail instead of the full buffer; the last 40ms still get faded).
Validated A/B on the same prompt 3 used in the recent benchmark:
prompt: "Je me sens un peu triste aujourdhui…"
seg 0 first audio: 14 485 ms → 10 936 ms (−3.5 s)
end-to-end first audio (LLM trigger → audio): 16.2 s → 12.7 s
Stream LLM total: 33 234 ms → 28 594 ms (−4.6 s)
Short segments (<SEQ_LEN codes) and the legacy non-streaming callers
(generateSegmentAudioVC, decodeChunked, multi-segment pipelines, etc.)
are untouched. The new path is gated behind USE_STREAMING_DECODE so it
can be reverted by flipping a single const if a regression is found.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
#2 BigVGAN GPU experiment: ORT-QNN GPU EP loaded the v2_decoder_conv ONNX
model successfully (session creation 463 ms, no fallback warnings) but
per-phrase inference jumped to ~3.5 s vs ~2 s on CPU 8-thread. The GPU/CPU
memory transfer cost dominates for this conv-heavy decoder, and the
optimization went the wrong way. Comment block updated to record both the
HTP and GPU paths as tried-and-rejected so future passes don't re-walk the
same ground.
LLM streaming filter: extend the lookahead-based <think>…</think>
suppressor to also strip singleton special tokens (<|im_start|>,
<|im_end|>, <|endoftext|>). Previously the closing <|im_end|> at end of
the assistant's turn leaked into the SentenceStreamer and ended up as a
spurious sentence at the end of the TTS output. Same lookahead-buffer
trick handles split tokens.
Validated end-to-end: 'Bonjour, comment vas-tu ?' → "Bonjour ! Je vais
bien, merci. Comment vas-tu ?" → seg 0 "Bonjour !", seg 1 "Je vais bien,
merci." (no <|im_end|>), BigVGAN back to 1.8 s/phrase.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The verbose 55-token system prompt was the cheapest TTFT win on the
kv-only path (52 ms per prefill token). Compacting it to 25 tokens while
keeping the three load-bearing constraints — Kazeia identity, French only,
short replies, /no_think — measurably improved end-to-end latency.
Validated 'Bonjour, comment vas-tu ?' on tablet:
Before: prompt_tokens=80, TTFT=4202ms, total=5716ms
After: prompt_tokens=53, TTFT=2865ms, total=4034ms (-1.3s, -32% TTFT)
Reply quality preserved: "Bonjour ! Je vais bien, merci. Comment vas-tu ?"
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Even with /no_think in the system prompt Qwen3 still emits an empty
<think>…</think> wrapper before the real answer. Without filtering, the
SentenceStreamer treats '<think>' as a sentence boundary and feeds three
tokens of XML into the TTS, producing audible parasites at the start of
each reply.
The new in-callback filter buffers a small lookahead (just enough to span
"</think>"), suppresses everything between the open and close tags, and
flushes the surrounding prose to onToken in order. With the lookahead, tags
that arrive split across decoded pieces ("<thi"+"nk>") still match.
Validated end-to-end: prompt 'Bonjour, comment vas-tu ?' now streams
sentence-by-sentence to the TTS — first segment "Bonjour !" reaches the
talker at 4.6 s, no <think> sneak-through.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
End-to-end validation on OnePlus Pad 3 with stream_llm intent:
Prompt: 'Bonjour, comment vas-tu ?'
Response: 'Bonjour ! Je suis là pour t'écouter. Comment vas-tu aujourd'hui ?'
TTS: Talker(PTE) 37ms/step, CP(PTE) 73ms/step, audio synthesized.
No su, no Magisk prompts.
Two fixes since the previous commit:
1. ExecuTorchLlmEngine: pass echo=false to LlmModule.generate() — by default
the runner echoes the prompt tokens back via the callback, which fed the
ChatML wrap (<|im_start|>user …) into the SentenceStreamer and TTS.
2. jni_layer_llama.cpp: pick Runner<uint8_t> vs Runner<uint16_t> based on the
model's get_kv_io_bit_width metadata, mirroring qnn_llama_runner.cpp main().
The hard-coded uint16_t was wrong for our Qwen3-4B export (which uses 8-bit
KV I/O) and produced fluent-looking but completely random tokens
("blocked罩ug darkestSOLEQuotes作者本人 …") — same symptom whether greedy or
sampled, the smoking gun for a width-mismatched KV cache reinterpretation.
Other tweaks:
- temperature=0.0 in the QNN_LLAMA branch of jni_layer_llama.cpp (greedy,
matches the working qnn_llama_runner --temperature 0 invocation)
- shared_buffer=true (same as binary defaults)
- Kotlin chat template mirrors qnn_llama_runner.cpp's get_formatted_prompt for
Qwen3 (user-first, then optional system, then "<|im_start|>assistant" with
no trailing newline — that quirky ordering is what the .pte was trained on)
TFTT is ~4 s for a 77-token prompt on kv-only mode (sequential prefill, one
forward per token). To get a sub-second TTFT we'd need to re-export the model
in --model_mode hybrid which adds a parallel prefill_forward graph; not
required for the conversational use case.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The root cause of the previous su-c requirement was that Qualcomm's FastRPC
kernel driver rejects processes spawned via ProcessBuilder fork+exec because
they lose supplementary GIDs on exec. Zygote-forked app processes retain the
proper init-configured credentials and are accepted by the adsprpcd service,
which is why ORT-QNN (Whisper, in-process) worked while the subprocess
qnn_llama_runner did not. Running the LLM in-process via ExecuTorch's
LlmModule bypasses the fork+exec path entirely.
What this commit does:
- ExecuTorchLlmEngine now uses org.pytorch.executorch.extension.llm.LlmModule
with MODEL_TYPE_QNN_LLAMA=4 (routes to example::Runner in jni_layer_llama.cpp,
the same C++ runner that qnn_llama_runner embeds).
- All su, ProcessBuilder, file-based prompt/response plumbing, and run_llm.sh
gone. ChatML template is built in Kotlin; tokens stream in via LlmCallback.
Supporting changes under executorch-patches/llm_in_process_jni.patch:
1. backends/qualcomm/CMakeLists.txt — gate PyQnnManagerAdaptor on NOT ANDROID.
The original guard (CMAKE_SYSTEM_PROCESSOR MATCHES x86_64) misfires in a
nested scope during Android cross-compile and tried to build the host
Python bindings.
2. extension/android/jni/jni_layer_llama.cpp — hardcode decoder_model="qwen3"
(was "llama3") and pass eval_mode=0 (EvalMode::kKVCached) + shared_buffer=true
to match our hybrid_llama_qnn.pte which only contains kv_forward, not
prefill_forward.
Build: scripts/build_android_library.sh arm64-v8a with QNN_SDK_ROOT pointing
to /opt/Kazeia/qnn_sdk_242/qairt/2.42.0.251225 and EXECUTORCH_BUILD_QNN=ON.
Produces libexecutorch_jni.so (192 MB) with QNN v2.42 backend + the llama
runner code, plus libqnn_executorch_backend.so. Both staged in jniLibs.
Validated on OnePlus Pad 3: LlmModule.load() completes in 4.2 s, no su
prompts, Pipeline ready with STT(WhisperHybridEngine) → [VoiceCommands →
LLM] → TTS(Qwen3TtsEngine). TTS .pte still loads with the upgraded v2.42
runtime — no regression.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Commit de sauvegarde avant la tentative d'unification QNN SDK v2.37 et
suppression du su -c pour le LLM. État actuel fonctionnel :
- LLM Qwen3-4B via su -c qnn_llama_runner (v2.42 dans /data/local/tmp/kazeia-et/)
- TTS talker + CP via ExecuTorch .pte JNI (v2.31 dans jniLibs)
- STT Whisper via ORT-QNN 1.24.3
Le rapport kazeia-no-root-report.md documente en détail les tentatives de
no-root et leurs échecs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ExecuTorchLlmEngine: system prompt forces French, 1-2 short sentences,
/no_think so the full budget goes to the answer (Qwen3 was consuming
120+ tokens on <think>); eval_mode 0 matches our kv-mode export.
- Qwen3TtsEngine.generateSegmentAudioVC: when the Hexagon talker socket
isn't open, fall back to runInterleavedPteFromEmbeds so the Stage 3
streaming session still produces audio. Without this the session opened,
accepted sentences, and silently emitted empty PCM.
Documents the QNN SDK version-skew pitfall in ExecuTorchLlmEngine.kt
ahead of the upcoming migration to a unified v2.42 toolchain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- ExecuTorchLlmEngine: eval_mode 0 (our .pte is kv-mode, not hybrid)
- KazeiaService: call llm.load() after TTS init; try/catch falls back
to echo mode if the runner or .pte are missing.
Pipeline on device: STT(WhisperHybridEngine) → [VoiceCommands → LLM] → TTS(Qwen3TtsEngine).
Validated on OnePlus Pad 3: LLM ready in ~8 s, gen 21.3 tok/s, RSS 1.76 GB in the
qnn_llama_runner subprocess (out-of-process from the Kazeia app).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the fixed maxGen + length-based boost with a fully dynamic
end-of-utterance detector that watches the model's own EOS logit rank.
End result on the Baer 3-segment monologue, validated by user as
"FORMIDABLE" / "impeccable" with both Damien and Zelda voices:
- All 3 segments terminate via EOS (no maxGen cap hit)
- No "page beg beg" filler tail
- No abrupt cuts between segments
- Audio durations 5-8 s per segment, matching Python within ~10 %
How it works (runHexGenWithPrefill, in tts/Qwen3TtsEngine.kt):
1. At every decode step, compute the rank of CODEC_EOS in the
repetition-penalised logits. Mid-utterance the rank sits at
150-700 (model is committed to producing speech). Approaching
the natural end, the rank dips toward top-50.
2. Arm the boost only when EOS rank stays below eosRankTrigger=60
for THREE consecutive steps. The 3-step requirement filters out
transient single-step dips that occur during low-energy phonemes
mid-sentence (without it, short sentences would terminate after
~3 s). Arming is also gated by eosBoostMinStep (50 % of expected
speech length) so we never arm in the very first frames.
3. Once armed, the boost increments monotonically: each subsequent
step adds boostStepsActive * eosBoostScale to the EOS logit. The
accumulated boost lifts EOS above top-1 within 1-3 steps, the
argmax check fires, and the loop breaks. Scale=4 gives the model
a small natural decay before termination; scale=5 was perfect-but-
slightly-clipping, scale=3 wasn't strong enough to outpace the
growing top-1 logit.
Other tweaks bundled in this commit because they all contribute to
the clean output:
* Inter-segment gap 120 → 250 ms — gives the listener a perceived
sentence boundary instead of a hard concatenation.
* fadeOut(audio, 40) on every segment — cosine roll-off over the
last 40 ms so the EOS-clipped tail decays naturally instead of
sample-clipping.
* top_k 50 → 200 in the fallback sample call — wider pool to keep
EOS reachable when the boost just fails to hit argmax.
Voice swap is a 45 KB file push (damien_voice_prefix.bin and
damien_voice_suffix.bin). Successfully tested today with Elodie
(female, norm 10.12) and Zelda (norm 9.39) using Damien (norm 10.36)
as the baseline — same Kotlin code, no rebuild needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two small changes:
* export_tts_text_embeddings.py now takes the voice wav as an optional
second CLI arg (defaults to damien_15s_24k.wav). Lets the same script
capture voice-prefix+suffix for any speaker wav without editing the
source — used today to test Elodie alongside Damien.
* synthesizeTextStreaming + generateSegmentAudioVC only run the
trimTailLowEnergy trim when n >= maxGen. The trim's 35%-of-peak
threshold is tuned to catch "page beg beg" filler after the talker
fails to emit EOS — but it was cutting valid speech when EOS fired
early (observed on Elodie seg 1: 10.08 s → 2.92 s, a 4-second over-
trim). With the guard it's a no-op on converging generations and
only fires on the ~15% of segments that hit maxGen.
Validation after the fix (Elodie, Baer monologue):
- seg 1: 126 tokens = maxGen → trimmed 10.08 s → 8.88 s (1.2 s cut,
the filler tail)
- seg 2: 105 tokens < 138 maxGen → no trim, 8.4 s kept as-is
- seg 3: 69 tokens < 96 maxGen → no trim, 5.6 s kept as-is
Voice prefix/suffix shape is speaker-invariant except position 7 (the
xvector). Confirmed by capturing both Damien and Elodie and diffing:
positions 0-6 and 8 identical within 1e-8, suffix identical within
1e-8, only pos 7 has a different xvector embedding (norm 10.36 vs 10.12).
That means swapping speakers on-device is a 45 KB file push — no app
rebuild, no re-export of the 297 MB vocabulary table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the four per-sentence TTS entry points (pipeline.speak, REPEAT
voice command, echo-mode TTS, LLM-response TTS) with a single shared
pipeline.speakText() that:
* opens a Qwen3TtsEngine streaming session when the TTS backend is
Qwen3 (voice-cloning path);
* feeds the whole response through a SentenceStreamer so the first
sentence starts playing as soon as it's decoded;
* falls back to the old one-shot synthesizeAndPlay for non-Qwen3 TTS
engines (AndroidTts, Chatterbox) that don't expose a session API.
KazeiaPipeline.speakText is now public so KazeiaService can use the
same dispatch — previously each call site re-implemented the
"streaming-or-fallback" logic or just called synthesizeAndPlay and
waited for the full synthesis.
Enabling the real on-device LLM is a separate task (task #48): the
existing llama-cli binary has ggml-hexagon linked in and fails to
init the DSP (0x80000406) when the TTS Hexagon runners hold the
session. Needs either a CPU-only llama-cli build or the restored
ExecuTorch qnn_llama_runner setup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes the loop on on-device conversational TTS. The LLM's token stream is
now consumed by a SentenceStreamer which fires a callback the moment a
terminal-punctuation boundary appears; each sentence is enqueued to a
persistent TTS streaming session that generates and plays audio through a
single shared AudioTrack. Sentence N's audio plays while sentence N+1 is
being generated on Hexagon+CP — no per-sentence AudioTrack init gap, and
no "wait for full response before hearing anything".
Mocked-LLM validation on the 3-sentence prompt:
"Bonjour. Je suis là pour vous écouter. Comment allez-vous aujourd'hui."
- First sentence detected: 1 ms
- Seg 0 prefill (Hex): 567 ms
- Seg 0 generated: 4 200 ms (18 tokens, 1.4 s audio)
- Seg 1 generated: 9 100 ms (42 tokens)
- Seg 2 generated: 11 000 ms (46 tokens)
- Session closed: 33 500 ms (all audio drained)
Changes:
* tts/SentenceStreamer.kt — 50-line helper that buffers tokens and
fires onSentence when a "." "!" "?" ";" or "\n" appears. minChars = 4
so "Oui." / "Bonjour." count as real sentences; higher thresholds
swallowed conversational openers into the next segment and delayed
first audio. flush() for the final partial sentence.
* Qwen3TtsEngine.startStreamingSession / enqueueSentence / endStreamingSession
triplet. startStreamingSession opens a 30-second MODE_STREAM
AudioTrack plus a background worker coroutine that pulls sentences
from an unlimited Channel. enqueueSentence is non-blocking; the worker
serialises generation so audio order matches enqueue order.
generateSegmentAudioVC is the per-sentence body (tokenize → prefill
build → Hexagon gen → decode) without the WAV-save side effects that
the /stream_text intent path does.
* KazeiaService new intents:
- stream_llm : real LLM path (needs LLM loaded; currently the
debug build runs echo-mode so this path is
shipped but requires production config to
exercise).
- stream_llm_mock : fakes the LLM stream by splitting the given
text on spaces with 50 ms per "token" —
matches the ~20 tok/s rate the on-device LLM
produces and lets Stage 3 be validated without
flipping the LLM on.
Architectural notes:
- AudioTrack buffer is 30 s so generation can run ahead of playback
without blocking writes. RTF on Snapdragon 8 Elite is ~3 for short
sentences, so for a 2-3 sentence response the buffer actually drains
between segments and the user hears a short gap — expected, not a
bug. Masking that gap requires RTF < 1 which is out of scope.
- Hexagon KV is reset between sentences (hexReset) so the talker
doesn't see stale context. Prefill observed cb0 = 1995 on every
sentence that starts with a capital letter, matching the Python
greedy reference — confirms prefill reconstruction is stable across
segments within a session.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Removes the PC-side prepare_tts_segments.py dependency for day-to-day
generation. The tablet now tokenizes, embeds, and voice-clones any
French (or Qwen3-supported) text with no network, no ADB push per
phrase, and quality that matches Python's reference on "Bonjour, je
suis Kazeia, je suis là pour vous écouter." — user validation:
"impeccable".
Three pieces that compose the path:
1. Qwen3BpeTokenizer.kt — byte-level BPE matching Qwen2/Qwen3's
Python implementation bit-for-bit. UTF-8 + GPT-2 byte encoder,
Qwen regex with \p{IsAlphabetic}/\p{IsDigit} (Android's regex
lacks UNICODE_CHARACTER_CLASS — caught in testing). Produces
identical token IDs to HF's Qwen2TokenizerFast on the test phrase:
[81581, 11, 4759, 35631, 730, 9832, 685, 11, 4759, 35631, 37915,
4914, 9012, 90229, 2676, 13].
2. export_tts_text_embeddings.py — one-time PC export of:
* Full projected text embeddings for the entire 151936-token vocab
as fp16 (297 MB). Sanity check: live vs stored max abs diff
1.15e-4 on token 1043. Mmap'd on-device so it stays off the
Java heap and leaves room for the 125 MB cp_embeddings alloc.
* Damien voice PREFIX (9 × 1024 fp32) — positions 0..8 of a
Python voice-clone capture, text-invariant across segments.
* Damien voice SUFFIX (2 × 1024 fp32) — positions nP-2..nP-1
of the same capture. Also text-invariant (diff = 0.0 across
3 different-text segments). Without it the talker never sees
"text ended" and decode falls into page/beg repetition.
* Qwen3 tokenizer vocab.json + merges.txt.
3. Qwen3TtsEngine.kt:
* mmap loader for the embeddings table + buffered fp16→fp32
lookup (halfToFloat covers subnormals/inf/NaN so pathological
tokens don't become 0).
* Stage 2 assets detected at init; missing file transparently
falls back to legacy 1050-token reduced-vocab path.
* synthesizeTextStreaming(text, onSegmentReady) — new public API:
sentence-split → BPE → build prefill as
[voice prefix] + [text_proj(id) + codec_pad] × N + [voice suffix]
(exact structure Python emits; verified bit-for-bit by matching
captured Baer prefill positions against text_projection(tok)+
codec_embedding(CODEC_PAD)) → runHexGenWithPrefill → decode
each segment through the existing BigVGAN pipeline → callback.
* runHexGenWithPrefill — Hexagon prefill + interleaved CP decode
loop. Feeds tts_eos once, tts_pad thereafter (same schedule as
Python's voice_clone). Degeneracy guard stops when 9 identical
cb0 in a row appear — catches the rare "page beg beg beg" tail
when EOS never fires. maxGen = ids.size*4 + 10 matches the
typical 3.3 codec-frames-per-text-token that Python produces.
* Prefill build uses the speaker's captured prefix/suffix rather
than the legacy in-code buildPrefillEmbeddings that puts only
one text token in prefill — the structure mismatch produced
garbled audio in the first attempt of this commit.
4. KazeiaService.kt: new stream_text intent extra wires text input
to synthesizeTextStreaming with an AudioTrack MODE_STREAM consumer.
First-audio latency on the "Bonjour..." test: ~23 s on Snapdragon
8 Elite (prefill + 74-token decode), vs a 3-phrase sentence batch
that was 65 s pre-streaming — streaming + on-device text together
unblock the MVP chat loop.
Known caveats:
* 297 MB on-device footprint for the embedding table. Acceptable on
OnePlus Pad 3; can be quantized further (int8 per-row) if storage
becomes tight.
* First init adds ~3 s for BPE vocab + merges load (151k × 2 hash-
maps). Happens once per process.
* maxGen cap means extremely long sentences may truncate. The
sentence splitter already keeps segments ≤120 chars so this
hasn't been observed in practice.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a streaming multi-segment pipeline on top of the Hexagon talker + ONNX
CP backend. First audio arrives at ~20s (vs ~65s for the full phrase
non-streamed) on the Baer 16.56s reference (3-segment split). Voice cloning
is preserved per segment because each segment now ships its own full prefill.
Changes:
* Qwen3TtsEngine.generateFromEmbedsHexagonStreaming(path, onSegmentReady)
reads single- or multi-segment embeds, runs prefill + generation + VQ
decode + BigVGAN per segment, and fires the callback with each
segment's ShortArray the moment it's ready. Saves per-segment WAVs
(kazeia_stream_seg{N}.wav) plus the concatenated kazeia_stream_full.wav
for offline inspection. Extracted the common generation loop into
runHexSegmentFromEmbeds(prefill, trailing, idx) so single-segment and
streaming paths share exactly the same code (no quality drift between
modes). Added hexReset() between segments so segment 2's prefill logits
don't contain segment 1's KV state.
* vqDecode buffer overrun fix: when the talker samples CODEC_EOS as cb0
it stores a vocab id > CODEBOOK_SIZE, which vqDecode then used as a
codebook row index — reading past the 2048-row buffer. The short Baer
probe never hit this; longer phrases do. Clamp any out-of-vocab code
to 0 at allCodebooks build time.
* KazeiaService: new stream_pipeline intent extra wires the callback
to an AudioTrack MODE_STREAM instance, writing each segment's audio as
soon as it comes back. Logs time-to-first-audio.
* prepare_tts_segments.py: the previous version only captured 1-token
decode calls and substituted a generic 9-embed "prefill_base" pulled
from an unrelated single-segment file — dropping the per-segment
xvector conditioning AND the text-encoded embeddings, so Hexagon
produced garbled mixed speech for segments 2..N. Now captures the
multi-token prefill call too (like prepare_tts_voiceclone.py) so each
segment is self-contained.
Limitation (documented, not fixed in this commit): RTF ~4.4 > 1 on the
Snapdragon 8 Elite with current config means each segment takes longer to
generate than it takes to play, so audible gaps between segments remain.
Removing the gaps requires either (a) producer/consumer parallelism across
two coroutines (doesn't help if RTF stays > 1), or (b) faster CP (the
~180ms/step ONNX MLAS CP is the bottleneck; Hexagon HMX has a known NaN bug
and the .pte path contends with Hexagon talker on the DSP).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kazeia Team