chore(stt): retrait du backend STT "lib" (non utilisé/non validé)
Le backend STT "lib" (KazeiaSttAdapter → com.kazeia.stt.SttEngine → libkazeia_stt) n'était jamais le défaut (sttEngine="prod"=Whisper) et restait non validé (transcriptions vides/drift, A/B non mergé). Retiré : - KazeiaService : dispatch STT simplifié → WhisperHybridEngine seul. - archivés : KazeiaSttAdapter.kt, stt/SttEngine.kt, BenchQnnHarness.kt + libkazeia_stt.so (/opt/Kazeia/archive/2026-06-18_cosyvoice-bascule/stt-lib/). - KazeiaApplication : trigger bench_qnn retiré. - ConfigStore.sttEngine documenté vestigial. - manifest jniLibs re-figé (18→17). Gardés : WhisperHybridEngine (NPU, gelé), MelExtractor (mel_extractor), core.SttEngine. Build vert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kazeia Team
parent
a1c785bdf8
commit
ecda7b5715
|
|
@ -46,19 +46,6 @@ class KazeiaApplication : Application() {
|
|||
// requires its own initialization sequence. Loading libs out of order causes
|
||||
// native crashes. Unity handles lib loading internally in onCreate().
|
||||
|
||||
// Bench QNN options : déclenché par `touch <files>/bench_qnn.trigger` puis relance app.
|
||||
// Charge WhisperHybridEngine + transcrit /files/bench_audio/*.wav (3×, médiane).
|
||||
val trig = java.io.File(getExternalFilesDir(null), "bench_qnn.trigger")
|
||||
if (trig.exists()) {
|
||||
trig.delete()
|
||||
Thread {
|
||||
com.kazeia.stt.BenchQnnHarness.runBench(
|
||||
ctx = this@KazeiaApplication,
|
||||
audioDir = java.io.File(getExternalFilesDir(null), "bench_audio"),
|
||||
modelDir = "$MODELS_DIR/whisper-small-sm8750"
|
||||
)
|
||||
}.start()
|
||||
}
|
||||
// Bench LLM (lib CPU-only) : `touch <files>/bench_llm.trigger`, relance app.
|
||||
// Charge EngineLlmEngine + 3 prompts FR thérapeutiques.
|
||||
val trigLlm = java.io.File(getExternalFilesDir(null), "bench_llm.trigger")
|
||||
|
|
|
|||
|
|
@ -61,9 +61,9 @@ class ConfigStore private constructor(private val file: File) {
|
|||
/** Synthèse vocale active. Si false : pas de TTS, réponse affichée
|
||||
* progressivement (token par token). Pilotable depuis l'app admin. */
|
||||
val ttsEnabled: Boolean = true,
|
||||
/** Backend STT : "prod" = WhisperHybridEngine gelé, "lib" = libkazeia_stt
|
||||
* (Kazeia-Engine unifié, A/B identique sur set 6 audios mais à valider plus large).
|
||||
* Pilotable depuis l'app admin / provider. Defaults safe = "prod". */
|
||||
/** Backend STT : VESTIGIAL depuis le retrait du backend "lib" (2026-06-18,
|
||||
* non validé). KazeiaService charge toujours WhisperHybridEngine (NPU, gelé).
|
||||
* Champ conservé pour compat config/télémétrie. */
|
||||
val sttEngine: String = "prod",
|
||||
/** Backend LLM : VESTIGIAL depuis la refonte 2026-06-17. Le dispatch ne passe
|
||||
* plus par cette valeur : KazeiaService charge tout LLM via UnifiedLlmAdapter
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
// Harness LLM ADB-pilotable, mirror de BenchQnnHarness STT.
|
||||
// Harness LLM ADB-pilotable (bench LLM in-app).
|
||||
// Déclenchement : `touch <files>/bench_llm.trigger`, relance app.
|
||||
// Valide le chemin refonte 2026-06-17 : UnifiedLlmAdapter (magic-byte) sur
|
||||
// 1) le GGUF par défaut (Qwen3.5-4B, CPU i8mm + session cache-préfixe)
|
||||
|
|
|
|||
|
|
@ -480,18 +480,13 @@ Pas d'introduction, pas d'explication. Juste les 4 lignes en francais. /no_think
|
|||
// pendant TTS (caché derrière le playback) et release pendant
|
||||
// les phases LLM/TTS où Whisper ne sert pas.
|
||||
_loadingState.value = LoadingState(25, "STT Whisper (lazy)…")
|
||||
// Backend STT pilotable via ConfigStore.sttEngine. Default "prod" =
|
||||
// WhisperHybridEngine gelé. "lib" = KazeiaSttAdapter (libkazeia_stt.so).
|
||||
val sttBackend = runtimeConfig.sttEngine
|
||||
val npuStt: com.kazeia.core.SttEngine = when (sttBackend) {
|
||||
"lib" -> com.kazeia.stt.KazeiaSttAdapter(
|
||||
"$modelsDir/whisper-small-sm8750"
|
||||
) { msg -> log("[STT] $msg") }
|
||||
else -> WhisperHybridEngine(nativeLibDir) { msg -> log("[STT] $msg") }
|
||||
}
|
||||
// STT = WhisperHybridEngine (NPU, gelé) — SEUL backend depuis le retrait
|
||||
// du backend "lib" (KazeiaSttAdapter/libkazeia_stt) non validé, 2026-06-18.
|
||||
// ConfigStore.sttEngine est désormais vestigial (compat config).
|
||||
val npuStt: com.kazeia.core.SttEngine = WhisperHybridEngine(nativeLibDir) { msg -> log("[STT] $msg") }
|
||||
sttModelPath = "$modelsDir/whisper-small-sm8750"
|
||||
stt = npuStt
|
||||
log("Whisper créé backend='$sttBackend' (lazy load au 1er PTT)")
|
||||
log("Whisper créé (lazy load au 1er PTT)")
|
||||
addMessage(ChatMessage(
|
||||
role = ChatMessage.Role.SYSTEM,
|
||||
text = "STT: Whisper-Small NPU"
|
||||
|
|
|
|||
|
|
@ -1,104 +0,0 @@
|
|||
// A/B harness : WhisperHybridEngine (prod) vs SttEngine (Kazeia-Engine lib unifiée)
|
||||
// sur les mêmes WAVs PCM16 mono 16 kHz. 3 runs/audio, médiane.
|
||||
// Déclenchement : `touch <files>/bench_qnn.trigger`, relance app.
|
||||
package com.kazeia.stt
|
||||
|
||||
import android.content.Context
|
||||
import android.util.Log
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import java.io.File
|
||||
|
||||
object BenchQnnHarness {
|
||||
private const val TAG = "BenchQnn"
|
||||
|
||||
fun runBench(ctx: Context, audioDir: File, modelDir: String) {
|
||||
val wavs = audioDir.listFiles { _, n -> n.endsWith(".wav") }?.sortedBy { it.name }
|
||||
?: run { Log.e(TAG, "no WAVs in $audioDir"); return }
|
||||
|
||||
// -------- A/ WhisperHybridEngine (prod patché QNN) --------
|
||||
Log.i(TAG, "=== A/ WhisperHybridEngine (prod) ===")
|
||||
val prodTimes = mutableMapOf<String, Long>()
|
||||
val prodTexts = mutableMapOf<String, String>()
|
||||
run {
|
||||
val eng = WhisperHybridEngine(nativeLibDir = ctx.applicationInfo.nativeLibraryDir) { Log.i(TAG, it) }
|
||||
runBlocking { eng.load(modelPath = modelDir) }
|
||||
if (!eng.isLoaded()) { Log.e(TAG, "WhisperHybridEngine load FAILED"); return@run }
|
||||
wavs.firstOrNull()?.let { readWavPcm16(it)?.let { p -> runBlocking { eng.transcribe(p, "fr") } } }
|
||||
for (f in wavs) {
|
||||
val pcm = readWavPcm16(f) ?: continue
|
||||
val t = mutableListOf<Long>(); var text = ""
|
||||
repeat(3) {
|
||||
val t0 = System.currentTimeMillis()
|
||||
text = runBlocking { eng.transcribe(pcm, "fr") }.text
|
||||
t.add(System.currentTimeMillis() - t0)
|
||||
}
|
||||
t.sort()
|
||||
prodTimes[f.name] = t[1]; prodTexts[f.name] = text
|
||||
Log.i(TAG, "[PROD] ${f.name}: median=${t[1]}ms '${text}'")
|
||||
}
|
||||
eng.release()
|
||||
}
|
||||
|
||||
// -------- B/ SttEngine (libkazeia_stt) --------
|
||||
Log.i(TAG, "=== B/ SttEngine (libkazeia_stt) ===")
|
||||
val libTimes = mutableMapOf<String, Long>()
|
||||
val libTexts = mutableMapOf<String, String>()
|
||||
val libSteps = mutableMapOf<String, Triple<Int, Int, Int>>()
|
||||
run {
|
||||
val eng = try { SttEngine(modelDir, useHtp = true, nThreads = 6) }
|
||||
catch (e: Exception) { Log.e(TAG, "SttEngine load FAILED: ${e.message}"); return@run }
|
||||
wavs.firstOrNull()?.let { readWavPcm16(it)?.let { p -> try { eng.transcribe(p, "fr") } catch (_: Exception) {} } }
|
||||
for (f in wavs) {
|
||||
val pcm = readWavPcm16(f) ?: continue
|
||||
val t = mutableListOf<Long>(); var r: SttResult? = null
|
||||
repeat(3) {
|
||||
val t0 = System.currentTimeMillis()
|
||||
try { r = eng.transcribe(pcm, "fr") } catch (e: Exception) { Log.e(TAG, "transcribe fail ${f.name}: ${e.message}"); return@repeat }
|
||||
t.add(System.currentTimeMillis() - t0)
|
||||
}
|
||||
if (r == null) continue
|
||||
t.sort()
|
||||
val rr = r!!
|
||||
libTimes[f.name] = t[1]; libTexts[f.name] = rr.text
|
||||
libSteps[f.name] = Triple(rr.melMs, rr.encoderMs, rr.decoderMs)
|
||||
Log.i(TAG, "[LIB ] ${f.name}: median=${t[1]}ms mel=${rr.melMs} enc=${rr.encoderMs} dec=${rr.decoderMs} ntok=${rr.nTokens} '${rr.text}'")
|
||||
}
|
||||
eng.release()
|
||||
}
|
||||
|
||||
// -------- A/B récap --------
|
||||
Log.i(TAG, "=== A/B récap ===")
|
||||
var prodTot = 0L; var libTot = 0L; var audioTot = 0L
|
||||
var same = 0; var near = 0
|
||||
for (f in wavs) {
|
||||
val a = prodTimes[f.name] ?: continue
|
||||
val b = libTimes[f.name] ?: continue
|
||||
val durMs = (readWavPcm16(f)?.size ?: 0) * 1000L / 16000
|
||||
prodTot += a; libTot += b; audioTot += durMs
|
||||
val ta = (prodTexts[f.name] ?: "").trim().lowercase()
|
||||
val tb = (libTexts[f.name] ?: "").trim().lowercase()
|
||||
val isSame = ta == tb
|
||||
val isNear = !isSame && ta.replace(Regex("\\W+"), "") == tb.replace(Regex("\\W+"), "")
|
||||
if (isSame) same++ else if (isNear) near++
|
||||
val mark = if (isSame) "≡" else if (isNear) "~" else "≠"
|
||||
Log.i(TAG, " ${f.name.padEnd(14)} prod=${a}ms lib=${b}ms Δ=${b - a}ms $mark")
|
||||
Log.i(TAG, " prod='${prodTexts[f.name]}'")
|
||||
Log.i(TAG, " lib ='${libTexts[f.name]}'")
|
||||
}
|
||||
if (audioTot > 0) {
|
||||
Log.i(TAG, "TOTAL prod=${prodTot}ms (RTF ${"%.3f".format(prodTot.toDouble() / audioTot)}) lib=${libTot}ms (RTF ${"%.3f".format(libTot.toDouble() / audioTot)})")
|
||||
}
|
||||
Log.i(TAG, "ACCURACY identique=$same/${wavs.size} proche(ponct.)=$near diff=${wavs.size - same - near}")
|
||||
Log.i(TAG, "=== Bench end ===")
|
||||
}
|
||||
|
||||
private fun readWavPcm16(f: File): ShortArray? = try {
|
||||
val b = f.readBytes()
|
||||
var off = 44
|
||||
for (i in 12 until b.size - 8)
|
||||
if (b[i] == 'd'.code.toByte() && b[i+1] == 'a'.code.toByte() &&
|
||||
b[i+2] == 't'.code.toByte() && b[i+3] == 'a'.code.toByte()) { off = i + 8; break }
|
||||
val n = (b.size - off) / 2
|
||||
ShortArray(n) { i -> ((b[off + 2*i].toInt() and 0xFF) or (b[off + 2*i + 1].toInt() shl 8)).toShort() }
|
||||
} catch (e: Exception) { Log.e(TAG, "wav fail ${f.name}: ${e.message}"); null }
|
||||
}
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
package com.kazeia.stt
|
||||
|
||||
import com.kazeia.core.SttEngine as CoreSttEngine
|
||||
import com.kazeia.core.TranscriptionResult
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
|
||||
/**
|
||||
* Adapter qui expose la lib unifiée Kazeia-Engine STT
|
||||
* (com.kazeia.stt.SttEngine, façade native de libkazeia_stt.so) derrière
|
||||
* l'interface prod com.kazeia.core.SttEngine. Sert au swap réversible piloté
|
||||
* par le flag config `sttEngine` ("prod" | "lib").
|
||||
*
|
||||
* Mesuré 2026-06-01, A/B sur 6 audios FR 3 s mono 16k vs WhisperHybridEngine :
|
||||
* - accuracy 6/6 identique au texte prod
|
||||
* - RTF global −21 % (gain decodeur-lourd, mel DFT 400 plus cher que prod FFT 512)
|
||||
* - RAM ~−30 % (≈379 MB vs 545)
|
||||
* Cf [[project-stt-engine-ab-in-app]] pour le détail.
|
||||
*/
|
||||
class KazeiaSttAdapter(
|
||||
private val modelDir: String,
|
||||
private val useHtp: Boolean = true,
|
||||
private val nThreads: Int = 6,
|
||||
private val onLog: ((String) -> Unit)? = null
|
||||
) : CoreSttEngine {
|
||||
private var engine: SttEngine? = null
|
||||
|
||||
override suspend fun load(modelPath: String?) = withContext(Dispatchers.IO) {
|
||||
if (engine != null) return@withContext
|
||||
val t0 = System.currentTimeMillis()
|
||||
val dir = modelPath ?: modelDir
|
||||
engine = SttEngine(dir, useHtp = useHtp, nThreads = nThreads)
|
||||
onLog?.invoke("KazeiaSttAdapter loaded in ${System.currentTimeMillis() - t0}ms (path=$dir)")
|
||||
}
|
||||
|
||||
override fun isLoaded(): Boolean = engine != null
|
||||
|
||||
override suspend fun transcribe(audioData: ShortArray, language: String): TranscriptionResult =
|
||||
withContext(Dispatchers.IO) {
|
||||
val e = engine ?: throw IllegalStateException("KazeiaSttAdapter not loaded")
|
||||
val r = e.transcribe(audioData, language = language)
|
||||
TranscriptionResult(
|
||||
text = r.text,
|
||||
confidence = 1.0f,
|
||||
language = language,
|
||||
durationMs = r.totalMs.toLong()
|
||||
)
|
||||
}
|
||||
|
||||
override fun release() { engine?.release(); engine = null }
|
||||
}
|
||||
|
|
@ -1,132 +0,0 @@
|
|||
package com.kazeia.stt
|
||||
|
||||
// Engine STT Whisper-Small (Qualcomm AI Hub HfWhisper KV-cache) in-process.
|
||||
// Backend : ONNX Runtime + QNN ExecutionProvider (HTP V79 sur SM8750).
|
||||
// Empreinte RAM : ~545 MB (encoder ctx 201 MB + decoder ctx 345 MB).
|
||||
//
|
||||
// Bench prod (Whisper-Small, FR, audio 1.6s) : ~825 ms, RTF 0.51.
|
||||
// Bench port C++ (audio 5s, FR) : ~2000 ms, RTF 0.40 — voir CLAUDE.md/STT.
|
||||
//
|
||||
// Le bridge natif est dans dist/jni/kazeia_stt_jni.cpp ; build via
|
||||
// dist/build_kazeia_stt.sh.
|
||||
|
||||
class SttJni {
|
||||
external fun nativeLoad(
|
||||
modelDir: String,
|
||||
useHtp: Boolean,
|
||||
nThreads: Int,
|
||||
maxDecodeSteps: Int
|
||||
): Long
|
||||
|
||||
/** Renvoie une chaîne : "err|text|mel_ms|enc_ms|dec_ms|total_ms|n_tokens|rtf" */
|
||||
external fun nativeTranscribe(
|
||||
handle: Long,
|
||||
pcm: ShortArray,
|
||||
sampleRate: Int,
|
||||
language: String,
|
||||
forceTranscribe: Boolean
|
||||
): String
|
||||
|
||||
external fun nativeFree(handle: Long)
|
||||
|
||||
external fun nativeVadNew(
|
||||
sampleRate: Int, frameSize: Int, rmsThreshold: Int,
|
||||
minSpeechFrames: Int, silenceEndFrames: Int
|
||||
): Long
|
||||
|
||||
/** 0 = silence, 1 = speech in progress, 2 = end_of_speech declenched */
|
||||
external fun nativeVadPush(handle: Long, pcm: ShortArray): Int
|
||||
external fun nativeVadReset(handle: Long)
|
||||
external fun nativeVadFree(handle: Long)
|
||||
|
||||
companion object { init { System.loadLibrary("kazeia_stt") } }
|
||||
}
|
||||
|
||||
data class SttResult(
|
||||
val text: String,
|
||||
val melMs: Int,
|
||||
val encoderMs: Int,
|
||||
val decoderMs: Int,
|
||||
val totalMs: Int,
|
||||
val nTokens: Int,
|
||||
val rtf: Float
|
||||
)
|
||||
|
||||
class SttEngine(
|
||||
modelDir: String,
|
||||
useHtp: Boolean = true,
|
||||
nThreads: Int = 6,
|
||||
maxDecodeSteps: Int = 200
|
||||
) {
|
||||
private val jni = SttJni()
|
||||
private val h = jni.nativeLoad(modelDir, useHtp, nThreads, maxDecodeSteps)
|
||||
init { require(h != 0L) { "SttEngine: load FAILED (modelDir=$modelDir, useHtp=$useHtp)" } }
|
||||
|
||||
/**
|
||||
* Transcrit un buffer PCM16 mono. Le sample rate doit être 16000 (resample externe sinon).
|
||||
* @param language "fr", "en", "auto", ... (cf Whisper langues, 99 codes ISO).
|
||||
* @param forceTranscribe true -> override <|translate|> en <|transcribe|>.
|
||||
*/
|
||||
@Throws(IllegalStateException::class)
|
||||
fun transcribe(
|
||||
pcm: ShortArray,
|
||||
language: String = "fr",
|
||||
sampleRate: Int = 16000,
|
||||
forceTranscribe: Boolean = true
|
||||
): SttResult {
|
||||
val raw = jni.nativeTranscribe(h, pcm, sampleRate, language, forceTranscribe)
|
||||
// Format : "err|text|mel|enc|dec|total|n_tokens|rtf"
|
||||
// limit=8 conserve un éventuel '|' dans le texte (au cas où) dans le 2e champ.
|
||||
val parts = raw.split('|', limit = 8)
|
||||
if (parts.size < 8) error("SttEngine.transcribe: format inattendu '$raw'")
|
||||
val err = parts[0].toInt()
|
||||
if (err != 0) error("SttEngine.transcribe err=$err raw='$raw'")
|
||||
return SttResult(
|
||||
text = parts[1],
|
||||
melMs = parts[2].toInt(),
|
||||
encoderMs = parts[3].toInt(),
|
||||
decoderMs = parts[4].toInt(),
|
||||
totalMs = parts[5].toInt(),
|
||||
nTokens = parts[6].toInt(),
|
||||
rtf = parts[7].toFloat()
|
||||
)
|
||||
}
|
||||
|
||||
fun release() { jni.nativeFree(h) }
|
||||
}
|
||||
|
||||
/**
|
||||
* VAD RMS énergie compatible drop-in avec VadStage.kt prod.
|
||||
* Paramètres par défaut : 16 kHz, frame 100 ms (1600 samples), seuil PCM16 brut 150,
|
||||
* 3 frames pour déclencher parole (300 ms), 8 frames pour fin (800 ms).
|
||||
*
|
||||
* Usage typique :
|
||||
* val vad = SttVad()
|
||||
* audioRecord.read(buf) { pcm ->
|
||||
* when (vad.push(pcm)) {
|
||||
* SttVad.SILENCE -> { ... }
|
||||
* SttVad.SPEECH -> { ... // bufferise }
|
||||
* SttVad.END_OF_SPEECH -> { stt.transcribe(buffer) ; vad.reset() ; buffer.clear() }
|
||||
* }
|
||||
* }
|
||||
*/
|
||||
class SttVad(
|
||||
sampleRate: Int = 16000,
|
||||
frameSize: Int = 1600,
|
||||
rmsThreshold: Int = 150,
|
||||
minSpeechFrames: Int = 3,
|
||||
silenceEndFrames: Int = 8
|
||||
) {
|
||||
private val jni = SttJni()
|
||||
private val h = jni.nativeVadNew(sampleRate, frameSize, rmsThreshold,
|
||||
minSpeechFrames, silenceEndFrames)
|
||||
fun push(pcm: ShortArray): Int = jni.nativeVadPush(h, pcm)
|
||||
fun reset() { jni.nativeVadReset(h) }
|
||||
fun release() { jni.nativeVadFree(h) }
|
||||
|
||||
companion object {
|
||||
const val SILENCE = 0
|
||||
const val SPEECH = 1
|
||||
const val END_OF_SPEECH = 2
|
||||
}
|
||||
}
|
||||
|
|
@ -12,7 +12,6 @@ d523468d62d9b603cb3354294d70d4b2feabf2c3f1e43b0c96c9aabf32813708 libc++_shared.
|
|||
b92ca83a53dab49e212fba411b48fcceca4331b05fe881e2b9220efed45e70b1 libkazeia_cosyvoice.so
|
||||
776f37e43433459a7350dc9d5d15dcb8eb3e17465a28fa78de92559bec3918bd libkazeia_engine.so
|
||||
f91df82c96d1efd3cbe6f7af6b9d261cd5c6f4a929c08bad49a4cfc1deff4477 libkazeia_pte.so
|
||||
fdd730af3daf132d0819b3fc99abd8aae79800aab7a9aad371e117c16bd3a3aa libkazeia_stt.so
|
||||
2ca77a28421fc0c680e0a5ddf0208a9471268370893f8dd4eba2a10a8aacb4ac libllama.so
|
||||
a207169a2696e8f0f7639090c2882152c734368b5696246a514d63b3ce0769c1 libomp.so
|
||||
95d01368b556f0c698cbbb6daa47494c666e0242a063cf1ca830202946435df4 libqnn_executorch_backend.so
|
||||
|
|
|
|||
Loading…
Reference in New Issue