From 655d65a65532588a596b149e661326b366dcb57c Mon Sep 17 00:00:00 2001 From: Richard Loyer Date: Tue, 9 Jun 2026 22:46:02 +0200 Subject: [PATCH] =?UTF-8?q?chantier=20RAG=20#2=20:=20mod=C3=A8le=20FR=20tr?= =?UTF-8?q?anch=C3=A9=20(e5-small=20voie=20a)=20+=20.so=20CPU-only=20+=20p?= =?UTF-8?q?atch=20converter?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Réponses aux 3 demandes du dev : #1 .so CPU-only livrée : dist/b-jni-cpu/libkazeia_engine.so (64 KB). DT_NEEDED = libllama/libggml/libggml-base (CPU-only dist/lib-cpu/) + système, AUCUNE dep libggml-hexagon -> pas de FastRPC -> pas de bloqueur SELinux. Validée tablette : charge e5+nomic, n_embd auto, déterministe, norme 1.0. #2 modèle FR tranché = VOIE A (multilingual-e5-small), blocage XLM-R levé. Question décisive "petit patch ou trou architectural ?" -> PETIT PATCH, mesuré : - runtime C++ a déjà UGM (unigram/SPM), découplé de l'arch (tokenizer_model=t5 -> LLAMA_VOCAB_TYPE_UGM). Pas de trou. - converter a déjà _xlmroberta_set_vocab (écrit t5/UGM) mais seul NomicBert le câblait. e5 declare architectures=[BertModel]+tokenizer Unigram -> tombait en BertModel.set_vocab -> chemin BPE -> get_vocab_base_pre échoue (hash inconnu). - fix = wiring dans BertModel (3 méthodes : __init__ détecte Unigram, set_vocab route xlmroberta, modify_tensors choppe positions) calqué sur Roberta/Nomic. Patch : dist/patches/bert_xlmroberta_unigram.diff (~25 lignes, à appliquer dans le fork ql par le dev — non commité dans ql par respect AGENTS.md). PROUVÉ end-to-end : converti e5-small-f16.gguf (384-dim) via converter patché, chargé sur build CPU-only tablette : cos(dormir,insomnie)=0.909 > cos(dormir,tarte)=0.821, marge FR 0.089 (nomic 0.04 -> e5 2.2x mieux), ranking propre, norme 1.0, déterministe. Voie c (nomic stopgap) documentée si besoin de dérisquer l'E2E in-app d'abord. #3 interface FIGÉE confirmée : loadEmbedder/embedText/freeEmbedder inchangées. Bit-identique garanti sur le MÊME build (CPU vs HTP divergent ~1.5e-2 sur Q4_K_M ; sans effet car ingestion+requête tournent sur le même build in-app). Doc dist/RAG_INTEGRATION.md mise à jour (décision modèle, patch, reproduction GGUF, interface figée, .so CPU-only). GGUF e5 = artefact distribution (non versionné git). --- dist/RAG_INTEGRATION.md | 60 ++++++++++++++++++--- dist/b-jni-cpu/libkazeia_engine.so | Bin 0 -> 64448 bytes dist/patches/bert_xlmroberta_unigram.diff | 63 ++++++++++++++++++++++ 3 files changed, 115 insertions(+), 8 deletions(-) create mode 100755 dist/b-jni-cpu/libkazeia_engine.so create mode 100644 dist/patches/bert_xlmroberta_unigram.diff diff --git a/dist/RAG_INTEGRATION.md b/dist/RAG_INTEGRATION.md index 7f3f000..ec52f1a 100644 --- a/dist/RAG_INTEGRATION.md +++ b/dist/RAG_INTEGRATION.md @@ -30,20 +30,64 @@ emb.release() Handle **séparé** du LLM Speaker/TTS. CPU pur, aucun HTP. Déterministe (pas d'échantillonnage). +**Interface FIGÉE** (demande dev #3) — ces 3 signatures ne changeront pas, tu peux t'y brancher : +``` +loadEmbedder(ggufPath: String, nThreads: Int, pooling: Int): Long +embedText(handle: Long, text: String): FloatArray? +freeEmbedder(handle: Long) +``` + +**.so CPU-only livrée** (demande dev #1, prérequis in-app) : `dist/b-jni-cpu/libkazeia_engine.so` (64 KB). +- DT_NEEDED vérifié : `libllama.so libggml.so libggml-base.so` + système — **AUCUNE dep `libggml-hexagon`** → pas de FastRPC, pas de bloqueur SELinux. +- À pousser dans `jniLibs/arm64-v8a/` avec les libs CPU-only `dist/lib-cpu/` (libllama/libggml/libggml-base/libggml-cpu) + `libc++_shared.so`. +- L'embedder partage ces libs avec LLM/TTS CPU-only (mêmes fichiers) — pas de duplication. +- Validé sur tablette : charge e5 + nomic, n_embd auto (384/768), déterministe bit-identique, norme 1.0. + --- -## Modèle d'embedding — décision critique (point #1 du spec) +## Modèle d'embedding — TRANCHÉ : multilingual-e5-small (voie a) -| | Statut | -|---|---| -| **Recommandé** | `multilingual-e5-small` (384-dim, FR-capable, pooling MEAN) — **à sourcer/convertir + versionner** | -| Testé localement | `nomic-embed-text-v1.5` (768-dim) — **anglo-centré, marge FR fine** | +**Décision : `multilingual-e5-small` (384-dim, FR), converti via patch converter (voie a).** Mesuré meilleur que nomic en FR, et le blocage de conversion est levé. -**Mesure FR (nomic, sur tablette)** : `cos("mal à dormir","insomnie")=0.556` vs `cos("mal à dormir","recette de tarte")=0.517` — ordre correct mais marge de **0.04 seulement**. Avec un mauvais préfixe l'ordre s'inverse (0.477 vs 0.611). **Conclusion : nomic n'est pas assez robuste en FR pour un RAG thérapeutique. Shipper `multilingual-e5-small` comme le spec l'exige.** +### La question du dev : XLM-R = petit patch ou trou architectural ? → **petit patch** + +Réponse mesurée, pas devinée : +- **Runtime C++** : le loader a déjà le chemin **UGM (unigram/SentencePiece)** (`llama-vocab.cpp` : `tokenizer_model=="t5"` → `LLAMA_VOCAB_TYPE_UGM`). Le type de vocab est **découplé de l'arch** — un modèle arch `bert` + tokenizer UGM charge sans souci. **Pas de trou architectural.** +- **Converter** : `conversion/bert.py` contient déjà toute la machinerie XLM-R (`_xlmroberta_tokenizer_init`, `_xlmroberta_set_vocab` qui écrit `tokenizer_model="t5"` + vocab unigram). Mais seul `NomicBertModel` la câblait. `multilingual-e5-small` déclare `architectures:["BertModel"]` + tokenizer `Unigram` → tombait dans `BertModel.set_vocab` → chemin BPE → `get_vocab_base_pre()` échoue (hash inconnu). + +**Le trou = un wiring manquant dans `BertModel` (3 méthodes), pas une absence de chemin.** Patch livré : `dist/patches/bert_xlmroberta_unigram.diff` (~25 lignes) — détecte `model.type=="Unigram"` dans `__init__`, route `set_vocab` vers `_xlmroberta_set_vocab`, choppe la matrice de positions dans `modify_tensors` (exactement comme `RobertaModel`/`NomicBertModel` le font déjà). + +### Validé end-to-end (converti + chargé sur build CPU-only, tablette) + +``` +multilingual-e5-small-f16.gguf : 384-dim, norme L2 = 1.0, déterministe +cos("j'ai du mal à dormir", "insomnie") = 0.909 <- proche +cos("j'ai du mal à dormir", "recette de tarte") = 0.821 +cos("j'ai du mal à dormir", "le chat dort...") = 0.843 +cos("j'ai du mal à dormir", "2+2=4") = 0.831 +MARGE FR (insomnie - tarte) = 0.089 (nomic = 0.04 -> e5 2.2x mieux) +``` +Ranking propre : l'item pertinent (`insomnie`) ressort nettement au-dessus des 3 non-pertinents (cluster ~0.82-0.84). Utilisable pour top-k + seuil. + +> Note e5 : cosinus de base élevé (~0.82 même entre phrases non liées) = normal (espace e5 anisotrope). Ce qui compte = le **ranking relatif**, correct ici. Avec l'asymétrie `query:`/`passage:` côté retrieval, la marge réelle est encore meilleure. + +### Reproduire le GGUF + +```bash +# 1. Appliquer le patch converter (dans le fork ql) +cd /opt/Kazeia-engine/ql && git apply ../dist/patches/bert_xlmroberta_unigram.diff +# 2. Convertir (venv avec torch + sentencepiece + gguf) +/opt/Kazeia/qnn_venv/bin/python convert_hf_to_gguf.py /chemin/multilingual-e5-small/ \ + --outfile multilingual-e5-small-f16.gguf --outtype f16 +``` +Source officielle : `intfloat/multilingual-e5-small` (`architectures:["BertModel"]`, tokenizer `Unigram`). Impératifs (contrat de distribution) : -- **Le MÊME GGUF à l'ingestion ET à la requête** (vecteurs incompatibles sinon). Versionner le n° de modèle. -- **Préfixes e5 (`query:` / `passage:`) = côté appelant**, jamais dans le moteur. +- **Le MÊME GGUF à l'ingestion ET à la requête** (vecteurs incompatibles sinon). Versionner le n° de modèle. (Bit-identique garanti seulement sur le **même build** : un GGUF généré sur build CPU vs build HTP diverge de ~1.5e-2 ; ingestion et requête tournent sur le même build in-app → OK.) +- **Préfixes e5 (`query:` / `passage:`) = côté appelant**, jamais dans le moteur. Pooling = MEAN (1) ou -1 (le GGUF e5 porte la métadonnée MEAN). + +### Stopgap nomic (voie c) si besoin de dérisquer l'E2E in-app tout de suite +`nomic-embed-text-v1.5` convertit déjà sans patch (WordPiece) mais FR faible (marge 0.04, s'inverse avec mauvais préfixe). Acceptable pour valider le câblage E2E, à swapper par e5 pour la qualité. Le code moteur est agnostique (n_embd auto) — le swap est juste un changement de fichier GGUF. --- diff --git a/dist/b-jni-cpu/libkazeia_engine.so b/dist/b-jni-cpu/libkazeia_engine.so new file mode 100755 index 0000000000000000000000000000000000000000..1ddc45cdd016a164922e78376943ef41a53219cb GIT binary patch literal 64448 zcmeFadwf*Y)i=J+Ig{kfBtQtc6PO7BO#&+6nkbmb4HNE$QIXP4t}v0@6Cm7VAXKBF zmeI6YybzGq%xGG)SixcmVEbUv7SyWkV`~8Q83?wZAQ>*s`(5WUIhkb8KF{;}yuUx* z_!;-vYp=cb+H0@9_S$RjbJ&+a zpSl%h!iUOm|9T(&Muva%dH*z7LAR+LQ~0EAQ+wN9krB4N+du7Agsi4&^}^2@%ZQcl z%BMP33zd4uZ?Z33i)2B|HK55E-n?Qhy#An1HWi-r#!AAvv7Y2ae9{_F*0AC&e zzAga#kpS@R0pPm>z+Vgie-O#Vdy;Ku^My92<#4gep9#cr^6%>m%{0Pv>* zz&{EAKNA3cB>?0pPs>=v)tg zuL~Jm4u$}5Qvmq10PvXs;MoD-ivz%|0pQC6z#o(5#Sm6mB3r(>iUE8@f$vk`ixv0| zNuP(X#!?y31-nLndnElC!sgb>dD3J0D^uuPERwRUNx@&K(3zmn$(M`S2uc1!*w&RY zz^uT@{fS^hm|gJ)!_Xi4dqc9T!2h0{Kd#W%DfGh?`jK)myF-DGRp6?B7On7q@c|ie z0_+t1-J`%wivKwQ{gMB8CjZAF<&^&j()X-B5ys%~K8%4+eKiJ_aWI!^`~B0waAg?z zW*n=A+)U^I`8kwdujHvur9)`Y1h!g{t(@n5dBUOYla4@)EFM4lYtQ0LCKky|&&YYk z@|m64IXgd5!d(dPBK$61ef04 zZ!Kc6ql|jM^&N;)T&A!q^x`uI9yph2pHrC;#L<|l@rsY zRkaH3s+zL$N^4p9vf`DMjn>-Y`r@jFenr$zO-HOt$|@JCjFr|@LwJpYSMQU6)CQU@ zE3dC-%WCT@svDQ}YeN!6O=)q70&OU+s;$I8NF7JB?(tlFE1#azsQ=FR%$D*w>H)nS2Q*hl;-6DmQh?;iN+gi>IslH zt+=)p(#~5rr&M95u?FHNg~8xpU=%1Z4M0u(8nrJ{##&lgURGY+RrQHyO2ntU2H@RI>%-M2JBb$#5+i-%$>%B|&1 zrRBAa6*Xl2Ed`4j(F)EG>;B zTq@JbnIzdFG;{C3W~|Gbnyj_u^$j(&s5P#!uA0coDlsyWIkeW=%D{`YVHR7qta4?8 zjTM*F)HkwPFi;M=UcVZaZ-sSrJth9omDLioO6oF`GAl@%prmA5w3u(<`^TnDO#50w4}{LiQB|awPDl=EZmAyb2%NBIm>X;1d*htRH-e0yp`=n-qAmAH0YH zDfpY{2hV?0hNt_%mnv|xAG}h5=lj9eEAWMW@XA)1{$fA)9tB?H2S1>|ZGP~h3cS`2 zeo=up`N10>liRiX!P6DLy2TH!#$mfZm+PtH!v0InyZzv11%5sNyw@N8FPTp5`*O-b zA7bQ3>Hkd{c!maU*T8Srz*{u%J2h}MzDptTObvXS20u#!-=n~J@PjJ8zkM3GvPF=} zx-@X#76BzkHE>1MCH&(WxUyA};2&w=!4gFNU$+LXxaU%VTLTw0_~$iny$0T^fg3b% z_7^!r{SDQ?MGZVm0}t20hiKrj8n_yAr(%-^K1_q3tbx;>ME#qnfou2s=^A*1ij35( zfsfF@^EL2D4Sb;n9;JaV*1)4R@FER7Rs*+b;Bgvwtp={{lc=~!10Siuw`<@D8hDEa zPBCcpZ<_`_S_L9~N&~0(yZYCzftyqy(hdzgQ3KzjfsfI^_i5ls8hDomK2`%is)3Kw zz>jO-H)-G>Y2Y_&;N2Q{vIg$fz*98v^BQ=n2HvZIkJrH22Xcn=e}V=sYT&nM;Ncqh ztr~c&22L>x_0Ob%PgH?OlQr-(4Sb>oK1l;l*T5%h;ARbciUyvqflt-I7i!?sH1Ne5 z_;d}tNCQvTz-=1%9U6G820lXrZ_>bLYT$Ma+^m7OXyDlz_%;nZM+1LK1JBjK+coe! z4ZK4GpQVBC(ZKK0!1rn3`5Jha240|nAJxETYv9K<@HrZ|*UMG1Jy>z}aN%zbVV`na zxc7%JH;*~BRS@?MVK3=K050lg93O(9o9^u)>}Zgw8-cOl{FjEXkApwuJ|4oZiDKIj zwoM=Y)DX5;zZ6K@4CXyUShry@o_j+VE4~7%-RL9Etnk;#HWJUn*wZD zw=iW-Fx#yQdnuTGtP>H5J*S&^E|?t;GMx`*TY`z!)?h+!3*JI4p`B>98&A8r$fGY`xeX%C>Nr3i~*5 zpMgEahwU-2-8|NSeZ1IVU|oDTW5;>1-M~KLKgJTgR}4F9U~Vz&83Sw4>z*>Ot$Jc> zn}J9@W#}LliN~8UDhxmIn!wHxAJ6bp+H~wwer~V8&I<(Y6=)6ucbYPmk%)7@$%!q& z%+7D)wxY{|Xh%kui@^9bop>&YZ4HV%AH>>&XfUL@OWE6ae47j3ZD3pYNHFT=#TElQ zkFXk~oxIJ!jtaZEtzb7u++tvRgTznu>_E^$3}!F^patD}Xu*VY0P8U#AL$9BTVEv6 z4`To&h%?w&7(A&_YFP6}F!FJ<{)~#J`aWt|*`#Sqj=9}}EfwNzx0~|jW#{GQ<`tSs z8=FkCvI^#!lE*cq7-;LTvU)Y*YPXxlHJEO}{|qemK5JgSbzz=$*~;qD`tnLsZ4Dw$ z<@F|Mzloi)sk)}o)UdL)7F(h6GS-CV%F?E=e9OX#411YH#jDE8W?|>rSg1qpgSx zvePD|BLKJRE zEH6Q`rt&5XW&_Dh+w{t6pB|cSEvIM&BD7X)+EpP#uUDFi8%^UXo1jB!lkv;IU|^F! zDT8Ti7uQ!!n?fol#jyJop!B5bXU1{K6I zn(2MwXqp6Orn0)#<<*m`rRY+hiRo)dnfI!=2kPylzM1c{M+nd?uZBu|i#<|Jkk9&* zfh7Qa`6d{vGN=J59_An-IhB=bOvNV5O?KB^^X6Lf^A=~_wP3#Wo<(_9m<@lM<1?lR zDXwfNhrKc(*mnP%pu;mnpH?2mH)Kmh_q_BJg%*Ld&{_=&<7VC!Kl({8Z#BIdUIh?A6Jf=LdHV>!YqRZ~-mzDv(?V6lOgDFdBi`INRSZIZSF z1IAUEo8QUk`eqcJK7EH3US5`KYAM`2(Edz^O`V4RPrM<2w1j>)U2mEMf)nwZ3Zhf+ z!wQNCo53csRpq6y#H@)*A?}Gj#Z(PRAlr*+TCeeJXbh>}nEe&i;0&rFX=|}Gx+uwp zVWh#H$nH^8UzUN-1)?QDSwtJj`eUhtbD}*jyP-7hntzYAx_mj7lU3!4XYrjz?nbMw zDuWx@ksH-loG{-x?v3b_LEGgs!^&I9lR&__6n+Yh!}@tKHtmU_?tgfhA2}@B`+*q};(v*KMd_i&uVGUD! z9-yDkOwM8L%6fUa*1|!7@`Gp}KKXLe!fGn3fo&jT1Nx;l)&#AE@{;HG?UO<=$TxQ0 zuzcl7r#)p0nQUm6m0-LFPR2BN$(T50NN+UOIOHs+)AN<(HyY(880<2*?rc1^7|*gO zZWnX2WNtF|G`E=hF$>`iuqA8`XW>qBuW(`P7!&y0*b8hm_uc<{kQuph+)+5so#+0E z|DR^>v9oL|`-J@-(0uk+*3LdafZL1KcJ>V;(k_dyvfn7el#1&Ipc1aBW$G45vy=Mw9>=o(*?$6dZ2y!0D!DLzF8hWXhn~FA>-E0$ zmDj7k>h=B#>3?1Edf&h3^*XNuj^E_3z22V!_A5L`I2WBT)W->lco; zcY@bfAcHvYdjV(nzGW zNRyFnK{^9zJJN+n_aMzj5d8oGo=+j&ia_S4NXH`JI0HfSXAoet0nd%}8Kh0+c8kyav|vH*M_U5fNgq+5}mN4ghj0uEW;L^=`aIiyRGE=43}8>M%n z4T2X!W(M?U5!ym}&4T_Tee*uFh17yecHyDGM>-a18Pas59Z2UQ-G;%hMcTB4u{NYz zk#-<`2I&E$FCjfndEAAI4FgT26Of)mYDU_N)Pi*EQj8bU-9G^TNcSMs;SlFIE~m#L zU0jB?k+#>O4C(uI_#yqY0c{Qi537+!n!W~lHw^OGjIl*}^oQU9>9vPI7imHZ#wHx~ z9)aEwuBA5OMX2x%> z9}Hq;X276w(jVd3@FN&PUeX_xt;a7PFqH=?n228+eiLp3T>|6|HRXqmGCiStF8Eo| zU<)&v@`tkb{_*$RP~T%R(2f!+>R_n+>hNYCB8sT3i}=+7SIeKs$uPBVV}X+ZSQ+s0 z@!X$}S|3Tm%LU#uS+XoxzLwu2A0ElG5_o%oC$*b9G(0zKRQPlJvpS+d;oWN}qyG1A zYjHms6q?p$52Qi#DO^c3{OMDf8)X*X&F4U7~kj#pprP64k8%0pj+*^FObO-n7cfcjxL@1pd2d+9# zs4bibu#W(v`R9{?)Ye>8^io??|G0uH5-$Fz{-=Pwjry`~DdQ{EU)Zlc=}ov!Zl7qW zf7CAB)2BXk5KyBdZIX`gEh?e+-v7tn%|KZUd~=>W$0htexylNl6)TN?u13u04Vq%l zfJPH&oWrw3L!(EE7PV_fRUy+i%%EdN)lWeuVHRkqbo_Z~>eu$5ywG~M(4U5hRGREk zP=1&}L*yTS56n-7#@vM=z)S6;dFmwL#cfx%W zxNX3tX9?G5Sunogx&3qzM;y}n@6!bzF6@jSZbI;Pqf33r2QK*)s@@PUO99&o*thXT zyx?SpwE(vk&x7d;kNNAHh8Wj;pdCP+H&I7AJRzMt&MgfaRixUPCk;=9{cPy=@TVi5 ziF_{l*;syGm=PGN$@%c3zy9q%r3p>CCc^J%x(j_!ZC26Z!AMDjgoPaD_~L$qXadqa zSqM5^U~U_p{l|@0mJMm(P)vQW1K(7DRBiXheGt;o^kAKs0a|Z^mSr~PDeR%t_Cib8 zD6>`^&kcPx+#oFF!;A|peeBb`A>PA8*wMM)jdvR78Ni=0ufIRA5U`1Uurk2L0!DJC zKNEgUfLQ=@6M*mVEjp5d17yPQ8Q{di_X~tWJahr)W8i4#)B6gH%IS|%H((zDPlXXL z*8qDTu=9kp7%cSJg<#qCYbW(zq}7kNY4i{fqL&Nba{Yq7{t;|3U@rj{Mzrxquu8yo zYhcNM*#Ub7Fp_`&xr=o~GaoBa{E~ueSI2iCzk&t&;hQHC#|D1y01(!V{z%%{CCv%q zHvvBJOYohweV7Q?Zoo{WulUQyFBh;q@U2^5XQe(2K6jL9Xb@rtP9KR=qJ=|McJ5x# zrADQ7bRf&~GYAv`nb}F=)9<6hDN&I6%kt0*5YJ^_F6MRGDhMP3?SzP+WwdI=St9lq=rozFEi>VJOEn^-TW@&WeXX6yolCT z(AtVUj`S||w~oEg1~WEr=!qx%J_4=CRT>|(7J9T$YYTrC{#?Yfkp}d!R%^;hFsM)b zBn*LXjJ=km>%@;4Zfu`D4U?=Xe>BW!_W7|NQgtXsKM2vXf!0T$<&$&mjg~5dPzF)g z4p^&lLx~n{p)d>fel~T?+Hbh-2`112N}s`37CveGhhoiL3tI@f()ek|-(QFOtGHy- z%&0$Mo#I=9L;}&lAGdblTxmxc`pi2&mq7`%x15XTvG^4sP7#LnO8oK1Ug-}W^o?iw z9zh;%+AgSTw=9oF&`XGi-wJt%zw*(ua{-22m;r&H6#Gb-mNMoR(CNajeI)!Z%q#I> zAFUgXR#1L|LGlS`z_jd<7YKSZX0hO{8y5xX{X6kVAKgKCgvTN&rXc!p`8Q&d8F$f2AiYub+GIf|} zim<2WZ-#Cl73cV9+Ha_-24T+tE+uO&#{!mu*3lHuq_Iiz(HvB7WS7q%ETPDw|fHY1@(KFk6xEBm>gAtk>C&*eFH61Oh%utfu1`}>A%rOPo7cAP624rpLMM8 z7o$n|i}{~I`1j*kB;~sV>j#t~gc(C=5rzhFKNsVTAE;#~k*9jN+!Rh}_BkQ^Zlylu zkMzj->+4>3UaR4pSC;e6UaR2z`aYfW;({ROb%8m&g5iY7%$~0W-gOecQ-aa;m7q(# zC<5`>0%w9f~#w@sAjh+Y!lB(E&7_cb;DHS$9Mr#@wi%tC#nK4o*v;$DkB9nm?`sZU3A8Ly#F z@Pi`Vf-%)w%CAAlEYdyH{7PI91N}IcH;%O>3+7iyE|;4}INeWEA7g#}xQcS)is|Xp zwum;}M{+aH{A!WYuz_224((7so=P@##-UyJq8atFqu8}!+pXooZgrg4rj zfW3=$FfAglLWWZL#00rKAxK&*Qr=+-VtX%POeJ2Q6QpsZavG;UBPAU< zfwT)t`w^tC1~Uh~fzWdVeZ&$FIUVx2yfNC@k;9)jhH_h&y|De^^_?9;?7j}+d%b6x zgB(sFruPhV%KZfMke+pih@I`N>pEqK&$0_m15W=Fgj%BI~D z&R%tc&%G$~zR32wYs?;^WeUmabSJ6wxXv+x*+=h`z&fFaju%mfXch77)vwS;(gz!3 z)4HEGdpb}?yhwS_+l_p;Fv5ko>bSTd5A!z{^Ee0dIUDmj%Xxj>A3De;P(Pmqu1Uv+ z?F!l0DY8w4zs`x>w>Rn5lw_J84-f5Z7e-y%*}S2%<4D}Tlfn?>|30I;!9J7Z(j5{e z^}RbZ6nzeouF&`D&x$aOz1C}DuGq!Q8MBaclKyT4tSEwcHc=YQJR2uLexND#hD<*Mn@4?l0A<>~ z=s9~~puRZK&+pQo`$3b&jOP91g_p)1VDEFHszKky72{zWXOOavR|0-WGEU5X&t;M1f_m0=v=f8UKRw2S6T~gSf^+$BbxF$Wt4EJ>%mc-z0fspXT7_WWt<()_IE$AX@>XLETg-WIZk5jcoY0dvfMMy zL9j03S1OBjy`h$M4U@|DBTwzi>m;>LYiOLbPWFe>`WY+1y9MSrCDDs=ofaaowll{h zZ1voNIimu!T6yy;XG6^%vU$nRW_yacN4Hbos6T{5^0kp|0?sD1?=CgVwkgK!*+9Gk z_u_)PAXE4e_sxQB%7abHaYEJ(vP&wB4#uWaT{p&*>fru)8_}#(X?jDZiRyR~-ZvrB z)UlS@M7|qs7Ab9hjQZ|jao4V|<6sAQ*h2wr>oBH47}H?q^&IkV#1r|hJHvUf zc~q~7XH$$iRyRA6S*|>2e&YVi557k}z&A0hO%&r>Pogf33CXR!Iia@$bryk!jgM>X z5JvUV8b~%@glzTbKlVNjL#TOwSA^N4^0Wzl0Pz9)>iSfO@$oVfbdCm_J#P(_#_K`` z*Rmb9_d+c2i3chtUnmo4#}V#?tfMiha-Kio6qfd~pzKaQu4|@Ki0$R|_L(e*b?!XE z12@io3i6U=G!0ME+!~>qGEvy6BxM^B*hme+n?-6@E_LwDcIEZ)9l2?An=K{ z7b7LxdSx={ICI?&ALz<-=An5+c9dY-;Tw~D$cLr**3rDQm*B|~Ty&9+5-!I=+Jg9c|%5oiaWuTqCtg*AOv9#ZiY$(kcvOCnq zO`u19>8_AL_8YKa8RVM_Y`}ga0rht|~tQ6D}H*+tkVvYkDkaTYddg|ZhJU_XI(6!hG{p}x^tHxYDI zo2lB%E6a2H*-Wz8_%4-Xqp6v3kkdIWKX)gDp6)DrYo9bg;no++{p+(ON z=o#{$=Y{kP-Js`rcqZOdx)-8Zq#5fhmCZ(bX4vI?N@0^J->*##{~@45{OgG}__w1z zufTLj$$wPaAl_yDU0q1i@d`rD9!x|HK2vUS*I= zB51duU+}|HX;Bb|ee1rg$JxHMzdMOJN_5cU5Ly4B$EYv5+|W^VZAbW0qDlQF zUHKgN-W&mbS_sw`5$lW|HpGB+Clu>?7<@I(5d+=0B99e(H0HXB^?UkIq7&mHJ^3wY zb&);-b^y~h@lUUusW@E3L_Hb*Ai^Qa(@rcJxiQHg*;^Nre=}&x3vLE3J zMV&}7Qd!GKxX2#0H;?Y!i@HT9+bbA*uc8mVp=`=UAvX0&2=i>0>c+dSMM37l?1cW3 z`El4DSLP^o0pB9Dq##yc(y;}hX68u!V#DL2XlOkr7+gi0GCQ{>vI{FVF~?4?*OJK@ zdb+Tl=+S;E`eMjq`;45v=Mc)ep(|v&PooWa-Hey~j61;7Ghc21?+LDtePs!d3v*3D z8R_N+K6s=HFUGlU1>6|I>c%6^Kypb8Mw^J`9Gzs|Z=B5PQn}dBY`}V;SGCZ|2RK7( zHK%W_;UZhBhK05|4Te^iVQA~G48zb5qpO+^Z{35R$c2Ws@}b8ZsPoG;6M7jJ7P=LE z|I={h*vN(Tum{ribyuYAGvk+u-y!VRuRuQ@f)0O(xvh@jZ16{8$iDQx+68;^7}h$9 z2aw#Gur^-9GxX8!f=nKVjh$fZU0JkQxI_k#}toakb*N?dPV+S{}3pvQo zBR-M;1oeN;rx)+ z@XgGT^jKzRoiMcZPaOORZfMVxds*kC`CKP=-D}y%MfPm^(rbAU@*;j7_2XX$o=X+} ziT88BDOY%32Hqcz>Y7;#-sgaKlHnQf|0wu>3H*=PvSOh_Jp9*| z&(M$E^EX0IPi(m`|J^NL&Tm9to;Z=&Ir*W}^Y6iLEA;#20@nHKd$`U+;L{2|DPB|o zJqqOvp)amY-B+&gE#bKLFdm7gZ&C&cc)-3^lsFpJTfOa-3{A(=|#cm zeV%uo#hkeMmCkt*cAa#m1$vJ8GbIu}AI%2~@>daO!QCfGKSb#DQLGn#lWb9piv!+k zz@fDO^K-9cH(>us=XL_luYd3fnsX;HhhCQO5WfQ5JMp|uvRxxxci>s|gVOQb4W7*? zn~LX^5>AwB62WQzwO}r6*&Nuk*|2Q|uyOfV`|onX=1KddDGJ{=0soHc-dE3(k5|f0 z1PiQ<_QtdiCSJ+UqZH@eQW~wK?ae>vZ5P<9qm(@HIs&w=qEC|%$8>`?8&blf_!H5g z^~M04$zCsP=m=K`o(a$W4f03@q&4@Fpo8BpVD04*yO81vueu?3S|hTF2l(Cc+AHU2jl!N^@_=ZhRrFB~7U*7lhtQ%0-wr7CaHqwpnvewtgbX`5DBb`scrVn~PkrVQr&nMzRQ`#?JUruos z?fHZg^N02e9nGWD{m&=DQBU13>=QDb!0DyFtLGD8D8JG9gbC#x&D@DeNZUgmM11)N zy$6Mn`<%@`NOv_`dpm@<-VU_i{_q{0G2nq>rMrX;o!$JS+jlkJF73Tg@ipfVgsQ=c$qQnI{nk z?{3ZO><;CmcsRBF|B^DFACOu?N|pEW4?qbaXL3MQUhGjql=vi@W8}m- z5&9babNiAHSB+jo^$e&l+x8gA$1XwM$D_iZ8m}YX`r>ubzIff;sM8hHJG~utoA^n@ z{8>zNaws+zol5j+9hi>ox+2izk%%`8$@mH6lUOz=Raq32=VXN`y&H8-FJf4)3v9~$8_kce8Qwd6u@oDPb-fm$jg7`4-U^)qF2%;e zK!^L3-FB3ZZKZt(ot+&fy+ivc(64@c8i`N7)15E0f z)U)s(Ox!0kW@BH)Hx&0$^i~M_mg)h8~-H4-f3(OG&dDHye z8E@V%{)Nu!Bc-`ZFgh1|p5Q(ogw7_6h#$!F`!`>ZZyD|S8sl37eW+0E;ys8_bf0F9 z#Yo8xer+V*M)ql^Nj|z$@($VjsjNocLtsStSjQBhb`lmo|uox5h z>dyhM2>qbv9a5dXxK&OBi%xU9g`ld7K07NinyJ3pfc*{4nehkDxx;aoRJ$ zXUDu}#}TtRPP``0$o6ELvpv59j>`W*sl7$<+KKTo)A@?<|)$IXrp{bAt!UBEJvaK{8O!)prv-kcF9r1j~D-qA9g^sQxb4OW;J@2^w^E-wmGNLm+-T z!etu@8v?lv#!FEE9;x5*9z_$IhCTTTaU{^iddW+A)eE{bX1_(dT|sQhJ)rjieiXYR z_^Z@Uz|Dx=?Mh;>=@?(|2}Q)(LHp4Wu9xv_4q{WKeHZ#oYuuyo->Ki5Ap@MrVLcn+ zqPWi?{wMLO3=&}vI)M8Ka4G(?>6({b13Ydt34ck535H)0OMt ztgRI5^%3k3wPW)O=otApWfa%Zj2ZlgHnlH@?$GpQ&v>~nPl-}r>Xg0+=tnSqfyV8U zZ`^bkw0C5qALaTCV^G9reh^4bMB{d% z!C2i8tg|PRXl!Lz(Kp_yKAMOL`O_@A?w#r$fxYx~@7}@i)bZ}d_|kZsyoxvTQuzW|w1jQ=9~{|v^UPsY+&ucTLd=X1_iFb*`fULgYh zFz$`qM|1XwZ_Y|`#UAGRI@}AxJfQvwXrFYJ`pmc;+a=v;<{Sr=c|KU3uOV;R{{^Z; zciF-byWIvGOY@AzLA}!^(_`QPwloKG8$4vw_<PUk=ZrWVXY1=G1njg*Yq*5KBfCcB@*Ki>$vC;JuZ}TQoT~7Q^Q{i7*=iZtR<%C_ zJA-lw2Kv~mw9==Uh9e%vH5re>lw} zYKL?V46W}hzg_cRp5wI2Us(+%kFV~m-6scb+ zGQUO8x$El=^|M2&?;_!)$#^#4!KT!Ak7$X%A-*CbhJT_&i>LM}JVWl|0Df+f+wrd_ z=f42orWyu^ZenCBCIPaycGU5 z)wNF-!u|E*-f+d9<=pt}i|i)RFi#c2cO79TPQy;pei(7awr(AJ^|jLvIwk(52;skh zuZ8!%+cMxc+>T%Jx*Siz#vI9?cphuv9e}M|o#UCzMI9@``PKSwa-K*BKAm%tZ$LOAuUNb(mD&j#Kj(Fv_B{D{`Z)dJ`Wp6Ks0scrj2Aw2ee&M9KglDtmv z7=e4ExOaa&dEH58JcGppFGJ_(t^lohU(Yh{qc!jo*c8k~%h#|IZrn{3ahB=<-br3O zhWD8zTWI9C9>h;9@Y|2GqT6U)iJWFO_fDV49IL+ZzAC}jhdJ;bM5K%p?SRZ8X&ol} zbr`a34`#16mzXoY#(i9O1bo_3*$1Hb1?2DQz&S*{i2V_8XfNbnE(~E24^a6Z%o!A) z#2Kn58GGe+(4l)|v-BuiX3qF6t-05Qy0rppHG^lOLv>|d$(lPUIw|i5l<7**Jv)fmHe2~;7MMOqg`WRS7*T=V;H|v8aQN$dT}l@ z_6T%#bEzR)qWu;8pE-)GZJ56)8|3+$1b>luyofqi5wD8F7+gWS(>aVa=JG7mp}S8M zdn0}0k*EH(L^ zwz;=u5_3%B#8y2%>q7ebAmFjXac^$j=m^lLE5;p;m?-9OqmTM1b2{!{PVI)gX+Ggz zm8TB#fc#N9`=xWtqExn@Vk=fIx(9ayEOSwZ;vqkZGe6#$cx?NMQOx1w^sX>(=Jcvj z?Czn={y5P&_!+DF0(Y5@0mq&;Z|#arJL2i=JcSEQt+mJVH^xJJ`qmVrB8xW^BgGr77> z&@>isb?0Coj9F*{XJ3@(kWzmMm*heGRFtp`e99)q7{}GkBm7ZZT|1>|T-_$51Mruz zjL!*w9#?0kJqh}N`DtnTgUrJxRHo))-(CRw z$UUUjo8G#n?un0QSB|88&S4VlsB~{9|$a{CSJehHK8#l1D?wfbS z`)D*`zlyQTT+iyxQQE}n+Tnjwp4JvRJI)kQhsty;%}gok96~B~3l0VwrnW`+nRe_? z!Gmbl`N~97-*aXytHaqc%HhAxOKer^i>wbvEb4pq9|v_T6Mw52%kXG5PwyPimxT&^lUa22}+Lro{ z{vJht5qq!6G~vE6W1S|(b&|YTOV>=ymzg{(>=Y+vdAO}zGn0Wc2IYx5`%G1)4Ej4Z zOrDo)cR_|U2PC=m?P0WiBiU;A8}fMj#cYnmpP)UBu|uCh`#GvF$y?c<+rzQX#Xjm0 z-u(F4=Q$kZ>W`6K>zd7lpDg7a?icmWi}x|l42%b4HpQKZm=fYBr+};M16_1a`X}%s zi-xf2RM-4IA3hiO$?y|W@Dt%9q`*%jzmR-$iYso$9WO(W*+chnZ1Benh$GN5)?7;z zcq2OOT|WF)(D@18$tnVm#=-_(ff-Y46!_vi+nlHMG#7ydL1e9&1Da8CjE zGn7%Ra|_1x_ZV}NfP1mnpPxZ&tXuf~jEy&O9$P%;aldHZUoo2V=!@D5ZDTl3MIpc6 z{X&kj=u!L3jgx?LD`5BX`@Kiv2h~5~*Zwvi&VyglCz6eJtjIs4cZq0k<9f zLx2n2Og#l2DJI>OXfI3xpY#sE=jh*5q(skXVi&S#jlILXUw_NyLjA(m@&22B#ZK5% zjK_XDAJ^Mu7zY^XAimS)kob<~Jjp0;j6fe;`ivqzqO<5R`^-T7AY9uneMX@2l{YA( zdC(0VTOKOwuwjeYL%hv-7Vj8{%rj>v(lDeyM;eaQi8K;vC(>A?FC$Gr`YKY)k2(91 zjz#(g(qyEEkWN7Qd!!SQzJ+uO(sz)iBRzq12GYMEH6#5i(p;o}N1BiHB+|J^KSR0@ z=^3OJq+cLijPwH1rARL!Ekf!+T88vGQX5h(%M)6OGze)eQa#c}q(hK4Asvo%J<=$o zcBFAgw=mD_QCS}SZheNKh3tZ|u1fRz2Jgt`OgQ1jev|Gh$#G@QMduOEC-Emv!j~ld zzbpi$l0GfG2ERUpV&KDl^WgnZ*0vM)yMX^0;AHbCMoja7Vyt$=%T6kA+H?2fc^K}% z(*6Wz?bAArEJ;tse78M_c+eN?A2-5x-9>A#-t<>7i{@~2&-0k8-OY3+GQveR+k@v| z%)gY=f+P8P!STQ;!Qp;^bEXJ7NAeMRX3mr(ox`>Z&o}FwHeTnTcP{A6cl{Wh<7|@P zv>g^4v8Bv$6|}zo73O6O^Tfbzm=NRf=1h?8RcP;2sP7^kggnV!itypiEMM|U@+Ps> zwyLF-Jz~C|JS-k-)$nUea9ml{TZ}bL-lL7co+%n*6ie@_WJq7Uc1^8^6Seh6* zukpYr);Ix`~~M!R;P zzu*_wxLG9KHMB$bq5pe3e!V2U1MB$(?D5V6?~_E~FLeH<=64`}Ss3m*2|XoSPx^oP z*PLe-+NL>v4|t>a=OU!=FSpV9kZAMyU3;?k2;_I|DZLmW`Iqk-b*=M+c9TMVC%x*VhU(Fl^ z@FT`g%yQx#Mcn1q;SN0SD!^Wv?m<)kRv*5$u;YapP7(Xplw`BB<5x4CMc;7FeBe~D zuwxxB-08IK!d|yql*WekSR~VRnC}~punYDAoIRdqCq!;!tG?LmjDk;)$;I|Kg@oSS zx0|nYVII<5MmOR$2JGuP0Jpw@dt$dSODjI1b8hVeO)K zy~aLXJgh1bn@FpHgM`_+CmXE4jd ze*H>TaChB_JM$P1RrX^5Z;6O+eG7b1%w!ID8xNl5{A>MtRp4nWAKLmN7usqEPZPky zT<}x`p7huBPd7uCh@TxJut%MWd%0iclTSXiVwc%Tc7S}FmM_H&6K9{`G{>H>Lg#G}qvZ=T_d(`Y7Im zC_MV)VD|JI-y~-FP^pKpY?8)u2y@W<%@D*^nm1%ia>g*r zI|`mUzSZc9J)&q(GDlnwc`%9k>l%5 zqFq(~>(Q5MLVRi%cxoxUU9u|%Wqv&h9%ydSyjp>IM(-NYyprC3!W z1(1)p!SM7W;2-z5e0K2&{6D>4AMjGKB%f0w{(bq7PW};Xsq&wn-_Qt1bID&2gtTKQ>Xq zB^oDzk296NHZ7r2*&&4v&U^dHKE7EhD>lgO!>8>li`B{Xux=3D3EHxA;~nFa{2}x! z@u^@5m);*`z}4f7VvH95O}w*!ekST=T6DhFCzBMZ?4;68gtkl<>mdKrha0Y!aEYHp z^oQ zlG?co__H;2cJhJibSQOh2mVa_tkB_PhIcrz7d(gkUozi24SQxOwvRn@8scsi8e`f6 z&&A%7_K5_e{2}bM^AY!_v&$5neLo|g6MOli^vw&2U?EZCW4|^1S$qy6perbQ+Y zSAc(0g{{~Bz6;$k%S60@KQUMbzNk?v?| z@7Cg-S3j5yv}kPToTuNI!sb$*fd@KICLL&}F@g@X6E5nUq_M(yGRV}QE{$my=>?4` z@M%m3!y|ne3{GQ)y@f^GV1As&Z7>|ti9zAefx+PF*e0IUjO~Fa+P`lrJuaUQ;op~# z# Pbq#)MolR~{9Y1~eP^nV^{7Rob^uPCG3uAbv53~9Go~QsVnBKe zVoJ~nV;Xb>agpDX%%L}7kTK3-r_eJ8f1#W1ra-so8+#N}gkIwvKKa``Eu&e47^hoA zxVM1jczN9Fx)GzNb~*T0^xY(eGj!NHk1e4OA0Pa(;Eq%z#mtI`FYvbseweyXJ%f5Z z_-*9)9y4^1_7t>N)$S>1|4e7!>Yf5;1|I)C1@;r$8E0^5+kS`I1Ai01Ln8Rzf`9B5 zUrcQs3w|Vie`xOY=l2%q7XCSv7^!@mxoJQ-G@Ql}m1^O`z`-WjBhI8v&^DRrJe`cqAv z#J2<2xrjI_$@L-NW4}7UHtR7~h!6LeF{X51$PN2C2fAE@IM6YSuL1E$e_Lh+eiw9e zd{9@$192<@AGCI*A};XaH{NMf){R*3c)e1Vjxy{ik6o9`7?llC$}A|uyXigGartQKX6D}*X#W{ltXD5Y#G%Ho(~{8hOg&Cl_1N?8ZW5GR^*MJ}^LcgBxW%HBj7 z_Qfkaav9n0@rg=VH_FB#CebUGbzwft!QGRMQO0@>G*VN*C*;Adg}MUCgwFX@ec5zS zJ0_vBe!d+hjpL7%F~K(?+Qw?e@#GtW)UnHP)XhqrNZ`l##_JbBxtvbdIJ|f@37)wT8zx zHQ{UBKI+;+djEiY2HTHWFVT?dPRDu%+t-1%3LvxbL)hIL^Z2?B#K5*9emQU5il(jl zA+2^LewloT-HmTNNc&y-)&<$r@rSs(zrY%P5WEykW0qekF`7R^5C4_;_V%q9)3dOX zLt#&Uq{RKGJ&OH(0U!7raJq1pXe;b#=2X7!3yf{%Oup_M#x661uXAHfXosE3G8=MToc|_)ZdX92+<33mBtMnZ0vg0{9i@V!4Ke-itdyI?|OYe|cOr49ECABx|~>rY?&me!x~N=(ZId8zA<6LDLL zX(492-H7`hKN<@gi+e0@BmQE+$YQ#@>_k;d_b<2k6X+E?r19Y@jXA-?wgeSI_R3A4_A^q`9Y+Y)V4JD1?J%toFd_q83ZqC-hmX~ z@loO;TKN}XtZ6?l(SXgkpx_iWe=>uaSvA$g_rjF;m&}iCn0SoZqVe^`Hqnk-NOsxK z{>j1GflkxiKx(I7`{Ul)o;MgAb1qYX=R*dhKh&23#JXuu5{5M~6aG#YVpgDM<*n^WsvJ~V{@B?2%fWCr=%0=qoabY-*B{VF zx@W1O?Z!K_#K)6^(3a*`$c}hDhi|>W=02w2(45N0JQ6Y2NER&$T|c&2aZ5gyr{BJ%x!cNTbzo`H83c(is)c(_AHG>VRa2JY~Yjb-$VyK~8i#o6YY zS}o{@5qLy{Y%$@TRctZg<%cuN1HhZjk$&r4B!hg^!B}UI3}?yD`{_9c&nq!@bbs7? zBlAMYT<1Zh4*p%K zwh@{-ht>tIQ>WAsfFG=>lekt>2YWe*H{~q0OsRwW6Lh}VMtbMtuSrv{eYjjtmn8W( z_bT<=NPGP1?OZiTJ#n~HFJGzmH>7{}t9P<-ka{!Hqoj%zr3mLaOv$ z2KKjf7bXd3{3S!PoS)&G-yUrCe2%kax|f}6p4y7{y4$+(-I5ZV{osz|UMep|eDzb{ zex~5<08Up3o4%UIo*Vku{VW&$Gw>oJzN5!ay~~FCS3P6kYkPy)%R4bwXpX6O+e$Fs z4VWWVE~clL?zTzy+@gC#dd8VH-2->xEZd-|a~Shf#p}Bd7cJd~i&5^w*?-1`KLB1n z1n-0O5#MZeUB|w@3*RHb_pq^@fPFw8x}N01@g1MG!Rr`Z=TV0l*fhHLy$1d67W(ef zfiIi~AzlU&q=m+Av=i~A7f3IjHVE*qFjRC&)y`q1hKO_?o^tA64B>>O=y`pZR zYi8{I^1DUtKjOk;hAQFy<$eH54-e$TRIQX~H4=VSQNTzNm5ud;+M(~^dZh0k%HPD5zKL|C z?{0dl0e7Hach3r;E{bm>-g1F_U=wt1?hkYJ?Z8@1dRYlw+=_QajQBp@X_T#qGJC4H zFyx2ylpt@zxa)CWG|u%*;cK`H5R3i8&mMcJKI|6e3<6zz6ZUT0p+ABBOGN%v?-cig z=EomF+lCGMo=&ZJV4szXTLJ&WB97blbWy6=Nq0B(Xp`P4Ohmgp$F*i9^0+U8_nfYW zOmkzss6d+roGX)DlT$e7@8LhmcHn^XGWKC#-^b%#zMy>fKyc+S?ijrjxG{`fut7dn zhWC6hFA?`Xz7uOlqG_~);_Gx@%8s?;3g&krc(>6z4Uoej%saf3^sNEn;jeW64(ST$qCB2C=o@`2hTb!H)AwE?%J6;{zB{^$IY}ND7YvTqr&EEy zNe%z`f{pR5_p&Y9H|p5SBZu$!+t(q?^9et)XAJzeOF}U8jJ`+8Rf;)1V zY-G=7r1IUXD3{?Y+*`!A=G?dk_B`(DoJ^9>;?;LIM8rI(o!75;pX@+6-P`fUp?B0? zL3ySS+p`PbCQe{+tvm3XgXa#s^G5Hx*>t#D4%)O1Jc{w@!24^xXp`P|i^2PDc8n#_ zmEU)h?}gD_p=UrB{?p#)f$!gLFZgOhIljwD-+_o!=jCAEsm6Twy*F|h?H=XhkF8$D zNn^Se@5QahU6Komo_m|=oo5}s&liMm1_k5YlMuXvB0}f&(0P1o*_?#;!sK`FPd8Ja z=#FR*bTAgsyoN&y5pZ>Gic@q2)4|KjOPk;8|;fFD@v6yz(KpyCA<=kRyE;J{Rxt^S&{V?MdI+jffb# z9rMcw|2vF)cI;u^fm|rIY^FT?^LC_|(~%T+|APGMG@Nlz*)6akNEz;S`^WznKM>Xp zdq{SNVj`;VReZC?Cu$ld`-d0h_~)N6X5{l0!Df`=*G=z6jl*|SLC05r+#vO14U)h2 zJEi_Y)Ln$16@IBGvf;)ba?;rrt%YZ{T@6pZzhj>X{yfE{+hI3ZHGPR2e4RTNO6tzN(=!7FUJm-d`0lcS%*G<$G1pOP5x~mVLh} zzVZiE360jO(Jbq_QI~Z+d_B=IJB>W^@xye?$%k>^i3d*tc+!EVAn+6noW`#X>C;~?H}eE7ZDp731}4inySCt0gM zscZ?#4$Eay{s+jP(&S5!=SN6&rSfISkJ02GKt5ZOuSUK^ldng9vnIa^`CXd)TI3IF z@(&_^N|WD=JRjND{zJ%*(c~XRK3kK24EYjG{&D0tYw|xuewQY{9r?qW{By{k(&Tp` z&qwvO{|n^DX!0)Pvo-l&B447(zk>W`P5w8?@6zO7NB*!T|2yPQY4V4W=cD`D{{!-4 zH2FUwpRLLN3HcID{$1oZYw{l;ze|(<8}f%W`F|jPN|Qf@JRj57{^!V#(d5q}pRLLN z6ZsNfe$VhPk>9Mze~tVuP5uh0**@|!jJX~^%= zG!KSq zXGQumD-TLF)`Wv0`osIx(%)0>%c;@~j{W361&xUcVyFUF+r$1#`a25ell~A(kp4bU z2weI`?$2VS9Q$SI&;G8Qs{c+V))naw=S))ni(Y?Y^nlP#Nf*3zx#FX?Y!O5Dd;>Z?ach9yWe~L-gLj2ot?|)$K282nt@W}*DC`H zVce>MyJLg7!NS0X`6Ow!-hIOjzOg=7X*4QP*lM)nQaD&@)NV)`jR^0lIXT%IfE^1-IL{7C*9XR2D?7sl@% zn3|f9Qcu}fidSblul=trbOFFw&N^pcmpP*6xjFBbx$3=BpAUgoM42*-20siiLb zZi_Dp*Z8VR_Xs~|$$wHfo=HqCkK-oE8ocC0{iA+64t^FMFwR;2JSlv$@JqznYr=O4 z*Z7RT^}u$T!nGg20KIAzbO2;#b0N>&Uv<-wnHzGbx{8oEdD=YXJ8;L ze-V$Ta~+$%ojJzOLR$WE;d?ATAUv`7)xr;2{5s}qo!8f=?Yqn`bx!^$%}1H{p1*jh z0hZqL_fZ+QTAxeA+zFf@bY5uvm0ykt6>;^a^5-zIA+GgT{t3)Wh^wEc#s2q&tKV|M z+xRkM%~{Ls((`m2_3w8c(M^5o$A>My-3m_Q^P&C5aDn^PaYB(-F2y=~g@4Ng9K}vU z>`W~g^tv1|@oSxYl}Bf`~h8lOKcT>Z9D>^vr1{ifsXN#W`@9iM+GT>W;T*!dgyq2KuY z9{UIFwfy`8;S+a@0a@XDFD&bv;}-v+@IU-WTK;9>>whcF@w^z@dD!Ck-4pY#So{s) z&sd!PRyoz@?2poVd0E4GW}UOq;`pu&3-Xc0*9pJZ;`lC^u;BdtSelOte^v*q9KKQbL-(cS$AzyynC6qhH(LCC!V4DP zEqu}99}vFh{lXhEIK}NFUo+)u-2MlVPaaIy=Lf=&R=O@ zki)NueDe_jvU)@K<6la1{B1=nc)s%|x_w5WgA6vd+2U8y&EY07; z{OykOZ)sj+eyQ_-y#W%zN|Etz!Rd%l-$&{>zsAlJHk8UKKl=Ejta7 zf76n0Ghgd$A56FF!_0g0Bi>Pfr8hr5g?SF=LCFhGh&jB=i?37E4;l_GIiL8lKV>@O zOoEdgJ&z7>ymp*F5w7_~)5)I;*ZlH|l>2SrnqPDp^t^D*FUo%+T=UB%V&|vAHNPl7 z2N$|jPt7lzMg9umnqM{uzec#`7tOyn3fKIi@>9Y!zi3{*L%8M_m2V2y{L++i4}+7R zbzF@J{|@s@9UWKa2w#VSXuhiBO8NW2sl6Kyh@f;>MY-guL&E=7?C5w?JLg?w>eCsY za$pvm>}#IV>s`FxpSi}L2RI%%&Pm}Kf4(KaQI^OVE#a62Bhgxm4(ZsB&^dr-I?_dYA! zj(dL~+>U!s3%BFmGs5k-cT%_=_wM8IyUy8uOdKYwU;I_$Pn$m=@+U3%^X2-;=Jewf z>KB{S54+cNe-pt^a;J!yc`FAm2zAty) zFBy}Q8S>K^{FV&gR z`BDadB7=W3gFg?BRM@lpYli%fGdO(#*l+w?!+f3dzz0kbCnUa2XUKmcgX0}_eeEC3 z;Pm}zzxtfa;ICxxpJnii5n%h-zb=DMWboTFc$C5K$>3$=nTv5f^7q-S z;ZtUXmxvqrr-k1p{GXW1PS@(k8T>yo_}kVR`xlms5uJxz%e>35Y~CpH>qQ>T*q4$U&`SBEq11wMt?Kn9W7U5V$-j^BN=>K2H(RR=WpBC-E3CQu?%^7sI*@^ zIUm0uqOjPpY!jj9{r!DtX~zDQqTQ;?G4Pjx_>5@twhN-Yds`E%fFC=<1#`n1qYa zV`UN~tI%-<%hh_-^W)fG;u8H@wFL7GDx;AYd5FdhujvOtwO)Y~mowLDIrPkSL*fm09mdFpH0o@-a5z^k`wb7Ac4^Y03)z8BU} z!O+`VuMU=)O{!PU*)~3&@^Fsofa=~c<+}SsA#T^FM|TuiduGSZ*<{4?P=@Q~@nk`@ z?Um}Shl_4oq;6Y8x7ki~M@&!R z;3!Fq(FbaPfnfa0&2_o^$q+nVaCX=CSL?x4xDZAzh7bumZDw8*XG#d^MH(@SXhw>A zW!O zF1y8LzvST;w9b@gTFSb4*3GkSo^|uAn`hlT>khH*5bF+^%ChcI0b^z7rcR@#@&!bJ z!1#F<&#mCmm)4f0X+)6Xw3`ni_&JEZYQ5a3O)pS=hPd1zE_aB_9YVPnM_g`!%PnxZ z1unP13>v;*)iUp^I(k;uE^~4IK z61BW`Jw_O7C1LFC3zxtVsurt>sKws?*l#uwQVbR|veQiF?Ha;g zn#+PDdft>7BTVB|x2;$aw2_`Jnu(Zk!tEJDDkp%=xhuQBjK6<{VHStf|EIQArr05UC0){mx;D+3$l%l#>qX2~#H3NG z^8{vviqvoEI>XFBsH5L8mRl=max+nA7_21Oj6(#~NM!r=SUKHe<2_-09XeM$Jt;bxsenw; zF{kG1EzFHf5O(w3$!t~}^@mBmF5laf}0W;4l>w~&MRJ`hv9DrPnmt7)nSAxw8owxa8k z2o>LTOV^Ci&70&k-iGaX5@&!rIz@sv1p!X3)@;)+g}B1;+~FCSHMMY&SsN+f>>w{J zPd@u7x}&XK^VL?$mZfwC<>w2Cag&(KZlR~Y-&dcU)+!d~q~SEqFh!F+%0tV0bPOp! zbu_h~giTe%Cgxmz1(ztsqoj;y!%J{&xgiYxqWo_O@We*Cp_M8SMW+w8ssNF>>K8NG znZbTPE>mC9bg<-?<~bjBIe%z^B(_cz$LU~JM;3<>FiQLEo;ywDQCxGE%VGQ$)m>Ng zccZ&tgNG*WMR>R+LU${?a$vVAH@Ym^n6!n@o^G(~C2>60c3~oXvY<*!5^^ z6wU7Vr0im3qz3J2Hr>r>+cDvKizW(u9#;KmOJ6UZPFu$?4Q_>KOm|Z|ZL{jtdm_6o z#%+;feK1{|snzCr9@I6A`sn1KvwNzz#m$eAN@uXsbxXSIdOB2ZT4CPE*P{J(d!@R2 zD4)A&4%Y>KDd~<>6L|`@-EwCV$tLK+X4R~X;R>x9%!C!(1zI(uBj|x1Chb9R z1~J(k;>K^qY-6>n%Y*mlV;GKDHXN*8rxkFQp-IVTwFYWFM;3SC3cX#KpRHDA!lWJ7 z5e)L4RmdpkgULEhowm6Tl$8_uCwF&v?YJ0Z3vc{R7a1`Ljx8Bt8j^}o}7!#Xf${8~| zeh%k6)nV3O2q}57Z=^OTN`^*HokmoYt{UimvpqXHQ@^-lAyP^kS2V>%%+_1}TVcv*jk`i|`*z#|4&xQ(rB*3>F`#?%j}_U*Xm&-3Bh44rCjUaQ^jTsqx0xfHK=(f5?adJ-?q;x=G$p1!N0%L-h& zU^X9nExKl8$ahKk2=79dufIrMuf=GaxViGuBVBvQLLKE!c3>jcTWWz1@&y$U+(}$E z63dmA97tVfF_TN?pwCs-Zgb5w<5yc4f%HW!-#Fm8E-EfJnlZC&ZFS1!sEsdt{kcYr z^d*{G`MWE#(}CNc*;#`8MgHQ{!*i4k%?z%xnK zHWCMHq!K30D!%(zYGIf}PI-TfD_iZ?jt?hp-+9x`ljHJe#Tl*WaTh$Ygx`#qMSq8^ z>m$N-rOy}YJ|5$B@JLBV{~UQPMc22CmFl`o4|hBrLXUi(6u}!v)n$hrz`3NYOZJPm9*)jh4(M^)f4fD0i&)h6aOhg!!$JCZ)Vs(39vIb0?d$KR zb-kzuAVZ7ozYl?tex43u$qn$Tvwj56{dX@HcJE+&3IDsxKXHQ*(6v6Ft!j4p_gVV- z`+8k3S`X)_|J2@L(Wmi2i@rx-V8ZP0s+zTa+U0)+2dMt4-+7(;Y1jYDI6^K~{R7`Mq6g%}vCFgdzXoxtyXxz6!G}cu1!^1?ZJ+9^ z-M_#w5>fr;_ek1tnn%rggIRSxRbRnhW20MN-xG60^fz1M&(?ny`oy&SlmBQepM1)! zH{f~0?nRo@LC<5WyL|mUct^h@X*TTgUj*(hzw>*ouN(QzmV&kG{0L%He$I$5e~Lb} zuY1w@>-yhu&~g!rpAdaLkMXks=Oz5t@-_ax0j9oVMfo`ReY1ggf~OX1*I5J5Pyev= NLxktWx))pj{{VBmd$a%m literal 0 HcmV?d00001 diff --git a/dist/patches/bert_xlmroberta_unigram.diff b/dist/patches/bert_xlmroberta_unigram.diff new file mode 100644 index 0000000..538e59e --- /dev/null +++ b/dist/patches/bert_xlmroberta_unigram.diff @@ -0,0 +1,63 @@ +diff --git a/conversion/bert.py b/conversion/bert.py +index 8af6c53..6dfe463 100644 +--- a/conversion/bert.py ++++ b/conversion/bert.py +@@ -28,6 +28,16 @@ class BertModel(TextModel): + cls_out_labels = None + self.cls_out_labels = cls_out_labels + ++ # Certains modèles déclarent architectures=["BertModel"] mais utilisent un ++ # tokenizer XLM-RoBERTa SentencePiece/Unigram (ex. intfloat/multilingual-e5-*, ++ # CamemBERT). Détecter ici pour router le vocab + chopper la matrice de positions ++ # exactement comme RobertaModel/NomicBertModel le font déjà. ++ self._tokenizer_is_xlmroberta = self._is_tokenizer_xlmroberta() ++ if self._tokenizer_is_xlmroberta: ++ self._xlmroberta_tokenizer_init() ++ else: ++ self._position_offset = None ++ + def set_gguf_parameters(self): + super().set_gguf_parameters() + self.gguf_writer.add_causal_attention(False) +@@ -37,6 +47,8 @@ class BertModel(TextModel): + self.gguf_writer.add_classifier_output_labels([v for k, v in sorted(self.cls_out_labels.items())]) + + def set_vocab(self): ++ if getattr(self, "_tokenizer_is_xlmroberta", False): ++ return self._xlmroberta_set_vocab() + tokens, toktypes, tokpre = self.get_vocab_base() + self.vocab_size = len(tokens) + +@@ -91,6 +103,11 @@ class BertModel(TextModel): + return super().filter_tensors((name, gen)) + + def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]: ++ # XLM-R : chopper la matrice de positions de _position_offset (pad/bos), ++ # comme RobertaModel/NeoBert. No-op si _position_offset is None (BERT WordPiece). ++ if name == "embeddings.position_embeddings.weight" and getattr(self, "_position_offset", None) is not None: ++ data_torch = data_torch[self._position_offset:, :] ++ + if self.cls_out_labels: + # For BertForSequenceClassification (direct projection layer) + if name == "classifier.weight": +@@ -101,6 +118,20 @@ class BertModel(TextModel): + + yield from super().modify_tensors(data_torch, name, bid) + ++ # Détection robuste du tokenizer XLM-RoBERTa (SentencePiece/Unigram). ++ # Présent aussi sur NomicBertModel ; remonté en base pour que tout modèle ++ # déclaré "BertModel" mais à tokenizer Unigram (e5 multilingue, CamemBERT) marche. ++ def _is_tokenizer_xlmroberta(self) -> bool: ++ tj = self.dir_model / "tokenizer.json" ++ if not tj.is_file(): ++ return False ++ try: ++ with open(tj) as f: ++ toktyp = json.load(f)["model"]["type"] ++ except Exception: ++ return False ++ return toktyp == "Unigram" ++ + def _xlmroberta_tokenizer_init(self) -> None: + # we need the pad_token_id to know how to chop down position_embd matrix + if (pad_token_id := self.hparams.get("pad_token_id")) is not None: