Does anyone have a good local model that works best?
I tried Gemma 4 26B A4B QAT at Q4_K_M, and it works fine, but it has a hard time ending scenes, to a degree that it will sometimes just output "end_scene" as last respond, but not actually ending it.
I also tried Swift 1.5 at IQ3_S through Strata. Better at ending scenes, but it uses like 100% of my PC, and I can't fit ComfyUI with it. Also I feel it ends scenes too soon. And I know that I can extend them by interrupting dialog, but once the games goes to stat rewards I cannot go back, and I cannot save during LLM, as it does when the scene is about to end. So I either go slow and wonder if it's going to end now, to interrupt it earlier enough, or I have to wait for entire LLM generation to go through, so I can save. Also Swift rarely have problem generating proper JSON, giving me error that some comma somewhere is wrong. Though I can just try again.
Also as a feedback. Any plans to implement voice through OpenAI compatible API (v1/audio/speech) for local models? I use omnivoice-server with ST, and it's great and fast.