After running our thai-chat model (qwen38) on team hardware for a while, the question we had to answer with hard numbers was "is it good enough to replace z.ai?" Without numbers, there's no basis for the claim — so we set up a full 50-case test suite across 5 categories, ran both systems side by side, and had a model score the outputs blind to which system produced them.
Test Methodology
- bomLLM side: thai-chat model via llm.siam2r.com (our LiteLLM router on our own hardware)
- Cloud side: glm-4.5-flash directly through z.ai's API (thinking disabled, max_tokens 2000 per budget)
- 5 categories, 10 cases each: Thai LINE Q&A, signal summarization (RAGSA), category classification, translation, and strict-format edge cases
- Primary judge: qwen38 model scoring each case 1–10, with Gemini 2.5 Flash as a cross-grader on the 18 remaining quota cases
Overall Results
| Category | bomLLM | z.ai | ratio |
|---|---|---|---|
| LINE Q&A (Thai) | 8.93 | 6.33 | 1.41 |
| Signal Summary (RAGSA) | 8.53 | 5.80 | 1.47 |
| Category Classifier | 1.00 (accuracy) | 1.00 | Tie |
| Translation | 9.63 | 8.50 | 1.13 |
| Edge Cases (Strict JSON) | 8.10 | 8.47 | 0.96 |
| Total 50 Cases | 7.24 | 6.02 | 1.20 |
The latency gap is even wider: bomLLM p50 = 5.19 seconds (p95 = 19.9) vs z.ai p50 = 35.1 seconds (p95 = 85.6) — roughly 7x faster at the median, with bomLLM's error rate at 2/50 cases (both were strict-format cases that exceeded the 30-second runtime limit).
What We Need to Be Honest About
This benchmark has important caveats to read before making major decisions based on it:
- The judge and competitor are from the same family — qwen38 judging qwen38 outputs naturally favors them. The subset where Gemini judged independently shows a lower ratio of 1.28 — still winning, but not as dominant as 1.44
- Test prompts were short (~310 tokens) — real workloads sending thousands of tokens of context are a different story. We hit a real case where a 7,000-token prompt made our local hardware exceed EA's 90-second limit, forcing us back to cloud (see the 6,928-token prompt article)
- Edge cases are a genuine weakness — strict JSON and mixed-language cases were the only category we lost (0.96). Fixed-format work should route to fallback
Verdict for Our Team
For Thai chat, signal summarization, and translation work — bomLLM on our hardware wins decisively in both quality and speed, with zero per-token cost. For prompts exceeding 3–4K tokens or strict-format work where errors are unacceptable, the safest path right now is cloud + fallback, switching based on workload type — which is exactly the architecture we've built into our router.
I make the final call on any real system migration myself — benchmarks' job is just to surface real numbers, not to press buttons for me.
