· Marc Morel · Architecture · 5 min read
10,000 mobile app reviews: how a decision model and an LLM split the work
A decision model classifies 10,000 mobile app reviews for cents, an LLM writes only the synthesis, and self-hosted Laya keeps the data inside the company.
The case: Ten thousands free-form comments on a mobile app (store reviews, in-app feedback, surveys):
Since the last update I can’t log in with Face ID, I have to re-enter my passcode every time. And the instant transfer to my sister took 2 hours.
The questions are always the same: what do customers talk about, how do they feel, what changed since the last release, and what can’t wait for a monthly report?
The default is one LLM call per comment ending in “return JSON”. At 10,000+ comments that is slow, costly and fragile:
- The output is generated text, not a guaranteed answer: the same comment can come back as clean JSON, JSON behind a preamble, or a category nobody declared. Every call needs parsing, validation and a retry path.
- Categories drift with every prompt edit, and pasted IBANs go to the provider.
Split the work
A decision model (a System One model: Jev, or open alternatives such as Decider, Kev and Laya) takes a text and typed questions: a Choice among your options, a Score on your scale, a Noul (the probability a statement is true). It returns values with calibrated probabilities, not prose to parse.
| Step | Done by |
|---|---|
| Draft the taxonomy from a redacted sample of ~300 comments | LLM, then ratified by product and customer relations |
| Classify all 10,000 comments | Decision model |
| Escalate fraud reports, aggregate trends per release | Code |
| Explain what moved, with redacted quotes | LLM, a few dozen comments per topic |
Propose new topics from the other bucket after each release | LLM |
The LLM runs a few hundred times, only where writing is the job.
One call, eight questions
Each comment gets one call with eight questions, answered in parallel:
topic: a Choice over ten versioned categories (login and authentication, app stability, transfers, cards, account opening, fees, support, savings, features, other);sentiment: a Score;- six Nouls:
actionable,churn_risk,fraud_or_scam,complaint,contains_personal_data,is_spam.
Extra questions add almost no latency and cost only their own tokens.
For the quoted review, the answers are: topic = login_auth, with part of the probability on transfers; very negative sentiment; actionable; no fraud. It really is about both.
Then code, never the model, applies policy and counts:
- Escalate what can’t wait.
fraud_or_scam ≥ 0.5goes to the fraud team the same day. The low threshold is deliberate: a missed fraud report is the expensive error. Whether in-app complaints enter the formal complaint process is decided by compliance, not the model. - Trend by release. Summing probabilities gives expected counts with uncertainty bands, instead of hard labels. The report can say: “
login_authrose from 6% to 19% of comments on 5.12.0, iOS only.” That is a release-regression signal. - Keep personal data away from the LLM. Comments flagged
contains_personal_dataare redacted before the LLM writes any synthesis.
The gains over one LLM call per comment
- Cost. About 5M input tokens at Jev’s $0.042/M, with output free, comes to about $0.20. A frontier LLM costs tens of dollars for the same volume (illustrative prices). At that ratio, re-running after every release, or re-classifying the whole history when the taxonomy changes, is a non-decision.
- Speed. Vendor-reported latency is 70 to 500 ms per call, so the full set runs in minutes and the pipeline can process the live stream. Verify under load.
- No parsing. Answers can only be your declared options, your scale, or a probability: no invented category, no preamble, no malformed JSON, no retry loop. The validation layer that wraps every LLM call disappears, and so do its failures at 10,000 calls. An LLM’s structured-output mode fixes the shape of the answer, but not the value: a valid label can still change from one run to the next. Here the question itself sets what the answer can be.
- Stable categories. With versioned criteria and a pinned model version, “before and after 5.12” is a fair comparison.
- Arithmetic on probabilities. Expected counts, noise-aware alerts, per-risk thresholds and review queues all come from the same numbers. Check calibration on a labelled sample first.
Privacy: Laya, self-hosted
Mobile app comments might contain very private and sensitive information (health, financial, etc.). Sending them to a hosted API outside the EU raises GDPR transfer.
Laya (Convai Innovations, Apache 2.0) answers them. It has a multilingual checkpoint, runs on infrastructure you control, and serves the same /v1/systemone wire format as Jev. Point the SDK’s base URL at laya-serve, set model="multilingual", and no comment leaves the company. Open weights also settle the DORA exit strategy.
Before trusting it:
- Keep options short. A question’s option descriptions share a 192-token budget. Past about 20 categories, go two-level.
- Chunk long survey verbatims.
- Re-calibrate thresholds, starting with fraud.
The LLM steps only ever see a few hundred redacted comments, and can run on a model hosted inside the company if policy requires.
What changes
Customer feedback analysis is one small judgment repeated thousands of times, plus a little synthesis. Give the repetition to a decision model: fast, cheap, stable, and able to spot the fraud report hidden in a one-star review.
Give the synthesis to the LLM. When the customers are a regulated company’s, run the decision model on Laya, without changing a line of classification code.