# 10,000 mobile app reviews: how a decision model and an LLM split the work

> A decision model classifies 10,000 mobile app reviews for cents, an LLM writes only the synthesis, and self-hosted Laya keeps the data inside the company.

Source: https://www.bluewise.fr/system-one-meets-llm  
Author: Marc Morel  
Published: 2026-09-29  
Category: Architecture

The case: Ten thousands free-form comments on a mobile app (store reviews, in-app feedback, surveys):

> Since the last update I can't log in with Face ID, I have to re-enter my passcode every time. And the instant transfer to my sister took 2 hours.

The questions are always the same: what do customers talk about, how do they feel, what changed since the last release, and what can't wait for a monthly report?

The default is one LLM call per comment ending in "return JSON". At 10,000+ comments that is slow, costly and fragile:
- The output is generated text, not a guaranteed answer: the same comment can come back as clean JSON, JSON behind a preamble, or a category nobody declared. Every call needs parsing, validation and a retry path.
- Categories drift with every prompt edit, and pasted IBANs go to the provider.

## Split the work

A **decision model** (a *System One* model: Jev, or open alternatives such as Decider, Kev and Laya) takes a text and typed questions: a **Choice** among your options, a **Score** on your scale, a **Noul** (the probability a statement is true). It returns values with calibrated probabilities, not prose to parse.

| Step | Done by |
|---|---|
| Draft the taxonomy from a redacted sample of ~300 comments | **LLM**, then ratified by product and customer relations |
| Classify all 10,000 comments | **Decision model** |
| Escalate fraud reports, aggregate trends per release | **Code** |
| Explain what moved, with redacted quotes | **LLM**, a few dozen comments per topic |
| Propose new topics from the `other` bucket after each release | **LLM** |

The LLM runs a few hundred times, only where writing is the job.

## One call, eight questions

Each comment gets one call with eight questions, answered in parallel:

- `topic`: a Choice over ten versioned categories (login and authentication, app stability, transfers, cards, account opening, fees, support, savings, features, other);
- `sentiment`: a Score;
- six Nouls: `actionable`, `churn_risk`, `fraud_or_scam`, `complaint`, `contains_personal_data`, `is_spam`.

Extra questions add almost no latency and cost only their own tokens.

For the quoted review, the answers are: `topic = login_auth`, with part of the probability on `transfers`; very negative sentiment; actionable; no fraud. It really is about both.

Then code, never the model, applies policy and counts:

- **Escalate what can't wait.** `fraud_or_scam ≥ 0.5` goes to the fraud team the same day. The low threshold is deliberate: a missed fraud report is the expensive error. Whether in-app complaints enter the formal complaint process is decided by compliance, not the model.
- **Trend by release.** Summing probabilities gives expected counts with uncertainty bands, instead of hard labels. The report can say: *"`login_auth` rose from 6% to 19% of comments on 5.12.0, iOS only."* That is a release-regression signal.
- **Keep personal data away from the LLM.** Comments flagged `contains_personal_data` are redacted before the LLM writes any synthesis.

## The gains over one LLM call per comment

- **Cost.** About 5M input tokens at Jev's $0.042/M, with output free, comes to about **$0.20**. A frontier LLM costs tens of dollars for the same volume (illustrative prices). At that ratio, re-running after every release, or re-classifying the whole history when the taxonomy changes, is a non-decision.
- **Speed.** Vendor-reported latency is 70 to 500 ms per call, so the full set runs in minutes and the pipeline can process the live stream. Verify under load.
- **No parsing.** Answers can only be your declared options, your scale, or a probability: no invented category, no preamble, no malformed JSON, no retry loop. The validation layer that wraps every LLM call disappears, and so do its failures at 10,000 calls. An LLM's structured-output mode fixes the shape of the answer, but not the value: a valid label can still change from one run to the next. Here the question itself sets what the answer can be.
- **Stable categories.** With versioned criteria and a pinned model version, "before and after 5.12" is a fair comparison.
- **Arithmetic on probabilities.** Expected counts, noise-aware alerts, per-risk thresholds and review queues all come from the same numbers. Check calibration on a labelled sample first.

## Privacy: Laya, self-hosted

Mobile app comments might contain very private and sensitive information (health, financial, etc.). Sending them to a hosted API outside the EU raises GDPR transfer.

**Laya** (Convai Innovations, Apache 2.0) answers them. It has a multilingual checkpoint, runs on infrastructure you control, and serves the same `/v1/systemone` wire format as Jev. Point the SDK's base URL at `laya-serve`, set `model="multilingual"`, and no comment leaves the company. Open weights also settle the DORA exit strategy.

Before trusting it:

- **Keep options short.** A question's option descriptions share a 192-token budget. Past about 20 categories, go two-level.
- **Chunk long survey verbatims.**
- **Re-calibrate thresholds,** starting with fraud.

The LLM steps only ever see a few hundred redacted comments, and can run on a model hosted inside the company if policy requires.

## What changes

Customer feedback analysis is one small judgment repeated thousands of times, plus a little synthesis. Give the repetition to a decision model: fast, cheap, stable, and able to spot the fraud report hidden in a one-star review.

Give the synthesis to the LLM. When the customers are a regulated company's, run the decision model on Laya, without changing a line of classification code.