Suikō

When I started Suikō I was studying Japanese full time in Tokyo: four hours of class a day, then reading, podcasts and Anki on my own. Reading and listening fit into a daily routine. Writing didn't. A diary entry in Japanese was something I only got to on days when I had motivation to spare.

When I did write, I checked the entry with an AI chat. It usually sent back a corrected paragraph. The entry was fixed, but I hadn't done the fixing, and I'd read the corrections once and move on. The entries ended up scattered across notes and chat threads.

I wanted three things: a reason to write most days, feedback that shows where each mistake is and leaves the fix to me, and one place that keeps every page and every draft. I built it for myself first, as a one-user app, and then opened it to other learners who can already write a few sentences on their own.

The name is 推敲, the Japanese word for revising prose. It comes from a Tang dynasty story about the poet Jia Dao, who was so absorbed in choosing between 推 (push) and 敲 (knock) for one line of a poem that he walked into a governor's procession. Revising your own sentences is the part of the practice the app is built around.

I chose each model with side-by-side tests on my own real entries.

The grader. I ran Claude Opus 5 and Claude Sonnet 5, each at medium and low reasoning effort, on three real entries, and marked every note without knowing which setup wrote it. All 77 notes were useful and none were wrong about the Japanese. The difference was coverage: Opus caught 85% of all the issues any setup found, and Sonnet caught 52%. Reasoning effort made no measurable difference. Running five more entries through the real pipeline, low effort was 40% cheaper per round ($0.068 against $0.113) and took 28 seconds instead of 63. A minute-long wait, several times a sitting, is hard to sustain as a daily habit, so I use low.

The cost problem. Opus was the better grader but cost too much to run every round. Over full entries it came to $0.22 per finished page against $0.12 for Sonnet. At my own usage, about 35 entries a month, Opus would have cost more than a $9 subscription brings in after Stripe fees.

So I split the work in two. Repeated proofreading moved to a small, fast model, and Opus runs once per entry on the editorial read. That's also where it was most different from Sonnet: in the blind test, 42% of Opus's notes looked beyond the quoted phrase, against 21% of Sonnet's.

The proofreader. I compared three small models on the same three entries, with the same prompt, schema and validation, and ran the test twice:

The cost difference worked out to about 30 cents a month, so cost didn't decide it. I moved to Gemini because its API enforces the output schema, and picked 3.5 Flash-Lite because it was the fastest to show the first note, which matters most while a learner is waiting. In production the first note now appears in about 1.8 seconds. I couldn't tell the three apart on quality in these runs.

The biggest risk is the AI marking correct Japanese as wrong. Learners can't always tell, so they end up changing sentences that were fine.

Learners can give any note a thumbs-down. I review each one and label it:

Each real mistake becomes a regression test, written with fresh text, that I rerun after any prompt or model change. The app also logs the cost of every AI call.

One case I track: the sentence 「N2はほぼ満点を取りました」 is correct, but Gemini models flagged it five times across four runs, each time with a different wrong explanation. DeepSeek never flagged it.

Next, I'm building a hand-marked answer set so every model can be scored against the same ground truth, starting with false positives.