Design process: naive vs. refined system prompt
To make the reasoning behind the final design explicit rather than just showing the end result, this project documents two versions of the system prompt: a minimal, unconstrained v1 baseline, and a v2 refined version that addresses its specific shortcomings — role clarity, level adaptation, scope boundaries, anti-hallucination guardrails, a "guide, don't solve" policy for exercises, and a clear rule for when to use structured (JSON) output.
| v1 problem | v2 solution |
|---|---|
| Vague role/tone | Explicit role definition |
| No explanation level | Dedicated section, asks if unsure |
| No out-of-scope handling | "Scope" section with fallback |
| Hallucination risk | Explicit honesty-about-limits section |
| Direct answers to exercises | "Guide, don't solve" behavior |
| No structured output rule | Explicit prose vs. JSON rule |
A dedicated comparison script sends the same targeted test questions to both versions side by side. The chatbot uses v2 by default.
A note on sampling parameters and model choice
This project uses two models for two different purposes: the main chatbot uses Claude Sonnet 5, the current production model — as of Sonnet 5, temperature, top_p, and top_k are no longer configurable, so model behavior is guided entirely through the system prompt instead. A separate sampling demo uses Claude Haiku 4.5, which still accepts custom sampling parameters, specifically to demonstrate how temperature and top_p affect generated output.
This split reflects a real engineering trade-off: use the current production model where it matters, and isolate the parameter demonstration on a model that still supports it, rather than building the whole project against an older model just to keep a feature.
Structured output: study plan requests
Messages that look like a study-plan or roadmap request are routed to a structured JSON response instead of free text.
Example interactions
- Multi-turn context: a follow-up question ("Can you give me an example of the second one?") correctly resolved to "unsupervised learning" from the previous turn, without re-stating it.
- Structured output: a study plan request returned valid JSON matching the schema, with 6 ordered modules, per-module prerequisites, and suggested resources.
- Observed edge case: a simple out-of-scope factual question ("What's 2 to the power of 10?") was not rigidly declined the way the "write me a poem" test was — the model answered the trivial question directly, then noted it was outside its specialization. This wasn't an explicit rule in the v2 system prompt; it emerged from the model's own judgment about proportionality.
Context management and error handling
Conversation history uses a sliding window: only the most recent N turns are resent on each call, to bound token growth (and cost) in long sessions. API calls distinguish between transient errors (timeouts, connection errors, 5xx) and rate limits, which are retried with exponential backoff, and permanent errors (invalid request, auth failure), which are surfaced immediately rather than retried.