Autonomous prompt optimizer. Give it a system prompt and eval cases — it mutates, scores, and keeps improvements overnight while you sleep.
Built with FastAPI · LangGraph · Claude (claude-sonnet-4-6)
The same loop as autoresearch — but for prompts instead of model weights.
START
│
▼
baseline ── score your original prompt against all eval cases
│
▼
reflect ─── Claude analyzes what's working and picks a direction
│
▼
mutate ──── Claude generates one improved mutation
│
▼
score ────── run the candidate against all eval cases, get a score
│
▼
judge ────── if score > best → keep, else discard
│
├── loop back to reflect (next mutation)
│
└── END (after N mutations)
Each node streams results to the browser via SSE as it completes. You watch experiments appear live.
1. Clone and install
git clone https://github.com/shaadahmade/autoprompt
cd autoprompt
pip install -r Requirement.txt2. Run
python -m uvicorn main:app --reload3. Open http://localhost:8000
Enter your Anthropic API key in the UI — no .env file needed.
autoprompt/
├── main.py # FastAPI backend + LangGraph graph
├── index.html # UI (served by FastAPI at /)
├── Requirement.txt
└── Readme.md
main.py is the whole backend — ~200 lines. No separate files for graph, nodes, or config.
- Paste your API key — Anthropic key (sk-ant-...)
- Write your base prompt — the system prompt you want to improve
- Add eval cases — pairs of user messages + what a good response looks like
- Set mutations — how many experiments to run (5–20 is a good range)
- Click Run — watch experiments stream in live
When done:
- The best prompt is shown in the UI and ready to copy
- Every experiment is logged with its score and status
| Node | Model | What it does |
|---|---|---|
baseline |
claude-sonnet-4-6 | Scores your original prompt as the reference point |
reflect |
claude-sonnet-4-6 | Analyzes experiment history and picks a direction |
mutate |
claude-sonnet-4-6 | Generates one concrete prompt mutation as JSON |
score |
claude-sonnet-4-6 | Runs candidate against all eval cases, LLM-judges each |
judge |
— | Keeps if score > best, discards otherwise, loops or ends |
The quality of your eval cases determines the quality of your results. Bad evals → fake 95%+ baseline → optimizer finds nothing.
Weak eval case:
{
"user_message": "Help me",
"expected": "be helpful"
}Strong eval case:
{
"user_message": "I've been a customer for 10 years and this is the third time this has happened. I want a manager and compensation NOW.",
"expected": "acknowledge frustration, validate loyalty, escalate to manager, offer concrete compensation without being defensive"
}Hard evals with specific, measurable expected outputs give the optimizer room to find real improvements.
One endpoint:
POST /run/stream
Content-Type: application/json
{
"api_key": "sk-ant-...",
"base_prompt": "You are a helpful assistant.",
"eval_cases": [
{ "user_message": "...", "expected": "..." }
],
"num_mutations": 5
}
Returns a Server-Sent Events stream. Each event is a JSON object:
fastapi
uvicorn
anthropic
langgraph
langchain-anthropic
langchain-core
python-dotenv
Python 3.10+ required (uses TypedDict with list[...] syntax).
- Eval library — pre-built eval sets per industry (support, legal, sales)
- Parallel scoring — run eval cases concurrently to speed up scoring
- History persistence — save runs to SQLite, compare across sessions
- Export — download results as TSV or copy best prompt with one click
- Multi-model — use a cheaper model for scoring, smarter model for mutation