What a strong prompt is worth: GEPA tested on 3 tasks
Sachin Agrawal
Always building 5 projects, sometimes finishing them. Community builder.
Most teams try to buy performance with a bigger model. The cheaper lever sits closer: the prompt. A strong prompt makes a small model punch far above its price, and it makes a good model even better. GEPA searches for that prompt automatically; use it for cost savings, for peak quality, and ideally both.
We tested GEPA on three tasks, our own event extraction plus the famous benchmarks GSM8K and HotpotQA, against a deliberately weak prompt and an expert hand-written prompt, on gpt-4o-mini and gpt-4o. On all three tasks, the small model with a GEPA-evolved prompt beat the big model running the weak prompt. On event extraction it even edged out gpt-4o with the expert prompt (0.902 vs 0.897) at roughly 12x lower cost per request.
The problem: harness optimization
The model is one part of your system. The harness around it (prompts, tools, retries, output parsing) decides how much of the model's capability you get. Model choice moves you along the cost axis; the prompt moves you up the quality axis. That gives you two plays:
- Cost: keep quality, shrink the model. A small model with an optimized prompt replaces a big model with an average one.
- Performance: keep the model, raise quality. The same optimized prompt lifts a strong model further.
Weak prompts are an alignment problem
Without a precise prompt, the model guesses what you want. Output formats drift, edge cases land differently on every run, and two models give two different answers to the same input. In our extraction task the weak prompt returned dates in three formats and 'N/A' where the spec wanted an empty string. A precise prompt pins down intent: behavior stays consistent across runs, and across models.
What is GEPA?
GEPA (Genetic-Pareto) is reflective prompt evolution, available as an optimizer in DSPy. Your metric returns a score plus textual feedback ('date format wrong, expected YYYY-MM-DD'). A reflection model reads that feedback, rewrites the prompt, and Pareto-improving candidates survive. You start from a weak one-liner and let the optimizer discover the strong instruction.
Where GEPA pays off beyond this pilot:
- Model migration: a provider switch or model update re-optimizes the prompt automatically instead of a hand-tuning week.
- Multi-model consistency: one metric aligns several models to the same output contract.
- Encoding domain judgment: experts write the scoring rubric; the optimizer turns it into prompt wording.
- Agent pipelines: each module's instruction gets optimized against end-to-end feedback.
The setup: an expert prompt as the benchmark
Automatic optimization is only interesting if it competes with what a careful human produces. So every task has three prompts on gpt-4o-mini, plus two big-model references on gpt-4o:
- Weak prompt: one thin line like 'Solve the math problem.' This is the GEPA starting point.
- Expert prompt: a detailed hand-written rubric. This is the human benchmark.
- GEPA prompt: evolved from the weak line (medium budget for the headline numbers; we also ran heavy). Reflection model gpt-4o.
Tasks: extraction of six fields from messy EN/DE event blurbs (ours, 12 test examples), GSM8K grade-school math (exact match, 24 test), and HotpotQA multi-hop QA with distractor paragraphs (token F1, 24 test). The test sets are small; read the deltas as directional, not as leaderboard scores.
Results across three tasks
Event extraction (n=12): weak prompt 0.72, expert prompt 0.89, GEPA 0.90 (+0.01 vs expert, compile ~31 min).
GSM8K (n=24): weak prompt 0.79, expert prompt 0.96, GEPA 0.88 (−0.08 vs expert, compile ~74 min).
HotpotQA (n=24): weak prompt 0.17, expert prompt 0.75, GEPA 0.66 (−0.09 vs expert, compile ~33 min).
The pattern: starting from a throwaway one-liner, GEPA beat our expert prompt on event extraction and closed 50% (GSM8K) and 84% (HotpotQA) of the weak-to-expert gap on the famous benchmarks, with zero prompt-writing effort. The biggest lift comes where the weak prompt fails hardest: HotpotQA jumps from 0.17 to 0.66 because the evolved prompt learned to answer with the exact short span instead of a helpful sentence.
Small model + evolved prompt vs big model
That is the cost-savings play in one picture: on all three tasks, gpt-4o-mini with the GEPA prompt beats gpt-4o running the weak prompt. A better prompt beats a bigger model. On event extraction the small model with the evolved prompt (0.902) even edges gpt-4o with the expert prompt (0.897) at ~12x lower price per request. HotpotQA is the honest counterpoint: the big model with the expert prompt stays ahead (0.80 vs 0.66); when multi-hop reasoning is the bottleneck, prompt quality alone does not close it.
Real prompts as GEPA rewrites them
These are not paraphrases. Each block is a real instruction candidate from our heavy GEPA runs — the seed we started with, an intermediate rewrite GEPA proposed mid-search, and the final best-on-validation prompt. Longer prompts are truncated with an ellipsis; full text lives in scripts/experiments/gepa/results/<task>/heavy/prompt_evolution.json.
Event extraction: seed → intermediate → final
GEPA proposed 63 candidates on this task. Seed = candidate #0 (val 0.74); intermediate = candidate #31 (val 0.94); final = candidate #54 (val 0.95).
Extract event fields from the blurb.Task: Extract structured data from an event description blurb to standardize event details for improved data management and retrieval.
Instructions:
1. **Title**:
- Extract the main title of the event, incorporating any unique identifiers like event numbers or themes (e.g., "Builder Friday #42").
- Avoid including excessive organizational descriptions unless they add essential context.
- Refine the title based on implied suggestions in the blurb, ensuring it closely aligns with feedback expectations. Keep titles concise and relevant.
2. **Date (ISO format)**:
- Convert the date mentioned in the blurb into ISO format (YYYY-MM-DD).
- Check for various date formats including DD.MM.YYYY, MM-DD-YYYY, or verbal forms like 'Tue May 20 2025.'
- If only the date is mentioned without explicit hints on time, assume the event will occur at a standardized time (e.g., 00:00 hours).
3. **City**:
- Extract the city mentioned in the event blurb.
- If the event is entirely online or the location is unspecified, use 'online'.
- In a hybrid setup, use the main physical location provided in the blurb.
4. **Format**:
- Classify the event as 'in-person', 'online', or 'hybrid'.
- If there are both physical and online attendance options, mark it as 'hybrid'.
5. **Topics**:
- Identify the main themes of the event from the blurb and form concise, generalized kebab-case topic tags.
- Convert descriptive phrases into short, standardized tags—aligning with common feedback suggestions, such as converting both "artificial-intelligence" and "ai solutions" into 'ai'.
- Ensure tags encapsulate all primary themes indicated in the feedback, merging related concepts when necessary, e.g., 'genai' becomes 'generative-ai', 'research-spinouts' should instead be broken into 'research, spinouts' unless specified otherwise.
- Strive to cover both direct and implied subjects in the event to prevent topic mismatches, ensuring the tags are holistic and meet expected terminologies in the feedback, like 'ai, research, mobility, design'.
6. **Registration URL**:
- Extract the URL for event registration.
- If the registration URL isn't provided, deno
…Task: Extract and refine structured data from an event description blurb to standardize event details for improved data management and retrieval, emphasizing alignment with feedback suggestions.
Instructions:
1. **Title**:
- Extract the main title of the event, incorporating any unique identifiers like event numbers, themes, or edition types (e.g., "Builder Friday #42", "remote edition").
- Avoid including excessive organizational descriptions unless they provide essential context or clarity.
- Refine the title based on feedback expectations, ensuring it captures the essence and brand of the event as implied in the blurb.
2. **Date (ISO format)**:
- Convert the date mentioned into ISO format (YYYY-MM-DD).
- Look for varying formats such as DD.MM.YYYY, MM-DD-YYYY, or verbal forms like 'Tue May 20 2025'.
- If the time isn't specified, assume a standard start time (e.g., 00:00 hours).
3. **City**:
- Extract the city if mentioned in the blurb.
- For online events or if the location is not specified, use 'online'.
- In hybrid scenarios, specify the main physical location mentioned.
4. **Format**:
- Classify the event as 'in-person', 'online', or 'hybrid'.
- If both physical and online attendance options are available, mark it as 'hybrid'.
5. **Topics**:
- Identify and form concise kebab-case topic tags reflecting the main themes.
- Prioritize broader feedback-aligned terms over detailed phrases to encapsulate primary themes accurately.
- Transform terms using established mappings:
- "artificial-intelligence" and "ai solutions" to 'ai',
- "automotive startups pitch" to 'automotive, startups, demo-day',
- "ai agents" as 'ai-agents',
- Include specific terms found within the event context like 'claude', 'cheatsheet', or 'live-coding'.
- Ensure topic tags reflect the coherent main themes and intended context, avoiding mismatches with expected themes.
6. **Registration URL**:
- Extract the event registration URL.
- If the URL is not provided, denote it as an empty string ('').
Ensure outputs are consistent, accurate, and aligned with feedback-guided guidelines, enhancing future task accur
…GSM8K math: seed → intermediate → final
GEPA proposed 13 candidates on this task. Seed = candidate #0 (val 0.88); intermediate = candidate #7 (val 1.00); final = candidate #1 (val 1.00).
Solve the math problem.Task: Solve word problems that require basic arithmetic calculations.
Instructions:
1. Comprehend the Problem:
- Thoroughly read the entire problem statement to understand the context, information given, and what is being asked.
- Identify and note all numerical values, units, and relationships pertinent to the problem.
- Pay attention to keywords and phrases such as "total," "difference," "per," "left," "twice as much," or "half," which signal specific arithmetic operations.
- Pinpoint the specific question that needs an answer.
2. Organize and Analyze:
- Clearly list and define all known quantities and variables. If possible, outline them in a table to aid visualization.
- Establish a plan or sequence of arithmetic operations that will lead to a solution, which might involve setting up equations or performing direct calculations.
- Take into account any contextual peculiarities like time conversions, recurring events, or changing conditions like scaled increases (as seen with ships doubling in size).
3. Calculation Execution:
- Methodically carry out calculations, ensuring each step is clearly tracked.
- Ensure uniform use of units across calculations and consistently apply the correct order of operations (PEMDAS/BODMAS: Parentheses/Brackets, Exponents/Orders, Multiplication and Division, Addition and Subtraction).
- For monotonically increasing or decreasing sequences, recognize patterns or multipliers consistent with compounding growth or decay principles.
4. Verify the Solution:
- Review calculations at each step to affirm correctness and logical consistency.
- Confirm the final answer precisely addresses the posed question and falls within expected values or contexts.
- Adapt for special conditions and edge cases like rounding errors, integer restrictions, or event periodicity, especially in incremental or decremental problems.
5. Present the Answer Clearly:
- Provide the final answer in a clear, direct manner.
- Format the solution logically, employing complete sentences when necessary to maintain clarity.
- Relate the answer back to the problem context, including necessary units or explicit indication
…Task: Solve word problems that require basic arithmetic calculations.
Instructions:
1. Understand the problem:
- Read the question carefully to identify key information, such as quantities, relationships, and operations needed.
- Pay close attention to details about who, what, when, and how much.
2. Break down the problem into steps:
- Identify all known quantities and what needs to be calculated.
- Consider any mathematical operations required (e.g., addition, subtraction, multiplication, division).
3. Execute the calculations:
- Perform each arithmetic operation step by step, and keep track of intermediate results.
- For problems that involve multiple calculations, ensure that you follow the correct order of operations.
4. Verify the solution:
- Double-check your calculations for accuracy.
- Ensure the final result answers the specific question asked in the problem.
5. Present the answer:
- Provide the final answer succinctly, in complete sentences where appropriate.
- Ensure the answer directly corresponds to the question, including the correct units or context if applicable.HotpotQA multi-hop: seed → intermediate → final
GEPA proposed 19 candidates on this task. Seed = candidate #0 (val 0.19); intermediate = candidate #9 (val 0.66); final = candidate #10 (val 0.70).
Answer the question using the context.Your task is to answer questions based on the provided context by extracting a concise segment or providing a simple 'yes'/'no' response when appropriate. Each context consists of multiple paragraphs, each potentially covering different topics or events. Some paragraphs may include distractors irrelevant to the specific question. Implement the following guidelines to ensure clarity, accuracy, and relevance in your responses:
1. **Exact Match**: Prioritize the use of specific names, entities, or numerical data mentioned in the context. Your responses should directly reflect an exact match from the context, ensuring they are concise and precise.
2. **Relevance**: Focus on details that directly answer the question. Ignore any information that does not contribute to addressing the specific query, including potential distractors.
3. **Verification**: Verify numerical, factual data and names within the context to ensure your response's accuracy.
4. **Directness**: Provide the most direct answers possible. Avoid adding extra context or unnecessary explanations unless it is essential for standalone understanding. If applicable, reply with a simple 'yes' or 'no'.
5. **Synthesis Across Paragraphs**: If a question involves information that spans multiple paragraphs, identify and integrate these interrelated details to form a coherent response. Use systematic identification to ensure all relevant data are combined appropriately.
6. **Answer Simplicity**: Always provide the simplest possible form of your answer. This includes avoiding restatement of questions and excluding additional information beyond the exact answer.
7. **Feedback Reception**: Learn from given feedback to better identify exact matches, ignore irrelevant data, and improve extraction accuracy through insight into expected responses.
8. **Specific Instruction**: Based on prior feedback:
- When asked for item comparisons (e.g., "Which was started first?"), respond with just the exact name of the older entity.
- Where multiple names or identifiers exist, utilize any additional context provided to resolve potential ambiguities, ensuring the most complete factual identification.
- Extract nam
…Your task is to answer questions by extracting precise information from a provided context. The context will be made up of multiple paragraphs, each addressing different topics or events. Some parts may contain irrelevant information or distractors. Your goal is to sift through these sections to provide a relevant and exact answer to the question based on the given guidelines:
1. **Exact Match**: For names, entities, or numerical data related directly to the question, ensure your answer is an exact match of the wording in the context. Respond with the precise short span of text or phrase as it appears in the context.
2. **Relevance**: Focus solely on information that allows you to directly answer the question. Disregard any distractors or unrelated details.
3. **Verification**: Double-check numerical and factual data (e.g., names, dates, coordinates) within the context to confirm the accuracy of the information being used in your response.
4. **Brevity & Precision**: Avoid providing extended context unless absolutely necessary. If the context supports it, and unless stated otherwise, answer with a succinct, 'yes' or 'no'.
5. **Synthesis Across Paragraphs**: If pertinent information is spread over multiple paragraphs, discern and integrate these elements to formulate a coherent answer. Ensure information is logically combined to maintain accuracy.
6. **Feedback Adherence**: Past feedback indicates responses should stick strictly to the precise requested information without additional restatements or explanations unless they ensure standalone clarity. Provide only the necessary excerpt to satisfy the query.
By consistently applying these principles, you will ensure clarity, accuracy, and relevance in your responses.The pattern across tasks: the seed is a one-liner; the intermediate already adds failure-mode checklists; the final is a dense operating manual the small student can follow. Watch what the optimizer invents without being told — date-normalization on events, step-by-step arithmetic checks on GSM8K, and 'answer with the exact short span' on HotpotQA.
Higher effort: heavy GEPA budget on all three tasks
Medium was the headline setup. We also ran a heavy GEPA budget on the same three tasks and the same held-out tests. Heavy spends more reflection calls and more compile time (events ~49 min, GSM8K ~80 min, HotpotQA ~47 min).
Heavy vs medium GEPA scores: events 0.91 vs 0.90, GSM8K 0.83 vs 0.88, HotpotQA 0.73 vs 0.66. Heavy beat medium on 2 of 3 tasks.
We swept GEPA budgets on the extraction task before settling on medium. Light (~2 min compile) near-tied the expert prompt (0.881 vs 0.885). Medium (~28 min) beat it (0.916 vs 0.890). Heavy (~43 min) still beat the expert but landed under medium. On a 12-example holdout, more budget is not automatically better.
What this means for builders
- Treat the prompt as a tunable parameter of your harness, not a one-time writing task. A weak prompt plus GEPA beat our expert prompt on one task and closed 50–84% of the gap on the other two.
- Check the small-model-plus-optimized-prompt combination before paying for the big model. It beat the big model with the weak prompt on all three tasks at roughly a tenth of the price.
- Write the metric, not the prompt. The textual feedback in your metric is what GEPA turns into instructions; that is where domain knowledge belongs.
- Publish the budget curve. Medium is often the sweet spot; heavy sometimes helps and sometimes overfits a small holdout.
For enterprise teams
For our enterprise clients, we have seen GEPA do wonders when the harness is treated as a product: a real benchmark, a scored experiment, and a rollout that is checked both on the benchmark and outside it. The journey usually looks like this:
- Build a good benchmark. Capture the cases that matter: edge formats, bilingual copy, the failures your team already knows about.
- Set up the experiment. Wire a metric with textual feedback, pick a weak seed and an expert control, and let GEPA search.
- Test on the benchmark and outside it. Held-out scores tell you the curve; production traces tell you whether the prompt still holds when the distribution shifts.
If you want help bringing this into your stack, start here: Creators for enterprises or get in touch.
This article is also available in German: Deutsch
