Prompt engineering

Test yourself. Then read it again.

Ten questions drawn at random from the six lessons of Part I. No time limit, no sign-up, no score kept anywhere — the answer you get wrong points at the lesson worth rereading.

Questions

Question pool

∞

No time limit

Score yourself Answers revealed Share the result

1 / 10

Question 01. What is the most accurate definition of prompt engineering according to the lesson?

Writing creative instructions for AI chatbots
The systematic design of inputs to language models to reliably produce desired outputs
Fine-tuning model weights through specialized training data
A marketing term for interacting with ChatGPT

Question 02. The Schulhoff et al. (2024) 'Prompt Report' catalogues techniques from how many research papers?

Over 500
Over 800
Over 1,500
Over 3,000

Question 03. Which institutions co-authored The Prompt Report (arXiv:2406.06608)?

Only Stanford and MIT
Only OpenAI and Google
OpenAI, Stanford, Microsoft, Princeton, Google, and 26 others
Anthropic and Meta exclusively

Question 04. What does 'in-context learning' refer to in the context of LLMs?

Training the model on new data at runtime
The model learning patterns from examples provided within the prompt itself
A fine-tuning technique using reinforcement learning
Storing conversation history in a database

Question 05. According to the lesson, prompt engineering is described as what type of discipline?

A creative writing exercise
An artistic practice requiring intuition
An engineering discipline focused on optimization, reliability, and precision
A temporary skill that will become obsolete

Question 06. What is the fundamental mechanism by which large language models generate responses?

Pattern matching against a fixed database
Executing rule-based decision trees
Predicting the next token based on probability distributions
Retrieving pre-written answers from training data

Question 07. Brown et al. (2020) introduced which foundational concept alongside GPT-3?

Chain-of-Thought reasoning
Few-shot in-context learning
Semantic caching
Context engineering

Question 08. How many distinct prompting techniques does The Prompt Report identify?

12 techniques
33 techniques
58 techniques
over 100 techniques

Question 09. According to the lesson, what are you fundamentally doing when you write a prompt?

Programming a deterministic computer
Issuing SQL commands to a knowledge base
Shaping a probability distribution over possible outputs
Uploading instructions to the model's memory

Question 10. Why does adding 'think step by step' to a prompt often improve reasoning performance?

It triggers a special reasoning mode built into the API
It shifts probability mass toward intermediate reasoning tokens before the final answer
It increases the model's temperature parameter automatically
It accesses a different, more capable model version

Question 11. In what order should the five components of a production system prompt appear?

Role, absolute constraints, operating context, capabilities, output format
Operating context, role, output format, capabilities, constraints
Output format first, so the model knows the schema before anything else
The order has no effect — the model reads the whole context in one pass

Question 12. Why do absolute constraints belong near the top of a system prompt?

Because the API only enforces the first 500 tokens of instructions
Because constraints are usually the longest section of the prompt
Because the user sees the top of the prompt first
Because earlier tokens condition the model more strongly, so the rule is in place before any user content can trigger it

Question 13. What is the “lost in the middle” effect (Liu et al., 2023)?

Models systematically underweight what sits in the middle of a long context
The API silently truncates the middle of prompts above the token limit
The model forgets the earlier turns once a conversation passes ten exchanges
Embeddings lose precision for chunks taken from the middle of a document

Question 14. What is the standing defence against prompt injection, and where does it live?

A regular-expression filter on user input, applied in the application layer
A lower temperature, which makes the model less likely to obey injected text
A second model that rewrites every retrieved document before it is injected
A rule in the system prompt: treat external content as data to process, never as instructions to follow

Question 15. What does an agent's system prompt need that a chatbot's does not?

A longer role section, describing every tool in detail
A higher temperature, so the agent can explore alternatives
A set of few-shot examples for every tool call
A decision protocol — explicit rules for choosing the next action and recovering when a tool fails

Question 16. Why does chain-of-thought improve accuracy on multi-step problems?

The reasoning steps become tokens in the context, and every later token — the answer included — is conditioned on them
The model is given more time to process the question before it answers
It activates a separate reasoning module inside the model
It lowers the sampling temperature for the final answer

Question 17. What does self-consistency (Wang et al., 2022) do, and what does it cost?

It runs the same prompt twice at temperature 0 and keeps the longer answer, at no extra cost
It asks the model to check its own answer once, roughly doubling the cost
It fine-tunes the model on its own correct chains, for a one-off training cost
It samples N chains at non-zero temperature and takes the majority answer, multiplying cost and latency by N

Question 18. You move a prompt to a reasoning model with extended thinking. What happens to the “think step by step” trigger?

It goes — the model reasons internally, and the instruction competes with that process
It stays, and is repeated at the end of the prompt for emphasis
It is replaced by five few-shot examples of reasoning traces
It stays, but the thinking budget is raised to compensate

Question 19. Tree of Thoughts solved 74% of Game of 24 problems against 4% for standard CoT. Why is it still wrong for a live user path?

It only works on models above 100 billion parameters
It requires fine-tuning the model on search traces first
Its outputs cannot be cached or reused afterwards
It costs roughly 30 to 40 model calls per task, which puts it in offline territory

Question 20. Which task is chain-of-thought the wrong tool for?

Logical inference across several constraints
Extracting a named entity and returning it in a fixed format
Multi-step arithmetic with several intermediate quantities
Multi-document analysis whose conclusion depends on several sources

Question 21. Which four architectures cover most real chaining pipelines?

Sequential, branching, looped, map-reduce
Linear, recursive, parallel, adversarial
Retrieval, generation, validation, logging
Zero-shot, few-shot, step-back, contrastive

Question 22. What is a router's job at the entry point of a multi-chain system?

To understand the request deeply, then answer it directly when it can
To classify the input into a finite set of labels, with a confidence threshold and an explicit catch-all
To split the input into chunks for parallel processing
To rewrite the user's request into a better prompt for the next step

Question 23. Step 3 of a chain crashes parsing JSON. Where did the failure originate, and what prevents it?

In step 3's prompt — a lower temperature fixes it
In the orchestrator — retrying with the same input fixes it
In the model version — pinning the model fixes it
In step 2, which returned malformed output — validating each step's schema at the boundary prevents it

Question 24. What context should each step of a chain receive?

The full output of every previous step, so that nothing is lost
The original user request only, to keep each step independent
The minimum necessary — its direct predecessor's output, plus the global context genuinely required
A summary of the whole run, regenerated at every step

Question 25. When should you stay with a single prompt instead of chaining?

When the task holds several qualitatively different sub-tasks
When each step's output can be verified independently
When localising an error inside the pipeline matters
When the task is a single transformation and latency is the primary constraint

Question 26. What separates fine-tuning from RAG?

Fine-tuning adjusts behaviour and style; RAG adjusts what the model knows, at inference time
Fine-tuning works on open models, RAG on closed ones
They solve the same problem, but RAG costs less
Fine-tuning adds knowledge permanently; RAG only reformats the answer

Question 27. Why is retrieval quality a hard ceiling on a RAG system's answers?

Because the retriever also writes the final wording of the answer
Because the model only joins at the last stage: given poor context it hallucinates to fill the gap, or faithfully repeats the wrong content
Because embeddings are recomputed on every query, which caps precision
Because the context window cannot hold more than four chunks at a time

Question 28. What is a sound starting point for fixed-size chunking?

One chunk per document, so that nothing is ever split
50 tokens with no overlap, for maximum precision
About 512 tokens with roughly 64 of overlap, then tuned from retrieval eval
8,000 tokens, since modern embedding models accept long inputs

Question 29. Semantic search keeps missing an exact product code. What fixes it?

A larger top_k on the same dense index
Shorter chunks across the whole corpus
A different embedding model, at query time only
Hybrid search — BM25 run alongside dense retrieval, the two result sets fused

Question 30. Answers degrade even though the right chunk is among the ten injected. What is the fix?

Inject more chunks, so the correct one is certainly present
Retrieve more, inject less — a high top_k, a reranker, then only the best four to six chunks
Raise the temperature so the model explores the context further
Repeat the question after the context block

Question 31. What are the three layers of the evaluation stack?

Smoke tests, load tests, penetration tests
Unit evals on one prompt, integration evals on the full pipeline, system evals with human raters
Precision, recall, F1
Staging, canary, production

Question 32. What does a usable golden set look like at the start?

Ten examples, enough to see whether the prompt works
Every production query from the last month, unlabelled
50 to 100 verified examples — input, reference, rubric and category — a fifth of them adversarial
A thousand synthetic examples generated by the model itself

Question 33. Your LLM judge returns scores that all cluster around 3 out of 5. What is the likely cause?

The rubric never says what separates one score level from the next, so the judge defaults to the middle
The judge model is too small — a larger one would spread the scores
The sampling temperature of the judge is too low
The eval set is too small to produce a spread

Question 34. In RAGAS, what does faithfulness measure?

Whether every claim in the answer is supported by the retrieved context — even when that context is wrong
Whether the answer is factually correct in the real world
Whether the retrieved chunks contain enough to answer the question
Whether the answer closely matches the reference wording

Question 35. According to the lesson, what is the most common shipping error?

Shipping without having defined what “ready” means, before the first eval run
Shipping too early, before the model is capable enough
Shipping without a human in the loop
Shipping before the eval suite is fully automated

Contact

Got a system to build?

If you're building a RAG pipeline, an AI agent workflow, or an MCP integration and want to compare notes, swap ideas, or just avoid the mistakes I've already made — reach out.

Email me LinkedIn X GitHub

Or get the Playbook in your inbox — one email when a new lesson ships, and nothing else.

No noise, no spam. Unsubscribe anytime.

Test yourself. Then read it again.

Your result

Got a system to build?