There is nothing precise about crafting prompts and context—it's just that, a craft. Even if you do the right thing and check some fuzzy boundary conditions using autoscorers, the model can still change out from beneath you at any point and totally alter the behavior of your system. There is no formal language here. After all, mathematics exists because natural language is notoriously imprecise.
The article has some good practical tips and it's not on the author but man I really wish we'd stop abusing the term "engineering" in a desperate attempt to stroke our own egos and or convince people to give us money. It's pathetic. Coming up with good inputs to LLMs is more art than science and it's a craft. Call a spade a spade.
I think it's fair to question the use of the term "engineering" throughout a lot of the software industry. But to be fair to the author, his focus in the piece is on design patterns that require what we'd commonly call software engineering to implement.
For example, his first listed design pattern is RAG. To implement such a system from scratch, you'd need to construct a data layer (commonly a vector database), retrieval logic, etc.
In fact I think the author largely agrees with you re: crafting prompts. He has a whole section admonishing "prompt engineering" as magical incantations, which he differentiates from his focus here (software which needs to be built around an LLM).
I understand the general uneasiness around using "engineering" when discussing a stochastic model, but I think it's worth pointing out that there is a lot of engineering work required to build the software systems around these models. Writing software to parse context-free grammars into masks to be applied at inference, for example, is as much "engineering" as any other common software engineering project.
My thoughts exactly. The author is saying we should think strategically about the use of context. Sure. Yes. But for that to qualify as engineering we need solid theory about how context works.
We don’t have that, yet. For instance experiments show that not all parts of the context window are equally well attended. Imagine trying to engineer a bridge when no one really knows how strong steel is.
The only thing that passed the test of time,so far is specificity: if you ask for multiple things or vague things, you receive half-baked answers trying to cover all bases. If you ask for specific one thing and describe it, the answer quality goes up;e.g. LLMs creating multi-part content mix up the parts and qualities of them, so e.g. asking for Part 1*specific, will always get a better answer than "list all parts of X"(quality drops with length of list).
Any of the "design patterns" listed in the article will have a ton of popular open source implementations. For structured generation, I think outlines is a particularly cool library, especially if you want to poke around at how constrained decoding works under the hood: https://github.com/dottxt-ai/outlines
I know this is a bit of a non sequitur but, on my feed just below your comment, some asked for the RSS for this blog. The juxtaposition of the two comments here is just soooo HN
Yes, and we've also decided that they deserve the title "engineering" more than software engineering does.
Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic.
Software engineering is easy in comparison because computers always do exactly what you tell them to do.
The ways LLMs fail (and the techniques you have to use to account for that) have more in common than physical engineering disciplines than software engineering does!
Yep. Consider woodworking - the wood you use might warp over time, or maybe part of it ends up in the sun or the thing you’ll make gets partly exposed to water.
Can you make a thing that’ll serve its purpose and look good for years under those constraints? A professional carpenter can.
Classic shilling behavior of the insufferably embarrassing: redefining words to the benefit of those who pay your bills to the confusion of everyone else.
The definition of engineering, according to people outside the pocket of the llm industry:
> The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.
How do these techniques apply scientific and mathematical principals?
I would argue to do either of those requires reproducibility, and yet somehow you are arguing the less reproducible something is that the more like "physical engineering" it becomes.
I completely agree that much of software engineering is not engineering, and building systems around LLMs is no better in this sense.
When the central component of your system is a black box that you cannot reason about, have no theory around, and have essentially no control over (a model update can completely change your system behavior) engineering is basically impossible from the start.
Practices like using autoscorers to try and constrain behaviors helps, but this doesn't make the enterprise any more engineering because of the black box problem. Traditional engineering disciplines are able to call themselves engineering only because they are built on sophisticated physical theories that give them a precise understanding of the behaviors of materials under specified conditions. No such precision is possible with LLMs, as far as I have seen.
The determinism of traditional computing isn't really relevant here and targets the wrong logical level. We engineer systems, not programs.
>The ways LLMs fail (and the techniques you have to use to account for that) have more in common than physical engineering disciplines than software engineering does!
Ah yes, the God given free parameters in the Standard Model, including obviously the random seed of a transformer. What if just put 0 in the inference temperature? The randomness in llms is a technical choice to generate variations in the selection of the next token. Physical engineering? Come on.
The tools mechanical and civil engineers use are predictable. You're confusing the things these engineers design, which have tolerances and things like that, with the tools themselves.
If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work.
If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy.
The tool isn't what makes engineering a practice, it's the rigor and the ability to measure and then use the measurements to predict outcomes to make things useful.
Can you predict the outcome from an LLM with an "engineered" prompt?
No, and you aren't qualified to even comment on it since your only claim to fame is a fucking web app
There was a good series interviewing people that worked in both software engineering and traditional engineering: https://www.hillelwayne.com/post/are-we-really-engineers/. The conclusion was that yes, a lot of what we do as software engineers is engineering.
I'd say this shit is even worse that "good at googling". Literal incantation for stochastic machines is like just two notches above checking the horoscope.
Most of the inference techniques (what the author calls context engineering design patterns) listed here originally came from the research community, and there are tons of benchmarks measuring their effectiveness, as well as a great deal of research behind what is happening mechanistically with each.
As the author points out, many of the patterns are fundamentally about in-context learning, and this in particular has been subject to a ton of research from the mechanistic interpretability crew. If you're curious, I think this line of research is fascinating: https://transformer-circuits.pub/2022/in-context-learning-an...
Readit News
The article has some good practical tips and it's not on the author but man I really wish we'd stop abusing the term "engineering" in a desperate attempt to stroke our own egos and or convince people to give us money. It's pathetic. Coming up with good inputs to LLMs is more art than science and it's a craft. Call a spade a spade.
For example, his first listed design pattern is RAG. To implement such a system from scratch, you'd need to construct a data layer (commonly a vector database), retrieval logic, etc.
In fact I think the author largely agrees with you re: crafting prompts. He has a whole section admonishing "prompt engineering" as magical incantations, which he differentiates from his focus here (software which needs to be built around an LLM).
I understand the general uneasiness around using "engineering" when discussing a stochastic model, but I think it's worth pointing out that there is a lot of engineering work required to build the software systems around these models. Writing software to parse context-free grammars into masks to be applied at inference, for example, is as much "engineering" as any other common software engineering project.
If you are the cincinnatian poet Caleb Kaiser, we went to college together and I’d love to catch up. Email in profile.
If you aren’t, disregard this. Sorry to derail the thread.
We don’t have that, yet. For instance experiments show that not all parts of the context window are equally well attended. Imagine trying to engineer a bridge when no one really knows how strong steel is.
But: Interestingly, the behavior of LLMs in different contexts is also the subject of scientific research.
Dead Comment
Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic.
Software engineering is easy in comparison because computers always do exactly what you tell them to do.
The ways LLMs fail (and the techniques you have to use to account for that) have more in common than physical engineering disciplines than software engineering does!
Can you make a thing that’ll serve its purpose and look good for years under those constraints? A professional carpenter can.
We have it easy in software.
The definition of engineering, according to people outside the pocket of the llm industry:
> The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.
How do these techniques apply scientific and mathematical principals?
I would argue to do either of those requires reproducibility, and yet somehow you are arguing the less reproducible something is that the more like "physical engineering" it becomes.
When the central component of your system is a black box that you cannot reason about, have no theory around, and have essentially no control over (a model update can completely change your system behavior) engineering is basically impossible from the start.
Practices like using autoscorers to try and constrain behaviors helps, but this doesn't make the enterprise any more engineering because of the black box problem. Traditional engineering disciplines are able to call themselves engineering only because they are built on sophisticated physical theories that give them a precise understanding of the behaviors of materials under specified conditions. No such precision is possible with LLMs, as far as I have seen.
The determinism of traditional computing isn't really relevant here and targets the wrong logical level. We engineer systems, not programs.
But they really shouldn't because obviously scheduling and logistics is difficult, involving a lot of uncertainty and tolerances.
Ah yes, the God given free parameters in the Standard Model, including obviously the random seed of a transformer. What if just put 0 in the inference temperature? The randomness in llms is a technical choice to generate variations in the selection of the next token. Physical engineering? Come on.
I’m going to start a second career in lottery “engineering”, since that’s a stochastic process too.
If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work.
If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy.
The tool isn't what makes engineering a practice, it's the rigor and the ability to measure and then use the measurements to predict outcomes to make things useful.
Can you predict the outcome from an LLM with an "engineered" prompt?
No, and you aren't qualified to even comment on it since your only claim to fame is a fucking web app
There is no evidence offered. No attempt to measure the benefits.
As the author points out, many of the patterns are fundamentally about in-context learning, and this in particular has been subject to a ton of research from the mechanistic interpretability crew. If you're curious, I think this line of research is fascinating: https://transformer-circuits.pub/2022/in-context-learning-an...