I've reinvented my own wheel in a particular niche. I didn't set out to do that, but I rejected the existing state of the art as fundamentally misguided. Then, I attempted to divide-and-conquer my particular problem, something that is conventionally considered impossible.
Against all odds, I not only succeeded (mostly thanks to ignorance and stubbornness), but my wheel turns out to be unbelievably good at what it does. Possibly even world-class. After further experimentation, it also enables feats that can only be described as pure heresy with troubling ease. Time passes and some people from that niche start picking up my wheel. They all hold it wrong at the beginning because it's so alien, but once they get the hang of it they never go back.
I get bug reports and feature requests from all over the world for the oddest of use-cases and workflows. I have deep, in-depth technical discussions with brilliant people I would've never met otherwise. I've witnessed achievements done by others with my wheel beyond my wildest dreams. I discover things that keep me awake at night. I get kicks out of melting down the brains of my uninitiated coworkers and colleagues explaining what my wheel does and what I can do with it.
Don't be afraid to reinvent the wheel. You never know what crazy, wild path it might roll you down to.
It's a Ghidra extension that can export relocatable object files from any program selection. In other words, it reverses the work done by a linker.
I originally built this as part of a video game decompilation project, having rejected the matching decompilation process used by the community at large. I still needed a way to divide and conquer the problem, which is how I got the funny idea of dividing programs. That allows a particular style of decompilation project I call Ship of Theseus: reimplementing chunks of a program one piece at a time and letting the linker stitch everything back together at every step, until you've replaced all the original binary code with reimplemented source code.
It's an exquisitely deep and complex topic, chock-full of ABI tidbits and toolchains shenanigans. There's next to no literature on this and it's antithetical to anything one might learn in CS 101. The technique itself is as powerful as it is esoteric, but I like to think that any reverse-engineer can leverage it with my tooling.
In particular, resynthesizing relocations algorithmically is one of those problems subject to the Pareto principle, where getting 80% of them right is reasonably easy but whittling down the last 20% is punishingly hard. Since I refuse to manually annotate them, I've had to relentlessly improve my analyzers until they get every last corner case right. It's by far the most challenging and exacting software engineering problem I've ever tackled, one that suffers no hacks or shortcuts.
Once I got it working, I then proceeded in the name of science to commit countless crimes against computer science with it (some of those achievements are documented on my blog). Cross-delinking in particular, that is delinking an artifact to a different platform that it originates from, is particularly mind-bending ; I've had some successes with it, but I sadly currently lack the tooling to bring this to its logical conclusion: Mad Max, but with program bits instead of car parts.
Ironically, most of my users are using it for matching decompilation projects: they delink object files from an artifact, then typically launch objdiff and try to create a source file that, when compiled, generates an object file that is equivalent to the one they ripped out of the artifact. I did not expect that to happen at all since I've built this tool to specifically not do this, but I guess when everything's a nail, people will manage to wield anything as a hammer.
I learned a fun fact this year that completely changed how I think about wheels.
Ancient wheels supported weight through compression: the cart pressed down, the wheel transferred that force straight into the ground. Simple and solid.
Modern wheels? Totally different. Bicycle wheels hold weight through tension. If a bike is standing still, you could cut the bottom spokes and it would stay upright. Cut the top ones and it collapses—the rim is literally hanging from the hub.
Even car tires follow this principle. It’s not just a balloon of air under the wheel. The internal pressure forms a tensioned ring that holds the car up. The car is suspended from the tire, not resting on it.
So while the article encourages reinventing the wheel metaphorically, I can’t help but admire how we’ve already reinvented the wheel literally—by flipping the physics entirely.
Yes, as TFA alludes to, this expression is actually a bit wrong because the wheel has often been reinvented since ~4500 BCE. The latest "reinventions" are as recent as 19-20th century with the continuous track, monowheel, "hubless" wheel and mecanum wheel [1] [2].
It seems to me that most people say, "don't reinvent..." because they believe there is an existing, ubiquitous, sufficient solution (well, that's why they think it. Why they say it out loud as a response to someone's project, I have different theories on that).
But in ubiquitous solutions, we also find assumptions and trade-offs that require constraining the application to fit the solution. Thus it is there, in those assumptions and trade-offs, as these literal wheel examples demonstrate, where we find the ability to usefully reinvent.
I have to laugh a little seeing your reference to Mecanum wheels and your post being dated 6 hours ago. 12 hours ago, I rewatched the Star Trek reboot film. In it, there is a scene where the characters are all loading into shuttles to head off to their assignments. In the background, we are supposed to be impressed with the technical advancement of the machinery in view, e.g. how something as familiar as a forklift has "futuristic" details like strange wheels. And those wheels on that forklift are Mecanum wheels. In-story, that makes them something like ~300 year old technology.
One of the most important reasons to reinvent the wheel which is is not mentioned by the author is to avoid adding complexity through unnecessary dependencies.
100% this, and I'll add that libraries become popular because they solve an issue in many different scenarios.
That menas that almost by definition, if a library is popular, it contains huge amounts of code that just isn't relevant to your use case.
The tradeoff should be whether you can code your version quickly (assuming it's not a crypto library, never roll your own crypto), because if you can, you'll be more familiar with it and carry a smaller dependency.
“Never roll your own crypto” is just this year’s “never roll your own date library”. There will always be something. Could I code this? Even if it’s quick, there’s ongoing maintenance cost and you lose out on the FOSS community identifying and fixing vulnerabilities as well as new features that you may need to use. Yes, the library might be large and contain things you don’t need, but that’s the tradeoff. You can mitigate this (depending on the platform and language)—for example, with ESM tree-shaking.
I’d rather install date-fns or moment and let it decide what the fourth Sunday of a given month is in 2046, and also audit for the latest browser attack vectors.
I also agree to avoid adding complexity through unnecessary dependencies.
> if a library is popular, it contains huge amounts of code that just isn't relevant to your use case.
It is true that many libraries do contain such code, whether or not they have dependencies. For example, SQLite does not have any dependencies but does have code that is not necessarily relevant to your use. However, some programs (including SQLite) have conditional compilation; that sometimes helps, but in many cases it is not suitable, since it is still the same program and conditional compilation does not change it into an entirely different one which is more suitable for your use.
Also, I find often that programs include some features that I do not want and exclude many others, and existing programs may be difficult to change to do it. So that might be another reason to write my own, too.
Unfortunately, if you depend on any libraries, there's a decent chance one of them depends on some support library. Possibly for just one function. And then your build tool downloads the entire Internet.
"Never roll your own crypto" usually means "never devise your own crypto algorithms". Implementing an established algorithm yourself is OK provided you can prove your implementation works correctly. And... well, as Heartbleed showed, that's hard even with established crypto libraries.
An underrated middle ground, at least when it comes to open source, is vendoring the dependency, cutting out the stuff you don't need, and adapting the API so that it's not introducing more complexity than it has to.
This is also generally helpful when you have performance requirements, as often 3rd party code even when optimized in general, isn't very well optimized for any particular use case.
That's the main reason that I tend to "Reinvent the wheel."
Also, the dependencies often have a lot of extra "baggage," and I may only want a tiny bit of the functionality. Why should I use an 18-wheeler, when all I want to do, is drive to the corner store?
Also, and this is really all on me (it tends to be a bit of a minority stance, but it's mine), I tend to distrust opaque code.
If I do use a dependency, it's usually something that I could write, myself, if I wanted to devote the time, and something that I can audit, before integrating it.
I won't use opaque executables, unless I pay for it. If it's no-money-cost, I expect to be able to see the source.
Custom solutions while initially potentially less complex gradually grow in complexity. There might be a time when you it's worth it to throw out your custom solution and replace it with more general dependency. There's still a benefit because dependency introduced at this stage is used way more thoughtfully because you know the problem it solves inside out.
It might also change your psychological relationship with the dependecy. Instead of being disugsted by yet another external dependecy bringing poorly understood complexity into your project you are thankful that there exists a piece of code maintained and tested by someone else that does the thing you know you need done and lets you remove whole mess of complexity you yourself constructed.
Yeah, I built a library to run tasks based on a directed a-cyclical graph (DAG) and each task can optionally belong to a queue.
So I had to write a simple queue, but since I wanted demos to work in the browser it has a IndexedDB backend, and I wanted demos it to work in an Electron app, so there is a SQLite backend, and I’ll likely want a multi-user server based one so there is a Postgres backend.
And I wanted to use it for rate limiting, etc, so limiters were needed.
And then there is the graph stuff, and the task stuff.
There are a lot of wheels to-create actually, if you don’t want any dependencies.
I do have a branch that uses TypeBox to make and validate the input and output json schemas for the tasks, so may not be dependency free for the core eventually.
I'll agree with this, though in a lot of cases reinventing the wheel is a bad idea.
A previous coworker insisted om writing everything instead of using libraries so I had to maintains a crap undocumented buggy version of what was available in a library.
Thanks for this inspiring essay, I couldn’t agree more that “reinventing for insight” is one of the best ways to learn. I had a similar experience couple months ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. Kinda like reinventing the car using the wheel I reinvented :)
Reinventing the wheel is the best way to learn. But imo that's really the only context where you should.
I love my rabbit holes, but at work, it's often not viable to explore them given deadlines and other constraints. If you want your wheel to be used in production though, it better be a good wheel, better than the existing products.
It's not really the best way to learn because it's the most expensive and time-consuming. What needs to be learned just needs to be well-documented and possible to tinker with, and clarity of communicating knowledge is a problem on its own, but you shouldn't have to build the whole thing from scratch.
> It's not really the best way to learn because it's the most expensive and time-consuming.
The expense (time or otherwise) follows from how intimately you have to get to know the subject. Which is precisely why it's the best way to learn. It's not always viable, but when you can spare the expense nothing else compares.
That sounds like learning something only very superficially. Rewriting from scratch is the only way to really learn any topic of non-trivial depth and complexity.
I think the article subtley misinterprets the whole idea of it and the problem of it.
It's fine to invent your own wheel. But that doesn't mean you should put it in production or feel any entitlement for anybody else to use it there, just because you put personal effort into it. It's going to need to be at least as good, if not better, not just in whatever novel genius way you made it unique, but in all the boring ways - testing, documentation, supportability, etc.
This is a good point, but there is a countervailing one: Sometimes the already invented, tested, and documented "standard" thing is much bigger than what you need. Perhaps you need one feature out of ten. It can make sense to re-invent the wheel to make exactly what you need, and nothing more, even at the expenses you've mentioned. Not always -- you need to weigh all the factors for your own situation -- but choosing "re-invention" is not necessarily a merely-indulgent exercise of ego.
Sometimes the size doesn't really matter, maybe if you are working on a constrained space (embedded software, let's say). But if you are doing Web or using any modern language, unused code is usually optimized out of the compiled bundle and you don't care about it. We care much more about tested and battle proven behavior
A great friend of mine once told me the following quote from an unknown author: "Reinvent the wheel, not because we need more wheels but because we need more inventors."
That quote has brought my mind and heart to some modicum of tranquility at various occasions when I wanted to learn some concept and resolved to write my own "toy version" of a software library, framework et cetera.
Later on, when I learned about Feynman's quote “What I cannot create, I do not understand”, amalgamated the sentiment that it is okay to build something in order to learn one concept. I have thus far learned that in every new journey to reinvent the wheel, so to speak, often led me to paths where my intuitions about the initial concept got stronger and beyond that, I learned several other concepts.
I feel this article misses the main reason (unless it falls under "Build a better wheel") which is to build a wheel that is tailored and stays tailored to your purposes.
How often do I see people metaphorically trying to use a car tire on bicycle with thee excuse of not re-inventing the wheel. There can be great benefits for the parts of your system to be tailor made to work together.
- A Formula 1 wheel when all you need is a bicycle wheel, but the person in charge of choosing wheels chooses it on the basis of “if we want to be as good as Formula 1, then we need to use the same wheels as Formula 1”
Readit News
Against all odds, I not only succeeded (mostly thanks to ignorance and stubbornness), but my wheel turns out to be unbelievably good at what it does. Possibly even world-class. After further experimentation, it also enables feats that can only be described as pure heresy with troubling ease. Time passes and some people from that niche start picking up my wheel. They all hold it wrong at the beginning because it's so alien, but once they get the hang of it they never go back.
I get bug reports and feature requests from all over the world for the oddest of use-cases and workflows. I have deep, in-depth technical discussions with brilliant people I would've never met otherwise. I've witnessed achievements done by others with my wheel beyond my wildest dreams. I discover things that keep me awake at night. I get kicks out of melting down the brains of my uninitiated coworkers and colleagues explaining what my wheel does and what I can do with it.
Don't be afraid to reinvent the wheel. You never know what crazy, wild path it might roll you down to.
It's a Ghidra extension that can export relocatable object files from any program selection. In other words, it reverses the work done by a linker.
I originally built this as part of a video game decompilation project, having rejected the matching decompilation process used by the community at large. I still needed a way to divide and conquer the problem, which is how I got the funny idea of dividing programs. That allows a particular style of decompilation project I call Ship of Theseus: reimplementing chunks of a program one piece at a time and letting the linker stitch everything back together at every step, until you've replaced all the original binary code with reimplemented source code.
It's an exquisitely deep and complex topic, chock-full of ABI tidbits and toolchains shenanigans. There's next to no literature on this and it's antithetical to anything one might learn in CS 101. The technique itself is as powerful as it is esoteric, but I like to think that any reverse-engineer can leverage it with my tooling.
In particular, resynthesizing relocations algorithmically is one of those problems subject to the Pareto principle, where getting 80% of them right is reasonably easy but whittling down the last 20% is punishingly hard. Since I refuse to manually annotate them, I've had to relentlessly improve my analyzers until they get every last corner case right. It's by far the most challenging and exacting software engineering problem I've ever tackled, one that suffers no hacks or shortcuts.
Once I got it working, I then proceeded in the name of science to commit countless crimes against computer science with it (some of those achievements are documented on my blog). Cross-delinking in particular, that is delinking an artifact to a different platform that it originates from, is particularly mind-bending ; I've had some successes with it, but I sadly currently lack the tooling to bring this to its logical conclusion: Mad Max, but with program bits instead of car parts.
Ironically, most of my users are using it for matching decompilation projects: they delink object files from an artifact, then typically launch objdiff and try to create a source file that, when compiled, generates an object file that is equivalent to the one they ripped out of the artifact. I did not expect that to happen at all since I've built this tool to specifically not do this, but I guess when everything's a nail, people will manage to wield anything as a hammer.
Ancient wheels supported weight through compression: the cart pressed down, the wheel transferred that force straight into the ground. Simple and solid.
Modern wheels? Totally different. Bicycle wheels hold weight through tension. If a bike is standing still, you could cut the bottom spokes and it would stay upright. Cut the top ones and it collapses—the rim is literally hanging from the hub.
Even car tires follow this principle. It’s not just a balloon of air under the wheel. The internal pressure forms a tensioned ring that holds the car up. The car is suspended from the tire, not resting on it.
So while the article encourages reinventing the wheel metaphorically, I can’t help but admire how we’ve already reinvented the wheel literally—by flipping the physics entirely.
If you're visual like me, this video illustrates it perfectly: https://youtu.be/8I7QsgYm_NU?si=Vz1aqpHBAPjayqlG
[1] https://en.wikipedia.org/wiki/Hubless_wheel
[2] https://en.wikipedia.org/wiki/Mecanum_wheel
But in ubiquitous solutions, we also find assumptions and trade-offs that require constraining the application to fit the solution. Thus it is there, in those assumptions and trade-offs, as these literal wheel examples demonstrate, where we find the ability to usefully reinvent.
I have to laugh a little seeing your reference to Mecanum wheels and your post being dated 6 hours ago. 12 hours ago, I rewatched the Star Trek reboot film. In it, there is a scene where the characters are all loading into shuttles to head off to their assignments. In the background, we are supposed to be impressed with the technical advancement of the machinery in view, e.g. how something as familiar as a forklift has "futuristic" details like strange wheels. And those wheels on that forklift are Mecanum wheels. In-story, that makes them something like ~300 year old technology.
That menas that almost by definition, if a library is popular, it contains huge amounts of code that just isn't relevant to your use case.
The tradeoff should be whether you can code your version quickly (assuming it's not a crypto library, never roll your own crypto), because if you can, you'll be more familiar with it and carry a smaller dependency.
I’d rather install date-fns or moment and let it decide what the fourth Sunday of a given month is in 2046, and also audit for the latest browser attack vectors.
> if a library is popular, it contains huge amounts of code that just isn't relevant to your use case.
It is true that many libraries do contain such code, whether or not they have dependencies. For example, SQLite does not have any dependencies but does have code that is not necessarily relevant to your use. However, some programs (including SQLite) have conditional compilation; that sometimes helps, but in many cases it is not suitable, since it is still the same program and conditional compilation does not change it into an entirely different one which is more suitable for your use.
Also, I find often that programs include some features that I do not want and exclude many others, and existing programs may be difficult to change to do it. So that might be another reason to write my own, too.
popularity != bloat
This is also generally helpful when you have performance requirements, as often 3rd party code even when optimized in general, isn't very well optimized for any particular use case.
I find that in many cases you can cut out 80 percent of the code of the original library.
Most of the deleted code is flexibility and features we don't need.
It's surprising how small the essence of a solution can be.
Also, the dependencies often have a lot of extra "baggage," and I may only want a tiny bit of the functionality. Why should I use an 18-wheeler, when all I want to do, is drive to the corner store?
Also, and this is really all on me (it tends to be a bit of a minority stance, but it's mine), I tend to distrust opaque code.
If I do use a dependency, it's usually something that I could write, myself, if I wanted to devote the time, and something that I can audit, before integrating it.
I won't use opaque executables, unless I pay for it. If it's no-money-cost, I expect to be able to see the source.
But it is literally a one liner to declare a Pair<> type in java
``` record Pair<S, T>(S first, T second) {} ```
It might also change your psychological relationship with the dependecy. Instead of being disugsted by yet another external dependecy bringing poorly understood complexity into your project you are thankful that there exists a piece of code maintained and tested by someone else that does the thing you know you need done and lets you remove whole mess of complexity you yourself constructed.
So I had to write a simple queue, but since I wanted demos to work in the browser it has a IndexedDB backend, and I wanted demos it to work in an Electron app, so there is a SQLite backend, and I’ll likely want a multi-user server based one so there is a Postgres backend.
And I wanted to use it for rate limiting, etc, so limiters were needed.
And then there is the graph stuff, and the task stuff.
There are a lot of wheels to-create actually, if you don’t want any dependencies.
I do have a branch that uses TypeBox to make and validate the input and output json schemas for the tasks, so may not be dependency free for the core eventually.
[1] https://github.com/workofart/ml-by-hand
[2] https://github.com/workofart/ml-by-hand/blob/main/examples/c...
[3] https://github.com/workofart/ml-by-hand/blob/main/examples/g...
I love my rabbit holes, but at work, it's often not viable to explore them given deadlines and other constraints. If you want your wheel to be used in production though, it better be a good wheel, better than the existing products.
[1] https://www.lesswrong.com/w/chesterton-s-fence
True for life in general. We have limited lifespans. Aging and death really are the great grand daddy of all problems.
The expense (time or otherwise) follows from how intimately you have to get to know the subject. Which is precisely why it's the best way to learn. It's not always viable, but when you can spare the expense nothing else compares.
It's fine to invent your own wheel. But that doesn't mean you should put it in production or feel any entitlement for anybody else to use it there, just because you put personal effort into it. It's going to need to be at least as good, if not better, not just in whatever novel genius way you made it unique, but in all the boring ways - testing, documentation, supportability, etc.
A great friend of mine once told me the following quote from an unknown author: "Reinvent the wheel, not because we need more wheels but because we need more inventors." That quote has brought my mind and heart to some modicum of tranquility at various occasions when I wanted to learn some concept and resolved to write my own "toy version" of a software library, framework et cetera. Later on, when I learned about Feynman's quote “What I cannot create, I do not understand”, amalgamated the sentiment that it is okay to build something in order to learn one concept. I have thus far learned that in every new journey to reinvent the wheel, so to speak, often led me to paths where my intuitions about the initial concept got stronger and beyond that, I learned several other concepts.
How often do I see people metaphorically trying to use a car tire on bicycle with thee excuse of not re-inventing the wheel. There can be great benefits for the parts of your system to be tailor made to work together.
- wheels that force you to do things that way the maker wants, instead of the way that suits you;
- wheels that are fine going downhill ... but nowhere else;
- wheels require you to attach an ever more expensive widget;
- wheels that promise but never quite deliver;
- wheels that keep pushing you into services that you just know are going to be a pain.
Often your own wheel is the best.
- A Formula 1 wheel when all you need is a bicycle wheel, but the person in charge of choosing wheels chooses it on the basis of “if we want to be as good as Formula 1, then we need to use the same wheels as Formula 1”