In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch.
Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist.
They'll cleverly manipulate and convince the boss, against the better discretion of their elder developers, that they can do it, and if they're one of the better developers, the boss won't want to risk losing them so they'll agree to the escapade.
Then said escapade turns into a shambles, as predicted by the elder devs, and the developer who created the mess simply quits and moves to some other job, in search of more fun and greener pastures. Any developer with decades of experience has probably seen this same pattern multiple times.
This is a sentiment that I've seen expressed in comment sections many times. I've been programming professionally now for 10 years, and it just doesn't resonate with my experience. Problems with build systems for external dependencies, package managers, and underfeatured / overcomplicated / buggy third party dependencies have been by far the worse issue in my career, compared to problems with homebrewed systems.
I'm not saying you're wrong, I don't doubt that many people have the opposite experience. It just makes me feel a bit alien when I read comments like this.
Thanks for saying this, I feel this way all the time even though I know it’s against the prevailing wisdom.
My experience is that in the pursuit of not reinventing the wheel, I am frequently told to use a dependency that doesn’t allow us to solve the whole problem, or prevents us from making making the user experience fast or cannot be made to understand our data model. It’s all well and good to use a tool that exists, but using the wrong tool just because it exists is madness. Even worse is when dependencies are deprecated or our use cases become unsupported. Honestly I would prefer to just build everything above the database layer in house, that way we at least know what we can and can’t deliver, and have some chance of fixing things when they break.
Not targeting you, but the industry in general. In every other industry I've been in outside of software dev, 10 years is not considered elder. You're just now becoming not a greenhorn. You're just now getting your sea legs. It's amazing what additional experience happens after year 10.
To that effect, Rust (2015) is 9 years old, Go and Node are 15 years old. While Python (1991) is 33 years old. Just putting things in a different perspective
I've been there, on both sides, with homebrew ideas pushed from up and down, some that worked nicely, and some that were complete disasters...
And I agree with you. The problems with third party dependencies are way worse than any in-house complete disaster.
But that happens almost certainly because everybody is severely biased into adding dependencies. Make people biased into NIH again, and the homebrew systems will become the largest problems again.
I think this depends a lot on whether you're already using high level languages and lots of external libraries vs doing lower level programming using something like C/C++. I managed a large dev team in a Microsoft shop and it would never have occurred to anyone to ever create their own compiler. Even the most experienced programmers would have just continued to brute force things atop .Net's compiler until it eventually "worked". The result, combined with esoteric and poorly understood business requirements, was fragile spaghetti code few could parse for bugs or updates, but it was still several layers above the compiler.
This attitude is by far the most common among "enterprise developers", and one of the big differences between people building things from preexisting building blocks vs -- as witnessed from my 8 years at Google later -- people who think they're smart enough to build everything from the ground up, and do so, using primitive blocks and custom compilers created by similarly hubristic engineers who came before them.
Ymmv, but this has been my experience over the past 25 years.
It's people trying to generalise some rule over the wrong thing. The right thing is that, in both directions, how the project goes is simply a skill question.
You have unskilled, sloppy developers? The homebrew project AND the third-party integration will turn out a mess.
I've spent most of my career in the infrastructure space and I agree with this so much. These days prevailing wisdom is just to use 20 off the shelf open source components and spend your entire day debugging YAML integrations. I think we've lost our minds a bit because of this prevailing wisdom that building a simple wheel that does the 10% of this you actually need is somehow self-indulgence or negligent or both.
Strong agree here. I tend to try as hard as I can to write as much as I can in house so that when shit hits the fan, I have a great chance of being able to do something about it.
Shelling out to an AST parsing library that happens to be slow? Well, shit, that sucks. Guess your compilers just slow now.
The argument is not between NIH and external deps. The argument is over needless complexity and brittle unreliable bits (which can come through either channel) vs keeping things simple.
In my experience, younger developers will push both (in-house and external) directions at once, actually. Building out complex edifices with sharp corners over a maze of transitive dependencies that few understand.
It's the same thing: A fantasy that a framework will solve the problem, combined with a fantasy that they can develop said framework. It's an urge we all suffer from but some of us have learned the hard way to be careful about. (And others who are great at self-promotion have been rewarded for it by naive investors and managers.)
This kind of thing admittedly isn't as pervasive in the last decade as it was the two before, so if you've been a dev only since 2014 years you may not have seen it. The old people like me will get it tho.
Yes. And one of those vaunted differences between 'senior' and 'medior' is knowing the difference.
Because I can confirm what you said: Both experiences are real.
Brewing it up yourself can blow up in your face.
Reaching for external deps that solve the problem can blow up in your face.
Knowing which choice to make is _tricky_ and is hard to confirm. It doesn't sit well with programmers; either solution will _work_ (you can't write a unit test that 'fails' if you made the wrong choice here), and even if you're willing to accept highly suspect, Goodhart's law-susceptible metrics such as LOC, you still can't get anywhere because it's trading off more code you have to write and maintain without help from a larger community against having fewer lines _in total_ as part of the system.
I do not know of any way to do it right other than to apply a ton of experience. And it's really hard to keep yourself honest. Even if you're willing to wait 5 years and then spend some time looking back, how do you really know?
Anybody with a bunch of experience has seen enough homebrew stuff asplode in their face to be able to paint a picture with how utterly badly that choice could go. If you chose the 'build it on external deps' route you can easily tell yourself you did it right by painting a terrible picture of how it would have gone if you made the other choice.
But the reverse is just as true.
I think I'm really good at it. But, writing about it here, I don't have any real basis to make that claim. I look around at other dev shops that make products of similar complexity and it feels like they need 10x to 100x more resources, have more downtime, and have far larger dev teams. But no doubt bias is creeping in there too, and no 2 software products are 100% comparable in this sense.
I naturally trust homebrewers more because they tend to understand complex technical things better. Someone who can just glue libraries together is lost when I ask them to fire up a debugger and figure out why some interaction is not working. A hopeless NIH sufferer needs to be 'supervised' and their choices about what to write needs to be questioned, but, that's doable with supervision. "Just git gud and be technically proficient" not so much. But then maybe that's bias too - that leads to a codebase that is easier navigated when you're familiar with debuggers and reading code to understand it. Reaching for third party deps a lot leads to a codebase that is easier navigated when you're familiar with docs and tutorials. These are self fulfilling prophecies.
Well, foreign projects communicating with each other is always ground for a mess, but this is not an either-or question.
Also, your mileage may vary based on the niche you are working on - in case of, say java, the initial setup of the build system may not be "fun", but it will just work from then on.
The worse trash fires were the homebrewed systems, but maybe that's because I could dig in and see how bad they were.
But I'd actually agree with you - as bad as those were, I'd rather them than a shitty 3rd party something. At least I can theoretically do something about the in-house one, and, all the ones I've seen were smaller in scope than any SaaS product.
There’s a flip side to this which is building on top of which solves your problem but aren’t actually suitable. It “works”, but often at the expense of someone else. A great example is homebrew and GitHub. Or, making a shim between something that solves 80% of your problem rather than solving the problem yourself.
The mark of a 10x engineer IMO is getting the build vs buy question right consistently. My experience is that teams get it wrong often in both directions
> even when there are better things that already exist
That's a "big if".
Lots of times what's there is a nightmarish tangle of technical debt left by previous greenfield devs. The dev who gets to maintain and evolve this dreck is the sucker, scapegoated for ever slower development.
Canonical example: on-call AWS engineers working hellish overtime to close tickets on one of AWS's many terribad fragile codebases.
I have only seen this problem in elder devs. Some people simply seem to believe they are selected by god to hand out their frameworks for poor juniors to be forced to work inside. Sometimes they are just a founding engineer, they were the only devs in a startup, or one of five.
These senior devs often quit or are fired and leave the rest of the developers with their “good ideas”.
I have never seen a junior taking on something new that is inherently huge and complex. I have seen them go overboard with refactoring, because someone tricked them into thinking the Boy Scout rule is good, or that DRY is important, or that they need to think ahead and abstract/generalize for the future. Inevitably that is something they were “taught” by senior colleagues or teachers.
A corollary to this is the pandemic of phobia for NIH. A lot of developers really seem to prefer janky, undermaintained third party libraries with huge APIs over a quick home made hack to solve exactly the problem your team has, and you can maintain and test and just know everything there is to know about. Building your own stuff is good. It is the business we are in.
I think what you're calling "elder devs" is actually the "intermediate" devs. They're not junior in any sense if they're capable/allowed to create these huge balls of mud we're referring to. And the elders normally have seen way too much to fall into that trap... and definitely don't quit often like you're describing at all (my experience is that the younger you are, the more often you change jobs - which is good for you as it's been shown this is the best way to get a good paycheck, but bad for employers, of course). They're tired of that constant churn and have had more than enough time to find a place where they're comfortable. The OP is likely talking about those as well, but from the perspective of someone who probably is truly senior and has been doing this at least since the 90's... basically, they're talking about the devs who know just enough to be dangerous (some will enter this stage from around 3 years of experience to 10, others may stay there from 5y to 20y - so it's difficult to group them together in a neat group) just like you're doing, but to you, they look senior as well.
> A lot of developers really seem to prefer janky, undermaintained third party libraries with huge APIs over a quick home made hack to solve exactly the problem your team has
Sometimes it’s not even that they’re janky and undermaintained, it’s just the huge and unnecessary API. A good example is watching for file changes. inotify has been around forever, and is easy to reason about. The Python library inotify_simple [0] just wraps that. That’s it. It works extremely well, has no dependencies of its own, and provides nothing else. I once needed this functionality for a project, and had another teammate argue we should use watchdog [1] instead, because it had more stars, and more frequent commits. It took me longer than I thought it would to explain that sometimes, projects are complete and don’t need commits, and that we didn’t need or want any of the additional complexity provided by watchdog.
Another example is UUID generation. Python doesn’t yet natively do UUIDv7 generation, but if you read their source code and the RFC for UUIDv7, it’s fairly easy to write your own implementation. This was met with “please don’t write your own UUID implementation; use a library.” Baffling.
To drop an anecdatum into the fray, I am an older dev and I tend to see this from younger devs because, frankly, the elder devs are too tired and semi-burnt out from trying to stop the younger devs creating nonsensically pure houses of glass built on the Wise Words of The Bloggers using Technique du Jour where everything takes 4x as long and results in the most fragile and complex sugar-spun castles which collapse into unmaintainable slop after the first contact with enemy (customer) fire.
(I may or may not be bitter about this from previous jobs)
> Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist.
This is true; but after enough years in the industry you learn to correlate success with laziness. This is well-discussed and arguably obvious but on an emotional level it takes a long time to fully sink in. We were all once developers with outsized ambitions and awareness we can flee to greener pastures.
On the flip side there are many engineers who are so afraid to build anything that seems even a little bit difficult you end up with a million dependencies for things that could have been 100 lines of code.
A common thing to hear in frontend these days is “you’re just rebuilding X” “you mean you want to rebuild X” where X is some trivial flavor of the month library.
All of the famous tools, databases and frameworks you see today was built by someone who said they can do it better and then they built a community around it.
I'm working with SBoM, one fun side effect is that you can scan SBoM's for vulnerabilities. Suddenly hackers, your customers and your competitors starts do to this and you need to make sure your third party dependencies are updated.
This reveals the cost of dependencies (that often are ignored).
I hope that we in the future will have a more nuanced discussion on when it's okay to add a dependency and when you should write from scratch.
> In software development it's pretty important to know when to build "on top" of something else, and when to start from scratch.
Building some brownfield CRUD for a run of the mill org? Starting from scratch will almost always go horribly, just pick whatever enterprise'y solution fits the task at hand and be done with it.
Working for one of the big orgs on something interesting, and have the backing needed for being able to throw person-years at a problem until it crumbles under the collective engineering effort? Building from scratch might be a good choice sometimes.
Personal learning projects, side projects and the like? If you won't have to maintain it long term or at least don't think you'll have significant amounts of time or effort you can spare for that, then from scratch is okay (your own game engine to learn about the internals? your own implementation of something S3 compatible? maybe your own CMS for the hell of it?), otherwise consider treating it as a brownfield project (e.g. if you want to make and finish a game, or just store some files, or maybe just run a blog where the focus is on the content not how you made the thing it's running on).
What's my reasoning for this? Code is typically written to solve a particular problem. In business context, that typically means finishing some Jira issues and having deliverables. In large enough open source projects that typically also means having instructions on how to run and administer the thing, proper test coverage given the larger amount of various people working on it. Thus, the bus factor becomes larger and it won't be as much of a miserable experience of code archaeology as when the dev who wrote some custom CMS for a project at work leaves and literally only some code without even proper CI/CD is left behind, no proper comments, no ADRs, no code examples that aren't coupled to the logic, no documentation or even summary of the project, maybe an empty template for a README, no decoupling between the technical bits and the business rules (or just tight coupling in general), because again, they only wanted to ship. And even if they had better intentions, there were still deadlines and they were still one person (or a small team) that can't compete with any of the large multi-year projects out there.
In the last paragraph what you said is often what happens due to bad management too. A good developer can be given a task that they barely have time to get done, and as a result the unit tests, and the documentation, and even the architecture suffers, or gets omitted.
Often in shops where just cranking out new features and/or bug fixing is the goal of management, the software continues to degrade endlessly due to all the things you mentioned, because spending time in those areas isn't something the boss finds justifiable expenditure of developer time. Once all the developers who originally wrote the code have left or been fired then the deterioration in code quality can start to go down rapidly until some kind of "cleanup" effort is undertaken, where ZERO new features are created, but things are just cleaned up.
In projects with millions of lines of spaghetti code sometimes this cleanup is completely impossible, because a total rewrite would be easier.
> boss won't want to risk losing them so they'll agree to the escapade.
That's precisely it. A motivated engineer is almost always going to outperform a bored engineer/one who quits. Morale is miles more important than chasing after efficiency.
"Boss" here is likely making the correct decision after weighing the ups and downs.
Because at way too many places there are seven junior eng and one senior eng. If they’re lucky.
I’ve been that senior eng and you spend 130% of your time trying to find terrible decisions about to be made before it’s too late and reviewing 1,400-line PRs only to discover (and try to teach) that it could have been 40 lines. Enough junior devs without sufficient supervision can literally crank out endless quantities of negative-value work. And it’s a battle you’re constantly losing.
Lots of times it's just ordinary office politics, or the boss likes one person more than another, or isn't "technical" enough to know when he's being manipulated. Because often managers aren't developers themselves, so they don't know which developer is telling them the best advice, when two developers disagree.
It usually doesn't happen for me, but when it does, it's because the seniors are out of options.
The safe, tried-and-true way to build typical web crap is to stick the one, blessed database in the middle, and then dangle all the dependencies off that. Everything is synchronous because it's simpler, and it's what the seniors grew up with.
And then one day you won't be able to run your new history feature, because it's locking up the database for too long and new transactions are timing out.
The juniors only get to run the show and introduce exotic, non-boring technology (asynchronicity, event-sourcing, eventual consistency, CQRS etc.) after the seniors have admitted defeat.
I've seen it other places as well. Film/video post production has phases of the hot editor/colorist/director/etc. Then a new young hotness comes along because people feel the gray beards are too long in the tooth and impossible for them to be hip. Then the gray beards watch the newbie make the same mistakes over and over. It's called getting old. It's young thinking they are invincible and finding it impossible the olds can possibly know anything. It's human nature
I'm in that boat except the ages are reversed and the older guys are constantly trying to build things from scratch. They just refuse to spend any time looking for off-the-shelf solutions and only build on what they know, so we waste time and end up with a crappy result.
Example: For a parallel data processing pipeline they wanted to build a REST interface for submitting "jobs" to a cluster which would parallelize with MPI, instead of just using xarray+dask.
Another example: They wanted to store tabular data product metadata Postgres with URIs pointing to NetCDF files on disk, instead of just putting everything inside NetCDF.
I like to build from scratch, but I prefer maintain something on top of something. Less code to maintain since I can delegate part of my code to the technology built on top of it.
In that matters, for my professional work have a healthy long term life, I usually select on-top development style, and sometimes, if I have really good reasons to do, from scratch is the option. Of course, for learning and personal projects, from scratch is always a very fun choice!
+1 on this. Number one source of bugs at a recent job was a homebrew TLS / HTTP load balancer. First chance I got I replaced it with nginx and bugs shot down immediately. With tools like apache, nginx, haproxy and caddy available, it was pure madness to reinvent that wheel... But the dev wanted open source CV padding...
I've seen this a lot when someone wants to add "workflow automation" or "scripting" to their app. The most success I'd had is embedding either Lua or Javascript (preferably Lua) with objects/functions from the business domain available to the user's script. This is what games do too. I think it's a great way to dodge most of the work. For free you can support flow control, arbitrary boolean expressions, math, etc.
I find that the #1 reason people add those "simpler than Lua" homegrown languages in Enterprise is to allow non-programmers to program. This not only has a tremendous cost to develop (compared to something like embedding Lua) but it also creates the worst kind of spaghetti.
One of the most unhinged pieces of software I have ever seen was the one from a fintech I worked with. Visual programming, used by business specialists. Zero abstraction support, so lots of forced repetition. No synchronous function call, so lots of duplications or partitioning to simulate it. Since there were two failed versions, there are three incompatible versions of this system running in parallel and migration from one to the other must be done manually.
The problem is about 90% of the business rules were encoded into this system, because business people were in a hurry. People wanted a report but didn't want to wait for Business Intelligence? Let's add "tags" to records so they appear on certain screens, and then remove them when they shouldn't anymore.
In the end the solution was adding "experts" to use it, but the ones who actually knew or learned any programming would just end up escaping to other companies.
One pitfall that is so obvious it hurts (but I have seen people fall into it), goes a bit like this:
1. We have a python application
2. We need a configuration format, we pick one of the usual (ini/toml/yaml/...)
3. We want to allow more than usual to be done in this config, so let's build some more complex stuff based on special strings etc.
Now the thing they should have considered in step 3 is why not just use a python file for configuration? Sure this comes with pitfalls as you now allow people who write the config to do similar things than the application, but you are already using a programming language, why not just use it for your overly complex configuration? For in house stuff this could certainly be more viable than writing your own parser.
Because now, anything that wants to read that config has to be written in Python. You've chained yourself to a stack just for a dynamic config. I ran into this issue at a previous job, but with a service that leaned heavily on hundreds of Django models. It made it impossible to use those models as a source of truth for anything unless you used Python and imported a heavyweight framework. It was the biggest blocker for a C++ rewrite of the service, which was really bad because we were having performance issues and were already reaching our scaling limits.
Declarative configs are preferable for keeping your options open as to who or what consumes them. For cases where config as code is truly necessary, the best option is to pick something that's built for exactly that, like Lua (or some other embedded scripting language+runtime with bindings for every language).
This can sometimes be a good idea. But it isn't without downsides. Now your config file is Python and capable of doing anything Python can do (which isn't necessarily a good idea), it's no longer safe, you now have to deal with shitty Python tooling, you might have to debug crashes/lockups in your config files, you can no long switch implementation languages, etc. etc.
One improvement though is using Starlark, instead of directly Python, since it offers a lot of advantages for a more lightweight runtime and parallelism.
I've done both. I embedded VBScript/JScript in an app via Microsoft's Active Scripting[1] interfaces and wrote a template language that grew to contain typical programming language constructs.
Looking back it was the VBScript/JScript functionality that caused me the most problems. Especially when I migrated the whole app from C++ to .Net.
Reminds me of the pain of intentionally building a compiler for Java 2 (subset) to MIPS compiler by writing out each AST node class by hand. And, I did it twice, once in C++03 with bison and flex and again in Java 2 with CUP and JFlex... each was developed to build and run as a host portably across Solaris (sparc), Linux (x86), HP-UX (68k), SGI (MIPS), and Windows (x86) with compiled with targets run on the SPIM emulator. It did have dead code, dead string, and dead variable elimination, but that was as far my optimization passes went. I recall the only build tool I used for each was the portable subset of make without GNU extensions.
Speaking of reinventing the wheel, in 1998, I built a flexible almost framework for a "portable" generic installer using Java 2, JWT (native GUI controls), and JNI on Windows to create a program group and desktop shortcut icon. The hilarious part was shipping a full JRE on a CD. It took forever to load but the additional time seemed impressive for expensive, niche software in a way similar to the now fake "loading..." delayed progress bar.
If you have a user flow that requires interaction, hits an API that returns instantly, and then changes something fundamentally, it is better UX to make the user wait for a couple seconds with an animation and then some success state indication. If it all happens at once it can be a bit jarring or confusing.
I get the solution for this and I know what all the terms mean. But I don't understand the problem. Whether it's facetious or hyperbole or whatever, I just don't get who or what circumstances this is addressing.
This is written like a Jeopardy answer. I just don't know what the question is.
Is it just me, or would "single page applications" fit easily in the "examples" section there? Sounds kind of trollish when I write it out, but honestly, it fits pretty well, right? We threw away all the things the browser gives you for free and re-implement the back button, history, etc in JavaScript. (and it's somewhat fractal, as within the frameworks we use to do this you'll frequently see people re-implementing things the framework already does).
That makes total sense. I have been tempted to do this in the past. Fortunately time and resources constraints have kept it to costly sane and maintainable, performant configurations until I learned that I would never create the system I wanted and that it was probably better that I didn't anyways. I guess I've been lucky and didn't even know it.
When writing programs that take other programs as inputs, and/or produce other programs as outputs, it's tempting to treat the program as only slightly more structured than its textual representation.
The problem is that unless your use case is very limited and is guaranteed to stay that way, supporting more and more language constructs will quickly turn your code into a mess.
Compiler design as we learn it (lex/parse, syntax tree, semantic checks, transforms, lowering to codegen) is _the_ solution to the problem of dealing with computer programs as inputs and outputs. Trying to do something less is like solving a dynamic programming problem without knowing dynamic programming: it will only work for a restricted set of inputs.
I've come very close to be the "sir" mentioned in the article (for hobby stuff only though), luckily always I did catch myself and was able to stop before it was too late. I decided at some point that I do not want to build a compiler/transpiler, and if I do some day, I want for it to be a conscious decision and not an accident like in the article.
It starts innocently, e.g. doing some template files and replacing some simple values, then you start to have to do more replacements and more "smart" parsing and then at some point it's too late, as the article suggests.
TBF, I did put together a transpiler from PHP to JS, but I didn't build it, just found the different pieces that luckily fit together and hacked around it enough that it could run in the browser.
Indeed, the safer thing is to actually build a few toy compilers on the side so you can get a sense for what they are good for, and what level of effort is required to build and maintain it.
Keeping them locked up in the "scary CS" closet only ends up stunting your growth.
I like to write toy compilers or interpreters as an exercise when learning a new language. Usually for a Forth or Lisp or one of Turing Tarpit languages. It requires some of the most common bits of programming: I/O, lexing, parsing (both of source and of arguments to the compiler), file handling, and some common algorithms & data structures (can't have an AST without a tree).
I'd say the opposite. Building a compiler incrementally, driven by clear needs at each step, as described in the article, tends to work out better than trying to build a compiler from day 1.
I've written 7 compilers. If you don't know you're writing a compiler from the beginning and you've never written before, it is basically guaranteed to be a hot mess in the end, unless you accidentally make exactly the right call at every one of a dozen critical steps. It's extremely unlikely to just stumble on all the good architectural ideas from a field that is 70+ years old without any planning.
Granted, if you've got a really simple language then maybe there's an upper bound on how bad it can be. But in practice people tend to also want to grow the language as they go.
Don't get me wrong; there's a lot than can wrong if you over-engineer the compiler from the beginning, including ending up with a hot mess anyway.
BTW if you can't write down, even in three sentences, the requirements for said compiler from the beginning, congratulations, you have no idea what you are doing.
Oh no. What the article describes ends up with a real mess. If you set out to support a certain language you'll have a global view of how things should fit together, and what the implementation should do.
Is there a meta-library someone somewhere has already written for when I just want to write 20% of a compiler, and it more or less takes care for me 80% of the common compiler-building-related things I’m likely to need to do?
Having recently built 90% of a compiler by mistake, I felt like this post was written specifically about me. Hilarious writing, congrats to the author.
There is also the opposite, Ed Kmett is on record as saying that he had a million ideas for his personal programming language, and all of the cool stuff it was going to do, but then he ran into Haskell and said “Let’s be real, whatever amazing language I make isn't gonna be half as put together as this and I have this one already in front of me...”
Don't duplicate what already exists worse except as a brief experiment or for learning. For real use, it will just cost money and time to fix and support for no reason.
I don't think building compilers is that bad, tbh. It's very difficult to do this without realizing it.
I've written a dozen different programs that might be considered compilers; some very simple, others very complex and whose life continued once I left the organization. Writing a functional compiler that provides the needs of the organization where existing tooling doesn't takes discipline and focus on what you actually want to accomplish. I don't know what "defining a struct inside a loop" might mean and this strikes me as, very obviously, having no clue what you actually want to build.
Perhaps the issue is not building a compiler but rather the lack of focus to begin with.
Back in the earlier days of AI, not that early, but the late 80's I was the lead developer for an AI research program being jointly conducted by 3 business professors from MIT, Harvard, and Boston University. We were working on "frame based knowledge representation" - frame of reference based node links between nodes containing something: a number, a word, a sentence, or a "function that combines linked nodes into a new frame of reference".
Long story short, we thought we were making a new type of N-dimensional spreadsheet, but after 3 semesters of work one of the advisors at MIT told us we need to meet his colleague, and that guy informed us we had a working compiler for a hybrid of Lisp and C.
Readit News
Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist.
They'll cleverly manipulate and convince the boss, against the better discretion of their elder developers, that they can do it, and if they're one of the better developers, the boss won't want to risk losing them so they'll agree to the escapade.
Then said escapade turns into a shambles, as predicted by the elder devs, and the developer who created the mess simply quits and moves to some other job, in search of more fun and greener pastures. Any developer with decades of experience has probably seen this same pattern multiple times.
I'm not saying you're wrong, I don't doubt that many people have the opposite experience. It just makes me feel a bit alien when I read comments like this.
My experience is that in the pursuit of not reinventing the wheel, I am frequently told to use a dependency that doesn’t allow us to solve the whole problem, or prevents us from making making the user experience fast or cannot be made to understand our data model. It’s all well and good to use a tool that exists, but using the wrong tool just because it exists is madness. Even worse is when dependencies are deprecated or our use cases become unsupported. Honestly I would prefer to just build everything above the database layer in house, that way we at least know what we can and can’t deliver, and have some chance of fixing things when they break.
To that effect, Rust (2015) is 9 years old, Go and Node are 15 years old. While Python (1991) is 33 years old. Just putting things in a different perspective
And I agree with you. The problems with third party dependencies are way worse than any in-house complete disaster.
But that happens almost certainly because everybody is severely biased into adding dependencies. Make people biased into NIH again, and the homebrew systems will become the largest problems again.
This attitude is by far the most common among "enterprise developers", and one of the big differences between people building things from preexisting building blocks vs -- as witnessed from my 8 years at Google later -- people who think they're smart enough to build everything from the ground up, and do so, using primitive blocks and custom compilers created by similarly hubristic engineers who came before them.
Ymmv, but this has been my experience over the past 25 years.
You have unskilled, sloppy developers? The homebrew project AND the third-party integration will turn out a mess.
Shelling out to an AST parsing library that happens to be slow? Well, shit, that sucks. Guess your compilers just slow now.
In my experience, younger developers will push both (in-house and external) directions at once, actually. Building out complex edifices with sharp corners over a maze of transitive dependencies that few understand.
It's the same thing: A fantasy that a framework will solve the problem, combined with a fantasy that they can develop said framework. It's an urge we all suffer from but some of us have learned the hard way to be careful about. (And others who are great at self-promotion have been rewarded for it by naive investors and managers.)
Finding simple solutions takes humility and time.
Because I can confirm what you said: Both experiences are real.
Brewing it up yourself can blow up in your face.
Reaching for external deps that solve the problem can blow up in your face.
Knowing which choice to make is _tricky_ and is hard to confirm. It doesn't sit well with programmers; either solution will _work_ (you can't write a unit test that 'fails' if you made the wrong choice here), and even if you're willing to accept highly suspect, Goodhart's law-susceptible metrics such as LOC, you still can't get anywhere because it's trading off more code you have to write and maintain without help from a larger community against having fewer lines _in total_ as part of the system.
I do not know of any way to do it right other than to apply a ton of experience. And it's really hard to keep yourself honest. Even if you're willing to wait 5 years and then spend some time looking back, how do you really know?
Anybody with a bunch of experience has seen enough homebrew stuff asplode in their face to be able to paint a picture with how utterly badly that choice could go. If you chose the 'build it on external deps' route you can easily tell yourself you did it right by painting a terrible picture of how it would have gone if you made the other choice.
But the reverse is just as true.
I think I'm really good at it. But, writing about it here, I don't have any real basis to make that claim. I look around at other dev shops that make products of similar complexity and it feels like they need 10x to 100x more resources, have more downtime, and have far larger dev teams. But no doubt bias is creeping in there too, and no 2 software products are 100% comparable in this sense.
I naturally trust homebrewers more because they tend to understand complex technical things better. Someone who can just glue libraries together is lost when I ask them to fire up a debugger and figure out why some interaction is not working. A hopeless NIH sufferer needs to be 'supervised' and their choices about what to write needs to be questioned, but, that's doable with supervision. "Just git gud and be technically proficient" not so much. But then maybe that's bias too - that leads to a codebase that is easier navigated when you're familiar with debuggers and reading code to understand it. Reaching for third party deps a lot leads to a codebase that is easier navigated when you're familiar with docs and tutorials. These are self fulfilling prophecies.
Also, your mileage may vary based on the niche you are working on - in case of, say java, the initial setup of the build system may not be "fun", but it will just work from then on.
The worse trash fires were the homebrewed systems, but maybe that's because I could dig in and see how bad they were.
But I'd actually agree with you - as bad as those were, I'd rather them than a shitty 3rd party something. At least I can theoretically do something about the in-house one, and, all the ones I've seen were smaller in scope than any SaaS product.
https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
The mark of a 10x engineer IMO is getting the build vs buy question right consistently. My experience is that teams get it wrong often in both directions
That's a "big if".
Lots of times what's there is a nightmarish tangle of technical debt left by previous greenfield devs. The dev who gets to maintain and evolve this dreck is the sucker, scapegoated for ever slower development.
Canonical example: on-call AWS engineers working hellish overtime to close tickets on one of AWS's many terribad fragile codebases.
These senior devs often quit or are fired and leave the rest of the developers with their “good ideas”.
I have never seen a junior taking on something new that is inherently huge and complex. I have seen them go overboard with refactoring, because someone tricked them into thinking the Boy Scout rule is good, or that DRY is important, or that they need to think ahead and abstract/generalize for the future. Inevitably that is something they were “taught” by senior colleagues or teachers.
A corollary to this is the pandemic of phobia for NIH. A lot of developers really seem to prefer janky, undermaintained third party libraries with huge APIs over a quick home made hack to solve exactly the problem your team has, and you can maintain and test and just know everything there is to know about. Building your own stuff is good. It is the business we are in.
Sometimes it’s not even that they’re janky and undermaintained, it’s just the huge and unnecessary API. A good example is watching for file changes. inotify has been around forever, and is easy to reason about. The Python library inotify_simple [0] just wraps that. That’s it. It works extremely well, has no dependencies of its own, and provides nothing else. I once needed this functionality for a project, and had another teammate argue we should use watchdog [1] instead, because it had more stars, and more frequent commits. It took me longer than I thought it would to explain that sometimes, projects are complete and don’t need commits, and that we didn’t need or want any of the additional complexity provided by watchdog.
Another example is UUID generation. Python doesn’t yet natively do UUIDv7 generation, but if you read their source code and the RFC for UUIDv7, it’s fairly easy to write your own implementation. This was met with “please don’t write your own UUID implementation; use a library.” Baffling.
[0]: https://github.com/chrisjbillington/inotify_simple
[1]: https://github.com/gorakhargosh/watchdog
To drop an anecdatum into the fray, I am an older dev and I tend to see this from younger devs because, frankly, the elder devs are too tired and semi-burnt out from trying to stop the younger devs creating nonsensically pure houses of glass built on the Wise Words of The Bloggers using Technique du Jour where everything takes 4x as long and results in the most fragile and complex sugar-spun castles which collapse into unmaintainable slop after the first contact with enemy (customer) fire.
(I may or may not be bitter about this from previous jobs)
This is true; but after enough years in the industry you learn to correlate success with laziness. This is well-discussed and arguably obvious but on an emotional level it takes a long time to fully sink in. We were all once developers with outsized ambitions and awareness we can flee to greener pastures.
But at the same time since I spend almost ever waking hour of my spare time coding, the word lazy still isn't quite accurate in every way either.
A common thing to hear in frontend these days is “you’re just rebuilding X” “you mean you want to rebuild X” where X is some trivial flavor of the month library.
All of the famous tools, databases and frameworks you see today was built by someone who said they can do it better and then they built a community around it.
This reveals the cost of dependencies (that often are ignored).
I hope that we in the future will have a more nuanced discussion on when it's okay to add a dependency and when you should write from scratch.
Building some brownfield CRUD for a run of the mill org? Starting from scratch will almost always go horribly, just pick whatever enterprise'y solution fits the task at hand and be done with it.
Working for one of the big orgs on something interesting, and have the backing needed for being able to throw person-years at a problem until it crumbles under the collective engineering effort? Building from scratch might be a good choice sometimes.
Personal learning projects, side projects and the like? If you won't have to maintain it long term or at least don't think you'll have significant amounts of time or effort you can spare for that, then from scratch is okay (your own game engine to learn about the internals? your own implementation of something S3 compatible? maybe your own CMS for the hell of it?), otherwise consider treating it as a brownfield project (e.g. if you want to make and finish a game, or just store some files, or maybe just run a blog where the focus is on the content not how you made the thing it's running on).
What's my reasoning for this? Code is typically written to solve a particular problem. In business context, that typically means finishing some Jira issues and having deliverables. In large enough open source projects that typically also means having instructions on how to run and administer the thing, proper test coverage given the larger amount of various people working on it. Thus, the bus factor becomes larger and it won't be as much of a miserable experience of code archaeology as when the dev who wrote some custom CMS for a project at work leaves and literally only some code without even proper CI/CD is left behind, no proper comments, no ADRs, no code examples that aren't coupled to the logic, no documentation or even summary of the project, maybe an empty template for a README, no decoupling between the technical bits and the business rules (or just tight coupling in general), because again, they only wanted to ship. And even if they had better intentions, there were still deadlines and they were still one person (or a small team) that can't compete with any of the large multi-year projects out there.
Often in shops where just cranking out new features and/or bug fixing is the goal of management, the software continues to degrade endlessly due to all the things you mentioned, because spending time in those areas isn't something the boss finds justifiable expenditure of developer time. Once all the developers who originally wrote the code have left or been fired then the deterioration in code quality can start to go down rapidly until some kind of "cleanup" effort is undertaken, where ZERO new features are created, but things are just cleaned up.
In projects with millions of lines of spaghetti code sometimes this cleanup is completely impossible, because a total rewrite would be easier.
That's precisely it. A motivated engineer is almost always going to outperform a bored engineer/one who quits. Morale is miles more important than chasing after efficiency.
"Boss" here is likely making the correct decision after weighing the ups and downs.
why are the junior calling the shots over the elder developers?
I’ve been that senior eng and you spend 130% of your time trying to find terrible decisions about to be made before it’s too late and reviewing 1,400-line PRs only to discover (and try to teach) that it could have been 40 lines. Enough junior devs without sufficient supervision can literally crank out endless quantities of negative-value work. And it’s a battle you’re constantly losing.
The safe, tried-and-true way to build typical web crap is to stick the one, blessed database in the middle, and then dangle all the dependencies off that. Everything is synchronous because it's simpler, and it's what the seniors grew up with.
And then one day you won't be able to run your new history feature, because it's locking up the database for too long and new transactions are timing out.
The juniors only get to run the show and introduce exotic, non-boring technology (asynchronicity, event-sourcing, eventual consistency, CQRS etc.) after the seniors have admitted defeat.
Example: For a parallel data processing pipeline they wanted to build a REST interface for submitting "jobs" to a cluster which would parallelize with MPI, instead of just using xarray+dask.
Another example: They wanted to store tabular data product metadata Postgres with URIs pointing to NetCDF files on disk, instead of just putting everything inside NetCDF.
In that matters, for my professional work have a healthy long term life, I usually select on-top development style, and sometimes, if I have really good reasons to do, from scratch is the option. Of course, for learning and personal projects, from scratch is always a very fun choice!
Deleted Comment
One of the most unhinged pieces of software I have ever seen was the one from a fintech I worked with. Visual programming, used by business specialists. Zero abstraction support, so lots of forced repetition. No synchronous function call, so lots of duplications or partitioning to simulate it. Since there were two failed versions, there are three incompatible versions of this system running in parallel and migration from one to the other must be done manually.
The problem is about 90% of the business rules were encoded into this system, because business people were in a hurry. People wanted a report but didn't want to wait for Business Intelligence? Let's add "tags" to records so they appear on certain screens, and then remove them when they shouldn't anymore.
In the end the solution was adding "experts" to use it, but the ones who actually knew or learned any programming would just end up escaping to other companies.
1. We have a python application
2. We need a configuration format, we pick one of the usual (ini/toml/yaml/...)
3. We want to allow more than usual to be done in this config, so let's build some more complex stuff based on special strings etc.
Now the thing they should have considered in step 3 is why not just use a python file for configuration? Sure this comes with pitfalls as you now allow people who write the config to do similar things than the application, but you are already using a programming language, why not just use it for your overly complex configuration? For in house stuff this could certainly be more viable than writing your own parser.
Declarative configs are preferable for keeping your options open as to who or what consumes them. For cases where config as code is truly necessary, the best option is to pick something that's built for exactly that, like Lua (or some other embedded scripting language+runtime with bindings for every language).
It isn't a magic solution.
Looking back it was the VBScript/JScript functionality that caused me the most problems. Especially when I migrated the whole app from C++ to .Net.
[1] https://en.wikipedia.org/wiki/Active_Scripting
- https://arcan-fe.com/2022/10/15/whipping-up-a-new-shell-lash...
- https://arcan-fe.com/2024/09/16/a-spreadsheet-and-a-debugger...
I am not using it as a daily driver, because, emacs, but I keep an eye on it because, well, emacs.
It's not perfect, but it's clean and moves in this lua direction.
Reminds me of the pain of intentionally building a compiler for Java 2 (subset) to MIPS compiler by writing out each AST node class by hand. And, I did it twice, once in C++03 with bison and flex and again in Java 2 with CUP and JFlex... each was developed to build and run as a host portably across Solaris (sparc), Linux (x86), HP-UX (68k), SGI (MIPS), and Windows (x86) with compiled with targets run on the SPIM emulator. It did have dead code, dead string, and dead variable elimination, but that was as far my optimization passes went. I recall the only build tool I used for each was the portable subset of make without GNU extensions.
Speaking of reinventing the wheel, in 1998, I built a flexible almost framework for a "portable" generic installer using Java 2, JWT (native GUI controls), and JNI on Windows to create a program group and desktop shortcut icon. The hilarious part was shipping a full JRE on a CD. It took forever to load but the additional time seemed impressive for expensive, niche software in a way similar to the now fake "loading..." delayed progress bar.
</old-guy-high-school-glory-days-and-nobody-today>
This is written like a Jeopardy answer. I just don't know what the question is.
Can anyone enlighten me?
--> gmail
The problem is that unless your use case is very limited and is guaranteed to stay that way, supporting more and more language constructs will quickly turn your code into a mess.
Compiler design as we learn it (lex/parse, syntax tree, semantic checks, transforms, lowering to codegen) is _the_ solution to the problem of dealing with computer programs as inputs and outputs. Trying to do something less is like solving a dynamic programming problem without knowing dynamic programming: it will only work for a restricted set of inputs.
It starts innocently, e.g. doing some template files and replacing some simple values, then you start to have to do more replacements and more "smart" parsing and then at some point it's too late, as the article suggests.
TBF, I did put together a transpiler from PHP to JS, but I didn't build it, just found the different pieces that luckily fit together and hacked around it enough that it could run in the browser.
Keeping them locked up in the "scary CS" closet only ends up stunting your growth.
Granted, if you've got a really simple language then maybe there's an upper bound on how bad it can be. But in practice people tend to also want to grow the language as they go.
Don't get me wrong; there's a lot than can wrong if you over-engineer the compiler from the beginning, including ending up with a hot mess anyway.
BTW if you can't write down, even in three sentences, the requirements for said compiler from the beginning, congratulations, you have no idea what you are doing.
I've written a dozen different programs that might be considered compilers; some very simple, others very complex and whose life continued once I left the organization. Writing a functional compiler that provides the needs of the organization where existing tooling doesn't takes discipline and focus on what you actually want to accomplish. I don't know what "defining a struct inside a loop" might mean and this strikes me as, very obviously, having no clue what you actually want to build.
Perhaps the issue is not building a compiler but rather the lack of focus to begin with.
Long story short, we thought we were making a new type of N-dimensional spreadsheet, but after 3 semesters of work one of the advisors at MIT told us we need to meet his colleague, and that guy informed us we had a working compiler for a hybrid of Lisp and C.