As an ML-focused python dev I have never been able to break the habit of REPL-driven development, but I find it works really well for "building code that works" rather than coming up with a tower of abstractions immediately. A typical python development workflow for me is:
* Start with a blank `main` file and proceed linearly down the page, executing as I go.
* Gradually pull out visually awkward chunks of code and put them into functions with no arguments at the top of the file.
* If I need to parameterize them, add those parameters as needed - don't guess at what I might want to change later.
* Embrace duplication - don't unnecessary add loops or abstractions.
* Once the file is ~500 LOC or becomes too dense, start to refactor a bit. Perhaps introduce some loops or some global variables.
* At all times, ensure the script is idempotent - just highlighting the entire page and spamming run should "do what I want" without causing trouble.
* Once the script is started to take shape, it can be time to bring some OO into it - perhaps there is an object or set of objects I want to pass around, I can make a class for that. Perhaps I can start to think about how to make the functionality more "generalized" and accessible to others via a package.
This is literally the only way I've ever found to be productive with green field development. If my first LOC has the word "class" or "def" in it - I am absolutely going to ripping my hair out 12 hours later, guaranteed.
> * Gradually pull out visually awkward chunks of code and put them into functions with no arguments at the top of the file.
I have seen scientific code written in this manner written in both Python and Fortran.
This may be some intuitive way to start off, and even complete the task at hand.
But for people trying to read, understand, and realistically, debug your code, this complicates things.
Because each no-argument function can only work via its side effects.
Your script becomes a succession of state transitions, and to understand it, you have to keep the intended state after each step in your head.
And in case of a mistake, you can't even call these functions individually via passing their intended arguments using the REPL. You have to set up all the state beforehand manually, call your no-arg function, observe the state afterwards. Which becomes more awkward the further down you are in your script, since all your dependencies are implicit and possibly even completely undocumented.
Sure that's like the worst case scenario, but in practice the entire point is that as soon as it becomes difficult to reason about, that's when you start cleaning up the functions and add some obvious OO. You don't leave it as a mess.
> Embrace duplication - don't unnecessary add loops or abstractions
I’ll usually make a function or perhaps tiny class as soon as I start reusing bits of code.
Apart from that, agree as stated. At my previous job (Python shop), a lot of the data engineers came from a Java background, and had a tendency to think top-down. Many things were over engineered ‘just because we might need it’:
- Factory classes used only once or twice in entire code base
- Lets make an AbstractReaderInterface because we might want to abstract the file type or location later (while 100% of files are Parquet on S3)
I’ve really enjoyed using dataclasses and Pydantics BaseModels prolifically, and adding type hints (coupled with type checks in CI).
Model the data, write a well structured imperative workflow, set up CI, write unit tests, enforce typing. Add OOP if needed, then close ticket.
Agreed. The general principle is (and it's a hard balance, of course, to avoid just slapdash work): don't do work up front you might not need. You will make your current work take longer than it should, which you will (correctly) be blamed for, and any time you save in the future you won't get the credit for.
This isn't a cynical statement; it's just my experience. Use it to your advantage!
People are getting caught up on the loops thing. All I meant was in my line of work I often end up with many special cases of general processes. Writing a loop prematurely always bites me - I end up writing control flow for handling the one-offs, it somehow always becomes more obtuse than just listing things out literally.
> Embrace duplication - don't unnecessary add loops or abstractions.
> Once the file is ~500 LOC or becomes too dense, start to refactor a bit. Perhaps introduce some loops or some global variables.
I agree with most, but if I "embrace duplication" I can reach 500+ LOC in half an afternoon :P. It seems to have really paid off for me to start some degree of abstraction (not OO yet in general) early enough. Tbf, it is easier for me to tidy up the code with some abstractions, rather than ensure that the "main"-titled script runs from beginning to end each point of time with no errors, which imo can hinder experimentation more than abstraction. But also depends on what I do, I guess, the greener the field the more I feel this way.
Wow, not a ML person, but controls and robotics. Yet, this describes my workflow for a lot of things almost to a tee. Even down to the avoiding loops. I tend to do that when I want to run the same simulation or analysis for a couple variables or datasets. It's interesting, because in the past I was a lot more prone to turn that into a loop early on. But this makes your code brittle. You'll want to do something slightly different for the two datasets, which means a bunch of conditionals in your loop. It's actually really similar to the problems you get with boolean flags when you try to abstract into a function too soon. It actually takes disciple to for me to commit to copy past but I think it pays it off.
I too agree with pretty much everything you say. Just want to add that I pretty much solely use ptpython[0]. It can handle line breaks in pasted code, vim (or emacs) bindings and syntax highlighting, and much more.
Pretty much follow an identical process. When I do finally rewrite the code, after getting a working version, the duplication pretty much screams clean me up and simplify/generalize. I have never been able to just see the whole thing before I start. The process itself teaches you things.
I usually use VS code and the "interactive" python functionality (not jupyter). I highlight code and execute just that code with a hotkey. Works just as well with any kind of vim-slime like functionality.
I also primarily write ML-focused Python. For me, having originally learned R and C at the same time, nothing has ever surpassed RStudio as a dev environment. For the past several years my preferred setup has been tmux and Vim with vim-slime in one pane and IPython in the other.
(Personally, and speaking only for myself, I hate Jupyter notebooks with a burning passion. I think they are one of the worst things ever to have happened to software development, and definitely the worst thing ever to have happened to ML/data science.)
Not OP but I often code in the REPL for python as well. Sometimes I'll stub out my code and just drop into an interactive debugger where I'm writing the next section.
In the python debugger, if you type `interact`, it'll give you the normal python repl. This combined with the `help` and `dirs` are super useful for learning new frameworks/libraries and coding with your actual data.
The temptation to throw away all of your code - be it a prototype or a "grown" code base - arises often.
Often it is a bad idea. I get it, though, there is an inherent attractiveness in the idea of starting fresh from a clean slate. Except, more likely than not, you'll soon find yourself in a similar situation to the one you started from. The fundamental problem is that it is easy to underestimate the edge cases. Sure, the main functionality is easily understood and straight-forward to conceptualize. But did you remember to think through all the smaller aspects of your software, too?
Perhaps it's easier with a prototype implementation that in fact doesn't have a lot of features yet, but to completely replicate the functionality of a complex piece of software isn't an easy undertaking. Sure, getting 80% there is probably easy, but the last 20% is the part that's easy to overlook when considering a complete rewrite.
Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Throwing away an established code base is a completely different thing from throwing away a few-days-old prototype, and not what the article is suggesting. The points you mention don't apply to the prototype case.
This is such a HN phenomenon it infuriates me. Read article. Then read comment that misconstrues article to be something completely different, accompanied with loud critique. Read replies all in violent agreement to obviously self-evident strawman.
> Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Agreed. Though sometimes you need to refactor the core of an application, in a way which will touch the entire app. To do that I often make a new, empty project. Then I rewrite the core of my project into the new folder (in whatever new structure I’m trying out). When that works, I slowly copy the content from the old project into the new project - refactoring and testing along the way.
But it’s not perfect. About half the time I do this, I discover halfway through that I didn’t understand some aspect of the system. My new design is wrong or useless and I throw away all the new code. Or I figure out that I can just make the changes in place in the old project folder after all, and I bail on the new code and go back to a traditional refactor.
But no matter what happens, I don’t think I’ve ever regretted a refactoring attempt like this. I always come away feeling like I’ve learned something. How much you’ve learned from a project is measured by the number of lines of code you threw away in the process.
> The temptation to throw away all of your code - be it a prototype or a "grown" code base - arises often.
I want an editor plugin which allows me to mark sections of code as reviewed or "perfect" (depending on how honest I'm being). Then, when I'm tempted to rewrite everything, I can go through and mark what I think is good, and then focus on refactoring the rest until I think it is good as well.
I'm tempted to rewrite code because I lose track of what it's doing, or I've learned a lot since I wrote that old code and so I'm not sure if the old code is good anymore. It's not so much about rewriting the code as an exercise in getting familiar with the code I've already written. I want a tool to help me with this.
I would think a combination of git, unit tests, and comments would solve this problem?
Unit tests prove the code works as intended, and are basically examples of what the code is doing. Whether the code is actually "good" is a bit more subjective -- but tests give you the freedom to modify it without breaking it.
Checking into git frequently is also a way to give yourself some freedom. Commit at every milestone, like every time the next thing is "working". If you feel like refactoring, go for it -- you can reset back to working state in a few seconds.
And lastly, leave comments in. You can always clean it up before you push. You can even squash or interactively rebase your history so no one else sees the gory details how the hot dog was actually made.
If you could modularize code in such a way the sizable chunks could “just sit still!” then entire modules could be marked as perfect. But you wouldn’t have to since you wouldn’t naturally need to touch them (e.g. how many codebases are using a fork of core-utils, for example, … to exaggerate a conways law effect).
But unfortunately the modules don’t make themselves apparent at the start of the project. So it needs refactoring discipline.
I don't really feel that attraction to complete rewriting. I wonder if the people who do are very smart, and thus able to hold more state in their head, so ugly code bothers them more even if it's not actively a problem, because they are able to have background tasks in their mind to worry about it?
And at the same time, perhaps their code is less encapsulated, because they didn't optimize for abstraction, they optimized for beauty. A leaky abstraction doesn't bother them, because ALL abstractions are leaky to them, they probably have a sense of internal workings even whem using household appliances, but ugly code tucked away somewhere bothers them a lot, and they might dislike using popular large libraries even if they work great, just because they're not comfortable using what they don't understand deeply.
My evidence of this is the fact that suckless exists and people actually use it, I assume their experience of thought is very different from anything I have experienced.
Speaking personally my urge to rewrite at least partially comes from not truly understanding the problem and the solutions to it until I've written something reasonably functional. It doesn't matter how much time I put into sitting and theorizing, there's always things I didn't anticipate and assumptions that turned out to be incorrect.
This usually means that rewrites are significant improvements across the board, especially if they're done a relatively short time after the original is finished since it's all still fresh in my head.
This may be a weakness of sorts on my part though, I lack formal engineering training which might be why purely mental modeling (no code) doesn't work all that well for me.
ALL abstractions are leaky - this is an objectively true statement.
As others said it's not for beauty, it to make sure if there is abstraction it fits the problem. If there is encapsulation it doesn't get in the way. For some coders they can get it on the first try and there is no reason for them to rewrite code. For the rest of us mid coders we need to explore first as well as make sure all cases we desire for Lib/API work.
There is some point regarding changes on the code where rewriting is less costly or more costly than changing the current code. For me, a big part depends on whether I depend long term or short term on it and how deep I will have to go anyway using it.
Ok, perhaps don't throw it away. When I come up with an idea (for an app usually), I use "a few" A4-sheets (anything between 5 and 20), scribble and draw on them with a pencil, draw screens, buttons, data flows, activities, write notes (in various font sizes). Then I use my CamScanner and call this a v1.
Then email me the PDF and store the papers in a box, and a couple of days later I start the v2 (same process), then v3.
By v4 it's 'good'.
I also use the same method on my 9-5.
I take VERY seriously the Abraham Lincoln quote “Give me six hours to chop down a tree and I will spend the first four sharpening the axe” on almost everything.
I consider the v1, v2, v3 as the "sharpening the axe" and the v4 on the actual cutting.
In that spirit, the article has a similar approach, as the v1, v2, and v3 may take you down an (more than) imperfect path.
> Admittedly, there's nothing sexy about refactoring.
There's the "strangler fig" (my favorite) method of refactoring. It's where you rewrite just a small portion of the software, little bits at a time. Instead of doing a full rewrite.
IMHO, it's the best way to refactor. You can switch out both "old" and "new" versions at-will until you're 100% sure you've covered all the edge cases.
Game companies implicitly do this, throwing out old versions of their code (sort of). You make a game. Ship it. Start on a new game. Your previous game is now sort of a practice run for making a new game from scratch. Continue, ad infinitum.
It's weird: you rarely have to maintain something for 20+ years, and you get to always improve and iterate on how you did things last time. But, are you training yourself to write hard to maintain code, since you don't really have to maintain it past a certain period? Or does the learning-from-iteration actually make writing-maintainable-code easier?
I know some people do keep developing their games for decades, look at Dwarf Fortress, I'm just talking in general.
what you do you get good at. if you don't have to maintain there is no point in getting good at it or any reason to think you will gradually get better at it.
Granted Joel is taking about a wholesale rewrite of proven / established code. Code and has had had the advantage of time.
Personally, I end up rewriting parts of my code often. It usually takes a mile for me to find exactly the right way it should be based on how are people use it.
You absolutely should rewrite prototypes, and re-factor important chunks of code. I rewrote something 3 times today, it was better each time.
Of the reasons he says engineers want to rewrite the code, he says: badly organized code, slow processing time, and ugly code.
Maybe I’m not an engineer, because I bet perfectly pretty fast code is an indicator you should reorganize it to make it easier to adapt, add more capabilities even if it runs slow, and put some ugly fixes in there so it’s useful for the end user.
Agreed that wanting to over-engineer and perfect something that should be constantly evolving is not a good reason to rewrite it. That sounds like back office bubble pretend-work.
Rather than "throw away" I would "start from scratch" meaning I can refer to prototype but reimplement with respect to appropriate norms.
As a version control fanatic, very little gets really thrown away in my book. Just hidden.
Yeah normally when I do this, it’s because I picked bad abstractions. The main business logic I can bring over almost line by line, but I want to reshape the abstractions / data structures / interface to that logic
Nah. I think a prototype is something you definitionally should throw away. There's a huge freedom to knowing that instead of taking on tech debt, you're committed to declaring tech bankruptcy and throwing away the code. It lets you try things, cut corners, and generally experiment. It's great for thinking through what everybody really wants.
I do agree that it's generally a bad idea to just throw out a long-lived code base. But for me that's not an estimation issue. It's because the urge to throw it out is usually a response to upstream problems that haven't gotten fixed. For example, a lot of code bases are a mess due to time pressure. But if you don't fix the process problems that turn time pressure into mess, then you're just doing to end up with another mess. Possibly a bigger one, in that stopping all productive output in favor of a rewrite usually makes business stakeholders crazy, causing increased time pressure.
A possible alternative is to use a different programming language for doing the prototyping than the one which will be used in production. The urge to hang on to prototype code can be removed, because you are going to do a rewrite in a different language, regardless.
By using 2 different languages, there is more freedom to just make a demo, discuss, and then decide what to keep or what direction to go.
I've never respected this advice, even though I hear it often enough. The reasoning appears to come down to this:
> If you know that you're possibly keeping the code, you do things in a "proper" way, which means moving slower
combined with the idea that it's faster to redo work than it is to refactor.
I don't buy either of these propositions as a universal rule. For the first, it seems that a mindset of avoiding premature optimization, which you should be cultivating anyway, is enough to ward it off.
For the second, I see it as very context dependent, with my bias going towards refactoring instead of rewriting. I most often get things right enough the first time that it really only needs to be refactored going forward. Only when I end up with serious flaws do I decide to start over, often before I finish my implementation because the flaws are already apparent enough. Add to this that even very large changes can be refactored effectively and timely in my experience, and I feel like the bar for rewriting is really high to be cost-effective.
Unfortunately this essay and many others seem to take objections like mine for granted, and I should just believe the author that rewriting is faster with zero supporting evidence.
> I most often get things right enough the first time that it really only needs to be refactored going forward.
I think it really depends on your values and what you’re working on.
If you have the mindset of a product engineer, the thing you care the most about is how well the software works for your users. The code is an unfortunate necessity. When I build react apps, I tend to think like this. The code I write first go is often good enough to last until the next redesign of the UI.
By contrast, if you’re thinking of the code as a home you build for your work that you will live in, then having a tidy home becomes its own reward. Good decisions today (in terms of cleaning up or refactoring code) should lead to more velocity later. When I think about building a database, an OS kernel or a compiler, I think like this. When I start, I don’t know where my code should be rigid and where it should be flexible. My first guesses are usually very wrong.
Personally I prefer the latter kind of problem. I like it when there’s no obvious way to structure my code and I have to explore while I work. The code I’m most proud of writing has probably been rewritten 5 or more times before I landed on the “final” design. The “right” abstractions are rarely obvious at first glance and the search can be incredibly rewarding.
The rich history of application programming abstractions suggests it isn’t exempt from this either. It’s just, when you’re building a UI there have been an awful lot of explorers who came before you and can show you the way.
I agree that it depends on factors like what you describe which is why I am asking those giving the advice like this to consider the context at all. I also happen believe that the contextual bar should be placed high. I have to say, what comes to mind when I read your comment was "What about this does refactoring not accomplish?" In the metaphor of building a house, refactoring is what keeps it tidy and organized, and rewriting would only appear necessary in the case of your trash chute leading into a bathtub or some other critical design flaw that could only be addressed timely by redesigning the entire house.
If you are in a context where you are in danger of making critical design flaws often, or where circumstances make it hard to make changes after committing, then I absolutely agree that rewriting is effective. That's just what I mean by a high bar, one that I personally don't encounter very often in my work.
I definitely disagree with you on the first point.
Setting aside the part that's just fussbudgetry, I think "proper" coding is about doing things that pay off in the long term even if they feel burdensome now. As an example, I'm a huge fan of good automated tests for long-lived code bases. But if I'm just writing a quick, throwaway script, then building a bunch of automated tests are going to slow me down.
Throwaway prototypes are another area where you can save a lot of effort if you have really committed to throwing the code away once you've learned what you set out to learn. As a physical analogy, before the Long Now built out their bar/cafe space in San Francisco [1], we spent a day in the raw space building furniture and walls out of cardboard and tape [2]. It let us get a real sense of how the designs would work in practice for actual use of the space.
In the past for a number of projects I've done throwaway prototypes and then started fresh with "proper" engineering, which for me usually includes pairing, test-driven development, and continuous deployment. I've also done ones where we think we understand it enough and just start building. And either way, we usually end up evolving ways to prototype on top of the existing platform via feature flags, etc, so we can learn something from the real world and then decide whether or not we want to refactor to accommodate a major change.
Using boxes is great! The beauty of coding is that those boxes don't need to be discarded, we have the virtual power to transmute them into real chairs and tables via the power of refactoring. It's like Photoshop vs. watercolor. I'm going to sketch either way, it's just a matter of what the bar is for throwing it all out, and I think it's very often sufficient to decide in the moment in a virtual environment. That's all to say, going back to my first point, I don't need to decide ahead of time whether the code will be kept, I'm going to keep it loose either way.
And there are situations I'd go in assuming I'll probably rewrite the work, but I wouldn't set the criteria so broadly as to cover literally whatever the next large project I embark on will be, as the article suggests.
looking for gaping security holes is premature optimization. if you run into security problems down the line, use a security profiler and find the hot spots i.e. radioactive code
> looking for gaping security holes is premature optimization.
I won't approve a PR if I notice such a thing (and if it really is "gaping" I'd expect an automated tool to flag it and fail the build anyway).
I don't really see ensuring your code follows basic secure SDLC best practices as "optimisation" at all - in fact it often comes at a (hopefully small) cost of less than optimal performance.
Of course there may be areas of software development (hobby projects etc.) where security is not a priority at all, but if it's code you're being paid to write that runs on the cloud or your customer's machines then security is absolutely worth getting right from the get-go.
Usually I think it is unnecessary, but you might want to include the “/s” marker in this case. It is just close enough to believable that someone would think this, given the amount of insecure trash products that have been released…
I think this depends greatly on the product area and ease of updating. Removing problems categorically from designs early has been valuable to me in the past to reduce surface area, avoid impossible refactoring, and eliminate systemic bug classes.
I'm largely with you on this... do the simplest thing that gets a job done, that you assume will be replaced, and don't be surprised when it's still un use a decade or more later.
I like to build my first version in one file with as few abstractions as possible. No helper functions, unless absolutely necessary, just straight-forward code. Building good abstractions later on is a lot easier than starting with wrong ones in the first place.
This is an underrated way of doing things that I've come to appreciate over time. Heck, I'll even go live with the all-in-one-file thing until there's a reason to split it up. It's all peanuts anyway compared to bigger decisions like what API, DBMS, DB schema, other deps, libs, etc you use.
Very interesting approach. As someone who has overengineered last major project I worked on......maybe this is the way. I got so fixated on getting it "right" using current standards and abstractions that it became a bit of a mess. I despise working with that code base now.
I've always done it like this as well. Initially I did it for the most obvious reason: it gave me the working software that I needed the fastest. Then later on, I always found it to be the best way to find natural abstractions as you go.
Yet I find when rewriting from scratch there's a countering force of simplification where I find myself thinking "We spent way too much complexity on that part, and we can entirely do away with that other part, and we turned out never to use features X and Y."
> "...and we turned out never to use features X and Y."
THIS. Often, the best way to start the rewrite is to go straight to the database, and start asking "what allowed values for all these fields have never actually been used?".
EDIT: Whether it's rewriting, or refactoring, or adding some "if( Deprecated_Feature_Allow !== true ) ... ErrorDie( Feature_Deprecated )" logic here and there, or updating training material, or whatever - knowing that Software_Feature got ~zero real-world use is d*mned useful.
I rewrote a mission critical application. The original app was something else. One function had 1500+ lines of code. It was the ugliest thing I have seen in my life. We wrote cleaner code in early college years. However, that code ran the business for 12 years with almost 0 modifications. People created work around for some stuff. Error handling was poor. But the goal was achieved and business continued to grow.
I think devs in general sweat too much about code quality.
Some disputes about quality are simply differences of opinion or style.
There's not really a book or standard on opinionated code quality that satisfies a majority of the people in the field. There's way too many ways to build things for that to exist.
This leaves it up to individual teams to approach a shared consensus (with some wiggle room) within their domain and tool set.
One thing that is universal is how strict the enforcement of a team's quality standards. So as a developer, you have to decide whether you like strict or lax enforcement of quality standards. Everyone can decide that but I think most don't.
> However, that code ran the business for 12 years with almost 0 modifications
We have code like that, and it requires everything around it to twist and contort to comply with the beast of unmaintainable at the center of it all, simply because at this point it has so poor test coverage and is so incredibly mission critical now.
Code quality isnt that important if you check this code "once a year" and dont have time pressure to change somewhere in there. Or if you are seeing this code daily and know all the quirks.
Decades ago I spent a whole day adding a serious new feature to my software. Had the first version all polished up and ready, and through a misunderstanding with my version control software (probably RCS, maybe CVS, probably not SCCS), I lost that work. Like maybe I did a checkout instead of checkin, something like that. I went to bed quite annoyed.
The next day I got up and started from scratch. The new implementation took me half a day, and was decidedly a better implementation.
This would sound like questionable advice except that the best code I wrote in college resulted from an accident…
I’d sat down all night and written some C for a graphics class. Everything was great, yay, victory. Proceeded to tar it up per class guidelines and did something like this:
tar cvf assignment.c assignment.h data.txt readme whatever.cuz
And as many of you no doubt recognize, I’d forgotten to supply a destination name before listing the source files.
So the code was gone. My brain was gone, too, so I explained what happened and sent what I had, including a working executable, to the TA. He gave me an extra day to finish.
So the next day I sat down and wrote it all again and as mentioned above, it was some of the best code I’d written. Doing it again from the top saved all sorts of dead ends and mistaken approaches.
This is exactly the type of story I tell when people are anti pair programming. I ask them if they’ve ever lost a days code (usually asking if they ever used source safe) and that when they finally knuckled down to write it again, it took about 20 minutes. As the writing the code wasn’t really the hard part or the bottleneck, solving the problem was the hard part. Hence pair programming having two heads on one problem to solve it better/faster, find the edge cases etc.
Out of curiosity, was it really fast for you to do it in that extra day or did it take all day?
Readit News
* Start with a blank `main` file and proceed linearly down the page, executing as I go.
* Gradually pull out visually awkward chunks of code and put them into functions with no arguments at the top of the file.
* If I need to parameterize them, add those parameters as needed - don't guess at what I might want to change later.
* Embrace duplication - don't unnecessary add loops or abstractions.
* Once the file is ~500 LOC or becomes too dense, start to refactor a bit. Perhaps introduce some loops or some global variables.
* At all times, ensure the script is idempotent - just highlighting the entire page and spamming run should "do what I want" without causing trouble.
* Once the script is started to take shape, it can be time to bring some OO into it - perhaps there is an object or set of objects I want to pass around, I can make a class for that. Perhaps I can start to think about how to make the functionality more "generalized" and accessible to others via a package.
This is literally the only way I've ever found to be productive with green field development. If my first LOC has the word "class" or "def" in it - I am absolutely going to ripping my hair out 12 hours later, guaranteed.
I have seen scientific code written in this manner written in both Python and Fortran. This may be some intuitive way to start off, and even complete the task at hand.
But for people trying to read, understand, and realistically, debug your code, this complicates things.
Because each no-argument function can only work via its side effects. Your script becomes a succession of state transitions, and to understand it, you have to keep the intended state after each step in your head.
And in case of a mistake, you can't even call these functions individually via passing their intended arguments using the REPL. You have to set up all the state beforehand manually, call your no-arg function, observe the state afterwards. Which becomes more awkward the further down you are in your script, since all your dependencies are implicit and possibly even completely undocumented.
1. Use comments to split visual awkward part into chunks.
# a lot of code...
## ===
# more a lot of code...
2. Use inner functions if that chunk need to be reused
3. Only move the chunk to top-level function if you think it's worth to take time to make its required state into parameters/return value
> Embrace duplication - don't unnecessary add loops or abstractions
I’ll usually make a function or perhaps tiny class as soon as I start reusing bits of code.
Apart from that, agree as stated. At my previous job (Python shop), a lot of the data engineers came from a Java background, and had a tendency to think top-down. Many things were over engineered ‘just because we might need it’:
- Factory classes used only once or twice in entire code base
- Lets make an AbstractReaderInterface because we might want to abstract the file type or location later (while 100% of files are Parquet on S3)
I’ve really enjoyed using dataclasses and Pydantics BaseModels prolifically, and adding type hints (coupled with type checks in CI).
Model the data, write a well structured imperative workflow, set up CI, write unit tests, enforce typing. Add OOP if needed, then close ticket.
This isn't a cynical statement; it's just my experience. Use it to your advantage!
It would be unnecessary to have a loop over keys of a dictionary to call function xyz when you can just repeat the xyz calls (it would look nicer too)
Unless the dictionary is huge and dynamically loaded.
> Once the file is ~500 LOC or becomes too dense, start to refactor a bit. Perhaps introduce some loops or some global variables.
I agree with most, but if I "embrace duplication" I can reach 500+ LOC in half an afternoon :P. It seems to have really paid off for me to start some degree of abstraction (not OO yet in general) early enough. Tbf, it is easier for me to tidy up the code with some abstractions, rather than ensure that the "main"-titled script runs from beginning to end each point of time with no errors, which imo can hinder experimentation more than abstraction. But also depends on what I do, I guess, the greener the field the more I feel this way.
[0] https://github.com/prompt-toolkit/ptpython
Just noting functions, d-structure and some flow usually helps to arrive at something worthwhile...
(Personally, and speaking only for myself, I hate Jupyter notebooks with a burning passion. I think they are one of the worst things ever to have happened to software development, and definitely the worst thing ever to have happened to ML/data science.)
In the python debugger, if you type `interact`, it'll give you the normal python repl. This combined with the `help` and `dirs` are super useful for learning new frameworks/libraries and coding with your actual data.
Dead Comment
Often it is a bad idea. I get it, though, there is an inherent attractiveness in the idea of starting fresh from a clean slate. Except, more likely than not, you'll soon find yourself in a similar situation to the one you started from. The fundamental problem is that it is easy to underestimate the edge cases. Sure, the main functionality is easily understood and straight-forward to conceptualize. But did you remember to think through all the smaller aspects of your software, too?
Perhaps it's easier with a prototype implementation that in fact doesn't have a lot of features yet, but to completely replicate the functionality of a complex piece of software isn't an easy undertaking. Sure, getting 80% there is probably easy, but the last 20% is the part that's easy to overlook when considering a complete rewrite.
Admittedly, there's nothing sexy about refactoring. And often it may seem like it's less work to just simply scrap everything and start over. However, that's fallacy a lot of times.
Agreed. Though sometimes you need to refactor the core of an application, in a way which will touch the entire app. To do that I often make a new, empty project. Then I rewrite the core of my project into the new folder (in whatever new structure I’m trying out). When that works, I slowly copy the content from the old project into the new project - refactoring and testing along the way.
But it’s not perfect. About half the time I do this, I discover halfway through that I didn’t understand some aspect of the system. My new design is wrong or useless and I throw away all the new code. Or I figure out that I can just make the changes in place in the old project folder after all, and I bail on the new code and go back to a traditional refactor.
But no matter what happens, I don’t think I’ve ever regretted a refactoring attempt like this. I always come away feeling like I’ve learned something. How much you’ve learned from a project is measured by the number of lines of code you threw away in the process.
I want an editor plugin which allows me to mark sections of code as reviewed or "perfect" (depending on how honest I'm being). Then, when I'm tempted to rewrite everything, I can go through and mark what I think is good, and then focus on refactoring the rest until I think it is good as well.
I'm tempted to rewrite code because I lose track of what it's doing, or I've learned a lot since I wrote that old code and so I'm not sure if the old code is good anymore. It's not so much about rewriting the code as an exercise in getting familiar with the code I've already written. I want a tool to help me with this.
Unit tests prove the code works as intended, and are basically examples of what the code is doing. Whether the code is actually "good" is a bit more subjective -- but tests give you the freedom to modify it without breaking it.
Checking into git frequently is also a way to give yourself some freedom. Commit at every milestone, like every time the next thing is "working". If you feel like refactoring, go for it -- you can reset back to working state in a few seconds.
And lastly, leave comments in. You can always clean it up before you push. You can even squash or interactively rebase your history so no one else sees the gory details how the hot dog was actually made.
But unfortunately the modules don’t make themselves apparent at the start of the project. So it needs refactoring discipline.
And at the same time, perhaps their code is less encapsulated, because they didn't optimize for abstraction, they optimized for beauty. A leaky abstraction doesn't bother them, because ALL abstractions are leaky to them, they probably have a sense of internal workings even whem using household appliances, but ugly code tucked away somewhere bothers them a lot, and they might dislike using popular large libraries even if they work great, just because they're not comfortable using what they don't understand deeply.
My evidence of this is the fact that suckless exists and people actually use it, I assume their experience of thought is very different from anything I have experienced.
This usually means that rewrites are significant improvements across the board, especially if they're done a relatively short time after the original is finished since it's all still fresh in my head.
This may be a weakness of sorts on my part though, I lack formal engineering training which might be why purely mental modeling (no code) doesn't work all that well for me.
As others said it's not for beauty, it to make sure if there is abstraction it fits the problem. If there is encapsulation it doesn't get in the way. For some coders they can get it on the first try and there is no reason for them to rewrite code. For the rest of us mid coders we need to explore first as well as make sure all cases we desire for Lib/API work.
Maybe this is ok, given that time period?
or will the rewrite have all these extra bells and whistles?
or will the rewrite throw away the unneeded bells and whistles?
Then email me the PDF and store the papers in a box, and a couple of days later I start the v2 (same process), then v3.
By v4 it's 'good'.
I also use the same method on my 9-5.
I take VERY seriously the Abraham Lincoln quote “Give me six hours to chop down a tree and I will spend the first four sharpening the axe” on almost everything.
I consider the v1, v2, v3 as the "sharpening the axe" and the v4 on the actual cutting.
In that spirit, the article has a similar approach, as the v1, v2, and v3 may take you down an (more than) imperfect path.
There's the "strangler fig" (my favorite) method of refactoring. It's where you rewrite just a small portion of the software, little bits at a time. Instead of doing a full rewrite.
IMHO, it's the best way to refactor. You can switch out both "old" and "new" versions at-will until you're 100% sure you've covered all the edge cases.
It's weird: you rarely have to maintain something for 20+ years, and you get to always improve and iterate on how you did things last time. But, are you training yourself to write hard to maintain code, since you don't really have to maintain it past a certain period? Or does the learning-from-iteration actually make writing-maintainable-code easier?
I know some people do keep developing their games for decades, look at Dwarf Fortress, I'm just talking in general.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
Granted Joel is taking about a wholesale rewrite of proven / established code. Code and has had had the advantage of time.
Personally, I end up rewriting parts of my code often. It usually takes a mile for me to find exactly the right way it should be based on how are people use it.
You absolutely should rewrite prototypes, and re-factor important chunks of code. I rewrote something 3 times today, it was better each time.
At the same time you should be wary…
Maybe I’m not an engineer, because I bet perfectly pretty fast code is an indicator you should reorganize it to make it easier to adapt, add more capabilities even if it runs slow, and put some ugly fixes in there so it’s useful for the end user.
Agreed that wanting to over-engineer and perfect something that should be constantly evolving is not a good reason to rewrite it. That sounds like back office bubble pretend-work.
https://devrant.com/rants/816880/i-ve-done-it-again
I do agree that it's generally a bad idea to just throw out a long-lived code base. But for me that's not an estimation issue. It's because the urge to throw it out is usually a response to upstream problems that haven't gotten fixed. For example, a lot of code bases are a mess due to time pressure. But if you don't fix the process problems that turn time pressure into mess, then you're just doing to end up with another mess. Possibly a bigger one, in that stopping all productive output in favor of a rewrite usually makes business stakeholders crazy, causing increased time pressure.
By using 2 different languages, there is more freedom to just make a demo, discuss, and then decide what to keep or what direction to go.
Deleted Comment
> If you know that you're possibly keeping the code, you do things in a "proper" way, which means moving slower
combined with the idea that it's faster to redo work than it is to refactor.
I don't buy either of these propositions as a universal rule. For the first, it seems that a mindset of avoiding premature optimization, which you should be cultivating anyway, is enough to ward it off.
For the second, I see it as very context dependent, with my bias going towards refactoring instead of rewriting. I most often get things right enough the first time that it really only needs to be refactored going forward. Only when I end up with serious flaws do I decide to start over, often before I finish my implementation because the flaws are already apparent enough. Add to this that even very large changes can be refactored effectively and timely in my experience, and I feel like the bar for rewriting is really high to be cost-effective.
Unfortunately this essay and many others seem to take objections like mine for granted, and I should just believe the author that rewriting is faster with zero supporting evidence.
I think it really depends on your values and what you’re working on.
If you have the mindset of a product engineer, the thing you care the most about is how well the software works for your users. The code is an unfortunate necessity. When I build react apps, I tend to think like this. The code I write first go is often good enough to last until the next redesign of the UI.
By contrast, if you’re thinking of the code as a home you build for your work that you will live in, then having a tidy home becomes its own reward. Good decisions today (in terms of cleaning up or refactoring code) should lead to more velocity later. When I think about building a database, an OS kernel or a compiler, I think like this. When I start, I don’t know where my code should be rigid and where it should be flexible. My first guesses are usually very wrong.
Personally I prefer the latter kind of problem. I like it when there’s no obvious way to structure my code and I have to explore while I work. The code I’m most proud of writing has probably been rewritten 5 or more times before I landed on the “final” design. The “right” abstractions are rarely obvious at first glance and the search can be incredibly rewarding.
The rich history of application programming abstractions suggests it isn’t exempt from this either. It’s just, when you’re building a UI there have been an awful lot of explorers who came before you and can show you the way.
If you are in a context where you are in danger of making critical design flaws often, or where circumstances make it hard to make changes after committing, then I absolutely agree that rewriting is effective. That's just what I mean by a high bar, one that I personally don't encounter very often in my work.
Setting aside the part that's just fussbudgetry, I think "proper" coding is about doing things that pay off in the long term even if they feel burdensome now. As an example, I'm a huge fan of good automated tests for long-lived code bases. But if I'm just writing a quick, throwaway script, then building a bunch of automated tests are going to slow me down.
Throwaway prototypes are another area where you can save a lot of effort if you have really committed to throwing the code away once you've learned what you set out to learn. As a physical analogy, before the Long Now built out their bar/cafe space in San Francisco [1], we spent a day in the raw space building furniture and walls out of cardboard and tape [2]. It let us get a real sense of how the designs would work in practice for actual use of the space.
In the past for a number of projects I've done throwaway prototypes and then started fresh with "proper" engineering, which for me usually includes pairing, test-driven development, and continuous deployment. I've also done ones where we think we understand it enough and just start building. And either way, we usually end up evolving ways to prototype on top of the existing platform via feature flags, etc, so we can learn something from the real world and then decide whether or not we want to refactor to accommodate a major change.
[1] https://theinterval.org/
[2] https://vimeo.com/73959127
And there are situations I'd go in assuming I'll probably rewrite the work, but I wouldn't set the criteria so broadly as to cover literally whatever the next large project I embark on will be, as the article suggests.
Deleted Comment
I won't approve a PR if I notice such a thing (and if it really is "gaping" I'd expect an automated tool to flag it and fail the build anyway). I don't really see ensuring your code follows basic secure SDLC best practices as "optimisation" at all - in fact it often comes at a (hopefully small) cost of less than optimal performance. Of course there may be areas of software development (hobby projects etc.) where security is not a priority at all, but if it's code you're being paid to write that runs on the cloud or your customer's machines then security is absolutely worth getting right from the get-go.
The problem with rewriting is that sometimes you face second-system syndrome[1] where you overengineer everything.
Believe me--it happened to me at Groove Networks after Lotus Notes (ask your grandparents about that).
[1] https://en.wikipedia.org/wiki/Second-system_effect
THIS. Often, the best way to start the rewrite is to go straight to the database, and start asking "what allowed values for all these fields have never actually been used?".
EDIT: Whether it's rewriting, or refactoring, or adding some "if( Deprecated_Feature_Allow !== true ) ... ErrorDie( Feature_Deprecated )" logic here and there, or updating training material, or whatever - knowing that Software_Feature got ~zero real-world use is d*mned useful.
I think it just boils down to "keep writing systems and you'll get better at it." But there's no bait on that kind of link.
I think devs in general sweat too much about code quality.
There's not really a book or standard on opinionated code quality that satisfies a majority of the people in the field. There's way too many ways to build things for that to exist.
This leaves it up to individual teams to approach a shared consensus (with some wiggle room) within their domain and tool set.
One thing that is universal is how strict the enforcement of a team's quality standards. So as a developer, you have to decide whether you like strict or lax enforcement of quality standards. Everyone can decide that but I think most don't.
We have code like that, and it requires everything around it to twist and contort to comply with the beast of unmaintainable at the center of it all, simply because at this point it has so poor test coverage and is so incredibly mission critical now.
I think if they sweated a little more then poor quality code wouldn't find itself into the codebase.
The next day I got up and started from scratch. The new implementation took me half a day, and was decidedly a better implementation.
Fred Brooks turned out to be right.
I’d sat down all night and written some C for a graphics class. Everything was great, yay, victory. Proceeded to tar it up per class guidelines and did something like this:
tar cvf assignment.c assignment.h data.txt readme whatever.cuz
And as many of you no doubt recognize, I’d forgotten to supply a destination name before listing the source files.
So the code was gone. My brain was gone, too, so I explained what happened and sent what I had, including a working executable, to the TA. He gave me an extra day to finish.
So the next day I sat down and wrote it all again and as mentioned above, it was some of the best code I’d written. Doing it again from the top saved all sorts of dead ends and mistaken approaches.
Out of curiosity, was it really fast for you to do it in that extra day or did it take all day?