I found that reading other people's code is very beneficial for my own coding. But I haven't found a resource that lists some great code in Python which is not a giant codebase. Any suggestions?
This is good advice. I learned Python long before ruff came on the scene, but I did the same with Pylint. I don't adhere rigidly to its recommendations any more, but I learned a lot about the language from trying. I fact, I think some of its recommendations are downright wrong, and what I learned was that I made my code harder to maintain by following them.
The value of this will vary wildly based on experience level. I wouldn’t suggest this outside of fairly senior developers.
Without a very good grasp of the language, it can be counterproductive to deep learning. The linters teach what to do, but not why, so it’s hard to grasp how the codebase is worse without following the linter, or when it’s appropriate to disable a linter vs dealing with it.
Ie I’m not a great JS dev. When the linter says “this should be an arrow function” I understand how to do that, but not why an arrow function is preferable there or in general. I would probably be a better JS dev if I hadn’t had the linter, felt whatever pain from not using arrow functions, and know why they’re important. Or never feel the pain and realize it’s a style choice more than a functional one.
I prefer using them to help me remember things I already understand. I know why mutating a copy of a string doesn’t mutate the caller’s copy, so I’m happy to have a linter point out when I’m trying to do that inadvertently.
Ruff specifically actually has a web page for every single check with a section on the rationale behind it.
I regularly use this when I see an error that I don't understand the purpose of, and am often convinced, though sometimes I recognize that the thing it's trying to protect me from doesn't apply in my particular case and so I disable it. Regardless. I feel it has had the effect of making me a better developer.
I agree that sometimes linters can enforce code styles that are more of hassle to deal with than offer any real concrete gain to new developers. But I disagree that only senior developers should use linters. Especially if you are learning a new language, it can introduce you to common conventions in that language, writing cleaner and more idiomatic code, and helps form good habits off the jump instead of building bad habits you will eventually have to change in a professional setting. Sure it can be overzealous at times, but I think on the whole it is a net positive.
To me, keeping the linter and the type checker happy is almost as important as actually writing good code.
If you don't know how to use the tools, you're missing out on time savings and error prevention, and once you write code that the linter doesn't like, you can't make it compatible without significant manual work.
I almost think Python would be better with mandatory types....
So, his code might not be a good place to find best patterns (for ex, I don't think they are fully typed), but his repos are very pragmatic, and his development process is super insightful (well documented PRs for personal repos!). Best part, he blogs about every non-trivial update, so you get all the context!
simonw might be one of the best and most down to earth Pythonistas of our time. He was one of the co-creators of Django, and that was almost 2 decades ago by this point - getting better all the whole. I second this recommendation heartily.
I think I've learned more reading bad code bases than reading good code bases.
The entire point is not to just mindlessly consume a code base, but instead form an idea of how to approach the problem and then see if your hypothesis is correct. Then comparing your approach to the actual approach.
This can show you things that you might've missed taking into account.
For example, gallery-dl's incidental complexity all lies in centralizing persistent state, logging, and IO through the CLI. It doesn't have sufficient abstraction to allow it to be rewired to different UIs without relying on internal APIs that have no guarantee that won't change.
Meanwhile a similar application in yt-dlp has that abstraction and works better, but has similar complexity in the configuration side of things.
It's a pain, but you can definitely learn a lot from fixing a bad codebase. For that, I recommend trying to write type annotations and get the whole thing to type-check. I've found that bad codebases end up having very complex type annotations because their authors actually contradict themselves. One of my personal favorites in Python is mixing strings and UUIDs as dictionary keys. This positively guarantees a fun afternoon.
Another learning point is being sensitive to your psychology / mental energy. I can start with high quality well named, well abstracted code.. but after two weeks I find myself writing shitty code.. and having a hard time realising I should stop, take a pause, take a step back instead of piling on.
I can't think of any (small) libraries I could recommend to learn best practices, but what does come to mind is click [0], the CLI library for Python. Their documentation is pretty great, there are tons of short example scripts to be found online, and in my experience making little apps with click can be a nice way to learn different python features like args/kwargs, decorators, string manipulation, etc.
I agree with others that code formatters like black or ruff might be helpful to you. The literature surrounding them, such as PEPs concerning code formatting, often include examples you may find useful.
Beware, the pallets people are decorator supremacists. Everything is a decorator even when it arguably shouldn't be. DDD --> decorator driven development. It's a nice technique, too a point. Only exaggerating a bit. ;-)
If you're looking for some best practices related but limited to machine learning application code, you could have a look at Beyond Jupyter (https://github.com/aai-institute/beyond-jupyter)
Here's an excerpt from the readme:
"Beyond Jupyter is a collection of self-study materials on software design, with a specific focus on machine learning applications, which demonstrates how sound software design can accelerate both development and experimentation."
Readit News
https://docs.astral.sh/ruff/
I just installed and enabled all the rules by setting select = [ "ALL" ]
And then looked at the 'errors' it showed me for my existing code base and then excluded some rules which don't interest me.
I also have it set up in my IDE, so it'll show me the linting errors while coding.
In a larger code base there will definitely be many 'errors', but using it cleaned up my code really good and it stopped me from some footguns.
Without a very good grasp of the language, it can be counterproductive to deep learning. The linters teach what to do, but not why, so it’s hard to grasp how the codebase is worse without following the linter, or when it’s appropriate to disable a linter vs dealing with it.
Ie I’m not a great JS dev. When the linter says “this should be an arrow function” I understand how to do that, but not why an arrow function is preferable there or in general. I would probably be a better JS dev if I hadn’t had the linter, felt whatever pain from not using arrow functions, and know why they’re important. Or never feel the pain and realize it’s a style choice more than a functional one.
I prefer using them to help me remember things I already understand. I know why mutating a copy of a string doesn’t mutate the caller’s copy, so I’m happy to have a linter point out when I’m trying to do that inadvertently.
Ruff specifically actually has a web page for every single check with a section on the rationale behind it.
I regularly use this when I see an error that I don't understand the purpose of, and am often convinced, though sometimes I recognize that the thing it's trying to protect me from doesn't apply in my particular case and so I disable it. Regardless. I feel it has had the effect of making me a better developer.
If you don't know how to use the tools, you're missing out on time savings and error prevention, and once you write code that the linter doesn't like, you can't make it compatible without significant manual work.
I almost think Python would be better with mandatory types....
https://github.com/simonw/datasettehttps://github.com/simonw/sqlite-utils
So, his code might not be a good place to find best patterns (for ex, I don't think they are fully typed), but his repos are very pragmatic, and his development process is super insightful (well documented PRs for personal repos!). Best part, he blogs about every non-trivial update, so you get all the context!
https://www.dabeaz.com/tutorials.html
https://youtube.com/@dabeazllc
I also like Raymond Hettinger.
The entire point is not to just mindlessly consume a code base, but instead form an idea of how to approach the problem and then see if your hypothesis is correct. Then comparing your approach to the actual approach.
This can show you things that you might've missed taking into account.
For example, gallery-dl's incidental complexity all lies in centralizing persistent state, logging, and IO through the CLI. It doesn't have sufficient abstraction to allow it to be rewired to different UIs without relying on internal APIs that have no guarantee that won't change.
Meanwhile a similar application in yt-dlp has that abstraction and works better, but has similar complexity in the configuration side of things.
Edit: speling
It's almost archaeological.
I agree with others that code formatters like black or ruff might be helpful to you. The literature surrounding them, such as PEPs concerning code formatting, often include examples you may find useful.
[0] https://click.palletsprojects.com/en/8.1.x/
Here's an excerpt from the readme: "Beyond Jupyter is a collection of self-study materials on software design, with a specific focus on machine learning applications, which demonstrates how sound software design can accelerate both development and experimentation."
This one came in handy not too long ago: https://github.com/openai/openai-cookbook/blob/main/examples...
Deleted Comment
Dead Comment