I feel jumping right into __init_subclass__ without explaining why metaclasses exist and what problems they typically solve excludes the majority of devs on HN. Thereby limiting any discussion only to the advanced Python developer echo chamber, while discussion across very different devs is usually far more interesting.
Simon keeps his TIL (today I learned) blog as a way to take notes mainly for himself, from what I understand.
I like the way he documents everything in Github issues[0]. I learn a new thing or two with every release of Datasette, because there's so much troubleshooting full of relevant links and solutions. Same with his TIL blog.
One use is for e.g. binding of database models for an ORM.
class FooModel(metaclass=Base):
x = Field(type=int, primary_key=True)
y = Field(type=reference(BarModel))
The Base metaclass will set it up to implement methods like save() by inheriting from parent classes, but it would also be nice for the library to have a list of all model types without the library user having to call a method like FooORM.register_type(FooModel). So the metaclass is being used in these classes to build up a dictionary of models when the class definition is encountered.
The metaclass is basically a class that itself builds classes, which means it can be syntactically convoluted.
However, with __init_subclass__ you can write a thing that looks like a regular class with regular parent methods, but instead just gets a method called each time the interpreter encounters a new subclass, which lets you do things like build up that dictionary for your ORM.
What classes are to instances, metaclasses are to classes.
This is from someone who has only ever read about, never used metaclasses, because they are widely regarded similar to git submodules. If you cannot really assert that you need them, you don't. They solve very specific problems, mostly found in library, not user code. A library can then allow user classes to be modified comprehensively. If you control the classes in the first place (not library code), you probably can do without metaclasses.
* Fluent Python — takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time
* Serious Python — deployment, scalability, testing, and more
* Practices of the Python Pro — learn to design professional-level, clean, easily maintainable software at scale, includes examples for software development best practices
Yeah this is great, __init_subclass__ comes along to make dynamic injection of methods and attributes on class creation easier, just as the entire practice is fast becoming fully obsolete because type checkers like pylance and mypy report these attributes and methods as errors.
That's why I resist python types. A lot of the usefulness and interestingness of Python is in its dynamic features which aid in rapidly developing complex systems. Hard to imagine a strictly typed future for Python, when you then have little to trade off for its bad performance and lack of real concurrency. JavaScript has seen success as a typed language because it has being "the exclusive frontend language" going for it, and v8 is high performance. And the dynamic capabilities of JavaScript were never that interesting or powerful.
> Hard to imagine a strictly typed future for Python
On the contrary, the dynamic nature of python can make it seem like inaccessible black-magic where you don't even know where to look in the code to see what's even possible. Stronger types will empower users to understand their ecosystems.
I've spent countless hours in rails/django land trying to figure out what's even possible with an object that the ORM or whatever gives me. The human-flavored docs kinda outline it, but if you're in IDE-land, context-switching to prose isn't productive.
Types help humans develop and debug, but they make library authors jump through acrobatics to express what their magic does. This is a limitation of the type-checker implementation(s) not having a strong syntax or notion to capture the runtime-dynamic polymorphism, but it doesn't mean that the concept of types is a flawed idea or not worth using when appropriate.
> Yeah this is great, __init_subclass__ comes along to make dynamic injection of methods and attributes on class creation easier, just as the entire practice is fast becoming fully obsolete because type checkers like pylance and mypy report these attributes and methods as errors.
The Python community and the set of people that treat mypy and pylance as dictators rather than tools to be used where appropriate and turned off where not are...not the same thing. (And the latter very much depends on tools built by the rest of the former that internally are wild and woolly, even if they present a nice cleanly typed interface to the consumer.)
There are a few ways to deal with that. One is to declare the properties directly in the subclass:
class Inject:
def __init_subclass__(cls):
cls.injected = 42
class Child(Inject):
injected: int
print(Child().injected) # 42
This technique isn't too useful in practice, though it has its place (e.g. when creating ctypes.Structures). But there's a clever way to simplify this. Remember that the subclass is a subclass -- it inherits from Inject:
import typing as t
class Inject:
injected: t.ClassVar[int]
def __init_subclass__(cls):
cls.injected = 42
class Child(Inject):
pass
print(Child().injected) # 42
(ClassVar[int] ensures that type checkers know it's a variable attached to the class, not each instance.)
yes you can do that, but if you want to do say, what dataclasses does, where it collects all the Field objects and creates a typed `__init__` method, typing can't do that without plugins. all the current typecheckers hardcode the specific behavior of "dataclasses" without there being any pep that allows other systems (like ORMs) to take advantage of the same thing without writing mypy plugins or simply not working for other type checkers that don't allow plugins.
Python is the most popular language on the planet because it’s a dynamic language, not in spite of it. If the type checker makes dynamic features difficult, ditch the type checker, not the dynamic features. I can’t wait for this pendulum to swing back the other way.
> Python is the most popular language on the planet because it’s a dynamic language, not in spite of it.
Disagree. It's popular because it has an uncluttered syntax - "executable pseudocode" - and simple semantics - "explicit is better than implicit", "there should be one obvious way to do it", and all that. Metaclasses were never particularly Pythonic (if anything they felt like a more Ruby-style way of doing things, and the ascent of Python over Ruby is a reflection of that difference).
Was there ever a point people were successfully building "backbone" systems in dynamically typed languages? I thought the pythons and Perl's of the world were always mainly doing scripts/interface applications.
That's my observation as well. If you write typed Python code (or Typescript) you need to constrain yourself to a small subset of what the dynamic language can do. In some cases it's beneficial but often it can be quite stifling, so not sure I'm a big fan of gradual typing anymore.
Yes, this. This is where type checking becomes potentially harmful: you have to start creating loads of explicit subclasses instead of one dynamic class. And then you're writing slow Java.
I'm at a point in my career where I see "magic" and wince hard.
This is so hard to reason about and fix at scale. If you let other engineers run wild with this and build features with it, the time will eventually come to decom your service and move functionality elsewhere. Having to chase down these rabbits, duplicate magic, and search large code bases without the help of an AST assisted search is slow, painful, and error prone.
I spent several years undoing magic method dispatch in Ruby codebases. Tracing through nearly a thousand endpoints to detect blast radiuses of making schema changes impacted by CRUD ops on lazily dispatched "clever code".
I'm sure there are valid use cases for this, but be exceedingly careful. Boring code is often all you need. It doesn't leave a mess for your maintainers you'll never meet.
Python users tend not to behave this way, but seeing posts like this requires me to urge caution.
Comments are underestimated. I'm not saying they should be used as an excuse to use "magic" esoteric code, but sometimes such solutions are needed and it's much better to document them for future maintainers.
Wasn't the idea behind this to make most code boring, and have only a small part of it exciting? Just like when structured programming was created, or even local variables introduced, etc. (Of course, the exciting part had to be moved into the compiler in those cases.)
“Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”
I realize this quote isn't meant to be taken literally, and it's a nice one-liner encapsulation of metaclasses, but it always bothered me. The people who "actually need metaclasses" were, at some point, learning about metaclasses for the first time and were "wondering whether they need them".
This quote ignores the middle ground of users who have a valid use case, but don't yet understand metaclasses. Not great advice for people who are trying to learn metaclasses.
I do sympathize with this sentiment, but there are times when "if you're wondering if you need it, you don't need it" is a very true statement, and Python metaclasses are in that boat. To see why, let me explain in more detail:
There's a "basic" [1] understanding of Python's object model, one which understands that most of Python is actually syntactic sugar for calling certain special methods. For example, the expression a[b] is "really" just calling a.__getitem__(b). Except that's not actually true; the "real" object model involves yet more dispatching to get things to work. Metaclasses allow you to muck with that "yet more dispatching" step.
So when do you need metaclasses? When you need to do that low-level mucking--the kind of mucking most won't know about until actually needed. If all you know about metaclasses is kind of what they are, then you very likely haven't learned enough about them to use them to actually need them. Conversely, if you've learned those details well enough to need to muck with them, then you've also learned enough to the point that you can answer the question as to whether or not you need metaclasses.
[1] I suspect most Python users don't even have this level of understanding, which is why I put it in scare quotes.
I suspect Tim's meaning was that ordinary users don't need to know how metaclasses work in detail. It is enough that people know they exist (i.e. do something at time class is defined). If you need them, you are doing deep enough magic that you can take the time to learn how they work.
Maybe he was also suggesting that people should generally not be using them. In my experience, it is exceedingly rare to need them. In 29 years of writing Python code, I think I've used them once or twice.
I think SQLAlchemy makes good use of metaclasses. It's the only thing I've seen that used them in a way that I thought made sense and was well justified (but I have been out of the Python world for a while).
In another language it would be done with code-gen, which would also be fine.
But SQLAlchemy is library code.
I took the time to understand how to use metaclasses, and concluded that I never would, especially not in code I expected another engineer to understand - I expect 90% of people who write Python haven't heard of metaclasses, and 99% have never written one.
Metaclasses / __init_subclass__ are generally only used if you try to define different / new behavior for how the body of a class is interpreted. So if you'd want to create e.g. an explicit interface system in Python, you might use them. If you're creating an ORM or some other kind of declarative property system, you might use them (but not necessarily, as descriptors are a thing). I think the only use in the standard library is in ABCs and probably the typing module.
I'm not fond of the example given in the post, but I've very happily used __init_subclass__ in a few places where I previously would have needed a decorator. Eg, for "plugin-like" things where I would have otherwise needed a decorator to keep track, or iterated over subclasses, now can just "register" subclasses without any extra work on the child classes. Things that I normally wouldn't expect to change the behavior of the subclass, but where having a no-effort hook at subclass creation time can be extremely useful.
I used to think so, but cls.__subclasses__() makes it possible to just walk the type tree at some later point. I find the lack of state to be a feature. (Normally with a hook like you're talking about, you'd add the class to some list somewhere, which can get out of sync.)
It's used very sparingly in the wild, for cases where you need a bunch of classes that are set up similarly, and ordinary subclassing still results in too much boilerplate.
For example... you might make a class for each HTML tag in an HTML processor, or each drawing command in an SVG processor. My experience is that it's not hard to debug, because all that you're doing is getting rid of some boilerplate.
Yep. As a current functional-language guy, this merely confirms my suspicion that those crazy functional people were onto something.
As it turns out, since the human mind is the real limitation here, anything that stops runaway complexity ("hiding" functionality behind "magic" is actually worse because you're just moving the complexity "away from your mental model" where it can fester and grow out of sight, even if what you see looks "simpler") results in fewer bugs, and more-debuggable bugs.
There are bugs in this code and I'm glad that I'm not the only one that has done it!
graph = {
key: {
p
for p in inspect.signature(method).parameters.keys()
if p != "self" and not p.startswith("_")
}
for key, method in cls._registry.items()
}
The first parameter to a bound method does not have to be called `self`. It's conventional, but not required. Is there a better way in the inspect module to filter these parameters? This comes up more often with classmethods where the naming convention `cls` is most common but I see `klass` somewhat frequently as well.
Readit News
I like the way he documents everything in Github issues[0]. I learn a new thing or two with every release of Datasette, because there's so much troubleshooting full of relevant links and solutions. Same with his TIL blog.
[0]: https://github.com/simonw/datasette/issues/1576
i have a text on the python object system and metaclasss here https://github.com/MoserMichael/python-obj-system/blob/maste... as part of my free python course https://github.com/MoserMichael/python-obj-system/blob/maste... (didn't cover __init_subclass__ yet, will have to extend it. You never stop learning with python...)
Edit: another user linked their blog https://news.ycombinator.com/item?id=29813349
The metaclass is basically a class that itself builds classes, which means it can be syntactically convoluted.
However, with __init_subclass__ you can write a thing that looks like a regular class with regular parent methods, but instead just gets a method called each time the interpreter encounters a new subclass, which lets you do things like build up that dictionary for your ORM.
This is from someone who has only ever read about, never used metaclasses, because they are widely regarded similar to git submodules. If you cannot really assert that you need them, you don't. They solve very specific problems, mostly found in library, not user code. A library can then allow user classes to be modified comprehensively. If you control the classes in the first place (not library code), you probably can do without metaclasses.
I’m a pretty good Python dev, but I feel in order to advance my skills I need to get into extensions … which I don’t have the time for.
Any suggestions on advanced Python resources?
* Fluent Python — takes you through Python’s core language features and libraries, and shows you how to make your code shorter, faster, and more readable at the same time
* Serious Python — deployment, scalability, testing, and more
* Practices of the Python Pro — learn to design professional-level, clean, easily maintainable software at scale, includes examples for software development best practices
On the contrary, the dynamic nature of python can make it seem like inaccessible black-magic where you don't even know where to look in the code to see what's even possible. Stronger types will empower users to understand their ecosystems.
I've spent countless hours in rails/django land trying to figure out what's even possible with an object that the ORM or whatever gives me. The human-flavored docs kinda outline it, but if you're in IDE-land, context-switching to prose isn't productive.
Types help humans develop and debug, but they make library authors jump through acrobatics to express what their magic does. This is a limitation of the type-checker implementation(s) not having a strong syntax or notion to capture the runtime-dynamic polymorphism, but it doesn't mean that the concept of types is a flawed idea or not worth using when appropriate.
Don't throw the baby out with the bath-water.
The Python community and the set of people that treat mypy and pylance as dictators rather than tools to be used where appropriate and turned off where not are...not the same thing. (And the latter very much depends on tools built by the rest of the former that internally are wild and woolly, even if they present a nice cleanly typed interface to the consumer.)
Disagree. It's popular because it has an uncluttered syntax - "executable pseudocode" - and simple semantics - "explicit is better than implicit", "there should be one obvious way to do it", and all that. Metaclasses were never particularly Pythonic (if anything they felt like a more Ruby-style way of doing things, and the ascent of Python over Ruby is a reflection of that difference).
This is so hard to reason about and fix at scale. If you let other engineers run wild with this and build features with it, the time will eventually come to decom your service and move functionality elsewhere. Having to chase down these rabbits, duplicate magic, and search large code bases without the help of an AST assisted search is slow, painful, and error prone.
I spent several years undoing magic method dispatch in Ruby codebases. Tracing through nearly a thousand endpoints to detect blast radiuses of making schema changes impacted by CRUD ops on lazily dispatched "clever code".
I'm sure there are valid use cases for this, but be exceedingly careful. Boring code is often all you need. It doesn't leave a mess for your maintainers you'll never meet.
Python users tend not to behave this way, but seeing posts like this requires me to urge caution.
That kind of code is great if:
(a) you wrote it
(b) you knew what you were doing when you wrote it and
(c) you also wrote an extensive set of tests to cover the entire domain of whatever you were doing.
Comments are underestimated. I'm not saying they should be used as an excuse to use "magic" esoteric code, but sometimes such solutions are needed and it's much better to document them for future maintainers.
— Tim Peters
This quote ignores the middle ground of users who have a valid use case, but don't yet understand metaclasses. Not great advice for people who are trying to learn metaclasses.
There's a "basic" [1] understanding of Python's object model, one which understands that most of Python is actually syntactic sugar for calling certain special methods. For example, the expression a[b] is "really" just calling a.__getitem__(b). Except that's not actually true; the "real" object model involves yet more dispatching to get things to work. Metaclasses allow you to muck with that "yet more dispatching" step.
So when do you need metaclasses? When you need to do that low-level mucking--the kind of mucking most won't know about until actually needed. If all you know about metaclasses is kind of what they are, then you very likely haven't learned enough about them to use them to actually need them. Conversely, if you've learned those details well enough to need to muck with them, then you've also learned enough to the point that you can answer the question as to whether or not you need metaclasses.
[1] I suspect most Python users don't even have this level of understanding, which is why I put it in scare quotes.
Maybe he was also suggesting that people should generally not be using them. In my experience, it is exceedingly rare to need them. In 29 years of writing Python code, I think I've used them once or twice.
I get that it's a dynamic language but if I need metaclasses for something it's more likely than not to be a hidden technical debt generator.
In another language it would be done with code-gen, which would also be fine.
But SQLAlchemy is library code.
I took the time to understand how to use metaclasses, and concluded that I never would, especially not in code I expected another engineer to understand - I expect 90% of people who write Python haven't heard of metaclasses, and 99% have never written one.
Edit: Enums use them as well (makes sense).
For example... you might make a class for each HTML tag in an HTML processor, or each drawing command in an SVG processor. My experience is that it's not hard to debug, because all that you're doing is getting rid of some boilerplate.
As it turns out, since the human mind is the real limitation here, anything that stops runaway complexity ("hiding" functionality behind "magic" is actually worse because you're just moving the complexity "away from your mental model" where it can fester and grow out of sight, even if what you see looks "simpler") results in fewer bugs, and more-debuggable bugs.
Dead Comment
https://twitter.com/dabeaz/status/1466731368956809219
Which is a warning that people shouldn't buy his new book if they want a deep dive on Metaprogramming:
https://www.amazon.com/dp/0134173279/
Simon does a nice job demonstrating how "the __init_subclass__ class method is called when the class itself is being constructed."
Deleted Comment