While functools.wraps does propagate __annotations__ by default, be aware that not all IDE-integrated type checkers handle that properly. It's easy in PyCharm, for example, to use functools.wraps such that the wrapper function is treated by the IDE as untyped.
Underneath, this is because many (most?) type checkers for Python aren't actually running the code in order to access annotation information, and are instead parsing it "from the outside" using complex and fallible techniques of variable reliability. That said, it's a testament to JetBrains' excellent work that PyCharm's checker works as well as it does, given how crazily metaprogrammed even simple Python often turns out to be.
Pycharm has the worst type checker that exists today.
It may have been the best a few years back but others have suppressed it considerably.
I recently switched from Pycharm to vscode which uses pyright and it's night and day on the amount of type errors it catches, it considerably improved the quality of my code and confidence during refactoring.
And to add insult to injury Pycharm doesn't even have a pyright plugin and the mypy plugin is extremely slow and buggy.
Interesting, looks like they ended up having to introduce typing.Unpack, to differentiate the ambiguity with the the TypedDict referring to the type of all the kwargs, vs just Mapping[str, TypedDict]
The ability of **kwargs to leave behind no proper documentation and silently swallow any invalid arguments has made us remove them entirely from our codebase. They're almost entirely redundant when you have dataclasses.
What about decorators, or wrappers around third-party code whose contracts change frequently (or even second party code when interacting with functions provided by teams that don't follow explicit argument typing guidelines, if you have that sort of culture)?
Usually the solutions range from a culture of “just don’t” to tests/mypy that have become increasingly stricter over the years, every time we’ve come a step further up the ladder. But I admit, it has taken quite some bridging to get there.
Moving to static Python in most places has dramatically improved the code and language.
For everybody reading this and scratching their head why this is relevant: Python subclassing is strange.
Essentially super().__init__() will resolve to a statically unknowable class at run-time because super() refers to the next class in the MRO. Knowing what class you will call is essentially unknowable as soon as you accept that either your provider class hierarchy may change or you have consumers you do not control. And probably even worse, you aren't even guaranteed that the class calling your constructor will be one of your subclasses.
Which is why for example super().__init__() is pretty much mandatory to have as soon as you expect that your class will be inherited from. That applies even if your class inherits only from object, which has an __init__() that is guaranteed to be a nop. Because you may not even be calling object.__init__() but rather some sibling.
So the easiest way to solve this is: Declare everything you need as keyword argument, but then only give **kwargs in your function signature to allow your __init__() to handle any set of arguments your children or siblings may throw at you. Then remove all of "your" arguments via kwargs.pop('argname') before calling super().__init__() in case your parent or uncle does not use this kwargs trick and would complain about unknown arguments. Only then pass on the cleaned kwargs to your MRO foster parent.
So while using **kwargs seems kind of lazy, there is good arguments, why you cannot completely avoid it in all codebases without major rework to pre-existing class hierarchies.
For the obvious question "Why on earth?"
These semantics allow us to resolve diamond dependencies without forcing the user to use interfaces or traits or throwing runtime errors as soon as something does not resolve cleanly (which would all not fit well into the Python typing philosophy.)
This does restrict all of your keyword arguments to the same type. If you have keyword arguments of different types, you're right back to no type safety.
Well, if you want to type your kwargs and use newer versions of python, you can use Unpack with typed dicts to achieve that. But the footgun there is that you can't redefine fields when extending them, so no Partial<SomeType> for you.
True, but there are a couple of mitigations available: you can express the types of selected kwargs (by leaving them out of the * residual), and you can use typing.Union/| to express product types for values in the residual as well.
That is an entirely different use-case than a function signature allowing arbitrary keyword arguments. Arbitrary keyword args are different than arbitrary positional args like you have in your example.
GP is suggesting that one should only ever use explicit keyword-only args (anything listed after `*,` in the signature) versus arbitrary keyword args implicit via `**kwargs`.
e.g. (omitting type hints for clarity):
def sum(*args, **arbitrary_kwargs):
...
vs
def sum(*args, some_keyword_only_arg):
...
In my opinion if one finds themselves writing code that uses arbitrary kwargs, they've got a design problem.**
It's pretty common when wrapping a function that has a large number of config options.
The wrapper is usually some shorthand for building a handful of those args or adding some side-effect, while still allowing the caller access to the remaining config options via kwargs.
But why would you want that doesn't that make for a more confusing api? Would it not be better to just have everything as a kwarg? You would get better types that way
A required positional OR kwarg as you’ve done it. Its closer to an optional kwarg if you expand the type declaration to also allow None and set a None default.
But there are times when you want to leave the number and names of kwargs open (one example is for a dynamic wrapper—a function that wraps another function that can be different across invocations.)
In my experience it's generally because Python developers make functions with an insane number of keyword arguments, and then wrap those functions. They don't want to type them all out again so they use kwargs.
subprocess.run() is an example of that. Also check out the functions in manim.
The inability to properly static type kwargs using TypedDict is probably the biggest flaw in Python's type hint system (after the fact that hardly anyone uses it of course).
> In the function body, args will be a tuple, and kwargs a dict with string keys.
This always bugs me: why is `args` immutable (tuple) but `kwargs` mutable (dict)? In my experience it’s much more common to have to extend or modify `kwargs` rather than `args`, but I would find more natural having an immutable dict for `kwargs`.
> This always bugs me: why is `args` immutable (tuple) but `kwargs` mutable (dict)?
Because python didn’t (still doesn’t, but at this point even if it did backward compatibility would mean it wouldn’t be used for this purpose) have a basic immutable mapping type to use.
(Note, yes, MappingProxyType exists, but that’s a proxy without mutation operations, not a basic type, so it costs a level of indirection.)
In Python, except for mutability, is there any difference between tuple and list? In my experience: Pure Python people get so excited about tuples ("oh, it's so Pythonic"); others: much less.
Readit News
If your function just wraps another you can use the same type hints as the other function with functools.wraps https://docs.python.org/3/library/functools.html#functools.w...
Underneath, this is because many (most?) type checkers for Python aren't actually running the code in order to access annotation information, and are instead parsing it "from the outside" using complex and fallible techniques of variable reliability. That said, it's a testament to JetBrains' excellent work that PyCharm's checker works as well as it does, given how crazily metaprogrammed even simple Python often turns out to be.
I recently switched from Pycharm to vscode which uses pyright and it's night and day on the amount of type errors it catches, it considerably improved the quality of my code and confidence during refactoring.
And to add insult to injury Pycharm doesn't even have a pyright plugin and the mypy plugin is extremely slow and buggy.
https://docs.python.org/3/library/typing.html#typing.ParamSp...
https://peps.python.org/pep-0612/
Not ideal but not too bad either.
https://peps.python.org/pep-0692/#keyword-collisions
Forge: forge (python signatures) for fun and profit
https://python-forge.readthedocs.io/
https://github.com/dfee/forge
Moving to static Python in most places has dramatically improved the code and language.
Kwargs everywhere, often only defined for a type at runtime by spooky voodoo action at a distance metaclass shenanigans...
Dead Comment
Essentially super().__init__() will resolve to a statically unknowable class at run-time because super() refers to the next class in the MRO. Knowing what class you will call is essentially unknowable as soon as you accept that either your provider class hierarchy may change or you have consumers you do not control. And probably even worse, you aren't even guaranteed that the class calling your constructor will be one of your subclasses.
Which is why for example super().__init__() is pretty much mandatory to have as soon as you expect that your class will be inherited from. That applies even if your class inherits only from object, which has an __init__() that is guaranteed to be a nop. Because you may not even be calling object.__init__() but rather some sibling.
So the easiest way to solve this is: Declare everything you need as keyword argument, but then only give **kwargs in your function signature to allow your __init__() to handle any set of arguments your children or siblings may throw at you. Then remove all of "your" arguments via kwargs.pop('argname') before calling super().__init__() in case your parent or uncle does not use this kwargs trick and would complain about unknown arguments. Only then pass on the cleaned kwargs to your MRO foster parent.
So while using **kwargs seems kind of lazy, there is good arguments, why you cannot completely avoid it in all codebases without major rework to pre-existing class hierarchies.
For the obvious question "Why on earth?" These semantics allow us to resolve diamond dependencies without forcing the user to use interfaces or traits or throwing runtime errors as soon as something does not resolve cleanly (which would all not fit well into the Python typing philosophy.)
Deleted Comment
To do otherwise would require some form of vararg generics which is uncommon.
def variable(n:str, nn:str, nnn:str, *, a:int, b:int, c:int)
Anything after,*, is a kwarg.
GP is suggesting that one should only ever use explicit keyword-only args (anything listed after `*,` in the signature) versus arbitrary keyword args implicit via `**kwargs`.
e.g. (omitting type hints for clarity):
vs In my opinion if one finds themselves writing code that uses arbitrary kwargs, they've got a design problem.**The wrapper is usually some shorthand for building a handful of those args or adding some side-effect, while still allowing the caller access to the remaining config options via kwargs.
Here's one example of that in the wild https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
[0] actually 3 positional-or-keyword which is even more widely divergent
I would rather:
Than a million argsA required positional OR kwarg as you’ve done it. Its closer to an optional kwarg if you expand the type declaration to also allow None and set a None default.
But there are times when you want to leave the number and names of kwargs open (one example is for a dynamic wrapper—a function that wraps another function that can be different across invocations.)
subprocess.run() is an example of that. Also check out the functions in manim.
The inability to properly static type kwargs using TypedDict is probably the biggest flaw in Python's type hint system (after the fact that hardly anyone uses it of course).
Deleted Comment
https://stackoverflow.com/questions/47060133/python-3-type-h...
what a disaster
https://peps.python.org/pep-0612/
This always bugs me: why is `args` immutable (tuple) but `kwargs` mutable (dict)? In my experience it’s much more common to have to extend or modify `kwargs` rather than `args`, but I would find more natural having an immutable dict for `kwargs`.
Because python didn’t (still doesn’t, but at this point even if it did backward compatibility would mean it wouldn’t be used for this purpose) have a basic immutable mapping type to use.
(Note, yes, MappingProxyType exists, but that’s a proxy without mutation operations, not a basic type, so it costs a level of indirection.)