Decorators are one of those language features that I want to use more of, but every time I attempt to, I realize that what I'm doing could be achieved more easily another way.
Have felt the same way till I used one for fastapi , One can literally implement production grade* Machine Learning pipeline in 15 lines of code using transformers in a single python file;
from fastapi import FastAPI
from pydantic import BaseModel, constr, conlist
from typing import List
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
model="models/distilbert-base-uncased-mnli")
app = FastAPI()
class UserRequestIn(BaseModel):
text: constr(min_length=1)
labels: conlist(str, min_items=1)
class ScoredLabelsOut(BaseModel):
labels: List[str]
scores: List[float]
@app.post("/classification", response_model=ScoredLabelsOut)
def read_classification(user_request_in: UserRequestIn):
return classifier(user_request_in.text, user_request_in.labels)
*: Production grade if used in combination with workers, A python quirk I felt is not relevant to the topic of decorators.
@production looks risky, compared to checking an environment variable. Hopefully that host_name function is guaranteed to return the same result every time it is called, and can never fail or raise an exception.
That looks amazing, thank you! I wouldn't want it in my production code but it seems like it would be great for something that I'm working on in the REPL.
The decorator pattern is a well known one, where one "decorates" a function by passing it into another function. GP expresses that they would avoid the pattern with these decorators.
The decorator operator is essentially prefix notation of the form `f1 = f2(f1) = @ f2 f1` which is what the GP alluded to, i.e. that f2 is a higher order function since it takes a function and produces another function. In-fact, the @ operator is a higher order function as well since it takes 2 higher order functions.
I am strugeling a bit with the @production decorator. I never had the issue of only wanting to run a function on prod. Often I want slightly different behaviour, but then id use env variables. (say using a Prod and a Dev API adr or DB adr).
Wouldnt it also be better design to keep dev and prod as close as possible?
There's more environments than just dev and prod. Even if the environments differ having some extra telemetry in a test environment often makes sense. So a @telemetry or @production decorator that does something in a non-production environment is an easy way to add that capability to functions.
I've found I like a decorator better than some envar testing custom logging/telemetry. If I have a custom_log function I use everywhere, I have to use it everywhere I might want it. With a decorator I can add it to only a few functions and get far less noise.
@redirect could be handy when the print statements are embedded in someone else's code.
We use a library that is very...chatty (some function calls send a screenful of info/progress to the screen), and I think I'm going to steal this to make it quieter.
> Not everything has to be designed for a super critical prod environment with >10 coders working non stop on it.
You don't need a super critical prod environment to have decent code. Half of these are hardcoding environment configuration, others have hidden side effects that the caller of the function cannot control at all, and others are badly reimplementing things that already exist (@redirect -> you want logging for this, @stacktrace -> use a debugger)
> Not everything has to be designed for a super critical prod environment with >10 coders working non stop on it.
And even when it does, cargo-culting rules-of-thumb is generally the wrong way to do that. Best practices are better treated as the Pirate Code than the Divine Writ.
I find them creative too - and I didn't say they are terrible. Actually, I'm also from data background and that's exactly the type of stuff I would come up with too.
But as I'm recently trying to improve my software skills, I notice that while those are indeed useful in the short term, in the long term they are not worth the price. The @production one, seems like a disaster waiting to happen.
>Those decorators are exactly what data scientists would do, while software engineers would be terrified.
Actually, when I read the post I'd guessed this is what an ex Software Engineer who is now a Data Scientist would do. And looking at the author's LinkedIn confirmed it.
You have to have a software engineering background to come up with this stuff in the first place.
I often use my custom timer decorator to time execution of functions/methods. This is not the only way I can do that, but I think it's a very convenient option.
Readit News
1. @dataclass
https://docs.python.org/3/library/dataclasses.html
2. @cuda.jit
https://numba.pydata.org/numba-doc/latest/cuda/kernels.html
Which means your dev and production environments are quite different, increasing risks of letting stupid mistakes slip into prod.
Besides switching between something like production and test you can catch a case where an envar isn't set or is an unexpected value.
@reloading to hot-reload a function from source before every invocation (https://github.com/julvo/reloading)
%load_ext autoreload %autoreload 2
now all your imports will be automatically reloaded.
So good that I have it in ipython startup scripts
https://reloadium.io
Great for prod if an error is unnacceptable
(or more realistically, web scraping where you just don't care about one off errors)
I just wouldn't use them as decorators in proper code.
Applying these functions as decorators, i.e. with @, means you can't run the non parallel version, or the test not in production, etc.
In the end, decorators, though nice on the first day of usage, reduce composability by restricting usage to whatever you wanted on that same day.
(this is not a general remark, it doesn't apply to DSLs that use decorators, e.g. flask)
Dead Comment
The decorator pattern is a well known one, where one "decorates" a function by passing it into another function. GP expresses that they would avoid the pattern with these decorators.
The decorator operator is essentially prefix notation of the form `f1 = f2(f1) = @ f2 f1` which is what the GP alluded to, i.e. that f2 is a higher order function since it takes a function and produces another function. In-fact, the @ operator is a higher order function as well since it takes 2 higher order functions.
Dead Comment
I've found I like a decorator better than some envar testing custom logging/telemetry. If I have a custom_log function I use everywhere, I have to use it everywhere I might want it. With a decorator I can add it to only a few functions and get far less noise.
Those decorators are exactly what data scientists would do, while software engineers would be terrified.
At least surprised. There are solutions to these problems already.
Author seemed to have learned decorators and is enthusiastic about abusing them, instead of learning the stdlib.
The capturing of print statements. Why not use the Logging machinery, instead?
Or the @stacktrace, it seems what they really want/need is a debugger.
But anyway, if these solutions fit their programming style better, so be it.
We use a library that is very...chatty (some function calls send a screenful of info/progress to the screen), and I think I'm going to steal this to make it quieter.
It would be more interesting to point out the parts you feel are so terrible.
Not everything has to be designed for a super critical prod environment with >10 coders working non stop on it.
You don't need a super critical prod environment to have decent code. Half of these are hardcoding environment configuration, others have hidden side effects that the caller of the function cannot control at all, and others are badly reimplementing things that already exist (@redirect -> you want logging for this, @stacktrace -> use a debugger)
And even when it does, cargo-culting rules-of-thumb is generally the wrong way to do that. Best practices are better treated as the Pirate Code than the Divine Writ.
But as I'm recently trying to improve my software skills, I notice that while those are indeed useful in the short term, in the long term they are not worth the price. The @production one, seems like a disaster waiting to happen.
I don't believe I've ever encountered this. Can you elaborate on what you mean?
Actually, when I read the post I'd guessed this is what an ex Software Engineer who is now a Data Scientist would do. And looking at the author's LinkedIn confirmed it.
You have to have a software engineering background to come up with this stuff in the first place.
Deleted Comment