DAGWorks is based on Hamilton, an open-source project that we created and recently forked (https://github.com/dagworks-inc/hamilton). Hamilton is a set of high-level conventions for Python functions that can be automatically converted into working ETL pipelines. To that, we're adding a closed-source offering that goes a step further, plugging these functions into a wide array of production ML stacks.
ML pipelines consist of computational steps (code + data) that produce a working statistical model that a business can use. A typical pipeline might be (1) pull raw data (Extract), (2) transform that data into inputs for the model (Transform), (3) define a statistical model (Transform), (4) use that statistical model to predict on another data set (Transform) and (5) push that data for downstream use (Load). Instead of “pipeline” you might hear people call this “workflow”, “ETL” (Extract-Transform-Load), and so on.
Maintaining these in production is insanely inefficient because you need both data scientists and software engineers to do it. Data scientists know the models and data, but most can't write the code needed to get things working in production infrastructure—for example, a lot of mid-size companies out there use Snowflake to store data, Pandas/Spark to transform it, and something like databrick's MLFlow to handle model serving. Engineers can handle the latter, but mostly aren't experts in the ML stuff. It's a classic impedance mismatch, with all the horror stories you'd expect—e.g. when data scientists make a change, engineers (or data scientists who aren’t engineers) have to manually propagate the change in production. We've talked to teams who are spending as much as 50% of their time doing this. That's not just expensive, it's gruntwork—those engineers should be working on something else! Basically, maintaining ML pipelines over time sucks for most teams.
One way out is to hire people who combine both skills, i.e. data scientists who can also write production code. But these are rare and expensive, and in our experience they usually are only expert at one side of the equation and not as good at the other.
The other way is to build your own platform to automatically integrate models + data into your production stack. That way the data scientists can maintain their own work without needing to hand things off to engineers. However, most companies can't afford to make this investment, and even for the ones that can, such in-house layers tend to end up in spaghetti code and tech debt hell, because they're not the company's core product.
Elijah and I have been building data and ML tooling for the last 7 years, most recently at Stitch Fix, where we built a ML platform that served over 100 data scientists from various modeling disciplines (some of our blog posts, like [1], hit the front page of HN - thanks!). We saw first hand the issues teams encountered with ML pipelines.
Most companies running ML in production need a ratio of 1:1 or 2:1 data scientists to engineers. At bigger companies like Stitch Fix, the ratio is more like 10:1—way more efficient—because they can afford to build the kind of platform described above. With DAGWorks, we want to bring the power of an intuitive ML Pipeline platform to all data science teams, so a ratio of 1:1 is no longer required. A junior data scientist should be able to easily and safely write production code without deep knowledge of underlying infrastructure.
We decided to build our startup around Hamilton, in large part due to the reception that it got here [2] - thanks HN! We came up with Hamilton while we were at Stitch Fix (note: if you start an open-source project at an employer, we recommend forking it right away when you start a company. We only just did that and left behind ~900 stars...). We are betting on it being our abstraction layer to enable our vision of how to go about building and maintaining ML pipelines, given what we learned at Stitch Fix. We believe a solution has to have an open source component to be successful (we invite you to check out the code). In terms of why the name DAGWorks? We named the company after Directed Acyclic Graphs because we think the DAG representation, which Hamilton also provides, is key.
A quick primer on Hamilton. With Hamilton we use a new paradigm in Python (well not quite “new” as pytest fixtures use this approach) for defining model pipelines. Users write declarative functions instead of writing procedural code. For example, rather than writing the following pandas code:
df['col_c'] = df['col_a'] + df['col_b']
def col_c(col_a: pd.Series, col_b: pd.Series) -> pd.Series:
"""Creating column c from summing column a and column b."""
return col_a + col_b
def col_d(col_c: pd.Series) -> pd.Series:
# logic
What we think is cool about Hamilton is that you don’t need to specify an “explicit pipeline declaration step”, because it’s all encoded in the function and parameter names! Moreover, everything is encapsulated in functions. So from a framework perspective, if we wanted to (for example) log timing information, or introspect inputs/outputs, delegate the function to Dask or Ray, we can inject that at a framework level, without having to pollute user code. Additionally, we can expose "decorators" (e.g. @tag(...)) that can specify extra metadata to annotate the DAG with, or for use at run time. This is where our DAGWorks Platform fits in, providing off-the-shelf closed source extras in this way.
Now, for those of you thinking there’s a lot of competition in this space, or what we’re proposing sounds very similar to existing solutions, here’s some thoughts to help distinguish Hamilton from other approaches/technology: (1) Hamilton's core design principle is helping people write more maintainable code; at a nuts and bolts level, what Hamilton replaces is procedural code that one would write. (2) Hamilton runs anywhere that python runs: notebook, a python script, within airflow, within your python web service, pyspark, etc. E.g. People use Hamilton for executing code in batch tasks and online web services. (3) Hamilton doesn't replace a macro orchestration system like airflow, prefect, dagster, metaflow, zenML, etc. It runs within/uses them. Hamilton helps you not only model the micro - e.g. feature engineering - but can also help you model the macro - e.g. model pipelines. That said, given how big machines are these days, model pipelines can commonly run on a single machine - Hamilton is perfect for this. (4) Hamilton doesn't replace things like Dask, Ray, Spark -- it can run on them, or delegate to them. (5) Hamilton isn't just for building dataframes, though it’s quite good for that, you can model any python object creation with it. Hamilton is data type agnostic.
Our closed source offering is currently in private beta, but we'd love to include you in it (see next paragraph). Hamilton is free to use (BSD-3 license) and we’re investing in it heavily. We’re still working through pricing options for the closed source platform; we think we’ll follow the leads of others in the space like Weights & Biases, and Hex.tech here in how they price. For those interested, here’s a video walkthrough of Hamilton, which includes a teaser of what we’re building on the closed source side - https://www.loom.com/share/5d30a96b3261490d91713a18ab27d3b7.
Lastly, (1) we’d love feedback on Hamilton (https://github.com/dagworks-inc/hamilton) and on any of the above, and what we could do better. To stress the importance of your feedback, we’re going all-in on Hamilton. If Hamilton fails, DAGWorks fails. Given that Hamilton is a bit of a “swiss army knife” of what you could do with it, we need help prioritizing features. E.g. we just released experimental PySpark UDF map support, is that useful? Or perhaps you have streaming feature engineering needs where we could add better support? Or you want a feature to auto generate unit test stubs? Or maybe you are doing a lot of time-series forecasting and want more power features in Hamilton to help you manage inputs to your model? We’d love to hear from you! (2) For those interested in the closed source DAGWorks Platform, you can sign up for early access via www.dagworks.io (leave your email, or schedule a call with me) – we apologize for not having a self-serve way to onboard just yet. (3) If there’s something this post hasn’t answered, do ask, we’ll try to give you an answer! We look forward to any and all of your comments!
What the heck are all these ML and data platforms, how do they benefit me, and how do I evaluate the gazillion options that seem to be out there?
For example, I recently came across DStack (https://dstack.ai/) and have had an open browser tab sitting around waiting for me to figure out WTF it even does. DAGWorks seems like it does something similar. Is that true? Are these tools even comparable? How would I choose one or the other? Is there overlap with MLFlow?
> Data scientist here, stuck in the Dark Ages of "deploying" my models by writing bespoke Python apps that run on some kind of cloud container host like ECS. Dump the outputs to blob storage and slurp them back into the data warehouse nightly using Airflow. Lots of manual fussing around.
Oh, man, been there! So, first, I want to say there are a lot of ML/data platforms -- largely because there are so many problems to solve, and they're not one-size-fits-all solutions.
> What the heck are all these ML and data platforms, how do they benefit me, and how do I evaluate the gazillion options that seem to be out there?
You probably don't need all of them, or all that many. As in everything, it depends on your pain-points. Given that you have a lot of manual fussing around, you probably want something to reduce it. We've found airflow to be painful, but a lot of dev teams/DS have airflow already integrated into their platform, so we wanted to build something that allows data scientists to plug into it. So for people who don't like airflow, the idea is that you could express your dataflow in Hamilton and DAGWorks can ship it to airflow (or any other orchestration system).
> For example, I recently came across DStack (https://dstack.ai/) and have had an open browser tab sitting around waiting for me to figure out WTF it even does. DAGWorks seems like it does something similar. Is that true? Are these tools even comparable? How would I choose one or the other? Is there overlap with MLFlow?
DStack is definitely a different approach, similar space. Hamilton is organized around python functions and dstack is more of a high-level workflow spec (reminds me of something we had at my old company). So Hamilton can model the "micro" of your workflow, whereas DStack models the "macro" -- managing artifacts. DStack could easily run Hamilton functions. What nothing we've found out there do (except perhaps kedro) is model the "micro" -- E.G. the specific fine-grained dependencies so you can take a look at your code and figure out how exactly it works.
Re: MLFlow -- DAGWorks + MLFlow are pretty natural connectors. Hamilton functions can produce a model that DAGWorks would be able to save to mlflow. DAGWorks is more on the data transform side, and doesn't explicitly say how to represent a model.
Just wondering then @elijahbenizzy - how does Hamilton differ from Kedro?
Deleted Comment
We aim to build the most easy tool to run ML workflows - without making you use the UI of any vendor, or hustling with Kubernetes, custom Docker images, etc.
MLFlow doesn't do what dstack does (automatic infrastructure provisioning) - unless you use Databricks.
For this, you could use for example https://flyte.org/, if you have platform engineers that could set it up for you. But I don't think their website makes a good job of explaining what it is and what it does.
What I try to do, is understand who created the platform, and understand the environment of that company. Then that will give you a better idea of whether it makes sense for you or not.
> acquiring and moving data around,
yep with Hamilton we provide the ability to cleanly separate bits of logic that's required to change and update this. For example you'd write "data loader functions/modules" that are implementations for say reading from a DB, or a flat file, some vendor. If they output a standardized data structure, then the rest of your workflow would not be coupled to the implementation, but the common structure which Hamilton forces you to define. That way you can be pretty surgical with changes and understanding impacts.
Regarding assessing impacts, Hamilton provides the ability to "visualize" and query for lineage as defined by your Hamilton functions. We think that with Hamilton we can make the "hey what does this impact?" question really easy to answer, so that when you do need to make changes you'll have more confidence in doing them.
> managing the data over the lifetime of a model’s use,
Hamilton isn't opinionated about where data is stored. But given that if you define the flow of computation with Hamilton and use a version control system like git to version it, then all you need to then additionally track is what configuration your Hamilton code was run with, and associate those two with the produced materialized data/artifact (i.e. git SHA + config + materialized artifact), you have a good base with which to answer and ask queries of what data was used when and where. Rather than bringing in 3rd party systems to help here, we think there's a lot you can leverage with Hamilton to help here.
For example, we have users looking at Hamilton to help answering governance concerns with models produced.
> and serving adjacent needs (e.g. post-deployment analytics).
If it's offline, then you can model and run that with Hamilton. The idea is to help provide integrations with whatever MLOps system here to make it easy to swap out.
For online, e.g. a web-service, you could model the dataflow with Hamilton, and the build your own custom "compilation" to take Hamilton and project it onto a topology. During the projection, you could insert whatever monitoring concerns you'd want. So just to say, this part isn't straightforward right now, but there is a path to addressing it.
While Hamilton/DAGWorks is mainly for expressing the pipeline/abstracting away the infrastructure, DAGWorks can help make the model lifecycle easy as well:
- Hamilton pipelines can be run anywhere, including in an online setting
- Breaking into functions can make it modular/easy to annotate and gather post-hoc analysis
- Hamilton pipelines specify the data movement in code -- providing a source of truth
That said, I think the ecosystem for doing this is much cleaner/easier to manage than it used to be -- the MLOps stack is far more sophisticated. Scalable/reliable compute (spark, Modin), easy storage (snowflake, new feature store technology), and more model experiment-as-a-service type systems (mlflow, model-db, etc...) have made these less of a difficult problem than it was in the past. As these permeate the industry and we develop more standards, I think that it pushes the problem up a level -- rather than figuring out exactly how to solve these, the difficult part is looping together a bunch of systems that all do it fairly well but (a) require significant expertise to manage and (b) often result in code that's super coupled to the systems themselves (making it hard to test). DAGWorks wants to decouple these from the pipeline code, enabling you to choose which systems you delegate to and not have to worry about it.
Furthermore, we think that smaller pipelines are actually super underserved in ML/data science community -- E.G. pipelines that don't have a lot of the "moving data around" problems but can be run on a single machine. I've seen these suffer from getting too complex/being difficult to manage, and we think Hamilton can solve this out of the box.
Thoughts?
Like Stefan mention in the OP, Hamilton works well with tools like Metaflow which can help with many other concerns you mentioned. How you define your data transformations for ML is an open question that Hamilton addresses neatly.
See here for an example of Metaflow+Hamilton in action: https://outerbounds.com/blog/developing-scalable-feature-eng...
> These functions then define a "dataflow" or a directed acyclic graph (DAG), i.e. we can create a “graph” with nodes: col_a, col_b, col_c, and col_d, and connect them with edges to know the order in which to call the functions to compute any result.
This 'new paradigm' already exists in Polars. Within the scope of a local machine, you can write declarative expressions which can then be used pretty much anywhere for querying instead of the usual arrays and series (arguments to filter/apply/groupby/agg/select etc), allowing it to build an execution graph for each query, optimise it and parallelise it, and try to only run through the data once if possible without cloning. Eg the example above can be written simply as
It is obviously restricted to what is supported in polars, but a surprising amount of the typical data munging can be done with incredible efficiency, both cpu and ram wise.To be clear, the specific paradigm we're referring to is this way of writing transforms as functions where the parameter name is the upstream dependency -- not the notion of delayed execution.
I think there are two different concepts here though:
1. How the transforms are executed
2. How the transforms are organized
Hamilton cares about (2) and delegates to Polars/pandas for (1). The problem we're trying to solve is the code getting messy and transforms being poorly documented/hard to own -- Hamilton isn't going to solve the problem of optimizing compute as tooling like polars, pandas, and pyspark can handle that quite well.
I think this design is a local maximum for languages that don't have first-class symbols and/or macros like R and Julia. I like to see convergence in this space.
It's also interesting because this style of API is portable more or less unchanged to just about any programming language, from C# to Idris.
Yep I think a declarative syntax is quite portable and can be reimplemented easily in other languages.
On the portable note, where portable we mean swapping dataframe implementations, it's even conceivable to write "agnostic" logic with Hamilton and then at runtime inject the right "objects" that then do the right thing at runtime. E.g. the following is polars specific:
I think with Hamilton you could be more agnostic and enable it to run on both Pandas and Polars -- with TYPE here a placeholder to indicate something more generic... So at runtime you'd instantiate in your Driver some directive to say whether you're operating on pandas or with polars (or at least that's what I imagine in my head) and the framework would take care of the rest...Deleted Comment
But in short yes. Especially if you take the perspective they're both trying to help you do operations over tabular data, where the result is also something tabular.
Duckdb is "A Modern Modular and Extensible Database System" (https://www.semanticscholar.org/paper/DuckDB-A-Modern-Modula...). So it has a bit more to it than polars, as it has a lot of extensibility, for example, you can give it a pandas dataframe and it'll operate over it, and in some cases, faster than pandas itself.
But otherwise at a high-level, yes you could probably replace one for the other in most instances, but not true for everything.
https://hamilton.readthedocs.io/en/latest/reference/api-refe...
coincidentally, my PR to the dbt viewpoint was closed by the docs team as "closed, won't do" [1]
I really like the convention of data plane (where you describe how the data should be transformed) and the control plane (i.e. the configuration of the DAG, do this before this). In this paradigm, I believe that the control plane should be as simple as possible, and even perhaps limited in what can be done with the goal of pushing the user to take data transformation as tantamount. Maybe this is why I fell in love with dbt in the first place is because it does exactly this.
"spicy" take: allowing users to write imperative code (e.g. using loops) that dynamically generates DAGs are never a good idea. I say this as someone who personally used to pester framework PMs for this exact feature before. While things like task groups (formerly subDAGs) [2] appear initially to be right answer, I always ended up regretting them. They're a scheduling/orchestration solution to a data transformation problem
Can y'all speak to how Hamilton views the data and control plane, and how it's design philosophy encourages users to use the right tool for the job?
p.s. thanks for humoring my pedantry and merging this! [3]
[1]: https://github.com/dbt-labs/docs.getdbt.com/pull/2390 [2]: http://apache-airflow-docs.s3-website.eu-central-1.amazonaws... [3]: https://github.com/DAGWorks-Inc/hamilton/pull/105
Said with all love: you're definitely wrong and it's not hard to demonstrate.
This is a thing that Airflow famously couldn't do until 2.3 or 2.4. Luigi could always do it. Prefect, Nextflow, and Martian(?) can do it. DBT can't do it, though the concept doesn't exactly apply (the DBT version is more like "I want to generate models at compile time" which is much more debatably sane... I have this problem currently and our solution sucks, but mostly I wish I didn't have the problem!)I see where you're coming from, I think, but my experience has been the opposite. In any major project, I absolutely need my DAGs to have a dynamic shape. Forcing them to have a static shape just pushes the required dynamism outside the tool, likely into some nasty code-generation step.
In my experience, you want to decouple the task-level orchestration mechanism from the nature of the data itself. A queuing system with multiple consuming threads/processes or a distributed system like spark is kind of meant for this type of task, and can do it more efficiently. So, why rely on the orchestrator to handle it when you potentially have more sophisticated tooling at hand?
To make it more concrete, say your upstream files change to have 10x smaller chunks, and 10x more files -- does the same orchestration system make sense? Are you going to start polluting the set of tasks?
If you did want to rely on the orchestrator for parallelism, an alternative strategy could be to chunk and assign the files to each task -- E.G. 3 files per task, and use a stable assignment method to round-robin them (basically the same as the queuing system). Might not get everything done at quite the parallel pace, but the DAG would be stable.
Your case is particularly tricky as they take so long to process, but it seems to me that this might be better suited for a listening/streaming tool to react when new files are added, then they can add the semi-processed data to a single location, and your daily orchestration task could read from that and set a high-watermark (basically a streaming architecture).
Anyway, not 100% sure how I feel about this but wanted to present a counter-point...
> "spicy" take: allowing users to write imperative code (e.g. using loops) that dynamically generates DAGs are never a good idea.
Can you give some examples of when it was a bad idea? Otherwise to clarify, with Hamilton, there is no dynamism at runtime. When the DAG is generated, it's generated and it's fixed. The operator we have for doing this `@parameterize` requires everything to be known at DAG construction time. It's really just short hand for manually writing out all the functions. So I don't think it's quite the same story - it is more a "power user feature" - but when used, it makes code DRY-er, at the cost of some code readability.
> While things like task groups (formerly subDAGs) [2] appear initially to be right answer, I always ended up regretting them. They're a scheduling/orchestration solution to a data transformation problem
Yep. Hamilton has a concept of `subdag` too. It's really short hand for "chaining" Hamilton drivers. We take the latter approach (I believe) since it is there to help you more easily reuse parts of your DAG with different parameterizations. Since Hamilton isn't concerned with materialization boundaries we don't have to make a decision here so how it impacts scheduling/orchestration can be punted to a later time :)
> Can y'all speak to how Hamilton views the data and control plane,
Hamilton is just a library. There is no DB that needs to be run to use Hamilton. The only state required is your code. So at the simplest micro-level, Hamilton sits within a task, e.g. creating features, and replaces the python script you'd run there. So at this level, I'd argue data plane vs control plane doesn't really apply, unless you view code as the control plane and where it runs as the data plane... At the macro-level, e.g. a model pipeline pulling data, transforming it, fitting a model, etc., you can logically describe the dataflow with Hamilton, without breaking it up into computational tasks a priori. I'd say Hamilton here tries to be agnostic and provide the hooks you need to help you coordinate your control plane and facilitate operation on your data plane. Note: this is where we see DAGWorks coming in and helping provide more functionality for. E.g. with Hamilton you don't need to decide whether everything runs in a single task say on airflow, or multiple. It's up to you to make that decision. The beauty of which, is that conceptually, changing what is in a task, is really just boilerplate given all the information you have already encoded into your Hamilton DAG.
> and how it's design philosophy encourages users to use the right tool for the job?
With Hamilton, we believe python UDFs are the ultimate user interface. By using Hamilton we force you to chunk logic, integrations, into functions. We also provide ways to "decorate" function logic which gives the ability to inject logic around the running of said functions. So we're really quite agnostic to the tool, but want to provide the hooks to be able to easily and cleanly add, adjust, remove them. For example, to switch between Ray and Dask, our philosophy is that ideally you can write code that is agnostic to knowing about the implementation. Then at runtime add those concerns in. As another example, the ability to switch/change say observability vendors, should not force a large refactor on your code base. We have an extensible `@check_output` decorator that should constrain how much you "leak" from the underlying tools. In short: (1) write functions that don't leak implementation details, they should just instead try to limit to just expressing logic; (2) the Hamilton framework should have the hooks required for you to plug in "tool" concerns. Does that make sense? Happy to elaborate more.
I've got recent experience with data eng / pipleine startups and wondering if you are hiring for your first engineers at this time.
1: https://docs.dominodatalab.com/en/latest/user_guide/bc1c6d/s...
DAGWorks is an opinionated way to write code that allows you to abstract it away from the infrastructure, whereas Domino is more about making it easy to deal with infrastructure, manage datasets, etc... Also has a large notebook/development focus. Nothing in is built to make it natural for the code to live for a while/keep it well-maintained, which is the problem DAGWorks is trying to solve.
The idea is that we can allow you to plug into whatever infrastructure you want and not have to think about it too much when you want to switch, although we're still building pieces of that out.