Looks very useful. I'm not aware of any open source alternative (although I could just be ignorant here!)
This is the biggest one: https://feast.dev/
Readit Newsdundun commented on Chronon, Airbnb's ML feature platform, is now open source medium.com/airbnb-enginee... · Posted by u/vquemener
dundun commented on Chronon, Airbnb's ML feature platform, is now open source medium.com/airbnb-enginee... · Posted by u/vquemener
Author. Happy to answer any questions.
How does this relate to Zipline and Bighead? Does it replace those projects or is it a continuation of them?
dundun commented on Winning a hackathon, losing my sanity jero.zone/posts/meal-plan... · Posted by u/jer0me
Cool project and write up.
An aside - while I love the snark and making fun of these "legacy" systems, it has given me a window into my own maturity as an engineer. I was absolutely this cavalier and cocky about poorly implemented systems I've been a user or admin of in the past. But having now spent nearly a decade and a half getting paid for this work and seeing a lot of stuff and the evolution of best practices, I have much more empathy for the organizations and authors of these systems. There are very very few programs that ever achieve something like elegance and beauty when they collide with the real world.
There's a ton of dunning-kruger going on, but I think as college kids riding a high of winning a contest and learning a ton, it can be excused.
dundun commented on Meta to release open-source commercial AI model zdnet.com/article/meta-to... · Posted by u/maskil
What's the monetization model here? Is this a closed-source version of their open-source model? (That's suggested by the phrase in the article, "a commercial version of LLaMA, its open-source large language model".)
Google opensourced Tensorflow because they believed it would help with the hiring process: if researchers could use the same framework to do their PhDs as Google used in their production systems, that was seen as an advantage.
Maybe that's Meta's play here? Maybe the idea is that the ecosystem around a model could be as valuable or more valuable than the model itself too, so an OSS model could benefit Meta a lot more by gaining more of the ecosystem mind share?
Or Maybe Yann LeCun is just a hippie that dreams of free love, hard drugs and open-source models?
dundun commented on How Spotify ran a large Google Dataflow job for Wrapped 2019 labs.spotify.com/2020/02/... · Posted by u/jhatax
dundun commented on Spotify's Kafka-Based Event Delivery System labs.spotify.com/2016/02/... · Posted by u/vgt
I wish they went more in-depth into the particular events. They seem to imply the events are user actions, which I took to mean user played this song, user viewed this playlist, and other analytics.
However do they, or would you, use a system like Kafka as an event source instead of the db. So you would also capture events like user added this song to their playlist that would get persisted to the event source db and then eventually a view model instead of directly to a relational db. It doesn't sound like they do that due to the delays, but I can't find a lot of real usage blog posts about how much to put into something like kafka.
You certainly can use a db for an event source. This article does a really good job of explaining how: http://www.confluent.io/blog/turning-the-database-inside-out...
As mentioned in the post, we've pushed Kafka to at least 700,000 events per second. We have room to push it to much more, but stay in tune for post 2 and 3 to see what we're doing instead.
dundun commented on Spotify moves its back end to Google Cloud news.spotify.com/us/2016/... · Posted by u/dmichel
I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?
Nothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal.
With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle.
[spotifier]
dundun commented on Petuum, a distributed machine learning framework petuum.github.io/?... · Posted by u/brendano
Browsing through the code, this looks a bit rough around the edges and untested. I'd wonder how it compares to similar packages for Spark, Hadoop or Flink. Lack of numbers and users make me wonder if it's anything more than someone's PhD project at CMU.
Where in NY are these stores closing?
I've personally witnessed three blatant thefts in the last few years from my local Duane Reade (that closed down in April). Every time the clerks are like "pretty sure that was the same guy from yesterday". It's never violent or scary. It's just like watching a fight between homeless people in a subway station -- you look, think that's odd, and move on.
> Where in NY are these stores closing? 4 different pharmacies that have closed down since the pandemic just on my path to work, including two a stone's throw from the NY stock exchange. https://maps.app.goo.gl/fJcHCgjVacP5pEuHAhttps://maps.app.goo.gl/kmDXnjHruMCvS2CA6
I suspect it's not all shrinkage though. I imagine continued trends where we buy more and more things via online retailers like Amazon and the growth of online/by mail pharmacies has contributed too. CVS/Duane Reade are still opening new locations too, so it can't be all that bad.