I wrote this post setting out how to work with cloud data in Polars. Although I've been using Polars for almost 3 years I still learned from writing this. I cover topics like how a CSV scan is different from a Parquet scan, authentication and query optimisations. Interested to hear your feedback!
Readit Newsbraaannigan commented on Reading and writing files in the cloud with Polars rhosignal.com/posts/readi... · Posted by u/braaannigan
There are some big questions around OpenAI's Advanced Data Analysis. Is it cutting edge data science or just bad Medium articles all the way down? Can you trust it? And will it let Larry from Sales take your job?
I've shared my experience in this blog post: https://www.rhosignal.com/posts/advanced-data-analysis/
tldr: the exploratory analysis was nice, the text summaries are oft wrong, the choice of ML models was poor IMO and it gains a lot from expert guidance
> physical models are the most reliable basis for prediction
Simpler models can yield better predictions than complex ones, even when the complex model is more "realistic". The many tunable parameters and complex feedback loops can increase uncertainty compared to a simple model.
See for example "To explain or to predict?" by Shmueli (2010, https://doi.org/10.1214/10-STS330)
Many of the physical models are of similar complexity to the model in this paper - the sort of analytical model that you run in a few minutes on your laptop, not on a super-computer. It's not a question of complexity it's about whether a statistical model that is not constrained by well-understood constraints is a high-value model
The publication is in Nature Communications, which is a bit of a different (lesser) beast than Nature.
Thanks for pointing this out, I missed that
I've got a background in this field and I am very surprised to see this published in Nature. The model presented is purely statistical with no representation of the underlying physics. When we are dealing with a phenomenon that is driven by well-understood physical laws (e.g. geophysical fluid dynamics, radiation physics etc) then these physical models are the most reliable basis for prediction.
When I say physical models here by the way I'm referring to physically-based mathematical models as well as numerical models.
It seems that the authors have done a good job in developing their model. My issue is with Nature deciding to publish it. If this paper was not published in Nature it would receive little attention within or without climate science - in fact many such statistical models are published each year without much comment. However, Nature have published a paper that I think many ocean scientists would feel draws dramatic conclusions from a weak basis but will now inevitably draw much more attention than more insightful papers.
MIT professor Carl Wunsch accused Nature in 2010 of near-tabloid science with a tendency towards sensational papers built on weak foundations. However, I've felt that Nature's choice of publications on climate in recent years has been high quality. This paper feels like a big step-down from that standard.
Some thoughts about gathering trends in data science - in particular the area of data processing that's been dominated by pandas for the last 10 years. I'd love to hear what you think!
Waymo is like the most courteous, respectful driver you can possibly imagine. They have infinite patience and will always take the option which is the safest for everyone. One thing which really impressed me is how patient they are at crosswalks. When I'm jogging, a Waymo will happily wait for me to cross - even when I'm 10 feet away from even entering the crosswalk! I don't know if I even have that much patience while driving! I've had a number of near misses with human drivers who don't bother checking or accelerate for no reason after I'm already in the crosswalk. Can you imagine a Waymo ever doing that?
If I see a Waymo on the street near me I immediately feel safer because I know it is not about to commit some unhinged behavior. I cannot say enough good things about them.