cpard (u/cpard) - Readit News

cpard commented on AI will create a near-term socioeconomic gap among knowledge workers ericdodds.com/blog/ai-wil... · Posted by u/ericdodds

cpard · 7 days ago

It's great to see content coming from people with real experience of the impact of AI.

The balance between more productivity versus less people is probably something that each team and company has to figure out but there has to be some kind of limit in terms of both.

Would be great to understand more that part, this will make it easier to reason about the longer term impact of AI in the market.

cpard commented on Bitmapist: We built an open-source cohorts analytics tool that saved millions doist.dev/bitmapist/... · Posted by u/amix

cpard · a month ago

User behavior analytics have created some interesting specialized data systems.

It's interesting that the authors chose to use Redis but how does it scale with a lot of events?

A few other interesting projects from the past that either have to do with user behavior analytics or using bitmaps.

TrailDB: https://github.com/traildb/traildb old but still a fascinating project in my opinion. Not related to bitmaps but they've done some very clever things on the storage level to compress and query the events in a way that fits well this particular workload.

FeatureBase: https://github.com/FeatureBaseDB/featurebase this one was built on top of bitmaps but they didn't market it as a solution specifically for behavioral analytics, although I'm sure it was used for that.

Of course there are the Mixpanels and Amplitudes of the work too that they had to build specialized storage and query engines for this particular workload.

Regardless of how fascinating I find these systems though as specialized compute engines for a specific workload, it seems that the use case itself is not lucrative enough to sustain companies built around them. I'm not sure why is that but it's interesting to see companies building infrastructure for this particular case on top of Spark for example which ends up being so painful in the mid-long run.

(edit: apparently they answered my question already while I was writing this!)

cpard commented on Experimental surgery performed by AI-driven surgical robot arstechnica.com/science/2... · Posted by u/horseradish

qgin · a month ago

LASIK is essentially an automated surgery and 1-2 million people get it done every year. Nobody even seems to care that it’s an almost entirely automated process.

cpard · a month ago

Makes total sense, I think robotic surgeries are happening for quite a while now not only for eye surgeries.

And I think it’s another great example of how automation is happening in the medical practice.

cpard commented on Experimental surgery performed by AI-driven surgical robot arstechnica.com/science/2... · Posted by u/horseradish

neom · a month ago

Uhmmm... I'm sorry but when Waymo started near everyone I talked to about it says "zero % I'm going in one of those things, they won't be allowed anyway, they'll never be better than a human, I wouldn't trust one, nope, no way" and now people can't wait to try them. I understand what you're saying about the trusted side of the house (surgeons are generally high trust) - but I do think OP is right, once the data is in, people will want robot surgery.

cpard · a month ago

Of course they will. I don’t argue that they won’t.

I just say that the path to that and the way it’s going to be implemented is going to be different and Invisalign is a better example to how it will happen in the medical industry compared to automotive.

cpard commented on Experimental surgery performed by AI-driven surgical robot arstechnica.com/science/2... · Posted by u/horseradish

austinkhale · a month ago

If Waymo has taught me anything, it’s that people will eventually accept robotic surgeons. It won’t happen overnight but once the data shows overwhelming superiority, it’ll be adopted.

cpard · a month ago

I think Waymo is a little bit different and driving in general. Because you have an activity that most people don’t trust how other people perform it already. It’s easier to accept the robo driver.

For the medical world, I’d look to the Invisalign example as a more realistic path on how automation will become part of it.

The human will still be there the scale of operations per doctor will go up and prices will go down.

cpard commented on Hyatt Hotels are using algorithmic Rest “smoking detectors” twitter.com/_ZachGriff/st... · Posted by u/RebeccaTheDev

cpard · a month ago

From their FAQ:

   "Is it worth the investment?

Absolutely. Hotels equipped with Rest have seen an 84x increase in smoking fine collection. Plus, our smoking detection technology helps prevent damage to rooms and reduce a number of future violations."

Apparently there are way more people smoking than we thought there are or the sensor just generates a lot of false positives.

The language they are using all over the site is very interesting though, see here an example:

From how it works:

"Automatically charge

If smoking is detected, your staff gets notified, simplifying the process of charging smoking fees."

With a system with false positives, it makes total sense to use real time notifications to staff to go and check what's going on, that would be legit, but then on top saying that you automatically charge?

It almost feels like they are selling a way to fraud to their customers while covering themselves against any litigation by using the right copy in there to support that it's the responsibility of the Hotel staff to go and check in real time that the violation is actually happening.

cpard commented on Batch Mode in the Gemini API: Process More for Less developers.googleblog.com... · Posted by u/xnx

YetAnotherNick · 2 months ago

Contrary to other comments it's likely not because of queue or general batch reasons. I think it is because that LLMs are unique in the sense that it requires lot of fixed nodes because of vRAM requirements and hence it is harder to autoscale. So likely the batch jobs are executed when they have free resources from interactive servers.

cpard · 2 months ago

that makes total sense and what it entails is that interactive inference >>> batch inference in the market today in terms of demand.

cpard commented on Batch Mode in the Gemini API: Process More for Less developers.googleblog.com... · Posted by u/xnx

3eb7988a1663 · 2 months ago

Think of it like you have a large queue of work to be done (eg summarize N decades of historical documents). There is little urgency to the outcome because the bolus is so large. You just want to maintain steady progress on the backlog where cost optimization is more important than timing.

cpard · 2 months ago

yes, what you describe feels like a one off job that you want to run, which is big and also not time critical.

Here's an example:

If you are a TV broadcaster and you want to summarize and annotate the content generated in the past 12 hours you most probably need to have access to the summaries of the previous 12 hours too.

Now if you submit a batch job for the first 12 hours of content, you might end up in a situation where you want to process the next batch but the previous one is not delivered yet.

And imo that's fine as long as you somehow know that it will take more than 12h to complete but it might be delivered to you in 1h or in 23h.

That's the part of the these batch APIs that I find hard to understand how you use in a production environment outside of one off jobs.

cpard commented on Batch Mode in the Gemini API: Process More for Less developers.googleblog.com... · Posted by u/xnx

jampa · 2 months ago

> who is actually getting value out of these batch APIs

I used the batch API extensively for my side project, where I wanted to ingest a large amount of images, extract descriptions, and create tags for searching. After you get the right prompt, and the output is good, you can just use the Batch API for your pipeline. For any non-time-sensitive operations, it is excellent.

cpard · 2 months ago

What you describe makes total sense. I think that the tricky part is the "non-time-sensitive operations", in an environment where even if you don't care to have results in minutes, you have pipelines that run regularly and there are dependencies on them.

Maybe I'm just thinking too much in data engineering terms here.

cpard commented on Batch Mode in the Gemini API: Process More for Less developers.googleblog.com... · Posted by u/xnx

vineyardmike · 2 months ago

It’s like most batch processes, it’s not useful if you don’t know what the response will be and you’re iterating interactively. It for data pipelines, analytics workloads, etc, you can handle that delay because no one is waiting on the response.

I’m a developer working on a product that lets users upload content. This upload is not time sensitive. We pass the content through a review pipeline, where we did moderation and analysis, and some business-specific checks that the user uploaded relevant content. We’re migrating some of that to an LLM based approach because (in testing) the results are just as good, and tweaking a prompt is easier than updating code. We’ll probably use a batch API for this and accept that content can take 24 hours to be audited.

cpard · 2 months ago

yeah I get that part of batch, but even with batch processing, you usually want to have some kind of sense of when the data will be done. Especially when downstream processes depend on that.

The other part that I think makes batch LLM inference unique, is that the results are not deterministic. That's where I think what the parent was saying about some of the data at least should be available earlier even if the rest will be available in 24h.