Looking forward to reading corroborating essays from other non-human species.
Readit Newsarkmm commented on We’re Not So Special: A new book challenges human exceptionalism democracyjournal.org/maga... · Posted by u/nobet
"There was one surprise when I revisited costs: OpenAI charges an unusually low $0.0001 / 1M tokens for batch inference on their latest embedding model. Even conservatively assuming I had 1 billion crawled pages, each with 1K tokens (abnormally long), it would only cost $100 to generate embeddings for all of them. By comparison, running my own inference, even with cheap Runpod spot GPUs, would cost on the order of 100× more expensive, to say nothing of other APIs."
I wonder if OpenAI uses this as a honeypot to get domain-specific source data into its training corpus that it might otherwise not have access to.
arkmm commented on How to sell if your user is not the buyer writings.founderlabs.io/p... · Posted by u/mooreds
This hits home. We're building business intelligence APIs around entity resolution, and the buyer/user split gets messy when you have engineering, product, and data science teams all involved.
Engineers immediately understand why matching messy company data is a nightmare, but executives just see delayed projects without grasping the technical complexity.
We're seeing more success lately with "your team burned N months on data matching that should've taken weeks" rather than explaining what entity resolution even is. We're talking to one company right now that's spent 10 years building their own entity resolution system and it still doesn't work well.
But even then, it depends on the company and what they're trying to do.
How are you guys reaching users with such a technical value proposition? Cold emailing engineers first and then expanding the conversation from there?
arkmm commented on Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs baseten.co/blog/sota-perf... · Posted by u/philipkiely
Let's say I want to run f2(f1(x)) where f1 and f2 are both a single pass through GPT4.
This takes 2 seconds time, assuming 1 second for every pass.
What I instead do is kick off f1(x) in another thread, and then run f2(g1(x)) where g1 is one pass through GPT-nano.
This takes 1 + 0.1 seconds, assuming gpt nano takes 0.1s for every pass. In this 1.1 seconds, the f1(x) that we kicked off in the 2nd thread would have finished (it takes 1 second).
So in 1.1 seconds we have available to us f1(x), f2(g1(x)), and we store the intermediate g1(x) as well
We compare g1(x) and f1(x)
If they were equal, i.e g1(x) = f1(x), then we have our answer = f2(g1(x)) in just 1.1s.
If they were not, we compute f2(output of f1(x) from 2nd thread) which takes 1 further second, bringing our total to 2.1s.
If the small model is equalling the big model in say 2/3 of cases, you will spend 2/3 * 1.1 + 1/3 * 2.1 = 1.433s on average for this computation. Without speculative decoding, it is always 2s.
This is a really great explanation.
arkmm commented on Ask HN: Have you ever regretted open-sourcing something? · Posted by u/paulwilsonn
Not the OP, but I have a similar dilemma. I'm currently sitting on a SOTA ML model for a particular niche. I'm trying to figure whether I should try selling it to the incumbents (in some shape or form), or if I should publish a paper on the techniques, and/or if I should OSS it.
IMO if you think you can sell to users within the niche, you can publish a blog post of benchmarks and that'll serve as strong technical marketing for your niche.
It also keeps open the option to sell to an incumbent (possibly helps maximize the value of that option as well).
Missing the California high-speed rail on their list of examples.
We have been fine-tuning models using Axolotl and Unsloth, with a slight preference for Axolotl. Check out the docs [0] and fine-tune or quantize your first model. There is a lot to be learned in this space, but it's exciting.
When do you think fine tuning is worth it over prompt engineering a base model?
I imagine with the finetunes you have to worry about self-hosting, model utilization, and then also retraining the model as new base models come out. I'm curious under what circumstances you've found that the benefits outweigh the downsides.
arkmm commented on ChatGPT agent: bridging research and action openai.com/index/introduc... · Posted by u/Topfi
I've been using OpenAI operator for some time - but more and more websites are blocking it, such as LinkedIn and Amazon. That's two key use-cases gone (applying to jobs and online shopping).
Operator is pretty low-key, but once Agent starts getting popular, more sites will block it. They'll need to allow a proxy configuration or something like that.
Automating applying to jobs makes sense to me, but what sorts of things were you hoping to use Operator on Amazon for?
arkmm commented on Attended Windsurf's Build Night 18 hours before founders joined Google DeepMind · Posted by u/schwentkerr
A bit of a minor detail, but this piqued my interest "DOM-aware browser integration" - could you say a little more?
arkmm commented on Andrew Ng: Building Faster with AI [video] youtube.com/watch?v=RNJCf... · Posted by u/sandslash
He doesn’t have to at this point, he just throws money at younger ones that will build it.
I want an Andrew Ng Agent.
Not affiliated, but someone's already working on that for you: https://www.realavatar.ai/