Love the benchmarks. Is better to use single LLM for performance or would always advise to add a self reflection step
self-reflection is very important for both humans and LLMs, indeed
Readit Newsadinagoerres commented on Show HN: LLMs suck at writing integration code… for now github.com/superglue-ai/s... · Posted by u/sfaist
adinagoerres commented on Show HN: LLMs suck at writing integration code… for now github.com/superglue-ai/s... · Posted by u/sfaist
Hey HN, I'm Adina, Stefan's co-founder at superglue. When we started working on LLM-powered integrations about a year ago, the models were barely good enough to handle simple mappings. We started benchmarking our performance as an internal evals project and thought it would be fun to open source it, to create more transparency around LLM performance. Our goal here is to understand how we can make agents production-ready and improve reliability across the board.
Hi HN! Adina here from superglue. Today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs.
tl;dr: LLMs suck at writing code to use APIs.
We ran 630 integration tests across 21 common APIs (Stripe, Slack, GitHub, etc.) using 6 different LLMs. Here are our key findings: - Best general LLM: 68% success rate. That's 1 in 3 API calls failing. Would you ship that? - Our integration layer scored a 91% success rate, showing us that just throwing bigger/better LLMs at the problem won't solve it. - Only 6 out of 21 APIs worked 100% of the time, every other API had failures. - Anthropic’s models are significantly better at building API integrations than other providers.
What makes LLMs fail hard: - Lack of context (LLMs are just not great at understanding what API endpoints exist and what they do, even if you give them documentation which we did) - Multi-step workflows (chaining API calls) - Complex API design: APIs like Square, PostHog, Asana (Forcing project selection among other things trips llms over)
We've open-sourced the benchmark so you can test any API and see where it ranks: https://github.com/superglue-ai/superglue/tree/main/packages...
Check out the repo, consider giving it a star, or see the full ranking at https://superglue.ai/api-ranking/
If you're building agents that need reliable API access, we'd love to hear your approach - or you can try our integration layer at superglue.ai.
Next up: benchmarking MCP.
Very cool! Love the demo.
thank you!
have you looked at Browser Use? https://browser-use.com/
they are in our YC batch! great product
this is VERY cool!
thank you!
something like this that runs as a browser agent, allowing me to extract structured data from websites (whitelisted) using natural language queries
huh interesting. we're exploring extraction from html
Doesn't someone own a trademark in that general area?
not that we're aware of!