Prompt testing should be simple and straightforward. All I wanted was a simple way to test prompts with variables and jinja2 templates across different models, ideally somthing I could open during a call, run few tests, and share results with my team. But every tool I tried hit me with a clunky UI, required login and API keys, or forced a lengthy setup process.
And that's not all.
Then came the pricing. The last quote I got for one of the tools on the market was $6,000/year for a team of 16 people in a use-it-or-loose-it way. For a tool we use maybe 2–3 times per sprint. That’s just ridiculous!
IMO, it should be something more like JSFiddle. A simple prompt playground that does not require you to signup, does not require API keys, and let's experiment instantly, i.e. you just enter a browser URL and start working. Like JSFiddle has. And mainly, something that costs me nothing if I'm or my team is not using it.
Eventually I gave up looking for solution and decided to build it by myself.
Here it is: https://langfa.st
Help me find what's wrong or missing or does not work from you perspctive.
P.S. I did not put any limits or restrictions yet, so test it wisely. Don't make me broke, please.
Maybe not obvious to users to collapse the panel.
Follow up, how are you handling actual calls to LLM?
In terms of calls to LLMs. I do not use any frameworks or LLM proxies like OpenRouter etc. Instead, I make the calls directly to LLM providers with a tiny thin proxy endpoint I created in Supabase.
One of the problems I had with other tools was the difficulty in understand the actual responses that particular playgrounds provided. Especially when it came to error responses. I guess that they are either built with the some Proxy providers like OpenRouter who handles and interprets errors internally before giving a response to the user, or they are using frameworks like LangChain with their abstraction hell.
In our case on Yola, it was crucial to have a playground that provided this raw type of experience that I have builtin.
2. Got an error:
"Unexpected Application Error! Cannot read properties of null (reading 'slice') TypeError: Cannot read properties of null (reading 'slice') at Hv (https://langfa.st/main.1510e80706059046a306.js:2:11907588) at hi (https://langfa.st/main.1510e80706059046a306.js:2:10922009) at Xa (https://langfa.st/main.1510e80706059046a306.js:2:10941715) at fs (https://langfa.st/main.1510e80706059046a306.js:2:10952350) at $c (https://langfa.st/main.1510e80706059046a306.js:2:10997432) at Gc (https://langfa.st/main.1510e80706059046a306.js:2:10997360) at Zc (https://langfa.st/main.1510e80706059046a306.js:2:10997202) at Nc (https://langfa.st/main.1510e80706059046a306.js:2:10993991) at yd (https://langfa.st/main.1510e80706059046a306.js:2:11006790) at Cd (https://langfa.st/main.1510e80706059046a306.js:2:11005523) Hey developer
You can provide a way better UX than this when your app throws errors by providing your own ErrorBoundary or errorElement prop on your route."
API calls are routed through a thin proxy on my side, this is how you get access to all the models with my API keys. I definitely would not want to store those keys in code of the JS bundle in the browser (:
Only if you want to spend more time managing an open source project than building a real world project. It is not easy and it can be a big distraction.
just one minor suggestion: It seems that the responses are not saved anywhere? even after signup, opening a new tab or just navigating within the app makes the responses disappear - if they're really not being stored, might be worth considering storing them. if they are, maybe the ux could make that more obvious (I couldn't find it). in any case, very useful project!
What tool was this?
From the BAML team, so it uses the BAML syntax (open source).
Any specific feedback so far?
Dead Comment