Congrats! Could this also be used to generate e2e test automations?
For scraping, how do you handle Cloudflare and Captchas? Do you respect robots.txt instructions of websites?
Readit Newsjackienotchan commented on Launch HN: BrowserBook (YC F24) – IDE for deterministic browser automation · Posted by u/cschlaepfer
jackienotchan commented on Launch HN: Webhound (YC S23) – Research agent that builds datasets from the web · Posted by u/mfkhalil
AI crawlers have lead to a big surge in scraping activity, and most of these bots don't respect any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits, user agents, etc.).
This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Webhound respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
The first two quotes are from founders of:
- BrowserUse - Founded 2024
- Greptile - Founded 2023
The third quote is from a VC who has never founded a startup himself and has a clear interest in pushing founders to trade work-life balance for his own quick returns.
So none of these people worked on anything longer than 2 years. I wonder what will happen if we check back in 5–10 years. Will they still be doing and promoting 996, or will they be burned out and have changed their minds? Make your bets.
jackienotchan commented on Launch HN: Reducto Studio (YC W24) – Build accurate document pipelines, fast · Posted by u/adit_a
I saw your recent $24M series A and was kind of surprised to only see you launching now, congrats!
YC seems to fund quite many document extraction companies, even within the same batch:
- Pulse (YC W24): https://www.ycombinator.com/companies/pulse-3
- OmniAI (YC W24): https://www.ycombinator.com/companies/omniai
- Extend (YC W23): https://www.ycombinator.com/companies/extend
How do you differentiate from these? And how do you see the space evolving as LLMs commoditize PDF extraction?
AI crawlers have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN (and experienced myself).
Do you have any built-in features that address these issues?
jackienotchan commented on Launch HN: Browser Use (YC W25) – open-source web agents github.com/browser-use/br... · Posted by u/MagMueller
AI agents have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN.
Do you have any built-in features that address these issues?
Would you mind sharing the story behind your pivot from on-demand photorealistic vision datasets[0] to browser automation?
[0] https://www.ycombinator.com/launches/Lbx-simplex-on-demand-p...
The record/replay is definitely and interesting direction. The browser automation space is getting super crowded though (even within YC), so curious to hear how you differentiate from:
- BrowserUse
- Browserbase
- BrowserBook
- Skyvern