If you're open to it, I'd love to hear what you think of what we're building at https://browserbase.com/ - you can run a chrome extension on a headless browser so you can do the semantic markdown within the browser, before pulling anything off.

We even have an iFrame-able live view of the browser, so your users can get real-time feedback on the XPaths they're generating: https://docs.browserbase.com/features/session-live-view#give...

Happy to answer any questions!

Have you compared markdown to just stripping the HTML down (strip tag attributes, unwrap links, remove obvious non-displaying elements)? My experience has been that performance is pretty similar to markdown, and it’s an easier transformation with fewer edge cases.

First I’ve heard of Semantic Markdown [0]. It appears to be a way to embed RDF data in Markdown documents.

The page I found is labeled “Alpha Draft,” which suggests there isn’t a huge corpus of Semantic Markdown content out there. This might impede LLM’s ability to understand it due to lack of training data. However, it seems sufficiently readable that LLMs could get by pretty well by treating its structured metadata as parathenicals

=====

What is Semantic Markdown?

Semantic Markdown is a plain-text format for writing documents that embed machine-readable data. The documents are easy to author and both human and machine-readable, so that the structured data contained within these documents is available to tools and applications.

Technically speaking, Semantic Markdown is "RDFa Lite for Markdown" and aims at enhancing the HTML generated from Markdown with RDFa Lite attributes.

Design Rationale:

Embed RDFa-like semantic annotation within Markdown

Ability to mix unstructured human-text with machine-readable data in JSON-LD-like lists

Ability to semantically annotate an existing plain Markdown document with semantic annotations

Keep human-readability to a maximum About this document

=====

[0] https://hackmd.io/@sparna/semantic-markdown-draft

Thank you for these.

I've been wanting to try the same approach and have been looking for the right tools.

We did something similar -na although in a somewhat different context.

Translating a complex JSON representing an execution graph to a simpler graphviz dot format first and then feeding it to an LLM. We had decent success.

I hope some of the opensource inference servers start supporting that endpoint soon. I know vLLM has added some "offline batch mode" support with the same format, they just haven't gotten around to implementing it on the OpenAI endpoint yet.

That's a great proposition by OpenAI. I think however that it is still one to two orders of magnitude too expensive compared to traditional text extraction with very similar precision and recall levels.

Yeah this was a phenomenal decision on their part. I wish some of the other cloud tools like azure would offer the same thing, it just makes so much sense!

The LLM is resistant to website updates that would break normal scraping

If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the data is in an unexpected format.

yours is the first one see that allows to scrape by selecting directly what to scrape. I always wondered why there was no tool doing that.

I wrote an in-house one for Ribbon. If there’s interest, will open source this. It’s amazing how much better our LLM outputs are with the reducer.

Jina.ai offer a really neat (currently free) API for this - you add https://r.jina.ai/ on the beginning of any API and it gives you back a Markdown version of the main content of that page, suitable for piping into an LLM.

Here's an example: https://r.jina.ai/https://simonwillison.net/2024/Sep/2/anato... - for this page: https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-us...

Their code is open source so you can run your own copy if you like: https://github.com/jina-ai/reader - it's written in TypeScript and uses Puppeteer and https://github.com/mozilla/readability

I've been using Readability (minus the Markdown) bit myself to extract the title and main content from a page - I have a recipe for running it via Playwright using my shot-scraper tool here: https://shot-scraper.datasette.io/en/stable/javascript.html#...

    shot-scraper javascript https://simonwillison.net/2024/Sep/2/anatomy-of-a-textual-user-interface/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"

We wrote something like this to power Skyvern: https://github.com/Skyvern-AI/skyvern/blob/0d39e62df6c516e0a...

It's adapted from vimium and works like a charm. Distill the html down to it's important bits, and handle a ton of edge cases along the way haha

Running it through Readability:

https://github.com/mozilla/readability

author here: I'm working on a follow-up post. Turns out, removing all HTML tags works great and reduces the cost by a huge margin.

Only works insofar as sites are being nice. A lot of sites do things like: render all text via JS, render article text via API, paywall content by showing a preview snippet of static text before swapping it for the full text (which lives in a different element), lazyload images, lazyload text, etc etc.

DOM parsing wasn't enough for Google's SEO algo, either. I'll even see Safari's "reader mode" fail utterly on site after site for some of these reasons. I tend to have to scroll the entire page before running it.

I wrote one for a project that captures a portion of the DOM and sends it to an LLM.

It’s strips all JS/event handlers, most attributes and most CSS, and only keeps important text nodes

I needed this because I was using LLM to reimplement portions of a page using just tailwind, so needed to minimise input tokens

That’s easy to do with BeautifulSoup in Python. Look up tutorials on that. Use it on non-essential tags. That will at least work when the content is in HTML rather than procedurally generated (eg JavaScript).

Not OP, but thanks for the suggestion. I’m starting to play around with LLMs and will explore locally hosted versions.

this! I've been following Kadoa since its very first days. Great team.

I second trafilatura greatly. This will save a huge amount of money to just send the text to the LLM. I used it on this recent project (shameless plug): https://github.com/philippe2803/contentmap. It's a simple python library that creates a vector store for any website, using a domain XML sitemap as a starting point. The challenge was that each domain has its own HTML structure, and to create a vector store, we need the actual content, removing HTML tags, etc. Trafilatura basically does that for any url, in just a few lines of code.