What percentage, or how many millions, of patients are accessible on the network today?

Following @isoprohplex, I'll be the fourth comment to say I also built a variant of this: https://hnss.alexwendland.com/

I built mine on top of an RSS feed I generate from Hacker News which filters out any posts linking to the top 1 million domains [1] and creates a readable version of the content. I use it to surface articles on smaller blogs/personal websites—it's become my main content source. It's generated via Github Actions every 4 hours and stored in a detached branch on Github (~2 GB of data from the past 4 years). Here's an example for posts with >= 10 upvotes [2].

It only took several hours to build the semantic search on top. And that included time for me to try out and learn several different vector DBs, embedding models, data pipelines, and UI frameworks! The current state of AI tooling is wonderfully simple.

In the end I landed on (selected in haste optimizing for developer ergonomics, so only a partial endorsement):

  - BAAI/bge-small-en as an embedding model
  - Python with
    - HuggingFaceBgeEmbeddings from langchain_community for creating embeddings
    - SentenceSplitter from llama_index for chunking documents
    - ChromaDB as a vector DB + chroma-ops to prune the DB
    - sqlite3 for metadata
    - FastAPI, Pydantic, Jinja2, Tailwind for API and server-rendered webpages
  - jsdom and mozilla-readability for article extraction

I generated the index locally on my M2 Mac which ripped through the ~70k articles in ~12 hours to generate all the embeddings.

I run the search site with Podman on a VM from Hetzner—along with other projects—for ~$8 / month. All requests are handled on CPU w/o calls to external AI providers. Query times are <200 ms, which includes embedding generation → vector DB lookup → metadata retrieval → page rendering. The server source code is here [3].

Nice work @jnnnthnn! What you built is fast, the rankings were solid, and the summaries are convenient.

[1] https://majestic.com/reports/majestic-million

[2] https://github.com/awendland/hacker-news-small-sites/blob/ge...

[3] https://github.com/awendland/hacker-news-small-sites-website...

I’ve been looking for something like this: eventually consistent syncing of DB content -> embeddings in a vector DB.

So far, I’ve been dealing with a tradeoff between latency + error handling in my API endpoints. I’ll either 1.) embed content + upsert into to the vector DB inside a transaction block for my main DB in the handler, which kills latency, or 2.) kickoff the embedding work separate from the main handler work, which risks data desynchronizing.

I’d much prefer a set-it-and-forget-it approach like Retake.

A few questions:

* If the “real-time server” goes offline temporarily, will it catch up on any newly added rows in the interim?

* Do you intend to emit any OpenTelemetry metrics? I’d like to monitor lag in production.

* Will I be able to deploy this as a single container on ECS/Kubernetes?

I needed a reader view library for a side project and decided to compare the most popular options (repo at https://github.com/awendland/readable-web-extractor-comparis...). Among cleanview, metascraper, @postlight/mercury-parser, and mozilla/readability I thought that mozilla/readability performed the best because of its consistent extraction of the primary content and minimal mangling of the semantic structure.

For a quick preview of each library on a random sample of 16 articles posted to HN, see https://github.com/awendland/readable-web-extractor-comparis... (you’ll need to expand a row to see its results).