Readit NewsOn a slightly related note-
I've been thinking about building a home-local "mini-Google" that indexes maybe 1,000 websites. In practice, I rarely need more than a handful of sites for my searches, so it seems like overkill to rely on full-scale search engines for my use case.
My rough idea for architecture:
- Crawler: A lightweight scraper that visits each site periodically.
- Indexer: Convert pages into text and create an inverted index for fast keyword search. Could use something like Whoosh.
- Storage: Store raw HTML and text locally, maybe compress older snapshots.
- Search Layer: Simple query parser to score results by relevance, maybe using TF-IDF or embeddings.
I would do periodic updates and build a small web UI to browse.
Anyone tried it or are there similar projects?
YaCy (https://yacy.net) can do all this I think. Cloudflare might block you IP pretty soon though if you try to crawl.
Yes, please reach out and repost.
I didn't get the feeling that lesswrong is the right place for such a post. Is it? What tag should I use? Anything else I should know when posting there for these this kind of content?
Lesswrong is a AI Safety related forum. There have been previous funding posts on lesswrong. Can check them out.
https://www.lesswrong.com/tag/grants-and-fundraising-opportu...
Also, it would be better if you reach out to the other blogs since they might have some doubts regarding this program of yours.
You might be able to reach the intended audience at lesswrong forum. Can also ask Zvi (https://thezvi.substack.com/), astralcodexten, Marginal Revolution to publicize this.
harias commented on ZLibrary domains have been seized by the United States Postal Inspection Service 3lib.net/... · Posted by u/lucia-wermer
Paper: https://www.nber.org/papers/w34388