Readit News logoReadit News
whatl3y commented on Epstein Files Smart Search – AI RAG pipeline, File explorer, Image gallery   search.epstein.ninja/... · Posted by u/whatl3y
whatl3y · 5 days ago
Smart search engine for the Epstein files from the DOJ built on a RAG pipeline that indexes and makes them searchable via natural language (i.e. LLM style). Documents (PDFs, HTML, images) go through text extraction with OCR as a fallback for scanned pages, then get sliced into ~500-token chunks with 50-token overlap. Each chunk gets a metadata prefix baked in (document title, source section) before being embedded into 1536-dimensional vectors through OpenAI's text-embedding-3-small. Those vectors live in PostgreSQL with pgvector, sitting behind an HNSW index with ef_search cranked to 400 (default is 40, which misses too much).

Queries hit the same embedding model, and the system pulls the top-K most similar chunks by cosine distance. There's a hybrid search mode too: it over-fetches 5x candidates from the vector index in parallel with keyword search (full-text search via a GIN-indexed tsvector column, falling back to trigram ILIKE when FTS returns few results). Results are merged using a slot reservation system: 60% of the final top-K comes from vector results ranked by cosine similarity, with up to 40% reserved for keyword-only matches that the vector search missed. Retrieved chunks get stuffed into a prompt with source metadata and sent to Claude Sonnet or GPT-4o with instructions to cite sources in bracket notation.

On the backend, pub-sub workers handle the indexing pipeline: text extraction, chunking, batch embedding in groups of 100, and firing off face detection through AWS Rekognition on images pulled from PDFs (very good in some cases, not so much in others). The query endpoint is free with some rate limiting, but also sits behind x402 micropayments ($0.50) that bypass rate limits when valid (it's not cheap to run these queries as of now). There's also an MCP server so AI agents can query directly as a tool.

Built with the help of Claude, so some of the tech (RAG via LLM, pgvector, etc.) is newish to me. Was a fun exercise!

whatl3y commented on Node.js 16 Available Now   nodejs.medium.com/node-js... · Posted by u/ilkkao
junon · 5 years ago
The problem is that this is more than just a syntactic difference. There's a chance it will take two ticks to resolve instead of one.

For documentation purposes, I recommend block-commenting the `async`:

const foo = /async/ () => new Promise(...);

whatl3y · 5 years ago
I did not know this and appreciate the insight! Will definitely store this back of mind and try and remember to use block comments when needing to explicitly return a promise.
whatl3y commented on Node.js 16 Available Now   nodejs.medium.com/node-js... · Posted by u/ilkkao
scrollaway · 5 years ago
Renaming a column is an operation which showcases the weakness in not using an ORM. Other operations of the sort do exist, anything that operates on columns across multiple arbitrary spots in the code.

IMO an hour for a change you have so little confidence about is not acceptable when the alternative allows you to do it in a second with full confidence.

whatl3y · 5 years ago
While this is fair and I don't disagree that changing a column name or possibly other schema changes using an ORM and it's subsequent migration scripts is potentially going to be a faster exercise than without one, changing a column name (or any schema change for that matter) in a database in my experience is rare. If you choose to use an ORM for reasons like it might (emphasis on might) be a bit faster and easier to change a column name than without one instead of for other more meaningful people or efficiency-oriented reasons of the day-to-day developer workflow is probably a poor approach.

Keep in mind I've installed and used an ORM in projects where the ORM is used only for migrations, but not used in application code and this is absolutely a fine reason to use one imo. But adopting an ORM for migration purposes and forcing or using in application code simply because it's installed isn't necessarily a good approach.

whatl3y commented on Prisma – ORM for Node.js and TypeScript   prisma.io/blog/prisma-the... · Posted by u/janpio
loloquwowndueo · 5 years ago
“ Application developers should care about data – not SQL”

Anyone who has been bitten by an ORM generating inefficient SQL (and subsequent having to learn and care about SQL) knows the above not to be true.

whatl3y · 5 years ago
While some people are hard lined enough on their anti-ORM stance that it sort of becomes weird, I agree with you that my beef with ORMs comes from being burned before a couple of times by a super inefficient aggregation ActiveRecord did on GROUP BY queries that it 1. not only took a really long time to figure out why a particular page in our app was loading slow but 2. we ended up having to write raw SQL to fix it.

I think the answer of whether to use one depends on the type and load/volume of app you're working with combined with the dynamics, size, and skill level of your team(s). I'm extremely comfortable writing, profiling, query planning, and debugging SQL queries. Others aren't, and therefore having an ORM to query data in the DB with the syntax of the language you're using in your projects makes way more sense, if nothing other in order to speed your team up.

whatl3y commented on Node.js 16 Available Now   nodejs.medium.com/node-js... · Posted by u/ilkkao
tayo42 · 5 years ago
This seems like a sweeping judgment your making saying don't use orms, or classes. But this seems like just an odd religious take people have so I'm out...
whatl3y · 5 years ago
While some people are hard lined enough on this that I agree it sort of becomes weird, I can tell you my beef with ORMs comes from being burned before a couple of times by a super inefficient aggregation ActiveRecord did on GROUP BY queries that it 1. not only took a really long time to figure out why a particular page in our app was loading slow but 2. we ended up having to write raw SQL to fix it.

I think the answer depends on the type and load/volume of app you're working with combined with the dynamics, size, and skill level of your team(s). I'm extremely comfortable writing, profiling, query planning, and debugging SQL queries. Others aren't, and therefore having an ORM to query data in the DB with the syntax of the language you're using in your projects makes way more sense, if nothing other in order to speed your team up.

whatl3y commented on Node.js 16 Available Now   nodejs.medium.com/node-js... · Posted by u/ilkkao
scrollaway · 5 years ago
How much work do you have to deal with if a column gets renamed? Be honest.
whatl3y · 5 years ago
1. Don't rename your columns, but if you have to

2. project search and replace on $COLUMN_NAME

If your column name is a common keyword, variable name, etc. in your code base and it's difficult to find using project search, that's unfortunate, but we organize our backend code and tests in a logical enough way that it's never taken longer than an hour to create a PR to create a migration to rename a column and update all places in code that reference it.

whatl3y commented on Node.js 16 Available Now   nodejs.medium.com/node-js... · Posted by u/ilkkao
motogpjimbo · 5 years ago
You don't need `async` here, since your function explicitly returns a promise.
whatl3y · 5 years ago
While this is true, I've found over time it's easier for me to quickly see in my code if I need to `await` on something if the function is an `async` function and not just that it returns a Promise.

I prefer

async function foo() { return await new Promise(...) }

as opposed to

function foo() { return new Promise(...) }

They're the same thing for the most part but the latter I have to potentially dig deeper into the function to confirm it returns a promise compared to the former.

whatl3y commented on Grumpy Cat NFT sold for nearly $80K   foundation.app/grumpycat/... · Posted by u/chrisma0
schrototo · 5 years ago
God this is infuriating. Apart from being an obvious scam, NFTs are actively doing harm to the environment. This is not funny, it's just bad. It’s a net negative for humanity.

As, of course, are any and all cryptocurrencies and blockchain-related get-rich-quick schemes. I can’t wait for this whole pyramid to collapse.

whatl3y · 5 years ago
I think it's necessary to put emphasis on the "get-rich-quick schemes" part, and not the "blockchain/cryptocurrencies" part.

I hope you're not implying blockchain tech and all cryptocurrencies don't/can't provide real utility value, but if so I beg to differ substantially. I built a hook to the Stellar blockchain to store SHA256 hashes of audit evidence that can be validated against at a later time as tamper-proofing evidence. This is just one extremely simple use case, not to mention the entire Defi ecosystem that's being developed as we speak.

u/whatl3y

KarmaCake day231December 7, 2016
About
Developer for a start up in Atlanta.

https://github.com/whatl3y

https://lance.to

[ my public key: https://keybase.io/whatl3y; my proof: https://keybase.io/whatl3y/sigs/LY68u61-jMyMVoAU-iLUMvuI72WBOe879q-XSLTwQUE ]

View Original