Readit News logoReadit News
colinchartier commented on Postgres Task Queues: The Secret Weapon Killing Specialized Queue Services?   fforward.ai/blog/posts/po... · Posted by u/olalonde
vosper · 2 years ago
What’s the max throughout anyone’s got using Postgres as a queue? And on what spec machine?

Could I conceivably put 200 messages per second through a Postgres queue given a big enough RDS instance?

Wondering if we don’t really need to move to Kafka at work…

colinchartier · 2 years ago
Postgres can be tuned to 10k inserts per second, Kafka is definitely overkill for 200/s
colinchartier commented on The Mystery of the Bloomfield Bridge   tylervigen.com/the-myster... · Posted by u/graypegg
colinchartier · 2 years ago
Reminds me of this video about the origin of the name Tiffany: https://m.youtube.com/watch?v=9LMr5XTgeyI

"I didn't fly across the Atlantic to... Of course I did"

colinchartier commented on The longest word you can type on the first row   rubenerd.com/the-longest-... · Posted by u/nafnlj
kristopolous · 2 years ago
Here's something you may not know, the *-insane dictionaries, which are giant, are functions of OCR output and are known to contain lots of errors.

I found a few earlier this year and I was going to file a bug so I did some research to find out this is a known and expected behavior.

If the computer say reads stubborn as stubbum, the smaller dictionaries are the ones that have cross checked and filtered those out. The insane ones do not. It's a good name. "Lack of sanity checks"

Here's an example word I found, "suabilities". You'll find it only on wordlist sites that used this wordlist and I guess, now here.

colinchartier · 2 years ago
Reminds me of the ghost Unicode character saga: https://www.dampfkraft.com/ghost-characters.html
colinchartier commented on Virtualizing development environments in 2023   hocus.dev/blog/virtualizi... · Posted by u/hugodutka
easton · 2 years ago
> Many cloud providers, like AWS inside VM-based EC2 instances, won't let you run a VM since they don't support nested virtualization

Does anyone know if there's a technical limitation to nested virt on EC2 or if it's just so people use .metal? I know Azure recently (1-2 years?) started offering it.

colinchartier · 2 years ago
Nested virt doesn't work with some things like OS snapshot/restore, so they might not support it to allow those features.
colinchartier commented on How to listen to database changes using Postgres triggers in elixir   peterullrich.com/listen-t... · Posted by u/pjullrich
stevencorona · 3 years ago
I've been using Elixir for the past 5-6 years for my startup. We use pg_notify extensively to broadcast changes between running nodes (basically, use Phoenix.PubSub locally in our apps, with a GenServer to subscribe+re-broadcast using pg_notify).

This has been a really elegant and low-complexity way to get distributed pubsub without the complexity of running a distributed erlang cluster (which seems a lil bit painful in a K8S+Continuous Deploy world)

There -are- some big downsides to be aware of though.

1. You can't use PgBouncer w/ LISTEN/NOTIFY. This has been really painful because of the high memory overhead of a pgsql connection + elixir keeping a pool of open pgsql connections. The tried and true method of scaling here is to just use PgBouncer. We've kicked the can on this by vastly over-provisioning our pg instance, but this has cost $10s of thousands on the cloud. Of course, it's solvable (dedicated non-pgbouncer connection pool just for LISTEN/NOTIFY, for example), but painful to unwind.

2. The payload has a fixed size limit (8KB, IIRC). This has bitten us a few times!

Even though I really like pg_notify, I think that if I were starting over, I'd probably just use Redis Pub/Sub to accomplish the same thing. Tad bit more complex if you're not already running Redis, but without the downsides. (Of course, w/ Redis, you don't get the elegance of firing a notification via a pg trigger)

colinchartier · 3 years ago
We got around this at my company by just pooling all of the LISTEN/NOTIFY streams into a single database connection in software, here's a sample implementation:

function software_listen(channel, callback):

  if not channel_listened(channel):

    sql("LISTEN " + channel)

  listeners[channel].append(callback)

function on_message(channel, data):

  for listener in listeners[channel]

    listener(channel, data)

function unlisten(channel, listener):

  listeners[channel].remove(listener)

  if len(listeners[channel]) == 0:

    sql("UNLISTEN " + channel)

Here's the actual go implementation we use:

https://gist.github.com/ColinChartier/59633c1006407478168b52...

colinchartier commented on Addressing the community about changes to our API   reddit.com/r/reddit/comme... · Posted by u/kotaKat
tedunangst · 3 years ago
Just from a technical perspective, what is Apollo doing that requires more than 100 queries per minute?

(Okay, so apparently client ID isn't user ID, but app ID. Explains a lot.)

colinchartier · 3 years ago
As far as I understand, that's 100 shared queries across all users.
colinchartier commented on I went down the rabbit hole of buying GitHub Stars, so you won't have to   the-guild.dev/blog/judgin... · Posted by u/Urigo
pizzaknife · 3 years ago
just being a stickler here but 10% is probably enough
colinchartier · 3 years ago
Paul Graham would say for exponential growth, 10x better is the bar
colinchartier commented on Show HN: Webapp.io - Free firecracker-based full-stack hosting   webapp.io/hosting... · Posted by u/colinchartier
tptacek · 3 years ago
Hey! Best of luck with this. Real quick: we don't use Docker "under the hood" at fly.io either. We take OCI containers as inputs, and transform them into VMs. The only "Docker" running here is the code that allows us to accept our users existing containers.
colinchartier · 3 years ago
Sure, I meant "docker" here as a simplification for the OCI format.
colinchartier commented on Show HN: Webapp.io - Free firecracker-based full-stack hosting   webapp.io/hosting... · Posted by u/colinchartier
Existenceblinks · 3 years ago
I can't even find their vm details (cpu/memory + bandwidth) and many other things.
colinchartier · 3 years ago
Will add those to the landing page - they got lost in the Figma design

u/colinchartier

KarmaCake day1097November 14, 2017
About
Co-founder @ https://webapp.io - a convenient solution for per-branch full-stack staging environments and automatic e2e tests.
View Original