This is also the university that develops RumbleDB[0]. It uses JSONiq as its query language which is such a pleasure to work with. It's useful for dealing with data lakes, though I've only experimented with it because of JSONiq.
Readit Newsjansommer commented on SQLite as an Application File Format sqlite.org/appfileformat.... · Posted by u/gjvc
It can be compressed, see https://sqlite.org/sqlar.html
Archive Files is for blobs as far as I understand. All your other data remains uncompressed?
Deleted Comment
jansommer commented on SQLite as an Application File Format sqlite.org/appfileformat.... · Posted by u/gjvc
Something to consider when using SQLite as a file format is compression (correct me if I'm wrong!). You might end up with a large file unless you consider this, and can't/won't just gz the entire db. Nothing is compressed by default.
jansommer commented on A race condition in Aurora RDS hightouch.com/blog/uncove... · Posted by u/theanomaly
Aurora doesn't use EBS under the hood. It has no option to choose storage type or io latency. Only a billing choice between pay per io or fixed price io.
Precisely! That's why RDS sounds so interesting. I get a lot more knobs to tweak performance, but I'm curious if a maxed out gp3 with instances that support it is going to fare any better than Aurora.
jansommer commented on A race condition in Aurora RDS hightouch.com/blog/uncove... · Posted by u/theanomaly
We've seen better results and lower costs in a 1 writer, 1-2 reader setup on Aurora PG 14. The main advantages are 1) you don't re-pay for storage for each instance--you pay for cluster storage instead of per-instance storage & 2) you no longer need to provision IOPs and it provides ~80k IOPs
If you have a PG cluster with 1 writer, 2 readers, 10Ti of storage and 16k provision IOPs (io1/2 has better latency than gp3), you pay for 30Ti and 48k PIOPS without redundancy or 60Ti and 96k PIOPS with multi-AZ.
The same Aurora setup you pay for 10Ti and get multi-AZ for free (assuming the same cluster setup and that you've stuck the instances in different AZs).
I don't want to figure the exact numbers but iirc if you have enough storage--especially io1/2--you can end up saving money and getting better performance. For smaller amounts of storage, the numbers don't necessarily work out.
There's also 2 IO billing modes to be aware of. There's the default pay-per-IO which is really only helpful for extreme spikes and generally low IO usage. The other mode is "provisioned" or "storage optimized" or something where you pay a flat 30% of the instance cost (in addition to the instance cost) for unlimited IO--you can get a lot more IO and end up cheaper in this mode if you had an IO heavy workload before
I'd also say Serverless is almost never worth it. Iirc provisioning instances was ~17% of the cost of serverless. Serverless only works out if you have ~ <4 hours of heavy usage followed by almost all idle. You can add instances fairly quickly and failover for minimal downtime (of course barring running into the bug the article describes...) to handle workload spikes using fixed instance sizes without serverless
Have you benchmarked your load on RDS? [0] says that IOPS on Aurora is vastly different from actual IOPS. We have just one writer instance and mostly write 100's of GB in bulk.
[0] https://dev.to/aws-heroes/100k-write-iops-in-aurora-t3medium...
jansommer commented on A race condition in Aurora RDS hightouch.com/blog/uncove... · Posted by u/theanomaly
> 3125 throughput
Max throughput on gp3 was recently increased to 2GB/s, is there some way I don't know about of getting 3.125?
This is super confusing. Check out the RDS Postgres calculator with gp3:
> General Purpose SSD (gp3) - Throughput > gp3 supports a max of 4000 MiBps per volume
But the docs say 2000. Then there's IOPS... The calculator allows up to 64.000 but on [0], if you expand "Higher performance and throughout" it says
> Customers looking for higher performance can scale up to 80,000 IOPS and 2,000 MiBps for an additional fee.
jansommer commented on A race condition in Aurora RDS hightouch.com/blog/uncove... · Posted by u/theanomaly
People who have experience with Aurora and RDS Postgres: What's your experience in terms of performance? If you dont need multi A-Z and quick failover, can you achieve better performance with RDS and e.g. gp3 64.000 iops and 3125 throughput (assuming everything else can deliver that and cpu/mem isn't the bottleneck)? Aurora seems to be especially slow for inserts and also quite expensive compared to what I get with RDS when I estimate things in the calculator. And what's the story on read performance for Aurora vs RDS? There's an abundance of benchmarks showing Aurora is better in terms of performance but they leave out so much about their RDS config that I'm having a hard time believing them.
jansommer commented on Baldur's Gate 3 Steam Deck – Native Version larian.com/support/faqs/s... · Posted by u/_JamesA_
What's gonna be super interesting is that I'm going to have an rpi zero 2 power up my machine when I press the controller's ps-button. That means I might need to solder and do some electrical voodoo that I've never tried. Crossing my fingers that the plan ChatGPT has come up with won't electrocute me.