Readit News logoReadit News
dav43 commented on Benchmarks for Golang SQLite Drivers   github.com/cvilsmeier/go-... · Posted by u/cvilsmeier
jitl · 3 days ago
For a project a while back, I needed to turn many-gigabyte Postgres CSV table dumps into SQLite databases. I turned to Go as its a great language for easy parallelism combined with enough memory layout control to get relatively good performance.

I quickly ruled out using database/sql drivers as the indirection through interface types added a bunch of overhead and stymied my attempts for reasonable memory layout. For my use-case, I found the crawshaw driver performed the best, but I ended up forking it as well as the Golang standard library CSV parser as I found defensive copying & allocation was the largest bottleneck. I ended up cycling several very large arenas among a CSV parser thread that filled the arena with column bytes and several threads writing to different temporary sqlite databases. Then at the end I ATTACHED them together and copied them into one big file (idk exactly why this is faster, but my profiles showed most cpu time spent in sqlite doing query binding things so MOAR CORES).

One notable optimization was exposing a way to bind borrowed bytes to query parameters without inducing a copy in either Golang caller code, or SQLite library code. The crawshaw driver upstream only exposes sqlite_bind_blob with SQLITE_TRANSIENT mode, which tells SQLite to copy the input to a private allocation before returning from the sqlite_bind* call. I added a version that passes SQLITE_STATIC, which means "trust me, I won't touch these bytes until the query is done, and I'll free them afterwards". This is safe in Rust who's "borrow" and "lifetime" concept models this perfectly, but I guess in Golang its dicey enough to not expose in your public package.

Here's the relevant commit in my fork: https://github.com/crawshaw/sqlite/commit/82ad4f03528e8fdc6a...

I'm curious how OP's https://github.com/cvilsmeier/sqinn would fare, I'm somewhat sus about copying 200GB to stdin but the benchmark results are pretty good so ¯\_(ツ)_/¯

dav43 · 3 days ago
Wouldn’t duckdb be a good fit for this?
dav43 commented on Streaming services are driving viewers back to piracy   theguardian.com/film/2025... · Posted by u/nemoniac
dav43 · 9 days ago
I pay for Apple TV and I’d still pirate so I don’t have to sit through their shit “look at our new tv show” crap ad at the beginning.
dav43 commented on Australia is introducing age checks for search engines like Google   abc.net.au/news/2025-07-1... · Posted by u/ahonhn
dav43 · a month ago
Australian politics is very much in favour of throwing around a lot of Red herrings. Eg get the conversation, media and public focussed on unimportant issues as a way to distract from the fact they are incompetent to deal with big issues.

Grew up in NSW for 25 years. Nothing has changed. A few extra toll roads.

dav43 commented on Hidden interface controls that affect usability   interactions.acm.org/arch... · Posted by u/cxr
dav43 · 2 months ago
Apple Photos has fallen into this trap as well.

As a user, you have no way to see if a photo has been "scanned" with smart features and what it has detected (e,g found person x, found dog, blue sky, beach etc).

Trips features, has this algorithm finished scanning your library? You have no idea, it's just hidden.

Faces, detection, has this completely scanned your library? You don't know. Photos that don't seem to have faces detect, was it scanned or failed or did it not scan yet?

The list is nearly endless - but in line with the rest of the direction of MacOS, getting worse.

dav43 commented on Anyone can push updates to the doge.gov website   404media.co/anyone-can-pu... · Posted by u/mahkeiro
pipes · 6 months ago
In what way is it rich people taking money?

Time will tell but there's evidence that some government staff grew inexplicably wealthy while in office which would suggest corruption. Corruption in government is terrible for the average citizen, ask anyone from a country that suffers from a lot of it.

I really fail to see why auditing government spending is a bad thing?

dav43 · 6 months ago
I agree that in principle, it's great that spending is being checked and payments "audited" (I have no clue if thats actually what is happening, I assume it's no KPMNG audit). However, are they really being audited? Is this the manner in which this should be done?

I am not a Trump voter. I agree with the outcome they have stated - reduce stupid spending - but I have no idea thats the true motivation, the true goal and I disagree with the manner in which they are doing it. Just because you agree with the dictator doesn't make it right?

dav43 commented on 1 in 5 online job postings are either fake or never filled, study finds   gizmodo.com/1-in-5-online... · Posted by u/belter
dav43 · 7 months ago
In Singapore it’s a lot higher than this. There are government mandates that roles must be advertised - so companies planning on hiring internally will advertise roles, interview candidates then justify why no candidates are good fit then hire the original internal candidate.

It’s laughable.

dav43 commented on Passengers Say Turkish Airlines Flights Have Unwelcome Guests: Bedbugs   nytimes.com/2025/01/01/tr... · Posted by u/whack
kylehotchkiss · 8 months ago
> the attendant dismissed their concern

I found Turkish airlines staff to be more stand-offish than others when I flew with them internationally a few years back. Is that anybody else's everyday experience with them?

dav43 · 8 months ago
had a instance with them where I need the bare minimum of customer service. There was none. Won't fly again.
dav43 commented on HDMI 2.2 is set to debut at CES 2025   tomshardware.com/tech-ind... · Posted by u/thunderbong
dav43 · 8 months ago
The tech industry bodies need to look in the mirror at themselves rather ferociously. These usb, smart, hdmi bodies are failing the consumers, failing sustainable practices and just smell like poorly run bodies.
dav43 commented on Replace Philips Hue Automation with Home Assistant's   blog.frankel.ch/home-assi... · Posted by u/Tomte
philjohn · 9 months ago
If you're going down the HomeAssistant route I'd recommend ditching any proprietary hubs and getting a Zigbee dongle instead, I've been using the Sonoff Zigbee dongle and even though it's connected to the server, in a rack, in the garage (so thick breezeblock walls between it and the rest of the house) there are enough always on mesh nodes in the lights themselves that the network is steady. Granted, the Dongle itself isn't inside the enclosed rack, it's on a USB extension cable and mounted on the top.

I've also gone the Zigbee2MQTT (ZQM) route instead of the ZHM built into HomeAssistant as it just supports a lot more, and a lot better in my experience.

Once you're in the open world of Zigbee, you can also go the Ikea Tradfri route as well for bulbs that are a WHOLE lot cheaper.

dav43 · 9 months ago
Have gone this route and exposed a variety of manufacturers devices to HomeKit through HA. Been surprised by the reliability. 3 years in.
dav43 commented on Show HN: Outerbase Studio – Open-Source Database GUI   github.com/outerbase/stud... · Posted by u/burcs
dav43 · 9 months ago
Looks nice and make be helpful. Keep in mind I can’t and most people can’t install this in a corp environment. If you get a pip install or npm install I’d be able to use it through corporate mirrors internally.

Make it as easy to run as something like datasette.

u/dav43

KarmaCake day309November 6, 2014
About
Not a programmer
View Original