Readit News logoReadit News
macdice commented on Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O   pganalyze.com/blog/postgr... · Posted by u/lfittl
nu11ptr · 7 months ago
Is this new async. I/O feature for Linux only?

I know Windows has IOCP and also now an IORing implementation of its own (Less familiar with macOS capabilities other than POSIX AIO).

https://learn.microsoft.com/en-us/windows/win32/api/ioringap...

Update: Most of the comments below seem to be missing the fact that Windows now also has an IORing implementation, as I mentioned above. Comparison article here:

https://windows-internals.com/ioring-vs-io_uring-a-compariso...

macdice · 7 months ago
I am not a Windows guy but I (with help) managed to get IOCP working for this in a basic prototype. Will share publicly soon. I also sketched out an IoRing version (if you are interested in helping debug and flesh that out let me know!).

Main learnings: the IOCP version can't do asynchronous flush! Which we want. The IoRing version can! But it can't do scatter/gather AKA vector I/O yet! Which is an essential feature for buffer pool implementation. So actually I am basically waiting for IoRing to add support for that before taking it too seriously (I can see they are working on it because the ops are present in an enum, it's just that the build functions are missing).

So my guess is that in a year or so we should be able to run all PostgreSQL disk I/O through IoRing on Windows. Maybe?

Another complications is that it really wants to be multithreaded (consuming completions for IOs started in another process requires a lot of hoop jumping, I made it work but...) This will resolve itself naturally with ongoing work to make PostgreSQL multithreaded.

The next problem is that IoRing doesn't support sockets! So in future work on async networking (early prototypes exist) we will likely also need IOCP for that part.

macdice commented on Waiting for Postgres 18: Accelerating Disk Reads with Asynchronous I/O   pganalyze.com/blog/postgr... · Posted by u/lfittl
kev009 · 7 months ago
A lot of work has gone into FreeBSD's aio(4) so it will be interesting to see how that works, because it doesn't have the drawbacks of Linux/glibc aio.
macdice · 7 months ago
BTW I have patches for PostgreSQL AIO on FreeBSD, which I will propose for v19. It works pretty well! I was trying to keep out of Andres's way for the core architectural stuff and basic features ie didn't want to overload the pipes with confusing new topics for v18 :-)
macdice commented on Of Psion and Symbian   abortretry.fail/p/of-psio... · Posted by u/klelatti
isr · a year ago
I feel your pain. I'm in the same boat as you (backed the astro slide - didn't get one).

I used the original Planet Computers device with psion-like keyboard, the Gemini, as my daily driver for a few years. The keyboard was no gimmick - it really felt like a quasi mini-laptop. A brilliant device, even considering the weak cpu (as compared to similarly priced phones)

Such a shame ...

(btw, had a psion 2, psion 3, psion 5 & sharp zaurus back in the day, so you can say I'm a sucker for these things)

macdice · a year ago
Surprisingly, they are still taking orders:

https://store.planetcom.co.uk/products/astro-slide

I suppose this means that they still have a firm intention to produce eventually, hopefully even with an updated chipset given that the original parts are lost ("ODM is effectively unwilling to release remaining produced stock and the pre-purchased chipsets"!) Obviously you know all that, fellow backer, but just in case anyone else is interested in this debacle:

https://www.indiegogo.com/projects/astro-slide-5g-transforme...

At this point I think of it as the Delorean of handheld computers...

macdice commented on Of Psion and Symbian   abortretry.fail/p/of-psio... · Posted by u/klelatti
macdice · a year ago
I had various Psion models in the 90s. Several of the people involved in the design of those things went to do modern-ish devices with Psion series 5-style keyboards, including the Cosmo Communicator. I was curious enough to back the Astro Slide 5G project, which attempted to "reverse the clamshell" to make a device with Psion series 5-style keyboard that closes to resemble a modern standard all-screen rectangular slab. It appears to have gone horribly wrong during final manufacturing at a Chinese factory that closed down during COVID, with thousands of IndieGoGo backers not having received their Astro Slide devices after several years (I am one). A shame. Anyway this article shows the Psion 5 and the Communicator side-by-side. That's a way out of date device now running ancient Android on a slow CPU, and sadly the refresh project seems to have died on the vine :-(

https://www.zdnet.com/product/cosmo-communicator/

[Edited for typos]

macdice commented on PostgreSQL Supported Platforms over Time   peter.eisentraut.org/blog... · Posted by u/ingve
sanxiyn · 2 years ago
Re: AIX. Why won't IBM pay for it? For example, from the case I know, IBM is paying for AIX support for Rust. I would think PostgreSQL deserves the same.

https://doc.rust-lang.org/rustc/platform-support/aix.html

macdice · 2 years ago
After it was dropped from PostgreSQL, a team from IBM showed up on the mailing list (see one of my other answers for link), so perhaps that is now going to happen! I think it's a case that they were using it, but didn't realise that the project was on the verge of dropping it for years due to lack of interested maintainer & resources. Open source is funny like that. Deleting it was certainly one way to get their attention.
macdice commented on PostgreSQL Supported Platforms over Time   peter.eisentraut.org/blog... · Posted by u/ingve
skissane · 2 years ago
Solaris runs on commodity x86-64 hardware, Oracle offers installers for download and a free license for development use, and there is an open source fork (Illumos) which is 99% compatible. Given all of this, it isn’t hard for open source developers to setup CI/dev/test environments to support Solaris x86-64 as part of their project. SPARC is harder (hardware availability), but if it builds on Solaris SPARC and tests fine on Solaris x86-64, it is probably going to run on Solaris SPARC too. There are some potential gotchas around memory model differences and endianness, but one can just punt those issues until a user reports them, and then say “we can’t fix it unless you let us use your hardware”

AIX only runs on non-mainstream hardware (POWER), plus IBM isn’t as open as Oracle is with providing access to it. This makes it significantly harder to support than Solaris is

Added to that, Solaris and Linux are closer to begin with. Historically, Linux copied a number of its APIs from Solaris; AIX was far less of an influence. Years ago, I had to port some psuedoterminal management code from Linux to both Solaris and AIX. I only had to make some minor changes to the Linux code to make it work on Solaris; the changes I had to make to get it to work on AIX were much more extensive

macdice · 2 years ago
Well said.

I've been able to debug PostgreSQL issues reported by [closed] Solaris users by booting illumos inside a virtual machine on my laptop in minutes...

I want to keep AIX support, for, well mainly irrational nostalgic reasons -- PostgreSQL used to run on pretty much the entire Unix family tree, and I wrote lots of code on AIX for a decade. But they make it hard. I don't know why an OS vendor wouldn't make an image of an OS available to developers as conveniently as possible (it is even possible to boot recent AIX versions on QEMU if you have the patience... the hypothesis is that they might have done work to make that possible, 'cause it didn't work in earlier versions; but you can't get an image of OS, so shrug).

macdice commented on PostgreSQL Supported Platforms over Time   peter.eisentraut.org/blog... · Posted by u/ingve
yjftsjthsd-h · 2 years ago
I'm not that shocked to see Solaris, but I am surprised to see Solaris and not AIX, which is also actively developed and I thought IBM specifically made an effort to keep assorted FOSS working on it.
macdice · 2 years ago
We (the denizens of pgsql-hackers@postgresql.org) didn't see any AIX users or developers or other interested parties in our community... There was a group in France, but they disbanded... However, after AIX was dropped in the PostgreSQL 17 development cycle, suddenly we heard from IBM who run products on it. So perhaps it will make a comeback in 18? That was the latest idea, anyway, let's see.

https://www.postgresql.org/message-id/flat/CY5PR11MB63921AB0...

macdice commented on PostgreSQL Supported Platforms over Time   peter.eisentraut.org/blog... · Posted by u/ingve
MBCook · 2 years ago
I’m a bit surprised to see Solaris. Is it reasonably popular in the business world still? Or is there just not a big burden to keeping it for now?
macdice · 2 years ago
POSTGRES was developed on SunOS (and its close cousin 4.3BSD or 4.2?). It's not so hard to support Solaris, as it was so influential that Linux uses a lot of similar things, including for example the ELF format and linker details, the sort of thing that developers have to maintain. So the burden is lower than (say) AIX. There is also someone from the Solaris community who feeds and waters a modern Solaris build farm animal (buildfarm.postgresql.org). Same goes for illumos (= forked from OpenSolaris, like Solaris 10), people run build farm animals and help test occasionally as required etc, which makes a big difference. That's my take, anyway (speaking as someone who works on PostgreSQL portability).
macdice commented on Open Sourcing DOS 4   hanselman.com/blog/open-s... · Posted by u/ndiddy
shanselman · 2 years ago
That's valid feedback. There is no clock, but there maybe should be. In this case, yes, Jeff and I had to PUSH. And that's a hassle. I'll ask around.
macdice · 2 years ago
This is fantastic work, thanks. Hmm, what else... let's see... Xenix also really, really wants to be free! What a magnificent piece of forgotten computer history it is. https://en.wikipedia.org/wiki/Xenix
macdice commented on ZFS 2.2.0 (RC): Block Cloning merged   github.com/openzfs/zfs/pu... · Posted by u/turrini
rossmohax · 2 years ago
Does ZFS or any other FS offer special operations which DB engine like RocksDB, SQLite or PostgreSQL could benefit from if they decided to target that FS specifically?

u/macdice

KarmaCake day556April 3, 2016
About
PostgreSQL and FreeBSD hacker
View Original