macdice

I am not a Windows guy but I (with help) managed to get IOCP working for this in a basic prototype. Will share publicly soon. I also sketched out an IoRing version (if you are interested in helping debug and flesh that out let me know!).

Main learnings: the IOCP version can't do asynchronous flush! Which we want. The IoRing version can! But it can't do scatter/gather AKA vector I/O yet! Which is an essential feature for buffer pool implementation. So actually I am basically waiting for IoRing to add support for that before taking it too seriously (I can see they are working on it because the ops are present in an enum, it's just that the build functions are missing).

So my guess is that in a year or so we should be able to run all PostgreSQL disk I/O through IoRing on Windows. Maybe?

Another complications is that it really wants to be multithreaded (consuming completions for IOs started in another process requires a lot of hoop jumping, I made it work but...) This will resolve itself naturally with ongoing work to make PostgreSQL multithreaded.

The next problem is that IoRing doesn't support sockets! So in future work on async networking (early prototypes exist) we will likely also need IOCP for that part.

BTW I have patches for PostgreSQL AIO on FreeBSD, which I will propose for v19. It works pretty well! I was trying to keep out of Andres's way for the core architectural stuff and basic features ie didn't want to overload the pipes with confusing new topics for v18 :-)

Surprisingly, they are still taking orders:

https://store.planetcom.co.uk/products/astro-slide

I suppose this means that they still have a firm intention to produce eventually, hopefully even with an updated chipset given that the original parts are lost ("ODM is effectively unwilling to release remaining produced stock and the pre-purchased chipsets"!) Obviously you know all that, fellow backer, but just in case anyone else is interested in this debacle:

https://www.indiegogo.com/projects/astro-slide-5g-transforme...

At this point I think of it as the Delorean of handheld computers...

I had various Psion models in the 90s. Several of the people involved in the design of those things went to do modern-ish devices with Psion series 5-style keyboards, including the Cosmo Communicator. I was curious enough to back the Astro Slide 5G project, which attempted to "reverse the clamshell" to make a device with Psion series 5-style keyboard that closes to resemble a modern standard all-screen rectangular slab. It appears to have gone horribly wrong during final manufacturing at a Chinese factory that closed down during COVID, with thousands of IndieGoGo backers not having received their Astro Slide devices after several years (I am one). A shame. Anyway this article shows the Psion 5 and the Communicator side-by-side. That's a way out of date device now running ancient Android on a slow CPU, and sadly the refresh project seems to have died on the vine :-(

https://www.zdnet.com/product/cosmo-communicator/

[Edited for typos]

After it was dropped from PostgreSQL, a team from IBM showed up on the mailing list (see one of my other answers for link), so perhaps that is now going to happen! I think it's a case that they were using it, but didn't realise that the project was on the verge of dropping it for years due to lack of interested maintainer & resources. Open source is funny like that. Deleting it was certainly one way to get their attention.

Well said.

I've been able to debug PostgreSQL issues reported by [closed] Solaris users by booting illumos inside a virtual machine on my laptop in minutes...

I want to keep AIX support, for, well mainly irrational nostalgic reasons -- PostgreSQL used to run on pretty much the entire Unix family tree, and I wrote lots of code on AIX for a decade. But they make it hard. I don't know why an OS vendor wouldn't make an image of an OS available to developers as conveniently as possible (it is even possible to boot recent AIX versions on QEMU if you have the patience... the hypothesis is that they might have done work to make that possible, 'cause it didn't work in earlier versions; but you can't get an image of OS, so shrug).

We (the denizens of [email protected]) didn't see any AIX users or developers or other interested parties in our community... There was a group in France, but they disbanded... However, after AIX was dropped in the PostgreSQL 17 development cycle, suddenly we heard from IBM who run products on it. So perhaps it will make a comeback in 18? That was the latest idea, anyway, let's see.

https://www.postgresql.org/message-id/flat/CY5PR11MB63921AB0...

POSTGRES was developed on SunOS (and its close cousin 4.3BSD or 4.2?). It's not so hard to support Solaris, as it was so influential that Linux uses a lot of similar things, including for example the ELF format and linker details, the sort of thing that developers have to maintain. So the burden is lower than (say) AIX. There is also someone from the Solaris community who feeds and waters a modern Solaris build farm animal (buildfarm.postgresql.org). Same goes for illumos (= forked from OpenSolaris, like Solaris 10), people run build farm animals and help test occasionally as required etc, which makes a big difference. That's my take, anyway (speaking as someone who works on PostgreSQL portability).

This is fantastic work, thanks. Hmm, what else... let's see... Xenix also really, really wants to be free! What a magnificent piece of forgotten computer history it is. https://en.wikipedia.org/wiki/Xenix

Reflinks and copy_file_range() are just landing in OpenZFS now I think? (Block cloning)

... but the BSDs did have support for SystemV binaries, through modules compat_svr4 and compat_ibcs2, so the grand parent comment made perfect sense to me. Not sure why there were two and I don't plan to go down that rabbit hole today, but perhaps due to SVR4 vs SVR3 differences, and ELF vs COFF, shrug.

Fun read. I'm pretty sure there were ways to run SCO SysV binaries unmodified on Linux (and I think FreeBSD and others), back in the day. Maybe something to do with ibcs2?

One technical problem is the WAL records changes to "shared" relations (stuff like the catalogues of databases and users that are global to the whole "cluster", not just one database), and those are mixed up with changes to per-database objects (ie most WAL activity).

Thanks! Author/developer here, happy to answer any questions about this topic.

The nostalgia is overwhelming. Learned BASIC, Z80 and 6502 assembler from these little cartoon robots.

Me too. But I think it led to more learning and less gaming! I did finally get a Speccy... recently!

(Author of the code here) Yeah :-)

FWIW I just removed pg_attribute_always_inline from those three comparator functions in tuplesort.c and it made no difference to the generated code under GCC 10 at -O2. If we can show that's true for Clang and MSVC too, maybe we should just take them out. Edit: But I don't really mind them myself, they do document the intention of the programmer in instantiating all these specialisations...

Speaking as the author of that change, and also as a long time user of C++ (in other projects), I hear ya! But I don't have the power to make PostgreSQL switch to C++. Years ago, std::sort() vs qsort() was almost a standard example of how C++ can be faster than C.

src/lib/sort_template.h is an example of a pseudo template function, but we also have at least one pseudo template container in src/lib/simplehash.h.

And if you want to see some "policy based design" (C++ community term) in PostgreSQL C code, check out src/backend/exec/nodeHashjoin.c. It uses forced inlining to "instantiate" two different versions of the hash join executor code, one with parallelism and one without, so that we don't have all the "am I in parallel mode?" branching/code size in the generated code.

It's interesting that uniq is spelled unique in that video. I wonder who changed that!

As a Firefox/FreeBSD user occasionally annoyed by this nonsense (not Chase but other things), but not being knowledgeable about modern web standards evolution, I wonder if https://wicg.github.io/ua-client-hints/ will fix this by killing User-Agent headers.

I haven't followed it or tried it, but supposedly you can boot up a pure open source Darwin system. Certainly it has been very useful for me as a cross-platform developer to be able to browse its source and understand quirks of system calls etc. Are you saying that key parts you'd like to review are closed source? Hmm, the new file system is closed source, which is a clue, but I don't know to what extent Darwin is "different" from what macOS is really running...

Some time in the early-ish 90s when I was a student I worked at a tiny place doing software development, targeting various commercial Unixes. Development was done on a lowly 486 running some variant of SCO System V UNIX, and I vaguely recall that the cost of licensing the system with compilers was thousands of dollars (you had to pay a lot of money for each little feature of the OS too, like TCP, SMP, ...). Or it would have been, if it hadn't been for some guys at the little ISP next door who gave us a copy of GCC, which was a gateway to running Slackware Linux, which was pretty soon where all future development was done before transferring it over to customers' HPUX boxes or whatever.

I was just a kid catching the tail end of all that, but there are still a couple of hard-to-get-your-hands-on big iron Unixes like that; I've worked on most of them (ie those and many that are gone), and I must say it's surprising to me that, by now, you still can't just download an ISO and run it on qemu or whatever (actually you can, but it's probably illegal). I could forget about them entirely, if I didn't have to keep the open source stuff I work on working on them too, with limited access and no CI...

I incorrectly assumed Norse and old English genders and cases ought to line up neatly and wouldn't have caused the whole system to be thrown out the window on contact. I stand corrected. Thanks!

https://en.wikipedia.org/wiki/Gender_in_English#Decline_of_g...

From a nearby reply there is a clue that it's a new Americanism. I have to resist the urge to send a complaint... I'm not ready to become an old geezer who complains about things changing. But I admit that I grit my teeth when people talk about their 'gas bill' and I genuinely don't know whether they're talking about petrol or heating gas, or say "gotten" instead of "got", etc etc; there's no way to say "but we're supposed to use British words here" without sounding like an arse...

Edited: I guessed wrong, see Tagbert's answer. It was Norse contact (erm, the ones in the north, not the Normans who were French-speaking ethnic Norse people in the south).

I thought it probably had more to do with the influence of Norman French. The Germanic gender and declension system couldn't survive the influx of French lexicon, as the two languages with completely incompatible grammar and genders coexisted and eventually merged into a new frankenlanguage.

There was a recent article I can't locate right now claiming that for some period of time, English scribes writing late middle English (?) used to add an extra 'e' (and maybe more endings) to various words following old grammatical rules that involve knowing the gender and case, but began to forget the rules and eventually it decayed into randomly adding 'e' to sound smart, which is the basis of modern joke forms like 'ye olde shoppe'.

As an English speaker it's interesting to compare with some of our closest cousins that have also lost a grammatical gender or two:

I don't know too much about Dutch, but as far as I can tell they went from the 3-gender German system to the 2-gender common/neuter system (like Scandinavia, yellow in the diagram) in the last few hundreds years, but their language is still full of frozen phases from the 3-gender system. I wonder, can a modern Dutch speaker actually create new phases using the words 'ten', 'ter' (= German zum, zur, "to the"), 'des', 'der' and maybe more? You'd have to know if a common gender noun was masculine or feminine, but I guess, unless you also know German, you won't be able to do it.

I believe the Afrikaans language of South Africa, a dialect of Dutch, is down to just one gender. IIUC they had to give up using Dutch school books some time in the past century, because (vast over simplification, I'm sure) they couldn't remember if nouns were common or neuter. I wonder if they also have frozen phrases that reflect the 2-gender, and perhaps even 3-gender grammar of the ancestral language.

The Scandinavians have a few pockets of dialects that still use 3 genders. I'm not sure how they maintain that, given that, as far as I know, the prestige forms and most literature are down to 2!

AFAIK the last people to speak a cousin language with more than one gender in the British Isles were the Norn speakers, who maintained a 3-gendered Norse dialect up until the mid 19th century.

I'm in New Zealand and I've never heard that; I feel like I might have heard it somewhere when I lived in England though. There are definitely oddities in NZ though; the NZ press uses the word "trespass" in a very strange way that seems to be somehow backwards: https://www.stuff.co.nz/timaru-herald/news/126753711/timaru-...

One I've noticed that is US vs UK/NZ/AU/...: "bring him with you" vs "take him with you", which has a directional component in British usage that I can't quite explain (away from the speaker maybe?) that the Americans don't do.

Linguistics nerd answer: sounds like "autotune" if you have the caught-cot vowel merger and alveolar flapping:

https://en.m.wikipedia.org/wiki/Cot%E2%80%93caught_merger https://en.m.wikipedia.org/wiki/Flapping

Interesting developments there:

https://docs.microsoft.com/en-us/windows/win32/api/ioringapi...