nigel_bree

Interesting. Both timely, given the recent discovery of the massive failure to disclose the asset hiding in his family trust by Carl Bates, yet also rendered less pertinent because of it - in reality a wide swathe, and possibly a majority of government ministers now are routinely not disclosing substantial gifts at all, let alone other forms of what are indisputably corrupt transactions.

The use of trust structures for tax efficiency is one thing, but they are so opaque they largely make a mockery of disclosure both because the size of the trust's asset base is opaque and often the trustee structures are pure paper fictions. It's still good to have this though, well done you.

> Are these people you know actively applying for better jobs and not getting them?

No, because the decades have worn them down. It's been a long time since there have ever been jobs in NZ advertised requiring hard development skills, and the tiny handful that do come up in public tend to be for very specialized verticals where they made hard demands on past experience in that niche area first and foremost over everything else.

> The other is happy with the chill job they have

I mean, I can't blame them for that - there are lots of toxic employers, ageism, credentialism, etc etc. If you're just going to be underemployed doing kiddie-level work anyway, better the devil you know, particularly if you have a family to take care of.

It's all just a big old mess of market failure, though. The problem isn't that the talent pool isn't there, it's that since there's no VC money around, the firms that _really_ need that talent can't pay what the hungrier younger people (who want more than anything else to get on the FAANG gravy train) want and the fantastically talented folks mostly can't earn any kind of premium remotely matching their business value and so make lifestyle choices it's hard to pry them out of.

> But paradoxically there is simultaneously a lack of top tier within New Zealand

Really? I'm aware of some extremely top-tier and wildly underemployed talent - the problem is that the NZ market has almost no companies that need or are interested in good people with hard skills - it's almost all very basic web dev. Pretty much all the veterans I know around my age are doing work beneath them to pay the bills or have switched out of development completely.

Symantec left our team - which was basically identical before and after acquisition - pretty much alone beyond adding in some (badly needed) release process and i18n requirements.

Almost the entirety of the growth in the size of the imaging executable, which did get hugely bigger, came from a constant drive to add capability to the NTFS support to match the FAT support, most crucially to allow the images to be edited in Ghost Explorer. The initial NTFS support that Ghost had prior to the Symantec releases was really crude, basically the content in the .GHO file wasn't files, but a raw-ish dump of used disk extents that it tried to always put back in the same place to avoid having to fix up attribute runs, whereas the FAT16/FAT32 content was basically a file archive where all the filesystem allocation metadata got recreated on the fly.

Customers wanted and pushed hard to have NTFS images editable, and that made life really hard - the approach that was ultimately taken meant creating a full read/write NTFS implementation, and those aren't small. And the design of that code interacted really badly with the C++ exception implementation in DJGPP (which before that work had begun, I had warned them about), so that eventually exception frame information was taking up ~25% of the on-disk size of the UPX-compressed binary!

The big thing I think with any book is to aim to inspire people (especially younger ones) to create for themselves, and while there's probably a way to do that with an exegesis of the existing source I don't know what it would be. The approach you take in your books is one that I think is inspiring to people to get them trying these things on their own - which is after all where the real hard learning happens.

That's particularly true because so many point design decisions made in commercial codebases like Coherent were conditional on a wider context - both technological and business - that just doesn't exist any more, and that I think gets in the way of helping younger people reaching a deeper understanding and appreciation of the design decisions made.

Recontextualising that code, and re-examining those decisions in the light of the modern day is just more interesting to me, and as well provides the chance to lead people through the process of solving the design problems in a way that's narratively easier to follow. Bootstrapping forces an order on what you do that gives a particular structure to the presentation, while at the same time we have aspirations about what future elements we want to achieve that shape and give context to all kinds of decisions as we work out how to get there.

After showing how to build something today, we can do a compare-and-contrast against the existing Coherent code that's more meaningful, especially because we can then measure the outcomes. Anyone can do (and lots of people do) ill-informed hand-wavy postulations about what-ifs, but the process of actually building things that do work forces your hand in ways that armchair enthusiasts miss.

Hey Nils, love your work - back around '90 when I joined Mark Williams I was a big Scheme fan, so your whole t3x.org site and "Scheme 9 from Empty Space" particularly puts a smile on my dial.

I do think the fact that Coherent really hewed 100% to the original V6 approach based on the basic design constraints of bootstrapped minimalism was part of the charm. Doing the 4.0 release process I really pushed myself hard to get as much POSIX compatibility in as humanly possible, but the balancing act was the size-coding part - making /bin/sh still be 16-bit while having $(()) and shell functions and able to process those insane MB-sized GNU autoconf scripts that embedded >64kb here documents, etc. Another big part of why the size-coding style mattered was keeping the system comprehensible since with only a few developers everyone was spread so thin, but that's again part of the fascination.

I've always wanted to give Coherent a bit of a conceptual reboot for modern x86 (particularly multicore, which means a complete new kernel) but staying to the 16/32-bit size and with a style similar to your books walking through the process of making it and how all the design choices need to interlock to make that possible is something I wish I had the free time for.

Definitely true. And as well, the level of innovation that brought, particularly in places like Australia/NZ or the (then) East Germany/Ukraine, the outrageous cost of components made for a whole generation of quite exceptional "frugal" engineers. UNIX itself was - despite being in the US - born out of a similar level of necessity.

One thing about Coherent (and why I periodically fiddle about with implementing replacement, modernized kernel designs for it, albeit not in C) is that exactly like V6 UNIX it is an exercise in that kind of design, but being for the x86 and partially modernised to an early-90's level it's considerably more accessible than V6 to anyone who wants to look at the source code today to learn about that.

The classic "Nautilus" manual was really interesting on two sides; one was the quality of the writing, which was all down to Fred Butzen - he just had a real knack for clarity and working with him on documenting the work I did during the 4.0 process was really enjoyable. Basically the process was that developers would write up drafts of everything to give to him, and he'd then edit them for clarity before passing them back for an iteration to ensure that none of the accuracy was lost. He just had an excellent sense of when there was going to be a better explanation than the classic dry manpage style.

The other notable thing about that manual was just the economics of getting a perfect-bound book of that sheer size printed and included in the USD99 price tag.

Yes, and also notably Dave Conroy of MicroEMACS fame. I do believe that as with QNX and MKS Systems and Watcom, Coherent was mostly built by former graduates of Waterloo university - their computing department is responsible for a great many notable products.

Yes, prior to 4.x processes were strictly limited to 64kb code and 64kb data - essentially the equivalent to what MS-DOS programmers would call the small memory model only. There are a bunch of reasons for this, all of which stem from the fact every part of Mark Williams C and the assembler/linker toolchain hewed so closely to the exact model set by V6 UNIX - including the same .o and .a formats, hence the same limitations all down the line.

Since so much my previous work before joining work on Coherent was on embedded toolchains and DOS development, the fact that so little use anywhere was made of x86 segmentation was surprising to me, but I knew all the V6 internals well enough to adapt back.

Even in 4.0 where the COFF and the iBCS2 syscalls all worked, actually most of the userland still stayed as 16-bit small model since the aim was to keep most of the installation process the same, which was that essentially the first install floppy was a reasonably full live system. Obviously things grew a ton and during the evolution of 4.0 we slowly had to move the bulk of the 32-bit userland components off to packages in later floppies.

It's not based on V6, although it was basically created as a clone of it (and on the PDP-11, no less - the x86 versions of Coherent until 4.x were basically just minimal ports that kept the PDP-11's classic 16-bit flavour). It was a very very good clone, enough that a lawsuit was threatened and Dennis Ritchie himself got pulled into checking that it wasn't derived from any UNIX source - see dmr's comment at https://groups.google.com/g/alt.folklore.computers/c/_ZaYeY4...

You'd likely find the experience pretty miserable, not so much because of the C compiler - Steve did a great job with that - but actually because of the filesystem. Coherent was around for a long time, remember; it used a very classic V6 UNIX filesystem, complete with odd endianness of 32-bit fields in the inode and most significantly, the classic 14-character file name length.

That right there will be an even bigger showstopper for the build scripts of most things than anything else. During the early 90's we could run GCC and Comeau C++ and all that kind of thing on Coherent fine because those codebases were still built for wide portability even though things like BSD FFS had opened that up - there were still plenty of machines running System III and System V code that most people took care not to create gratuitous collisions if truncation happened. Even shortly post the advent of Linux, that had changed dramatically and basically turned on its head and the effort involved in backporting to any flavour of classic UNIX (not just Coherent) quickly became almost impossible.

Yeah, just to confirm that for v7.1, we got a full source license of it for Symantec Ghost because Microsoft had withdrawn permission for us to distribute anything at all to do with MS-DOS, and we had a lot of customers still using DOS boot disks. Prior to that I believe we had a distribution license for PC-DOS but around that point IBM no longer wanted to maintain it and we had small things we needed to do, so we got the source license and just distributed builds we made as part of the Ghost Boot Wizard disk builder.

What was more interesting from a software archaeology point of view was when I found Symantec had acquired the assets of the then-defunct Quarterdeck; so the source for QEMM and Desqview eventually turned up in the corporate Perforce server and that was certainly fun to read, having made all kinds of things for QEMM back in the day, like a replacement overlay manager for Turbo Pascal (and Turbo C++) that would run overlay code segments directly out of the EMS page frame - this meant pretty much any program we built with the Borland compilers could be turned into a TSR with only the data segment resident.

The one thing that I never did find, though, was the source for the Whitewater Group's Actor. When it was announced that Symantec had acquired those assets from Borland I tried as hard as I could to see if the Actor source code was included but no-one I was ever able to find knew about that. Actor 3.0 was my first introduction to the Smalltalk way of building software, and I still sometimes fire it up in a VM to play enough with its VM format.

Not sure about the specific text the parent had in mind but "Principles of Interactive Computer Graphics" by Newman and Sproull (1979, but I bought it in '81 in the student edition and it influenced me a lot in my youth) made a bit of the connection with Renault in the chapter on Bezier Curves - many of its other examples of advanced graphics (at the time) were on flight simulators and such built by Evans and Sutherland, where the "Evans" was one of Alan Kay's mentors at the University of Utah.

I love this article, and have recommended it many times in discussion with young people learning programming in other forums (although it's rare for them to come back to me to say they'd read it, so...). There are some small differences in the social dynamics when it comes to mentoring in the workplace, but the vast majority of this is still incredibly on point. Thanks so much for writing it.

Anyone from HN who wants to can drop by on the way to a more traditional tourist spot like Mangawhai or further north (although over summer I'm often away at the weekends racing stockcars). The phone number for the library redirects to our house if it's not open.

The cheese shop is fine! I know Keith and Marita who have owned it for many years, they are good people - Ad Clarijs who founded it still occasionally runs cheesemaking classes, his place is not far from mine. Also worth a visit is the amazing Cafe Eutopia next door - https://g.page/eutopiacafe?share - which is a marvel.

It just boggles my mind how US firms don't realise how poor a value proposition - especially to people of an age to have a family or strong community ties, like the roots I have here - they offer to developers in AU/NZ when demanding that people relocate to work for them, and I'm not really optimistic that COVID-19 will really bring any significant change to their corporate cultures. I am hopeful that perhaps more firms in AU may cotton on to the fact that there's a substantial pool of exceptional veteran talent in NZ who are really underutilized, which really represents an opportunity for them if they were to make a serious attempt to hire here and let people work remotely.

I currently live in Kaiwaka not far from Mangawhai and have since Symantec bought out Ghost in '98, as that helped me afford a first house. It's a wonderful place to live. I did a lot remotely back then and once Symantec closed in NZ a decade ago I've been 100% remote from home ever since.

While I don't have an ocean view, and only about 2ha of land, it's a wonderful place to be. However, there are pretty significant drawbacks in getting gigs as there just aren't many local firms doing anything at all intellectually challenging and US firms still just apparently aren't interested in talent in this country who don't want to suffer being forced to relocate to awful places to live.

I don't see that changing, either; the fact is that people who really place proper value on a quality lifestyle tend to be older, and few firms really place much value on older talent either. Younger people are cheaper and more willing to suffer abuses such as moving to unpleasant, expensive metro areas in countries with alien values.

> I think I later upgraded to a 100MB IDE disk, but was still lucky.

Well, it really wasn't luck as much as keeping with the electrical specs. Do that, you'd never see a problem, and the later IDE "cable select" schemes did really help to mitigate a lot of the damage from improperly terminated cables.

> Anyway, thanks for contributing to a great OS!

Well, other than living in infamy due to introducing that bug, I didn't start there until the push to turn 4.0 from a tech demo into a real product. So really all the credit for 3.2 and earlier which set the foundation for Coherent belongs to the other guys, many of whom were long gone by the time I got there like Dave Conroy (who wrote the MicroEMACS I loved to use) and Randall Howard (who went on to found MKS). There were some great people from the earlier days still there though, like Norm Bartek, Hal Snyder and La Monte Yarroll who where there when I joined and of course Steve Ness who was the sole man behind the MWC C Compiler (much as Fred was responsible for that remarkable manual).

Also worth a mention, among all the other notable characters I remember fondly is one of the support/QA folks at MWC: Jim Leonard aka Trixter, who became a notable demoscene figure - https://trixter.oldskool.org/2015/04/07/8088-mph-we-break-al...

Yeah, cross-subsidised products like that were an issue. But it's worth noting that part of the fine line we had to walk was that lots of the people who bought Coherent had hardware on which something like Solaris would have been totally unusable in practice, at least unless they spent US$5K on brand-new machines with enough grunt.

For context, while we we working on 4.x I recall Norm Bartek having a crack at installing one of the early Windows NT 3.1 beta releases on one of the beefiest machines we had in the office. After grinding over half the day copy stuff to the install partition, then when it went to actually boot it just bluescreened because the machine had "only" 12Mb of RAM (which was the minimum spec for release NT 3.1, but the betas were built in debug mode and so needed more).

There was a big gap between what the "workstation" folks thought PC hardware was like and the reality of what most people could actually afford, hence why the minimum spec target for Windows 95 - a couple years later! - was still only down around 4Mb, even though that still tended to page hard.

So, for us, although it sometimes took fairly heroic efforts in size-coding, by keeping everything we possibly could still as 16-bit, our install floppy was actually a fully live system, without any tricks like compressed filesystems expanded to RAMdisk or anything. You booted off the install floppy to the 32-bit kernel, with the regular init and login and /bin/sh and everything - the installer was a shell script that just helped walk through fdisk and mkfs and then did an untar of the remaining stuff on later floppies (with things like the C compiler and libs) but right from the install floppy you could quit (or Alt-F2 to an alternate console TTY) to a root shell and have a usable system that happily ran in 1Mb of RAM.

So, there was still a niche for Coherent (and similarly, QNX), albeit that it wasn't a large enough one to bridge it through the growing pains of the changes at the time, especially given the forces that would shortly wipe out AT&T and DEC and almost all the other UNIX vendors despite their size.

It's interesting, given what I then went on to, to compare that niche to some of the customer response to Ghost. Even well after 2005 we had plenty of paying customers there who absolutely clung to being able to build and boot to DOS via floppies, even though we did have to use the RAMdisk trick once the UPX-packed Ghost binary exceeded about 2Mb by itself (a lot of which was due to a mistake I told the other devs not to make around the time Symantec acquired it, which was to rely on C++ exception handling; alas after several years, something like about 40% of the size of the binary was just GCC's exception metadata).

> What killed it is they, for some reason, spent time and treasure port XWindows to it instead of working on a TCP/IP stack

Just to be clear, there's a whole heaping helping of really big work required to get there either way, since Coherent was still essentially a 16-bit OS that started out as a V6 clone on the PDP-11 and hadn't really kept up during the 1980's with any of what was happening in the wider UNIX world beyond, oh, the Seventh Edition.

The key starting point for Coherent 4.x was essentially a single piece of work (almost all of it provided in a single .c file) that let a 3.x kernel boot into 386 protected mode and provided a system call handler that more-or-less emulated the call sequence specified in the Intel iBCS specification and mapped the iBCS system calls to the original 16-bit Coherent 3.x ones, and a very basic loader for statically linked SVR3 COFF files.

[ See https://en.wikipedia.org/wiki/Intel_Binary_Compatibility_Sta... which essentially formalized a subset of what the SCO port of System V did. Note, that omitted a number of system calls; I'm fairly certain that the iBCS manual didn't include any of the socket-related system calls anyway, so the public spec for what people took to mean "will run SCO binaries" left a lot to be desired. ]

Everything else around that was still, essentially, 16-bit code that was written around some variation of Sixth Edition or Seventh Edition, not System III/ or System V (so to be iBCS compliant the bible was the System V Interface Definition, aka SVID), let alone any of the things that were being added to POSIX.1 or POSIX.2

So, the work required for releasing the V4.x series of Coherent included a massive number of upgrades of almost everything, along with all kinds of awkward technical compromises to a) not break any existing things from the 16-bit userland, which stayed 100% supported, b) navigate between SVID and POSIX because while we on the one hand wanted to be able to run SCO binaries (hence, do things per the SVID) while we had all kinds of enthusiasts really wanting to use the ability to 32-bit code to run various shar archives of GNU project code unmodified.

So, I'd just note about that 16-bit userland from Coherent 3.x was that virtually everything on the (nominally 32-bit) version 4.x install floppies - all the standard utilities, pretty much everything that could be - still was built as 16-bit a.out binaries to keep the install size down. Pretty much everything we did in terms of C libraries and such had to be, and had to stay, buildable both 32-bit and 16-bit from the same source since both subsystems were needed. At some point we could conceivably have (at a technical level) cut over to all-32-bit, but just from the point of view of media duplication, doing that and still staying at a $99 price point wouldn't fly, and also the resulting increased memory footprint wouldn't have made existing customers happen since the kernel didn't have any support for demand paging, at a time when lots of people had just 1-2Mb (or 4Mb for a relatively beefy system) of RAM.

[ So for all the work required to let /bin/sh able to run successfully run multi-megabyte autotools scripts that tried relying on POSIX.1 shell features? Yup, everything involved in POSIX.2 support was all done with code that stayed 16-bit. ]

Without getting too much into the weeds of all the other various challenges involved getting a system like this out, for TCP/IP specifically the question was what kind of TCP/IP stack to pursue and most critically of all the question of where were the NIC drivers going to come from. There was not going to be any plausible way we could work with third-party SCO driver binaries, which were the most common type of UNIX driver that NIC vendors did provide at the time; SVR4.2, however, had STREAMS and the DDI/DDK so there was a plausible, reasonably portable driver model which seemed like a realistic future, but just licensing Lachmann and trying to bolt that into the kernel wasn't realistic at the $99 price point (and at the time, there weren't much of any third-party NIC drivers either).

To make 4.x happen we had to come from a very very long way behind, and getting to where we needed to go while staying within the constraints of a small staff and that $99 price point just made for an incredibly tight set of constraints. None of the resulting tradeoffs were fun to make. We did our best.

> The bug was triggered when your drive when into a very common thermal recalibration mode

As the person responsible (alas), my specific recollection of this particular bug was that the root cause wasn't thermal recalibration, but rather UDMA signalling errors.

Prior versions of Coherent using PIO mode had excruciatingly slow access, and when adding support for UDMA I also added support for the disk driver to recognise sequential access and issue multisector transfer requests; this boosted performance fairly massively, something like 3-4 times for some common things, and it was run for a fairly long time in-house and by beta testers with no trouble before it shipped.

The problem though, was a small - literally one line - arithmetic error when the drive end of things reported a UDMA transfer error had occurred in the middle of a multisector operation; the error-handling code that set up a retry of the operation didn't compute the start kernel address correctly when a whole bunch of transfers had been merged (and some subset had worked).

The primary problem with the UDMA modes was sensitivity to correct cable termination - see https://en.wikipedia.org/wiki/Parallel_ATA#Cable_select for some of that; basically, signal reflections from parallel ATA cable runs that didn't have terminating resistors made things electrically marginal and some systems would have really excessive numbers of UDMA CRC faults as a consequence, and given sufficiently high error rates and really bad timing that could end up polluting the buffer cache with stuff that was skewed by a sector :-(

The big thing (on top of not having any in-house hardware that triggered this specific bug) was the sheer volume of work required for those releases, since getting from what was basically a fairly vanilla Seventh Edition UNIX to where it needed to be to start running large pieces of third-party code expecting POSIX was a big lift. Since there weren't many people, everyone was having to wear lots of hats; for instance, aside from kernel work I did a huge amount of work for POSIX.1 and .2 compatibility and on top of doing the underlying code changes (which ranged all over the system, particularly for some of the stuff we ran into Autotools scripts relying on) all of those needed documenting, too.

[ Fred Butzen did amazing work writing the actual manpage text and making it really easy to understand - he justly deserved the credit for the quality of the manual in terms of its readability. But the scale of the changes needed to bring so many parts and pieces from V7 to POSIX meant lots and lots and lots of work trying to iterate over docs for technical accuracy at the same time as having to redesign all the affected parts and pieces. It was, in a word, exhausting. ]

Genuine Ghost - aka "Symantec Ghost Solution Suite" was indeed file-level, built on its own filesystem code, although the NTFS support was initially fairly rough and the image file format use for NTFS in .GHO containers wasn't a pure file archive as it was for FAT32. That original Ghost was developed in the mid-90's and purchased from Binary Research; development on that continued in Auckland until 2009 when everyone but myself and another engineer was laid off (he and I maintained the product and helped transition it to mothballed state, and then we were laid off mid-2010).

PowerQuest's Drive Image product was block-level, and that's the product sold as "Norton Ghost" and which has been cancelled (although like Symantec Ghost, everyone who worked on it has long been laid off; both ended up in maintenance-only mode looked after by separate teams in Symantec's facility in Pune).

Later on, "Norton 360" ended up with another separately acquired file-level backup product not related to either genuine Ghost or PowerQuest's product line (or any of the Veritas products either).

Neither the corporate Symantec Ghost - genuine Ghost, ex Binary Research - nor the consumer Norton Ghost - ex-PQ evolved out of Drive Image - shared any code at all, except for a small DLL used to allow the Symantec Ghost product to read from V2i images created by the consumer product and treat them the same as VHD or VMDK files when we added support for using those.

The branding confusion stems from when PowerQuest failed and was acquired by Symantec. One of the ex-PowerQuest managers was put in charge of not just the rump of PQ, but also PCAnywhere and genuine Ghost, and this resulted almost immediately in genuine Ghost (which made more net profit than PowerQuest had revenue) being cancelled in 2004, and most of the staff assigned to supporting PowerQuest's legacy Drive Image product or writing plug-ins for another acqusition done at the same time as PQ (On iCommand, later eventually withdrawn since it was unfit for sale). PowerQuest's V2i windows product, which included a volume snapshot driver for Windows 2000 source from StorageCraft (founded by ex-PowerQuest people), was rebranded and released as a consumer product called "Norton Ghost" version 9.

The genuine Ghost product was at the time sold to consumers as Norton Ghost 2003, and to businesses as Symantec Ghost Enterprise Edition 8. The choice of "Norton Ghost 9" as the branding for the ex-PowerQuest product was done to position it as not just consumer but also as the successor to the genuine business product (despite it having none of the capabilities of that) to leverage off our brand, although what it mostly accomplished was to poison our brand and confuse our customers.

[ Symantec Ghost 8 was rebranded with a minor update issued as "Symantec Ghost Solution Suite v1" when development was cancelled and it went to maintenance-only; we put out an update by borrowing ahead on our maintenance budget to get out a 1.1 update to that, and then after the Veritas merger genuine Ghost was uncancelled. When the new Veritas managers took over supervising both our team and the ex-PQ one and saw that we still made substantially more money than they did, they unwound some of the changes made to kill out line of business - but not all, and from then on our funding was kept too low for us to advance the product rapidly and we were unable to replace staff. As a result although Symantec Ghost Solution Suite v2 and 2.5 eventually followed, in April 2009 we were just about to enter beta for 3.0 when the team was disbanded and the site closed.]

Using the ex-PQ product as a consumer backup product was reasonable, as it was a dumb block-based product, but thanks to the storage driver they installed combined with VSS it had differentials, which genuine Ghost didn't. However, branding the consumer PQ product as Ghost was a disaster for the genuine Ghost product, both in terms of causing massive brand confusion, and thanks to the politics ensuring we never got properly funded from 2004 onwards.

An interesting experiment came from later, post-Veritas attempts to reduce the costly damage to the Ghost brand that "Norton Ghost" was causing; the exact same consumer product was released as "Norton Save and Restore 1.0", with a decent marketing push. Exact same product, it sold about 1/10 as much without the "Ghost" brand. Although the business Symantec Ghost did many times the revenue of the PQ consumer "Norton Ghost", this was enough to ensure that the (deliberately created) branding confusion was never properly fixed.