A very old, but still key lesson here for product makers: when you know that you're doing new, early-gen work with a kinda reasonably high probability of something faulting - you need to provide stellar, high touch support, whatever it costs. And you need to plan to be ready for that from the point when you ship your first thing.

(I think this applies both to hardware and software products.)

Of course, this is easier said than done, but ultimately this is the core function - are your customers happy or not?

As a CTO you need to prepare the company and your investors for this failure mode. You need to be able to explain to them that sometimes making sure that the product actually works and the existing customers are happy is more important than that sexy new feature of the month. It's not always a very easy balance.

In my experience, even semi-technical people with a more sales/marketing kind of focus in the company often get blindsided by this. It's on us to prep them for this.

It seems like this company learned it the hard way. The key thing is that they did learn.

For what it's worth, we do have a focus on making sure the hardware works properly etc.; I've always been pushing for that and haven't had any real problems there.

This particular board was a test escape. A very unique, thankfully singular (so far as I know) test escape, but there's always a chance for that -- the larger failure in my mind was the breakdown in the support process. With our new controls in place (including rapid escalation / RMA of "odd" faults observed by customer) I am confident this won't happen again.

And yes, we're updating tests to catch the newly discovered failure mode. It's an interesting one, the board is a sort of a "zombie" in that if you boot it up by looking at it just right, it'll run stably and pass our stress tests, but it should never have left our factory in that condition due to the other faults. Full stop. :)

That particular horror story led me to not equip our development team with a dozen POWER9 machines, which is a pity, because the combination of almost-total trustworthiness and enormous firepower was extremely attractive for our particular niche (plus, I used to hand-code PPC64 AltiVec “velocity engine” code for HFT funds back when Apple sold G5 Xserves).

I’m glad to hear that it was a one-off and that you have revised your support strategy, but... really, it made for a pretty frustrating read. It’s not chump change, and going out on that kind of a limb on... trust... which was clearly unwarranted at least at the time of that initial incident.

There’s nothing more frustrating than paying top dollar for something exotic and then being stonewalled by oblivious “tech support” when things don’t work as advertised, particularly when something is sufficiently novel or unusual that relatively few other users are out there to proffer help and the potential points of failure are plentiful and “unknown unknowns”.

As far as I’m concerned, that incident was extremely bad publicity. That board could’ve been shipped to me. I’d have been left to dangle from my solitary rope and squirm when management asked me what was going on.

That’s just not acceptable, sadly.

If I were you I'd be a lot more comfortable pulling the trigger after reading all of this.

Being worried over what might have been spoilt milk if you had gotten a broken board in the past seems counterproductive.

The new machines I eventually decided to go with were ordered more than a week ago.

Sorry to hear that. If you ever get tired of worrying about untrustworthy firmware phoning home, DRM, etc., please consider us for the replacement purchase? ;)

It also won't happen at this point, ever. Is there anything we can do to help you revisit this decision?

Nope, too late. The decision was made and orders have been placed. You don’t get do-overs, sorry.

I might still get a machine for my own amusement, but... I don’t know, reading of some poor guy left dangling for best part of a year just doesn’t build confidence and leaves me with a “there but for the Grace of God go I” feeling, you understand?

I understand. I also know this happened to no one else, and it was a pathological case in a support system that has since been completely overhauled. This is one reason why we started the self-serve RMA process -- to show that yes, we are committed to replacing defective hardware in a timely fashion. If the machine is defective, and we can't help via our online troubleshooting guide and Email / phone support, send it right back for replacement.

That being said, the assembled systems products are checked with a significant burn-in test etc. before they ship. We have never had an assembled system arrive defective anywhere, except if it was damaged in shipping (i.e. crushed box etc.).

I’m not trying to guilt-trip you or anything... I really like the idea of having non-monoculture machines and I’ve always been a sucker for unusual hardware (NeXT, BeBox, IBM RS/6000, dual Pentium machines to name but a few)... but if the one prominent story I find online is about a poor guy who is left to cry his eyes out regretting his courageous and expensive decision while the company that supplied him is apparently indifferent (this was before the happy if belated epilogue) it’s just an instant turn-off as far as I’m concerned. You’re producing something unusual and new, and users take a risk and pay far above market prices for the privilege to be on the bleeding edge... the least one can expect is competent and timely technical support. This poor guy went through “please reinstall your OS”-level indifference for months. That’s not the way one builds confidence. And yeah, I’m glad you’ve revised your policies, but in my particular case it came too late. This guy bought a Ferrari-analogue and got a Trabant with Lada-level tech support. He’s been pretty polite in writing up a follow-up detailing your company’s long-overdue resolution of his issues. I doubt he’s asked you to refund you the months of lost productivity and the time he spent essentially volunteering his time to serve as a free engineering consultant. That’s just not good enough. Somebody should’ve swooped in before this poor guy became so frustrated he wrote it up and submitted it to HackerNews.

No DRM, full-ownership computing is great, and the performance is literally astounding (particularly for our use-case), but at the time the decision was made, based on the evidence I had, the decision was clear. I do not regret deciding not to buy your product because based on the data available to me at the time as far as I knew I would’ve been alone and/or I would have spent a fortune sending faulty items back-and-forth through customs, paying import duty each time.

Whoever decided not to be proactive in this guy’s case procured your firm a sizeable amount of bad publicity.

Not to mention that the team I am part of would’ve probably, as a side-effect of the proprietary software we are developing, also contributed various patches and so forth to various parts of the underlying open-source stack.

No problem, I understand. In your shoes I may have even done the same thing.

Just in case you didn't see them (they're not exactly easy to find with Google etc. for some reason), there are a number of positive reviews or comments online from various people. I know one thing we haven't (yet) done is to start assembling all the success stories around our hardware; this is always challenging due to the number of people that use the hardware without posting online for privacy reasons etc.

Without further ado:

Slightly out of date (the software now available for POWER is significantly more robust than it was at the time -- we even have Fedora desktop ISOs showing up now for POWER):

http://tenfourfox.blogspot.com/2018/05/a-semi-review-of-rapt...

Various other (more recent) comments from our user base:

https://twitter.com/RobinBa40108750/status/11790782959813632...

https://twitter.com/tsundeoku/status/1171414555089494016

https://twitter.com/alnesbit/status/1170425263559254017

https://twitter.com/klauskiwi/status/1164273128588972032

https://twitter.com/notgrubles/status/1161438996586803200

https://mastodon.sdf.org/@jirka/102520125253219962

Just wanted to show a little bit of the other side of things, for context. :)

I think the point is well-made and flogging it doesn't really add much. Raptor can't change what happened, only what happens next.

I totally feel your pain. You do so much work to keep things super structured and still bizarre defects escape or only show in the wild.

In fact I'm debugging a device in the wild behaving very strangely at this exact moment.

> the larger failure in my mind was the breakdown in the support process.

100% agreed. This is where your company failed. And now has redeemed itself.