News for hackers. Stuff that matters.
Re:Oxide Computer releases distribution of illumos intended ... by refulgentis on Monday January 29, 2024 @02:17PM (#39181020)
Can you get me excited? I spent 20 seconds browsing the homepage and walked away with "so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?"

But immediately got myself to "what does a server OS do anyway, doesn't it just launch VMs? You don't need Linux, just the ability to launch Linux VMs"

Tell me more? :)

Re:Oxide Computer releases distribution of illumos intended ... by mustache_kimono on Monday January 29, 2024 @02:37PM (#39181316)
> so the idea is vertical integration for on-premise server purchases? On custom OS? Why? Why would people pay a premium?

As I understand it, re: vertical integration, the term is actually "hyperconverged". Here, that means it's designed at the level of the rack. Like -- there aren't per compute unit redundant power supplies. There is one DC bus bar conversion for the rack. There is an integrated switch designed by Oxide. There is one company to blame when anything inside the box isn't working.

In addition, the pitch is they're using open source Rust-based firmware for many of the core components (the base board management controller/service processor, and root of trust), and the box presents a cloud like API to provision.

If the problem is: I'm running lots of VMs in the cloud. I'm used to the cloud. I like the way the cloud works, but I need an on-prem cloud, this makes that much easier than other DIY ways to achieve (OMG we need a team of people to build us a cloud...).

Re:Oxide Computer releases distribution of illumos intended ... by steveklabnik on Monday January 29, 2024 @02:59PM (#39181625)
The terminology in this space is confusing, but "hyperconverged" isn't really what we're doing. I wrote about the differences here: https://news.ycombinator.com/item?id=30688865

(That said I think other than saying "hyperconverged" your broad points are correct.)

Re:Oxide Computer releases distribution of illumos intended ... by mustache_kimono on Wednesday January 31, 2024 @09:06AM (#39203910)
> but "hyperconverged" isn't really what we're doing.

This is fair, especially as it follows a discussion of how you're building at the rack level, which is more like a mainframe than current "hyperconverged" offerings. Recommend others read the linked post.

Re:Oxide Computer releases distribution of illumos intended ... by SteveNuts on Monday January 29, 2024 @02:31PM (#39181236)
It seems like the folks on HN tend to think the world runs on AWS (I'm not trying to say they don't have a huge market share), but many huge enterprises still run their own datacenters and buy ungodly amounts of hardware.

The products that are on the market for an AWS-like experience on-prem are still fairly horrible. A lot of times the solutions are collaborations between vendors, which makes support a huge pain (finger pointing between companies).

Or, a particular vendor might only have compute and storage, but no offering for SDN and vice-versa. This sucks because then you have two bespoke things to manage and hope they work together correct.

These companies want a full AWS experience in their datacenter, and so far this looks to be the most promising without dedicating huge amounts of resources to something like Openstack.

Re:Oxide Computer releases distribution of illumos intended ... by lijok on Monday January 29, 2024 @02:42PM (#39181376)
Wouldn't a "full AWS experience in their datacenter" be AWS Outpost?
Re:Oxide Computer releases distribution of illumos intended ... by mustache_kimono on Monday January 29, 2024 @02:44PM (#39181421)
> "full AWS experience in their datacenter"

... Including the bill!

Re:Oxide Computer releases distribution of illumos intended ... by lijok on Wednesday January 31, 2024 @10:32AM (#39205004)
Part of AWS managed services offering. AWS Managed Wallet
Re:Oxide Computer releases distribution of illumos intended ... by mardifoufs on Monday January 29, 2024 @02:49PM (#39181479)
Is AWS outpost truly a full AWS stack/experience? I thought it wasn't actually meant to be a "data center in a box" experience, but more so a way to run some workloads locally when you are already using AWS for everything else.
Re:Oxide Computer releases distribution of illumos intended ... by PeterCorless on Monday January 29, 2024 @06:49PM (#39184396)
Some data products will run successfully in AWS Outposts. Others will not. For example, AWS itself can't run DynamoDB in an AWS Outpost. It recommends users to run ScyllaDB in DynamoDB compatible mode.

e.g., https://www.scylladb.com/2020/09/15/scylla-cloud-on-aws-outp...

Disclosure: I worked at ScyllaDB.

Re:Oxide Computer releases distribution of illumos intended ... by adfm on Monday January 29, 2024 @02:46PM (#39181449)
With DHH and others promoting a post-SaaS approach (once.com, etc.) we might see hardware refresh as cost-cutting. Astronomical compute bills and lack of granularity bring all things cloudy into sharp focus.
Re:Oxide Computer releases distribution of illumos intended ... by threeseed on Monday January 29, 2024 @07:42PM (#39184879)
What they are doing is SaaS by stealth.

You buy their product once, but it only has bug and security fixes for 3 years.

Which means every business is going to need to upgrade on a cycle anyway.

Re:Oxide Computer releases distribution of illumos intended ... by mlindner on Monday January 29, 2024 @08:12PM (#39185136)
People don't usually throw out their server hardware after 3 years. After 3 years is up they'll probably sell service plans. And with the code being all open source some owners may go the self-supported route, though probably most will buy service plans.
Re:Oxide Computer releases distribution of illumos intended ... by threeseed on Monday January 29, 2024 @08:52PM (#39185453)
Was actually referring to DHH and the 37signals products.

Whilst I think we will see a trend back towards more on-premise hardware I don't think SaaS is going away anytime soon. And in fact it's arguably better for everyone because the software is being continually maintained.

Re:Oxide Computer releases distribution of illumos intended ... by refulgentis on Monday January 29, 2024 @04:17PM (#39182754)
The "(finger pointing between companies)" took me from confusion to 100% understanding, was at Google until recently. It was astonishing to me that it was universally acceptable to fingerpoint if it was outside your immediate group of ~80 people.*

Took me from "why would people go with this over Dell?" to "holy shit, I'm expecting Dell to do software and make nvidia/red hat/etc/etc etc/etc etc etc help out. lol!"

* also, how destructive it is. never, ever, ever let ppl talk shit about other ppl. There's a difference between "ugh, honestly, it seems like they're focused on release 11.0 this year" and "ughh they're usless idk what they're thinking??? stupid product anyway" and for whatever reason, B made you normal, A made you a tryhard pedant

Re:Oxide Computer releases distribution of illumos intended ... by _zoltan_ on Monday January 29, 2024 @02:51PM (#39181519)
OpenStack is pretty smooth sailing these days and I bet you it would be much cheaper to just get 3 FTEs for your OpenStack install than an Oxide rack
Re:Oxide Computer releases distribution of illumos intended ... by linksnapzz on Monday January 29, 2024 @03:41PM (#39182221)
Where, exactly, are you getting these 3FTEs qualified to touch production OpenStack infra, for more than a year, where their aggregate cost is less than a rack of equipment?
Re:Oxide Computer releases distribution of illumos intended ... by gtirloni on Monday January 29, 2024 @03:54PM (#39182432)
The rack doesn't require FTEs?
Re:Oxide Computer releases distribution of illumos intended ... by linksnapzz on Monday January 29, 2024 @04:44PM (#39183116)
Not three of them; it ought to be about as difficult to administer as a single rack of hw, +Vsphere, if that.
Re:Oxide Computer releases distribution of illumos intended ... by _zoltan_ on Monday January 29, 2024 @05:13PM (#39183477)
if you need OpenStack you're not running one rack, but a couple dozen.
Re:Oxide Computer releases distribution of illumos intended ... by KAMSPioneer on Tuesday January 30, 2024 @02:52AM (#39187434)
That...sounds like a market segment you've just discovered for Oxide.
Re:Oxide Computer releases distribution of illumos intended ... by milon on Monday January 29, 2024 @02:26PM (#39181148)
Having a solid on-prem rack product to me is a great thing. I like IaaS services a lot, don't get me wrong, and I think they're the right pick for a bunch of cases, but on-prem servers also have their "place in the sun", so to speak :) I could present any number of justifications that I don't think I'm qualified enough to defend, but the gist is that at the bare minimum, I'm glad the option exists.

As to why I'm personally excited: I enjoy the amount of control having such an on-prem rack would afford me, and there surely could be a great amount of cost-savings and energy-savings in many scenarios. Sometimes, you just need a rack to deploy services for your local business. I like the prospect of decentralizing infrastructure, applying all the things we've learned with IaaSes.

Re:Oxide Computer releases distribution of illumos intended ... by bionsystem on Tuesday January 30, 2024 @01:49AM (#39187106)
In the last 10 years and 6 different clients/employers I worked there is pretty much no way to run production on the cloud. Only 1 of them had some stuff running in the (GCP) cloud at all.

Of all of the 6 infrastructures I've seen, only 1 of them is half decent, with 6 dedicated teams around the datacenter working closely together (by dedicated I mean, nothing is required of them concerning the core software product that the company develops). Network, Unix/Virtu, Windows, Storage, PC, and datacenter. That's 30+ people just to run a couple big datacenters and a few more server rooms. The service was actually quite good with VMs/zones delivered under an hour and most tech issues solved in half a day. The other infrastructures were either bigger or smaller, with more or less people, and were all terrible, sometimes needing weeks of email exchanges with excel attached to get a single VM.

AWS was the dream everywhere I went for everybody. Oxide may be coming out with a product that will solve a LOT of issues. SmartOS/IllumOS has all the tech to be self-sufficient (virtualization, storage, SDN...), add support for networking and storage and you get a complete product that a handful of people can run (well, you still need a windows team in most cases but fine).

Re:Oxide Computer releases distribution of illumos intended ... by lijok on Monday January 29, 2024 @02:40PM (#39181354)
> Why would people pay a premium?

I would pay a premium just to not have to deal with HPE, DELL, etc

Re:Oxide Computer releases distribution of illumos intended ... by _zoltan_ on Monday January 29, 2024 @02:52PM (#39181524)
Dell's been nothing but fantastic for us (compute, not storage.)
Re:Oxide Computer releases distribution of illumos intended ... by sarlalian on Monday January 29, 2024 @05:22PM (#39183572)
Dell is a mixed bag depending on how well the individual region you are dealing with is doing overall. Things were great for us, but something changed and now getting good support for hardware failures has been a nightmare of jumping through hoops, time zone handoffs to other teams, and forced on-site techs to replace a stick of ram.
Re:Oxide Computer releases distribution of illumos intended ... by throwup238 on Monday January 29, 2024 @02:21PM (#39181079)
The best elevator pitch I've heard is "AWS APIs for on-prem datacenters". They make turn-key managed racks that behave just like a commercial cloud would with all the APIs for VM, storage, and network provisioning and integration you'd expect from AWS, except made to deploy in your company's datacenter under your control.
Re:Oxide Computer releases distribution of illumos intended ... by magnawave on Monday January 29, 2024 @03:48PM (#39182343)
I guess the wildcard is price.

AWS's pricing model works kinda at their OMG eyewatering scale - aka all the custom hardware they design is highly cost optimized, but just doing custom hardware has a notable cost. This is easily covered by their scale, to make for their famous margins. [during their low scale times, they did use a good bit of HP/Dell, etc]

Oxide seems to be no different (super custom hardware) only major difference being the "in your datacenter" part. Since you own the cost of your datacenter, Oxide has to come in a lot cheaper to even compete with AWS, but how do you do that with low volume [and from the look of it not-cost optimized, but instead fairly tank-like] bespoke hardware? Feels like the pricing / customer fundamentals are going to be pretty rough here outside perhaps a few verticals.

Re:Oxide Computer releases distribution of illumos intended ... by sarlalian on Monday January 29, 2024 @05:27PM (#39183629)
Datacenter costs are weird. The first big cost is having a datacenter. However once you have the space, power, cooling and that part makes sense, then the actual hardware going into it can have a pretty decent premium and still be highly competitive with AWS. It will also depend heavily on what you are doing and producing, if the answer to that is a large amount of data, and it needs to transit out of AWS, suddenly the cost of a pretty large datacenter is really cheap in comparison. AWS egress fees have a markup that will make your accountants panic. From a hardware standpoint, once you need GPU compute or large amounts of RAM, the prices get pretty dumb as well.
Re:Oxide Computer releases distribution of illumos intended ... by kaliszad on Monday January 29, 2024 @06:20PM (#39184135)
Oxide seems to be a lot more efficient than a rack full of 1U servers with each having 2 PSUs + 2 ToR switches + 1 management switch somewhere for all the OOBMs. All those little fans and power conversions eat a lot of power, the fans and the PSUs all cost something too. Also, have fun managing all of that in a secure manner or debugging anything at all. Once you add the VMware licensing you might end up with more or less the same cost up front and quite likely higher overall cost. And I am not even beginning to talk about racking/ stacking of the whole rack. I haven't seen much support even when Dell/EMC owned VMware and together produced the VXRail lineup and the company I used to work for was presented as the reference project in Saxony, Germany at that time. All of the boxes would add up to about 2 standard racks but it was representative of the other bigish customers in that area and time.

I imagine, some of the customers will order 1-2 racks half full and over a few years possibly add a few sleds, these will probably demand great GUI/ manual experience and possibly competitive Oracle/ SAP/ MSSQL benchmarks and I can imagine Veeam integreation. Other customers such as the DoE or some big enterprise customers will order whole rows of racks and demand perfect automation options. That is just a guess.

Re:Oxide Computer releases distribution of illumos intended ... by jjav on Thursday February 01, 2024 @03:29AM (#39213852)
> Oxide has to come in a lot cheaper to even compete with AWS

Which should be pretty easy. I don't know the exact costs but in a previous Oxide discussion the number 1M was thrown around. If that's roughly correct, that is comfortably less than a single year of AWS bills at most startups I've been in (except the very tiny ones < 15 people).

Haven't seen any performance numbers either so admittely estimating here, but from what I know about building racks of 1U servers and knowing that Oxide is more efficient, I can believe an Oxide rack should handily outperform the AWS VMs we (the startups) were paying >>>100K/mo for.

If these numbers are anywhere in the ballpark, an Oxide rack should easily be saving quite a bit of money already by year two.

Re:Oxide Computer releases distribution of illumos intended ... by kortilla on Monday January 29, 2024 @02:26PM (#39181138)
That’s the elevator pitch for open stack
Re:Oxide Computer releases distribution of illumos intended ... by steveklabnik on Monday January 29, 2024 @04:08PM (#39182618)
You are not wrong that OpenStack is sort of similar in a sense, but the difference is that Oxide is a hardware + software product, and OpenStack is purely software.
Re:Oxide Computer releases distribution of illumos intended ... by capitol_ on Monday January 29, 2024 @02:29PM (#39181199)
That just sounds like a bunch of api's on top of linux.
Re:Oxide Computer releases distribution of illumos intended ... by throwup238 on Monday January 29, 2024 @02:30PM (#39181222)
Just like Dropbox is a bunch of APIs on top of FTP.
Re:Oxide Computer releases distribution of illumos intended ... by adamnemecek on Monday January 29, 2024 @02:56PM (#39181594)
One company making both HW and SW generally leads to really good, integrated experiences. See e.g. Apple.
Re:Oxide Computer releases distribution of illumos intended ... by 0cf8612b2e1e on Monday January 29, 2024 @03:16PM (#39181872)
I am really hoping the broader industry takes note. By owning the platform, the Oxide team was able to dump legacy stuff that no longer makes sense.
Re:Oxide Computer releases distribution of illumos intended ... by throwawaaarrgh on Monday January 29, 2024 @02:36PM (#39181305)
It's a mainframe. If you can't get excited for mainframes it'll be hard to be excited about this.

IllumOS is the OS/360 to Oxide's System/360. (It won't get that popular but it's a fair enough comparison for illustrative purposes)

Re:Oxide Computer releases distribution of illumos intended ... by linksnapzz on Monday January 29, 2024 @03:42PM (#39182245)
It's a mainframe, for people who do not, actually, know what a mainframe is or does.
Re:Oxide Computer releases distribution of illumos intended ... by kaliszad on Monday January 29, 2024 @06:23PM (#39184166)
I bet it costs a fraction of what a similarly powerful mainframe would cost. However I don't think the customers for each overlap that much. If you need a mainframe, you need one and there is no discussion about possible alternatives because there are none.
Re:Oxide Computer releases distribution of illumos intended ... by panick21_ on Monday January 29, 2024 @03:06PM (#39181735)
Except that it use the same standard CPU as commodity machine. Doesn't have much of the extra reliability stuff. It can go from vertical to horizontal scaling. The OS is open source Unix. And yeah its not like a mainframe at all really.