They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.
[0] With built-in networking and storage.
This achievement is clearly worth a lot to people.
It's nice, it's just nothing new.
I believe the episode name was "Virtualizing Time"
"no cables no assembly just cloud" wtf is that
No cables, except for a few cables.
To be super clear about it, this is referring to not needing to cable up all of the individual sleds to the rack upon installation. It doesn't mean that we recommend connecting a rack of compute to your data center via wifi.
I think their description conveys what it does just fine for the target audience.
The comment you replied to was not questioning the value of integrated cabling. It was pointing out that the product description on the site does not make sense.
"Cloud computer" sounds like a server you rent from AWS. It's kind of like calling Rust "cloud compiler."
If you choose to use words that your audience doesn't understand, or even worse understands to mean the opposite of what you want them to mean, it's a good idea to explain these words immediately using conventional words with conventional meaning. The comments by throw0101a did that.
The product seems really cool, but there is no way I would've understood what it was from the website.
And for what it's worth, I don't think you need to explain what's happening to Steve, it seems to me that he understands perfectly well. To me you come across as being rather condescending and in my opinion Steve is being commendably polite in response.
Cloud as a term is pretty undefined/overloaded. It makes sense to me if I think about 'cloud' computing as API-based provisioning on pooled resources.
Size of doorways, weight bearing capacity of floors, electrical service parameters, environmental conditions, etc.
e.g. Does it actually handle electrical voltage fluctuations of +/- 1V, or whatever is advertised?
The guaranteed parameters of a fully set up machine:
Minimum performance metrics, software compatibility with whatever the sales department promised, maximum power draw, etc.
e.g. Can it reliably hit X metric (FLOPS, IOPS, Integer calculations, etc.)?
And so on.
Or does the customer need to take on some risk and hazard associated with installation, configuration, initial boot up, etc.?
e.g. If someone buys with the intention of using it up to X FLOPS, and the machine only delivers Y FLOPS once it's all said and done, what happens?
Otherwise I would imagine it would be a major selling point and be advertised publicly.
But if it is a complicated enough internal issue for that not to be the case, then my apologies for pestering.
> hyperconverged cloud infrastructure
> no cables no assembly just cloud
When people hear cloud, it means that aspects such as electricity costs, electricity stability, Internet, bandwidth, fire protection, safety, etc etc are abstracted away.
Oxide IS on-premise, right? The website is very vague and wishy-washy.
What a waste of resources.
If you have a rack like that you'd probably want it to run a real database or server backend.
Ehrm. What is that exactly?
Are you alluding to cute design, different user interface? Or ditching then common PC component modularity? "Thinking differently"?
One difference is that Oxide development was done in the open and they don't seem hell bent on creating a closed ecosystem. (Yet, at least)
You are also correct that we diverge from Apple in other ways, such as our commitment to openness, rather than secrecy.
The first iMac famously made it easy to connect to the Internet; The 'i' in iMac was for "Internet". Its setup manual was a couple of pages long, mostly pictures and IIRC, just 37 words.
Sun did that experiment:
* https://en.wikipedia.org/wiki/Modular_data_center
Also:
* https://www.deltapowersolutions.com/en/mcis/data-center-solu...
* https://atos.net/en/solutions/high-performance-computing-hpc...
* https://www.zelladc.com/zella-max
Search for "shipping container data centre" or "containerized datacenter".
I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.
Oxide feels like it'll be that "integrated" experience, but with the added benefit of software freedom.
Also https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.htm...
Does illumos run on ARM?
I wish IBM would shrink their LinuxONE Rockhopper 4 Rack Mount down to at least an "under the desktop" model. To my knowledge, IBM still makes quality products and has excellent customer service. They have fun names too (Rockhopper and Emperor are types of penguins!) and they even have 3D models of their rack mount cloud computers with shadows. [3] In fact, when I first read about Oxide a year or two, I searched for "IBM cloud server", and left it at that. So IBM, could you please send someone from the LinuxONE down to Boca Raton to create our new PC? :) Thanks!
P.S. I would even accept a shrunken 370. :)
[1] https://en.wikipedia.org/wiki/IBM_Personal_Computer
[2] https://archive.org/details/byte-magazine-1981-01/page/n313/...
[3] https://www.ibm.com/demos/it-infrastructure/ux/product.html#...
And the debugger is here: https://github.com/oxidecomputer/humility/blob/master/cmd/ri...
I also gave a talk at this year's Open Source Firmware Conference that covers a bunch of debug strategies: https://www.osfc.io/2023/talks/unplugging-the-debugger-live-...
(conference videos aren't online yet, but should be posted early next week)
I design them in Monodraw, pass it through a janky converter I wrote that converts text into a json grid of characters. I then render a number of layers that get combined, which is a mix of the static art layer, and others generated from functions that spit out a similar cell based frame.
If you're interested: https://gist.github.com/benjaminleonard/c913ddbf23fe7a70f9c2...
And for what it's worth there's this ASCII game: https://twitter.com/StoneStoryRPG
So, Humility is like MC/DC?
https://en.wikipedia.org/wiki/Modified_condition/decision_co...
1. The state or quality of being humble; freedom from pride and arrogance; lowliness of mind; a modest estimate of one's own worth; a sense of one's own unworthiness through imperfection and sinfulness; self-abasement; humbleness.
-- Webster's 1918 Dictionary
Our avionics software was originally spec-ed to not NEED a debugger it was to be of such high quality, but the designer added break point op-codes in the VM anyway. I guess he was aware of the concept of humility, too. ;) Like humor, humility seems to be in short supply these days.
People claiming to be humble are all over the fucking place these days and Millennials are addicted to wanting that word like crack. The result is that the people bragging about being humble the most are usually narcissists.
Hmm...maybe I should have referenced:
https://en.wikipedia.org/wiki/Sacred_cow_(idiom)
> The motto of the satirical magazine The Realist was "Irreverence is our only sacred cow".
Humor does seem to exist today, it just has a narrow band-pass filter, with apparent focus on "organic" product marketing.
When I see the concept of "extremely high quality" applied to software in the aerospace domain, I tend to think of methods like formal proofs and "semi-formal" methodologies like DO-178C (including MC/DC, not as a tool, but as a critical sub-process). It does appear that Humility adheres to some traditional ideas used in even "safety conscious", real-time embedded work:
> However, Hubris may be more interesting for what it doesn't have. There are no operations for creating or destroying tasks at runtime, no dynamic resource allocation, no driver code running in privileged mode, and no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems. [1]
Oxide surely didn't invent all these concepts, we were applying some of this (using C) in the early '90s on human-safety-critical embedded projects, and they were most likely used before that. The attack surface eliminated is more than just security, e.g. even self-induced "attacks" from task priority inversion. Maybe the concept of applying them to rack mount servers is novel, but I have no experience there. Techniques like DO-178C go farther and do things like trace requirements to object code and strict coverage criteria.
When you are considering where Hubris/Humility might be applied in aerospace, also note that it would most likely be denied at the proposal stage of talking to a regulatory body like the FAA if you said, "The methodology for our brake controller is cloning a Github repo written by 100X S/W engineers in the Rust language, then modding that to fit." It's a process you have to follow in a documented fashion, from the start. Who knows--I certainly don't know everything--if you do have a identify reproducible process that improves S/W quality, the FAA might be interested.
And as far as eliminating debuggers goes, Boeing did actually do a joint academic/industry project on a zero-bug reduced (subset) Ada compiler (Zbra), which itself would be qualified as a tool instead of the more laborious process of mapping requirements to object code directly. [2]
In the spirit of humor shown on this topic, I would suggest considering a reduced subset of Rust called Iron. :) Regards, and best of luck!
[1] https://oxidecomputer.github.io/hubris/
[2] http://www.sigada.org/conf/sigada2002/SIGAda2002-CDROM/SIGAd...
* Memory access is protected, but addresses are not virtually mapped. The lack of paging of course ties strongly to the inability to dynamically add tasks.
* They wish they could have read-only shared libraries (viable since there's only one address space) so they could truly be shared, but the current ecosystem assumes mutability is possible (even though in Rust mutable globals are rare).
Can't wait to read its source code, I was curious since reading this thread[2].
[1] https://github.com/oxidecomputer/helios (if this is not found for you, we're in the same boat)
I would expect nothing less from that crew. They did amazing things at Sun Microsystems, Inc. (RIP), and they continue to do even greater things now.
https://github.com/oxidecomputer/hubris
And, that one underscore-delimited folder name in this repo just catches the eye, huh?
With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!
[0] https://github.com/oxidecomputer/hubris/commit/651a9546b20ce...
Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I like it, but it doesn't seem like a big deal or revolutionary in any way.
When it breaks, good luck!
You're no worse-off with Oxide from that perspective. Their open source firmware means that thr opportunity to pay somebody else to support you at least exists.
Even small shops can use bad experience as leverage for credits and discounts, especially if the vendor has account managers. This is one of the (few) benefits of having a human involved in invoicing vs. self-serve.
> when any of that breaks, good luck!
The premise is that you don't need luck, you can call Oxide. As you said, they wrote all of it, so they own all the interaction so they can diagnose all of it.
When I call Dell with a problem between my OS filesystem and the bus and the hardware RAID, there's at least three vendors involved there so Dell doesn't actually employ anyone that knows all of it so they can't fix it.
Sure, Oxide now needs to deliver on that support promise but at least they are uniquely positioned to be able to do it.
The rest comes down to execution. Sure, we all have high hopes for Oxide. Sure, we all hate established players like Dell.
Just about any company will sell your company a support contract.
The more interesting question is, can they back it up with action when push comes to shove? I suspect most people have plenty of stories of opening support tickets with big name vendors that never get resolved. And through the grapevine you find out that they won't fix it because they can't fix it. They might not even have access to the source code or anyone on staff who has a clue about it because it came from who knows where. Sales is happy to sell you the support contract but it doesn't mean your problems can be fixed. BTDT.
From listening to the Oxide podcasts, my impression is that Oxide actually can technically fix anything in the stack they sell, which would make them vastly different from Dell et.al.
I used to work for a company targeting Fortune 500s. At that level of spend, when a client had a problem, somebody got on a plane. Only a fraction of those problems escalated all the way to R&D, which is where Oxide skills are. That's where VMWare etc are hard to beat.
Can an engineering first approach break into the cloud market? Hard to say as enterprise sales is very powerful, and the numerous "worse is better" forces always loom large in these endeavours. That said, enterprise sales driven companies are fat, slow and complacent. Oxide is lean and driven, and a handful of killer use cases and success stories is probably enough to sustain them and could be the thin end of the wedge on long-term success. We can hope anyway.
I think the question is how well they can do the management plane. Dealing with the "quirks" of a bunch of grey box supermicro stuff is always painful in one way or another. The drop shipped, pre-cabled cab setups are definitely nice but that's only a part of what Oxide is doing here. No cables and their own integrated switching sounds nice too (stuff from the big vendors like UCS is closer to this ballpark but also probably closer to the cost too).
I suspect cooling and rack density could be better in the Oxide solution too, not having to conform to the standards might afford them some possibilities (although that's just a guess, and even if they do improve there these may not be the bottlenecks for many).
Docs:
* https://docs.oxide.computer/api/guides/responses
See perhaps "This repo houses the work-in-progress Oxide Rack control plane."
On-prem buyers are doing cost reduction and cost reduction targets things like, as one example, the crazy cost of GPU servers on the CSPs. Your run of the mill stuff is very hard to cost reduce.
You can see their sort of lack of getting it by using Tofino2 as their switch. That’s just a very bad choice that was almost certainly chosen for bad reasons.
The cost case only works for GPU heavy workloads which this isn’t - wrong chassis, wrong network, etc.
Tofino2 is the wrong choice because even when they made that choice it would have been clear that it’s doa. Intel networking has not been a success center in, well, ever. That’s a selection that could only have been made for nerd reasons and not sensible business goals alignment or risk mitigation.
When you make an integrated solution you’d better be the best or close to the best at everything. This does not seem to be the best at anything. I will grant that it is elegant and largely nicer than the hyper converged story from other vendors but in practical terms this is the 2000s era rack scale VxBlock from Cisco or whatever Dell or HPE package today. Marginally better blade server is not a business.
They also make a big deal and have focused on things no one who actually builds data center pods cares about.
I actually hope they get bought by Dell or HPE or SuperMicro. Those companies could fix what’s wrong here and benefit a lot from the attention to detail and elegance on display.
My experience with the likes of Dell is that they'll deliver it but they won't support it.
Sure, there's a support contract. And they try. But while they sell a box that says Dell, the innards are a hodgepodge of stuff from other places. So when certain firmware doesn't work with something else, they actually can't help because they don't own it, they're just a reseller.
Charging for EBS local snapshots on your own Outpost S3 storage and Route 53 Resolvers on your own compute is a weird one. I don't know how they defend that. To me, it seems indefensible.
> You can purchase Outposts servers capacity for a three-year term and choose between three payment options: All Upfront, Partial Upfront, and No Upfront. … At the end of your Outposts servers term, you can either renew your subscription and keep your Outposts server(s), or return your Outposts server(s). If you do not notify AWS of your selection before the end of your term, your Outposts server(s) will be renewed on a monthly basis, at the rate of the No Upfront payment option corresponding to your Outposts server configuration.
https://aws.amazon.com/outposts/servers/pricing/
> You can purchase Outposts rack capacity for a 3-year term … either renew your subscription and keep your existing Outposts rack(s), or return your Outposts rack(s)
https://aws.amazon.com/outposts/rack/pricing/
There is no permanent purchase option.
I do wonder if AWS will eventually go down market if oxide gets any scale.
It seems like AWS has all the pieces to compete with Oxide if they care to.
Is this true? Can you set your own root of trust for the firmware signing key and build and deploy it yourself?
DC build out cost and effort
Power cooling requirements
Dark fiber bandwidth requirement
New headcount to support all this
No thanks , I have widgets to make and sell and a business to run.
And beyond that, there's all sorts of weird environments that need a lot of local compute. You'd be shocked how many servers are on a cruise ship for one example among many.
Prices are here: https://oxide.computer/sales ;)
I am not in sales and so I hesitate to speak on it in case I am incorrect, but the way that I personally think about it is that it is true that it is not an inexpensive product: there's a LOT of computer here. But the goal is to be competitively priced.
I wish there was a 4U version (10-25K but I don't think that they could come close to that price point - regardless even that is out of reach for me to ever get to play on one :-/ )
One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).
Their coupling approach will most likely leave them perpetually behind and more expensive.
But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.
This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.
And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.
It seems like they're trying to hit a middle ground between cloud vendors and fully decoupled server equipment companies.
Using Oxide is likely cheaper over the life of the hardware than using a cloud vendor. A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
True, but Oxide may find themselves competing against Dell or HP if they adopt Oxides software for their respective servers. Additionally, Oxide may find itself competing against consultants and vendors in specialized verticals (e.g. core Banking software + Oracle DB + COTS servers + Oxide software). Oxide, and their competitors are going for people who used to buy racks of Sun hardware.
> Their coupling approach will most likely leave them perpetually behind
This is a startup that took years to get their initial hardware developed. The time between this version and the version using the next version of AMD chips will be shorter than the time it took to develop this product. This is not an inherent issue with coupling vs decoupling.
Also, most servers are rarely running on the most recent cpus anyway. At least in companies I've worked at with on-site hardware they're usually years (sometimes even a decade) out of date getting the last life sucked out of them before too many internal users start complaining and they get replaced.
I'm sure they'll improve their processes over time but the lag will probably always be a non-zero value. Hopefully they'll be able to keep it low enough that it's not an important factor but as a customer it's certainly something one should consider.
It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
The talk here talks about that from about 32:15 : https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
As to your second point, unless AMD somehow becomes supply constrained and only wants to ship to their most important customers first I don't see a future where there would be any lag. Again, the delay this time is from how long it took from company start until product release. Future delays will be based on the time it takes from them getting early development parts to released products, which they could even possibly beat Dell to market on given the smaller company size and IMO more skilled employees.
> It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I mean they've already hit tons of nasty issues, for example finding two zero-day vulnerabilities in their chosen security processor. They've shown they can work around issues pretty well.
I just think your premise is wrong - most customers don't care about not having the absolute latest and greatest. Indeed they will often avoid them because
1. They are new so more likely to have as yet undiscovered issues ( hardware or drivers ).
2. If you buy top end, they sell at a premium well above their performance premium.
ie the customers who are perennially chasing the latest hardware are in the minority.
1. The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
2. Not sure where you're getting this from. There is almost always a price:performance calculation that results in current generation smashing the previous generation with server and switch hardware. Often this means not buying the flagship chips but still the current generation.
And a major reason to get off old generations of hardware is that they become unavailable relatively quickly. It's always easier to buy current generation hardware than previous generation hardware, especially a couple years into the current generation. This has nothing to do with chasing the latest hardware.
That's not in the customers interests per se- in fact it's a pain. Having control of their own stuff could mean they could offer a much longer effective operational life.
> The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
What breaks stuff is change - sure when they are starting up it's higher risk - but again if they can manage the lifecycle better, not have change for changes sake, then they could be much more reliable.
> Not sure where you're getting this from.
I was talking about not taking the flagship stuff - which is typically a few months ahead of the best price/performance stuff.
IDK about every use case, but slightly older generations of CPUs would affect me roughly zero. I'm sure there are things so compute-intensive that one would care very much, but a lot of people probably wouldn't bat an eye about that, and not because they're unsavvy.
To the extent that these things are supported as a whole by the vendor rather than a bunch of finger pointing though, that could be massive, specifically in terms of how many staff members you could "not hire" compared to if you had to employ someone to both build and continually maintain it.
I'm posting this not to invalidate what you're saying, just to say that a little predictable upfront amount of money (the premium) will be spent very happily by lots of people who value predictability and TCO over initial price.
A moderately large Haswell cluster is equivalent in power to a moderately powerful modern server.
How is it ironic?
Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
Their competition has open source firmware as well:
https://www.dell.com/en-us/blog/enabling-open-embedded-syste...
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
And Canonical played with a cluster-in-a-box all the way back in 2013-2014: https://www.zdnet.com/article/canonicals-cloud-in-a-box-unde...
You could turn it into an OpenStack cloud in ~20 mins with an automated Juju OpenStack install.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
And you'll be down for weeks or months while they do it.
Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
With Oxide, you'd be lucky to get same day service.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Neither am I.
>If you hit a "zero-day" bug that Dell has never seen it's going to take time.
If you hit a "zero-day" bug that Oxide has never seen it's going to take time.
>And somehow every large customer finds bugs that Dell certification didn't.
Yes, happens. And I'm sure the exact same will happen with Oxide, so it's not a differentiator.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...
[0] https://github.com/oxidecomputer/phbl
[1] https://github.com/oxidecomputer/illumos-gate/tree/stlouis
[2] https://github.com/oxidecomputer/illumos-gate/blob/stlouis/u...
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
Huh, then why did Sun, Oracle, and Veritas have to set up a shared tech support center in San Jose?
"Accelerated finger pointing", said a friend who had to do business with them.
At least with Oxide there is a glimmer of hope for a better future in this regard.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
GNU/Linux was also "risky" at some point.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Or Linux running underneath all the Java-y Android stuff.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Now imagine a multi-million dollar mission critical pile of computers running on premises, and your sysadmin being able to do so.
Oxide is closer to a rack of Supermicros than AWS.
Solaris is still Solaris, as of the latest release last month. OpenSolaris hasn't been OpenSolaris in a while and is Illumos, yes.
Could be! Seems too early to tell though, and remains to be seen whether it pencils. Which is the whole idea of starting a new venture, no?
One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.
FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.
These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].
[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.
Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search.
Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that.
[1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.
>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.
that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.
They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.