News for hackers. Stuff that matters.
Re:Oracle built a 1,060 node Raspberry Pi cluster by NohatCoder on Thursday November 21, 2019 @03:46PM (#21599404)
Now load any piece of Oracle software onto it and get charged for running 1060 CPUs. Whoops.
Re:Oracle built a 1,060 node Raspberry Pi cluster by MR4D on Thursday November 21, 2019 @04:27PM (#21599834)
“The only thing that scales better than an Oracle database is their licensing cost! “

- heard years ago from a friend. Not sure of the origination of the quote.

Re:Oracle built a 1,060 node Raspberry Pi cluster by walrus01 on Thursday November 21, 2019 @05:45PM (#21600567)
Whenever I see any technology news about Oracle, I mentally translate it as "Larry Ellison needs a bigger yacht".
Re:Oracle built a 1,060 node Raspberry Pi cluster by tyingq on Friday November 22, 2019 @08:06AM (#21605181)
He owns an entire Hawaiian island. It's like he's a real life Bond villian.

https://en.wikipedia.org/wiki/Lanai

Re:Oracle built a 1,060 node Raspberry Pi cluster by n0rbwah on Friday November 22, 2019 @09:13AM (#21605671)
Oh, and there are 3000 people living there? It's like he's some kind of feudal lord.
Re:Oracle built a 1,060 node Raspberry Pi cluster by bitwize on Friday November 22, 2019 @08:24AM (#21605304)
I saw an article written by an Oracle employee about "why Java is relevant in $CURRENT_YEAR". And what was the illustration accompanying the article? The coffee cup logo? A graph? A cartoon? Nay.

It was a photo of Larry Ellison's yacht.

And suddenly the intended audience of the article came into focus.

Re:Oracle built a 1,060 node Raspberry Pi cluster by rowanG077 on Thursday November 21, 2019 @04:00PM (#21599558)
Oracle doesn't charge per CPU it charges per core. Enjoy paying for 4240 cores!
Re:Oracle built a 1,060 node Raspberry Pi cluster by malux85 on Thursday November 21, 2019 @05:05PM (#21600218)
I don’t know why you’re getting downvoted, you’re correct!
Re:Oracle built a 1,060 node Raspberry Pi cluster by zepearl on Thursday November 21, 2019 @05:17PM (#21600331)
Aha - I stubled upon this https://jira.mariadb.org/browse/MDEV-10872 yesterday and thought "why would they want to achieve compatibility with Oracle DBs?" => now (I think that) I do understand that it would be to offer an alternative that would provide the same capabilities (but maybe with a different performance/reliability/whatever) for a different price, for any software that Oracle sells which wants to run on an Oracle DB, right?

Is the process "Google vs. Oracle about Java" (I understood that it's about the functions/methods declarations and not about the code itself) similar to a theoretical similar situation where DB XXXX would provide the same functions as the ones provided by the Oracle DB?

Re:Oracle built a 1,060 node Raspberry Pi cluster by tyingq on Friday November 22, 2019 @08:03AM (#21605162)
It's 4240 cores total, Oracle Enterprise DB is $47,500 per core. So ~$201M.

Java SE is priced per processor. $17.50 per month at that CPU count. And so, a relative bargain per year at 17.50 x 12 x 1060 = ~$222k / year.

Re:Oracle built a 1,060 node Raspberry Pi cluster by chrisseaton on Thursday November 21, 2019 @04:03PM (#21599592)
I believe there's a scaling factor, so lower powered cores like this are charged as a fraction of a Xeon you'd normally use.
Re:Oracle built a 1,060 node Raspberry Pi cluster by tyingq on Friday November 22, 2019 @08:30AM (#21605353)
ARM isn't listed on their table, so it slots into "all other multicore CPUs". Scaling factor is 1. So no help there.
Re:Oracle built a 1,060 node Raspberry Pi cluster by Yuioup on Thursday November 21, 2019 @03:15PM (#21599025)
That's what Azure feels like sometimes, running like a cluster of Pis.
Re:Oracle built a 1,060 node Raspberry Pi cluster by partisan on Thursday November 21, 2019 @03:44PM (#21599376)
I really do wonder why Azure is so slow. You have to optimize somewhat aggressively to squeeze average performance out of Azure SQL, for instance.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jermaustin1 on Thursday November 21, 2019 @04:06PM (#21599622)
Back in 2012 or 2013 I built an auto-scaling WordPress hosting off of Azure. It worked OK, better than shared hosting, but the $15 base VM at Azure (it was a cheaper time back then) couldn't compete with the base $10 droplet at Digital Ocean (it was a more expensive time back then).

On top of that Digital Ocean's API was so much better than Azure. So I spent a week and switched to Digital Ocean. Needed to scale the instances less, cost less, so more profit!

Unfortunately after 2 months of development, and half a year of marketing it, we only managed to attract a handful of customers, and had to shut it down.

Re:Oracle built a 1,060 node Raspberry Pi cluster by twobat on Friday November 22, 2019 @01:30AM (#21603309)
Why did try to scale when you had no customers?
Re:Oracle built a 1,060 node Raspberry Pi cluster by jermaustin1 on Friday November 22, 2019 @07:18AM (#21604882)
I guess I wasn't perfectly clear. The technology was to scale the customer's wordpress installation when certain metrics were met CPU or RAM > 75%, clone droplet and load balance.

The technology was basically an a beefy server for NFS, a beefy MySql server, and a beefy load balancer. When you started hosting with us, you paid $50+/mo and got what felt like a standard WordPress install on a dedicated digital ocean droplet, when metrics showed that you were hitting the scale threshold, we would create a new droplet run an initialization script to mount NFS, set up the nginx server, and then attach it to the loadbalancer.

When the combined CPU or RAM droped below the initial scale threshold, it would drop one VM from the load balancer at a time keeping it provisioned for an hour in case it needed to scale again.

Worked amazingly well. Sure the load balancer was a single point of failure, but in the 10 or so months that it was running, we never had a second of outage based on pinging the websites.

Re:Oracle built a 1,060 node Raspberry Pi cluster by apple4ever on Friday November 22, 2019 @09:27AM (#21605788)
Now that is very cool! I've always wanted to try something like that.

What was it written in?

Re:Oracle built a 1,060 node Raspberry Pi cluster by jermaustin1 on Friday November 22, 2019 @10:50AM (#21606641)
The main application I left hosted at Azure. It was an ASP.Net MVC app.

I used the Digital Ocean API to provision infrastructure, and SSH to remotely login and run the initialization bash script. The init script configured nginx, NFS (maybe it was SMB, i dont rmember now), a cron job to minutely heartbeat statistics back to the MVC app.

There are so many better ways to do it now, though. Off the shelf load balancers with letsencrypt and an api, hosted dns, managed & clustered databases, monitoring api. It would be a lot simpler today than it was just 2 years ago, let alone 7.

Re:Oracle built a 1,060 node Raspberry Pi cluster by neurostimulant on Friday November 22, 2019 @01:05PM (#21608166)
These days you can do it (relatively) easily by using wordpress docker image on kubernetes (or any other container platform). e.g. using Traefik for load balancer, a mysql container or an external mysql db instance (aws, google and do now offer managed mysql), and storage provider that can be mounted from multiple nodes (nfs, glusterfs, ceph, etc) and you're done! You can throw in an sftp container that mounts /var/www so you can manage the wordpress installation using an sftp client.
Re:Oracle built a 1,060 node Raspberry Pi cluster by dx034 on Friday November 22, 2019 @03:31AM (#21603867)
If you want to offer a completely automated process, auto-scaling is important. Otherwise it'll fail as soon as the first customers come in.
Re:Oracle built a 1,060 node Raspberry Pi cluster by vorticalbox on Thursday November 21, 2019 @03:46PM (#21599405)
Maybe because Windows updates and xbox love run through the network?
Re:Oracle built a 1,060 node Raspberry Pi cluster by weberc2 on Thursday November 21, 2019 @04:40PM (#21599979)
Pretty sure all of Netflix runs on AWS and I've got to think that's several orders of magnitude more traffic than Windows updates or xbl.
Re:Oracle built a 1,060 node Raspberry Pi cluster by ofrzeta on Thursday November 21, 2019 @04:44PM (#21600016)
Netflix hosts video content at ISPs' sites: https://openconnect.netflix.com/en/
Re:Oracle built a 1,060 node Raspberry Pi cluster by slenk on Thursday November 21, 2019 @06:45PM (#21601018)
Edge content yes, but everything else is in AWS
Re:Oracle built a 1,060 node Raspberry Pi cluster by GhettoMaestro on Thursday November 21, 2019 @06:58PM (#21601123)
The edge traffic makes up 99%+ of the traffic in question.
Re:Oracle built a 1,060 node Raspberry Pi cluster by slenk on Thursday November 21, 2019 @07:18PM (#21601238)
Do they publish how many ISPs participate? I am sure the big ones do but who knows if local Fiber is connected to someone who is behind an OpenConnect
Re:Oracle built a 1,060 node Raspberry Pi cluster by ivnubinas on Friday November 22, 2019 @12:13AM (#21602861)
Dominican here studying Telecommunications. I was giving a tour to the Datacenter of the 3rd largest telecom company in my country. They were using the Netflix box. I was giving a tour to the 4th largest telecom company, they too had the Netflix box. I was told the 1st and 2nd largest telecoms in the country also had the box.
Re:Oracle built a 1,060 node Raspberry Pi cluster by GhettoMaestro on Friday November 22, 2019 @04:32PM (#21610311)
My understanding is that Netflix will pretty much subsidize the cost of the CDN box to nearly any ISP above a certain minimum traffic level ingress from Netflix.

Great model to attack the incumbent [transport / transit] carrier costs - end run around the middle-men and drop your CDN boxes directly at the edge (last-mile) ISPs.

Re:Oracle built a 1,060 node Raspberry Pi cluster by walrus01 on Friday November 22, 2019 @06:32PM (#21611206)
ISP here, the netflix cache appliance boxes (a single 1RU server is the minimum) are free, but you have to provide rack space, power and cooling for them. For an ISP in a place like a caribbean island it's a great idea because their upstream/IP transit rates are significantly higher than in the US48 states.
Re:Oracle built a 1,060 node Raspberry Pi cluster by sanxiyn on Thursday November 21, 2019 @08:07PM (#21601525)
Non-participating ISPs get it from participating ISPs. Basically no one hits AWS.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jlgaddis on Thursday November 21, 2019 @04:43PM (#21600005)
FWIW, the video streaming part of Netflix does not run in AWS (but, AIUI, pretty much everything else does).
Re:Oracle built a 1,060 node Raspberry Pi cluster by weberc2 on Thursday November 21, 2019 @06:49PM (#21601061)
Oh, TIL I guess. Thanks for the correction.
Re:Oracle built a 1,060 node Raspberry Pi cluster by thu2111 on Friday November 22, 2019 @08:58AM (#21605564)
It's massively, massively overcommitted. They managed to actually run out of hardware in a whole US region recently, even though of course most customers had lots of unused quota. A typical Azure VM must be extraordinarily busy given some of the latencies and speeds I've seen on them.
Re:Oracle built a 1,060 node Raspberry Pi cluster by axegon_ on Thursday November 21, 2019 @02:56PM (#21598773)
It's incredible how powerful these small things are. I have a couple and I use them for just about anything, from access servers to access my home network, to running various applications inside containers on them. I did plan on making a cluster some time ago but for various reasons I've put that plan at the back of the line.

The real question here is did they use it to run some heavy applications on the cluster and how did it perform? Basically do they have a purpose for it or was it simply a "because we can" type of thing?

Re:Oracle built a 1,060 node Raspberry Pi cluster by shervinafshar on Thursday November 21, 2019 @04:20PM (#21599755)
I think an objective might have been demonstrating the benefits of Oracle Autonomous Linux regarding automatic patching and tuning; lower TCO being the advertised selling point.
Re:Oracle built a 1,060 node Raspberry Pi cluster by djaque on Thursday November 21, 2019 @03:36PM (#21599297)
I think this is cool, but engineering for engineering's sake aside I've seriously looked at building a raspi cluster a couple of times and have always concluded that it isn't worth it.

The real selling point for me is energy efficiency and price, but the reason I could never make it happen is because most of the applications I want to use are x86 only and I don't have the resources to be a trailblazer in that area. In addition sourcing becomes an issue with raspberry pis when you aren't a big company like Oracle. It's hard to buy more than a handful as an individual.

Re:Oracle built a 1,060 node Raspberry Pi cluster by TwoNineFive on Thursday November 21, 2019 @09:27PM (#21601913)
This is purely for education and show.

I worked on two top-50 supercomputer clusters for science applications. You learn a lot by doing. You find out what happens to your electrical power, TFTP, DHCP, and syslog servers, and other infrastructure when those hundreds/thousands of nodes boot up all at once. Your Cisco switch reloads when you add a secondary IP address to all of your network interfaces because it's CAM table overflows. You learn how to automate without stuff like Ansible and Chef because they don't exist yet.

Re:Oracle built a 1,060 node Raspberry Pi cluster by intpx on Friday November 22, 2019 @09:19AM (#21605723)
Ha, it was my experience with HPC that lead me to convince our CCIE network admin that CAM tables even could overload on a vlan of that size (we were using layer 3 switching to route tons of other vlans and this particular vlan was very active in traversing those boundaries and was also using multicast and lots of other weird things). We were seeing an issue when where computers on the same switch would lose their ability to see each other, even if they could see other machines on a different switch and everything about it smelled like the cam table was not accepting new entries. I finally convinced him to clear it out on one affected port and it worked.
Re:Oracle built a 1,060 node Raspberry Pi cluster by willis936 on Thursday November 21, 2019 @04:00PM (#21599563)
In case people reading play with raspis and aren’t aware: you can reduce power consumption by several percent by disabling peripherals. This wiki has some useful information for those looking to make raspi clusters.

https://www.pidramble.com/wiki/benchmarks/power-consumption

Re:Oracle built a 1,060 node Raspberry Pi cluster by jacquesm on Thursday November 21, 2019 @03:50PM (#21599451)
The real selling point for me would be physical isolation. If you run all of them single-core you wouldn't have to worry at all about snooping using the various side channel attacks.
Re:Oracle built a 1,060 node Raspberry Pi cluster by Skunkleton on Thursday November 21, 2019 @04:24PM (#21599803)
Yes, but how many RPis do you need to match the workload of a single x86 core? How much does it cost to cluster and maintain these systems? IMO you would probably have a better story all around if you based your cluster on low end AMD chips, or perhaps even old refurbished systems.

That's not to say this isn't cool engineering for engineering's sake. Just that if you are considering practical applications for this, maybe think twice.

Re:Oracle built a 1,060 node Raspberry Pi cluster by jacquesm on Thursday November 21, 2019 @07:00PM (#21601137)
When you need that guarantee of isolation you are picking your solution for different reasons than when you need a particular workload. So there is no point comparing the workload if that wasn't the deciding factor in the first place.
Re:Oracle built a 1,060 node Raspberry Pi cluster by Skunkleton on Friday November 22, 2019 @04:53PM (#21610512)
My point is that you can probably get physical isolation for less money on x86.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jacquesm on Saturday November 23, 2019 @10:21AM (#21614537)
You're going to have to explain a bit more on how you intend to achieve that economy. I'm not aware of a single SOC+peripherals based on x86 that would beat the Pi in price/performance, is there such a thing?
Re:Oracle built a 1,060 node Raspberry Pi cluster by mappu on Thursday November 21, 2019 @05:38PM (#21600512)
> most of the applications I want to use are x86 only

Really? What software in this space (headless server cluster) is still x86 only?

Re:Oracle built a 1,060 node Raspberry Pi cluster by djaque on Thursday November 21, 2019 @09:06PM (#21601826)
Unfortunately some proprietary scientific software is compiled for x86 only and we just get access to the binaries. I don't like it, but that's just how some fields work.
Re:Oracle built a 1,060 node Raspberry Pi cluster by giancarlostoro on Thursday November 21, 2019 @05:07PM (#21600242)
The Raspberry Pi Zero's are ridiculously affordable. For the cost of a regular Pi 3 you can get one thats wireless with a case with 3 covers for the case (1 for the Raspberry Pi camera). So maybe the Zero W's are more of what you might want. If you dont need them all to have wireless then you can just get the Zero's instead.
Re:Oracle built a 1,060 node Raspberry Pi cluster by djaque on Thursday November 21, 2019 @09:07PM (#21601831)
By sourcing, I wasn't arguing about their price. The problem is that it's hard to find anyone who will sell you more than a handful of them at low prices.

I haven't checked recently, but at least that's what I remember.

Re:Oracle built a 1,060 node Raspberry Pi cluster by charwalker on Thursday November 21, 2019 @06:33PM (#21600927)
I'd want a zero for projects. I have a 3B for messing around with and wish I already had PoE out so I could just set and forget a Pihole box or something. At the moment my Pihole instance is on an x86 Raspbian VM for simplicity but I think tonight or this weekend I'll set up the 3B do be a backup and point my router to it as a secondary DNs server. At least it'd be using the Pi vs it sitting in my drawer.
Re:Oracle built a 1,060 node Raspberry Pi cluster by imtringued on Friday November 22, 2019 @04:13AM (#21604058)
That's a pretty stupid reason. The real reason ist that most software doesn't run on a cluster at all and an 8 core Ryzen costs less than $170 and $340 for a full system without a GPU but is significantly faster than a 10 node rasperry pi cluster ( doesnt even include sd card power supply network gear)
Re:Oracle built a 1,060 node Raspberry Pi cluster by tra3 on Thursday November 21, 2019 @03:10PM (#21598961)
I've been curious about RPis for a while.

I have a file server running AMD Phenom II 955 that I'm thinking of replacing with a RPi. Is there a practical way to have RPis manage a software RAID? Performance is not a consideration. Is it even worth looking at RPi in this scenario? The advantages I can think of is small footprint, lower electricity cost.

Re:Oracle built a 1,060 node Raspberry Pi cluster by pizza234 on Thursday November 21, 2019 @03:54PM (#21599475)
I think the most effective approach in this case is to set the requirements and start from them.

This is not trivial advice :-) Requiring 4 drives in RAID10 has different consequences from 2 drivers in RAID1. Requiring a price < 100$ is different from < 200$. Hacked storage setup (eg. USB drives) versus proper (eg. storage controller with multiple SATA ports) is different.

For example, if you're willing to spend more, but not so much more, than say, 80$, why limiting to the RPi? The Odroid H2, for example, gives a lot of options.

Among the other things, I use it as file server with a COW-filesystem on two (cheap) SSD disks in RAID1. The board has two SATA ports, so it's a solid hardware setup. It has two ethernet ports, so it's ready to work as router. It's also x86, so it will have software support for a very long time¹. And it also runs at an incredibly cool temperature - I bought the fan, but it rarely spins. To be fair though, the H2 cases are expensive trash.

You can certainly do pretty much everything on a RPi4, as much as you can commute 50 km a day with a bike, which is... not ideal :-)

¹=RPi have an enthusiastic community, but ARM boards are second class citizens when it comes to Linux support. While Linux is formally compatible, it's still adviced to run the specific ARM distro.

Re:Oracle built a 1,060 node Raspberry Pi cluster by undersuit on Thursday November 21, 2019 @03:51PM (#21599457)
You could set up a couple of GlusterFS or Ceph nodes. With the RPi4 having USB3 and Gigabit Ethernet you no longer get cramped by the previous generation's internal USB 2.0 Hub.
Re:Oracle built a 1,060 node Raspberry Pi cluster by rb808 on Thursday November 21, 2019 @03:43PM (#21599372)
Usually you can get a new NAS really cheap on Black Friday. Even today you can get a 119j for $103, which isn't much more than a rpi with card and case.

https://www.newegg.com/synology-ds119j/p/N82E16822108418

Re:Oracle built a 1,060 node Raspberry Pi cluster by dwoozle on Thursday November 21, 2019 @04:45PM (#21600028)
A one-bay NAS... didn’t even know those existed
Re:Oracle built a 1,060 node Raspberry Pi cluster by rb808 on Thursday November 21, 2019 @06:06PM (#21600741)
Yeah I use as backup, and it syncs to the cloud. So I figure if it dies I'm still good as I have the original and cloud. It also means I dont have to pay for 2 disks or think about RAID. It is slow and there is a learning curve but mostly just works and is very low power - at it shuts down disk when not in use.
Re:Oracle built a 1,060 node Raspberry Pi cluster by blaser-waffle on Friday November 22, 2019 @12:36PM (#21607828)
> I have a file server running AMD Phenom II 955 that I'm thinking of replacing with a RPi. Is there a practical way to have RPis manage a software RAID?

I did this using an rpi2. It is non-trivial in terms of effort.

In a practical sense the rpi is limited in terms of power to the USB ports, so any build with 2+ HDDs and some sort of wireless USB keyboard and mouse dongle will use up too much power.

In other words, I had to buy a powered USB hub.

I used mdadm to do a software RAID, which more or less worked out of the box but took a while to clone the original drive. Throughput wasn't amazing for anything except occasional pulls and puts; I wasn't going to stream 4K video off of this, though I did have some luck with Subversion and music streaming.

It was clunky enough that I got a Synology box a year and a half later and never looked back.

Re:Oracle built a 1,060 node Raspberry Pi cluster by jandrese on Thursday November 21, 2019 @03:17PM (#21599043)
It's a bit difficult since the RPi doesn't have any form of storage controller, just a USB hub. So your system would have to be built out of externally powered USB drives, which would be slow even by your lax standards and likely finicky.

It's hard to see the advantage over a dedicated NAS device, since the savings on the controller gets eaten up by needing to buy a whole bunch of drive enclosures and power supplies.

Re:Oracle built a 1,060 node Raspberry Pi cluster by ansible on Thursday November 21, 2019 @06:21PM (#21600846)
I'm (still) in the process of re-purposing an old laptop for a home server. 2TB laptop hard drives are a reasonable price, and on older laptops with a CD/DVD drive bay, you can usually get an adapter caddy to put in a second drive. Pay attention to the allowable drive height though.

This system is going to be set up with software RAID.

I expect to have higher compute power, more RAM, and much better IO throughput than a RPi solution. Also, the laptop battery (which is still good) serves as a built-in UPS.

Re:Oracle built a 1,060 node Raspberry Pi cluster by whalesalad on Thursday November 21, 2019 @03:14PM (#21599014)
> Is there a practical way to have RPis manage a software RAID?

I can't think of any good reason to do use software RAID, let alone doing so on a Raspberry Pi. I would recommend hardware RAID, ZFS, or a Synology appliance.

If performance is truly not a concern, yes it is possible to do this with a Pi but I don't think you are going to be super impressed.

Re:Oracle built a 1,060 node Raspberry Pi cluster by tra3 on Thursday November 21, 2019 @03:33PM (#21599273)
Software RAID = simple. At least when I built this machine nearly 10 years ago. Since then it's been rock solid, if not blazing fast. There were also horror stories of hardware raids failing due to controllers failing; or subtle differences between different revisions of RAID controllers. That said I also had budgetary restrictiions at the time that made software raid attractive.

I looked at NAS appliances, but I think it's still cheaper to create one from scratch (assuming existing knowhow, and "free" time :)

Re:Oracle built a 1,060 node Raspberry Pi cluster by ComputerGuru on Thursday November 21, 2019 @03:17PM (#21599040)
ZFS is 100% software raid.
Re:Oracle built a 1,060 node Raspberry Pi cluster by whalesalad on Thursday November 21, 2019 @03:19PM (#21599064)
Sorta. At the end of the day, yes, the disks are being managed by your CPU and not a dedicated raid controller... but it's a completely different ballgame than something like mdadm.
Re:Oracle built a 1,060 node Raspberry Pi cluster by ownagefool on Thursday November 21, 2019 @04:05PM (#21599610)
Not "sorta", it's software raid and thus, like software raid, it's portable to any system that'll run the software and JBOD the drives.

Most would argue RAIDZ is better, but many of these reasons are functionality based over and about just redundancy, i.e. Copy on Write, et all.

Re:Oracle built a 1,060 node Raspberry Pi cluster by GhettoMaestro on Thursday November 21, 2019 @07:03PM (#21601153)
Friends don't let friends create RaidZ pools.

Bite the bullet of the cost of N*2 mirrors.

Re:Oracle built a 1,060 node Raspberry Pi cluster by whalesalad on Thursday November 21, 2019 @04:23PM (#21599790)
I say sorta because in most circles when one is speaking of 'software RAID' they are referring to a very specific configuration and are explicitly not referring to ZFS.
Re:Oracle built a 1,060 node Raspberry Pi cluster by colejohnson66 on Thursday November 21, 2019 @03:55PM (#21599497)
ZFS doesn’t support hardware RAID controllers?
Re:Oracle built a 1,060 node Raspberry Pi cluster by whalesalad on Thursday November 21, 2019 @04:22PM (#21599775)
It supports it, but ZFS itself is operating on the host CPU, not the RAID controller. Additionally, ZFS prefers that the RAID controller is disabled/defeated and operating in JBOD mode where disks are passed-thru to the OS without any caching.
Re:Oracle built a 1,060 node Raspberry Pi cluster by ComputerGuru on Thursday November 21, 2019 @10:34PM (#21602192)
Not just caching. The problem is that non-JBOD HBAs also swallow error codes and mask retries, meaning ZFS is potentially unaware of some read errors and is not in control of the latency as the controller firmware has its own timeouts and retry counters. Moreover, most will not pass through SMART or else fake it.
Re:Oracle built a 1,060 node Raspberry Pi cluster by icefo on Thursday November 21, 2019 @05:54PM (#21600645)
Not a Pi but the rockpro64 has a pcie port in which you can plug what you want (a sata card for example).

I made a nas with one 3tb drive and zfs. Zfs was a bit difficult to install but now it works really well and the power consumption is low.

You have to use the Linux images mades by ayufan (a third party) though. The manufacturer just list a bunch of distro that run on the cards and most of them seem to be made by some volunteer. I'm not a big fan of that.

Re:Oracle built a 1,060 node Raspberry Pi cluster by jacquesm on Thursday November 21, 2019 @06:50PM (#21601068)
There is nothing wrong with software raid. Synology uses SW raid in all their NAS products as far as I know, right up to the flagship models with > 10 drives and 100T+ of storage.
Re:Oracle built a 1,060 node Raspberry Pi cluster by imtringued on Friday November 22, 2019 @04:21AM (#21604097)
Hardware raid controllers are a single point of failure and when they fail the vendor sells a different product so you will always have to stock up and buy two at once.
Re:Oracle built a 1,060 node Raspberry Pi cluster by pjc50 on Thursday November 21, 2019 @05:05PM (#21600220)
Software RAID is easier to manage, easier to do disaster recovery, does not require specific driver support, and easier to move to other hardware.
Re:Oracle built a 1,060 node Raspberry Pi cluster by voltagex_ on Thursday November 21, 2019 @06:58PM (#21601126)
maybe the Pi 4 would be able to do this with a USB attached disk box, but you'd be much better off with something like a commercial NAS - and I say this as someone who has used ARM boards since the Sheevaplug.

It all depends - do you want a fileserver or a crash course in how to build a fileserver?

Re:Oracle built a 1,060 node Raspberry Pi cluster by newnewpdro on Thursday November 21, 2019 @03:25PM (#21599159)
Maybe if you could use something like AoE turning the network port into the backend storage interface a la CoRaid. Otherwise like others have said you have USB.
Re:Oracle built a 1,060 node Raspberry Pi cluster by hinkley on Thursday November 21, 2019 @03:55PM (#21599494)
There are one or two NanoPi models with an M.2 slot and full gigE, but that doesn’t get you an array unless there’s some m.2 breakout box I’m not familiar with.
Re:Oracle built a 1,060 node Raspberry Pi cluster by vorpalhex on Thursday November 21, 2019 @03:29PM (#21599220)
For RAID, no. The pi has a few strikes against it for that case. If you're simply after redundancy, ZFS may work for your case.
Re:Oracle built a 1,060 node Raspberry Pi cluster by SEJeff on Thursday November 21, 2019 @05:15PM (#21600310)
This explains so much about Ellison's vision for Oracle Cloud.
Re:Oracle built a 1,060 node Raspberry Pi cluster by tzs on Thursday November 21, 2019 @03:50PM (#21599444)
> At Oracle OpenWorld 2019, tucked away in the Oracle Code One area [...]

As someone not familiar with Oracle events, I'm rather confused about Oracle Code One. I found a FAQ [1] which says:

> Oracle Code One is the ultimate developer event where we will cover the technologies and programming languages developers have come to love over the years. Expect talks on Go, Rust, Python, JavaScript, and R and SQL along with more of the latest Java technical content.

but looking at the photos, I see in the background quite a few arcade video games. Is this some sort of break area for people to hang out in between talks, or something like that?

[1] https://www.oracle.com/code-one/faq.html

Re:Oracle built a 1,060 node Raspberry Pi cluster by SanderMak on Thursday November 21, 2019 @04:22PM (#21599774)
Oracle Code One is the successor of JavaOne. Since Oracle acquired Sun, the JavaOne conference was co-hosted with the much larger Oracle OpenWorld conference. Recently, JavaOne was rebranded into Oracle Code One, morphing it into a cross-technology conference rather than a Java-focused one.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jaaron on Thursday November 21, 2019 @07:07PM (#21601172)
I remember when JavaOne was definitely worth going to. I've not heard anything like that for years and just assume it's dead as "Code One." Anyone want to confirm?
Re:Oracle built a 1,060 node Raspberry Pi cluster by jlgaddis on Thursday November 21, 2019 @04:53PM (#21600113)
TFA says this cluster was part of "developor relations" and runs OAL and Java.

Perhaps the arcade games are also running on Java and Oracle is simply trying to "show off" a variety of uses of Java or something.

Re:Oracle built a 1,060 node Raspberry Pi cluster by nrki on Thursday November 21, 2019 @03:08PM (#21598931)
No benchmarks. Cool but just seems like an ad for Oracle.
Re:Oracle built a 1,060 node Raspberry Pi cluster by rhinoceraptor on Thursday November 21, 2019 @04:31PM (#21599881)
Of course, you're not allowed to benchmark Oracle publicly, it's in the EULA :)
Re:Oracle built a 1,060 node Raspberry Pi cluster by ineedasername on Thursday November 21, 2019 @10:11PM (#21602109)
Seriously? As someone with purchasing power, I could only assume they were slower than alternatives in that case. Anything they quoted me would have to be considered marketing fluff without independent numbers.
Re:Oracle built a 1,060 node Raspberry Pi cluster by rhinoceraptor on Friday November 22, 2019 @09:18AM (#21605717)
I can almost understand why they do it. Benchmarking is incredibly hard to do correctly, and it requires a very deep understanding of software and hardware.

For example, I remember hearing about Joyent's port of KVM to Illumos. They were testing the Illumos KVM against Linux KVM, and the Illumos KVM was faster. After a lot of debugging, they realized the Illumos machine was turboboosting better than the Linux machine, since it was in a colder part of the data center.

Re:Oracle built a 1,060 node Raspberry Pi cluster by _bxg1 on Thursday November 21, 2019 @03:08PM (#21598940)
I wonder what the cost-effectiveness is in terms of compute power. I'd assume not great, but who knows? Raspberry PIs pack a lot of punch for their size and price.
Re:Oracle built a 1,060 node Raspberry Pi cluster by asdfasgasdgasdg on Thursday November 21, 2019 @03:32PM (#21599260)
When it comes to cluster-scale computing, size is not the main issue. The main issues are: compute/$ and compute/watt.

There is also the matter of how much RAM and disk can be attached to each board, and the speed of the NIC. Gigabit ethernet is a little slow these days.

Re:Oracle built a 1,060 node Raspberry Pi cluster by willis936 on Thursday November 21, 2019 @04:04PM (#21599602)
GbE is slow, yes, but GbE for the amount of compute a raspi 4 can do is massive. Raspis don’t even have the oomph to max out GbE links. You hook 48 raspi 4 into a switch and using a 25 GbE link to a storage cluster.

Edit: It is conceivable that a raspi has the oomph to max out a GbE link but is limited to 300 Mbps due to the use of a USB 2.0 bridge in between the ethernet interface and the SoC. My mistake.

Re:Oracle built a 1,060 node Raspberry Pi cluster by esotericn on Friday November 22, 2019 @05:30AM (#21604411)
The RPi4 is capable of gigabit. I benchmarked mine at 900-950mbit via iperf yesterday. That was to a usb3 gigabit ethernet adapter on a laptop.
Re:Oracle built a 1,060 node Raspberry Pi cluster by willis936 on Friday November 22, 2019 @07:31AM (#21604958)
Ah thanks. My previous post is dated knowledge, applicable only to the raspi 3.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jandrese on Thursday November 21, 2019 @03:21PM (#21599087)
If they can leverage the GPU on the Pis it could be quite a bit of compute power spread across that array. Even if they're purely CPU bound that is 4240 cores which is not nothing.

Power wise each Pi going flat out burns around 5W, so all told you're burning ~6kW, which is quite modest for a big computing setup.

Re:Oracle built a 1,060 node Raspberry Pi cluster by willis936 on Thursday November 21, 2019 @04:16PM (#21599708)
It looks like you can get 400 single precision MFLOPS for those 5W. So 80 MFLOPS/W +/- 20%. It isn’t the worst and it isn’t the best, but it is respectable. It would be a big hassle to make and maintain a cluster of thousands of these.

https://www.arnabkumardas.com/platforms/nvidia/nvidia-jetson...

https://raspi.tv/2019/how-much-power-does-the-pi4b-use-power...

https://en.wikipedia.org/wiki/Performance_per_watt

Re:Oracle built a 1,060 node Raspberry Pi cluster by willis936 on Friday November 22, 2019 @03:18PM (#21609614)
Oh jeez. Out of curiosity I did a comparison against a gaming GPU. A GTX 1080 uses 200W of power for 9 TFLOPS, which is roughly 500x better performance per watt than a raspberry pi. Outside of the fun of it, and maybe a cheap way to prototype larger systems, I can't see a serious reason to make a raspi cluster.
Re:Oracle built a 1,060 node Raspberry Pi cluster by ramses0 on Thursday November 21, 2019 @05:52PM (#21600637)
...but you could experiment with "disposable computing" (ie: similar to original google datacenter racks: never replace a node, only replace a rack at a time if 51% is failing).
Re:Oracle built a 1,060 node Raspberry Pi cluster by jacquesm on Thursday November 21, 2019 @06:52PM (#21601088)
That and interesting configurations (systolic arrays, message passing architectures, Erlang on a large number of nodes and so on). Lots of fun with stuff like this.
Re:Oracle built a 1,060 node Raspberry Pi cluster by chapium on Thursday November 21, 2019 @03:19PM (#21599062)
Definitely not great but thats one of the tradeoffs.
Re:Oracle built a 1,060 node Raspberry Pi cluster by jimmcslim on Thursday November 21, 2019 @04:39PM (#21599969)
More info on the USB power supplies they were using would be interesting, seems likely it would have been rack mounted and probably custom?
Re:Oracle built a 1,060 node Raspberry Pi cluster by rasz on Friday November 22, 2019 @01:39PM (#21608524)
Re:Oracle built a 1,060 node Raspberry Pi cluster by salvagedcircuit on Thursday November 21, 2019 @05:37PM (#21600508)
Talk about a thermal design nightmare. Definitely just an "look-at-me" exhibit
Re:Oracle built a 1,060 node Raspberry Pi cluster by Havoc on Thursday November 21, 2019 @05:01PM (#21600177)
>an array of USB power supplies.

Surprised they didn't just bridge 5V to GPIO pins.

Re:Oracle built a 1,060 node Raspberry Pi cluster by jonplackett on Thursday November 21, 2019 @04:36PM (#21599944)
Not much info on what they can actually do with it
Re:Oracle built a 1,060 node Raspberry Pi cluster by tr33house on Thursday November 21, 2019 @03:28PM (#21599207)
This seems like Oracle's way of marketing to devs that they're a cool place to work... And it might have actually just worked
Re:Oracle built a 1,060 node Raspberry Pi cluster by tobyhinloopen on Thursday November 21, 2019 @03:27PM (#21599184)
Why...
Re:Oracle built a 1,060 node Raspberry Pi cluster by rvz on Thursday November 21, 2019 @03:37PM (#21599306)
Exactly, I just feel that Oracle just needed to do a huge flex to the world to show that they are relevant.

Weird flex but okay, I'll just carry on with my ARM64 Cavium kubernetes cluster that blows this toy out of the ocean.

Re:Oracle built a 1,060 node Raspberry Pi cluster by walshemj on Thursday November 21, 2019 @03:51PM (#21599453)
But they probably built it as test cluster to do initial work on before going for a big shot on a big boy cluster.
Re:Oracle built a 1,060 node Raspberry Pi cluster by faizshah on Thursday November 21, 2019 @03:57PM (#21599519)
I’ve seen a few of these RPi clusters (obviously none this big), I think the main reason people make them is to demo distributed systems or build a homelab cluster(sometimes they will argue its cause of the power efficiency). Also they are usually a nerd magnet so it makes for a great conversation piece as this probably was for their convention.
Re:Oracle built a 1,060 node Raspberry Pi cluster by yellowapple on Thursday November 21, 2019 @04:38PM (#21599961)
Why not?
Re:Oracle built a 1,060 node Raspberry Pi cluster by et2o on Thursday November 21, 2019 @05:27PM (#21600417)
Burning a few interns worth of money?