In contrast, I paid around $120 for a PCEngines APU2 (running PFSense), and it uses around 8W of power (measured).
Old servers are the same -- the amount of power they use for the performance makes them expensive to run 24x7.
What do you think of that?
I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores.
Anyone know where I could look to find someone I could contract to put something like this together?
Or dual epyc for a single box 64/128 system?
https://www.newegg.com/Product/Product.aspx?Item=N82E1681911...
From that thread: "Definitely not the threadripper - there were some benchmarks scattered around in other topics and it's actually slower at compiling and lighting builds than my 7820X, while costing more to build a system with."
But not sure if I should trust those opinions from the dev community as the benchmarks aren't too sophisticated.
I'll continue doing more research on those thank you! I think dual epyc could be interesting for the cost.
These days, one dual-socket server can give you that, for a pretty reasonable cost compared to the time, expense, and power usage of running a rack full of systems.
If you had to put together a copy of the Amazon X1 instance https://aws.amazon.com/ec2/instance-types/x1/ any recommendation on a mobo/memory/case to use?
As for contracting, there's lots of options, but everything depends heavily on your usage model. Let's say your render jobs take the shape of "press button, enqueue 10,000 CPU-minutes of work". How often do you press the button? How valuable are low completion times? How penalizing are high completion times? Are there ever times when there is no work?
Cloud computing is a good fit for some problems: you can press a button, run your code on 10,000 CPUs, and finish the job in a minute. Cloud computing is expensive for other problems: in terms of hardware, the $1500 machine above compares favorably versus a cc2.8xlarge instance which costs about $1500/month at on-demand pricing. Server hardware is not the only cost -- you still need space and power and disks and network and time for setup and ongoing maintenance -- but there's definitely a break-even point.
PaaS, IaaS, dedicated servers, colocation, and in-house datacenters all make sense for users seeking different tradeoffs. It's difficult to give useful advice without knowing more about the tradeoffs you want to make and the relative values you place on setup time versus run time versus money.
Only fear is that the time/cost it takes to spool up/down would not let us iterate as fast as ideally possible if we owned the hardware and had it running 24/7 for dedicated just in time use.
So rather than rent an X1 instance indefinitely we might just pony up the $ upfront and buy our own.
"Each X1 instance is powered by four Intel® Xeon® E7 8880 v3 (codenamed Haswell) processors and offers up to 128 vCPUs."
Anyone know if the X1 instance is 1 motherboard with 4 sockets or 2 separate mobos with 2?
IMO, if your core competency isn't sourcing and buying hardware in order to build render farms, you shouldn't do it. It's just not worth it. AWS brings a lot of value to the table, not the least of which is the fact that you won't have to maintain/fix hardware when you're just trying to render something. If the instance is broken, you can just kill it and spin up a new one.
Seriously, this will make your life so much easier.
> Anyone know if the X1 instance is 1 motherboard with 4 sockets or 2 separate mobos with 2?
This shouldn't matter. It's abstracted away from you by design.
Definitely agree on using other cast-off hardware, though.
Absolutely this. No need to run enterprise level hardware at home. You can easily run a huge environment with 128gb of ram and an 8-core Ryzen processor. If you want to get crazy you could even get an Epyc CPU with a SuperMicro motherboard, put it into a eATX whitebox case and run silent fans with a huge heatsink and still barely use any power. You can even run all the networking OS's in VM's, either with PFSense or some other vm template or use GNS3. No need to purchase a ton of hardware.
It really makes no sense to me why people drop all this money on R710's and make their house sound like the Hartsfield-Jackson Delta Terminal.
Anyway, the ZFS approach is to inherently trust the data that's in RAM over the data on the drive if there's a discrepancy, so ECC RAM is required to maintain the integrity of data on RAM.
That ZFS needs ECC moreso than other filesystems is an often repeated misconception. It's just that ZFS (and e.g. Btrfs) is often run on file servers, on which ECC is recommended.
This is absurd. Please read up (and understand) this before talking about it again. The benefits of ZFS and the benefits of ECC are orthogonal, though if your fileserver has both, it's a pretty darn robust system with respect to integrity -- assuming that (for example) the disk controllers don't lie about something being synced.
I've been wiring up 10G throughout the house, and managed to get a ER-8-XG (beta version) on this site for under $500. (8 10-gig SFP+). Wife made me get rid of it because it was too loud. :( Got a Dell T1700 (they're like giving them away on EBay). Quad core Xeon E3, plenty of headroom to do firewall/routing in software.
Do you know how many watts you consume? Are alternatives to old enterprise gear from eBay going to be cheaper from an electricity point of view?
-Cisco ASA 5510 firewall - 100W
-Dell R610 running ESXi - 400W
-Old Dell 2950 full of 2TB drives running FreeNAS - 400W
-Cisco 'Small Business' 50 port managed switch - 100W
-A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted -servers - 2 x 300W
So, maybe 1600W under medium load? A 15A circuit can deliver 1800W max., so it would be occupying one whole circuit.
Hardly worth it, considering the noise, heat, space, and power. Old enterprise hardware gets given away for a reason.
Apart from that, places like www.weirdstuff.com might exist in your area, and have stuff like this laying around for pennies on the dollar.
A few years ago, they sold 4 pallets of Extreme 10/100 switches for $20! The catch is they sell lots and you have to pick them up at one time.
I agree with you that they are great value but I find the noise and power consumption of that generation a bit much so just built my own boxes
42u IBM-branded cabinet
Cisco ASA 5510 firewall
Dell R610 running ESXi
Old Dell 2950 full of 2TB drives running FreeNAS
Cisco 'Small Business' 50 port managed switch
A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted servers
If you've got the space and can deal with the noise, I can't say too many good things about the R610s - available dirt cheap, and though they're about four generations old at this point, still perform decently since Intel/AMD have, until recently, pretty much stagnated on CPU performance.
Luckily I have a basement.