News for hackers. Stuff that matters.
Re:Redoing all my home networking by CommieBobDole on Sunday December 03, 2017 @05:18PM (#15839974)
I've found that the home lab is a great place for cast-off corporate equipment either bought from ebay or scrounged from data center upgrades. Mine, assembled at little to no cost, consists of:

42u IBM-branded cabinet

Cisco ASA 5510 firewall

Dell R610 running ESXi

Old Dell 2950 full of 2TB drives running FreeNAS

Cisco 'Small Business' 50 port managed switch

A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted servers

If you've got the space and can deal with the noise, I can't say too many good things about the R610s - available dirt cheap, and though they're about four generations old at this point, still perform decently since Intel/AMD have, until recently, pretty much stagnated on CPU performance.

Luckily I have a basement.

Re:Redoing all my home networking by Johnny555 on Sunday December 03, 2017 @07:41PM (#15840617)
I've found that enterprise gear is not exactly power friendly -- the Cisco 5510 is rated at 150W steady state, if it uses half that much power in practice (75W), that's around $90/year where I live.

In contrast, I paid around $120 for a PCEngines APU2 (running PFSense), and it uses around 8W of power (measured).

Old servers are the same -- the amount of power they use for the performance makes them expensive to run 24x7.

Re:Redoing all my home networking by craftyguy on Sunday December 03, 2017 @08:41PM (#15840859)
This is pretty much what I ended up doing. It's a collossal waste of power (and money) to run enterprise equipment in my area for a home network that is nowhere near enterprise-level in terms of utilization.
Re:Redoing all my home networking by stock_toaster on Monday December 04, 2017 @01:16AM (#15841836)
I also used to have more gear, but hav really scaled back and consolidated. I now have one fairly beefy server (storage array, decent zeon, lots of ram), run a few vms (dev, plex, etc), and export storage. Then I have a pair of NUCs doing various things.
Re:Redoing all my home networking by cm2187 on Monday December 04, 2017 @03:17AM (#15842419)
Plus hard drive capacity seems to have gone back on a steep increasing curve with 12TB drives coming to the market, offering further opportunities for consolidation.
Re:Redoing all my home networking by mentos on Sunday December 03, 2017 @05:51PM (#15840125)
Someone shared this website with me a while back to find used servers on ebay https://www.labgopher.com/

What do you think of that?

I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores.

Anyone know where I could look to find someone I could contract to put something like this together?

[0] - https://docs.unrealengine.com/udk/Three/Swarm.html

Re:Redoing all my home networking by AlphaSite on Sunday December 03, 2017 @11:33PM (#15841460)
Have you looked into one of these guys [1]? They go for ~800USD for 16/32 cores/threads. You can get a good high-end build for ~1500USD. It behaves like a modern system with good power profiles and single threaded and multithreaded performance.

Or dual epyc for a single box 64/128 system?

https://www.newegg.com/Product/Product.aspx?Item=N82E1681911...

Re:Redoing all my home networking by mentos on Sunday December 03, 2017 @11:48PM (#15841510)
I had done some very basic research on it and found that the performance for the 'Lightmass' program used in Unreal Engine called Swarm was not ideal https://forums.unrealengine.com/community/off-topic/1388243-...

From that thread: "Definitely not the threadripper - there were some benchmarks scattered around in other topics and it's actually slower at compiling and lighting builds than my 7820X, while costing more to build a system with."

But not sure if I should trust those opinions from the dev community as the benchmarks aren't too sophisticated.

I'll continue doing more research on those thank you! I think dual epyc could be interesting for the cost.

Re:Redoing all my home networking by JoshTriplett on Sunday December 03, 2017 @09:55PM (#15841158)
> I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores.

These days, one dual-socket server can give you that, for a pretty reasonable cost compared to the time, expense, and power usage of running a rack full of systems.

Re:Redoing all my home networking by mentos on Sunday December 03, 2017 @11:29PM (#15841444)
Yea I think the cost savings when it comes to cores may not make sense in the long run when it comes to setup/teardown and power usage.

If you had to put together a copy of the Amazon X1 instance https://aws.amazon.com/ec2/instance-types/x1/ any recommendation on a mobo/memory/case to use?

Re:Redoing all my home networking by willglynn on Monday December 04, 2017 @01:02PM (#15845951)
I put together a few dual E5-2683v3 machines for a colo lab earlier this year. Pricing looks a little less favorable today, but I'd ballpark around $1500 per 28 cores/56 threads before RAM and disk.

As for contracting, there's lots of options, but everything depends heavily on your usage model. Let's say your render jobs take the shape of "press button, enqueue 10,000 CPU-minutes of work". How often do you press the button? How valuable are low completion times? How penalizing are high completion times? Are there ever times when there is no work?

Cloud computing is a good fit for some problems: you can press a button, run your code on 10,000 CPUs, and finish the job in a minute. Cloud computing is expensive for other problems: in terms of hardware, the $1500 machine above compares favorably versus a cc2.8xlarge instance which costs about $1500/month at on-demand pricing. Server hardware is not the only cost -- you still need space and power and disks and network and time for setup and ongoing maintenance -- but there's definitely a break-even point.

PaaS, IaaS, dedicated servers, colocation, and in-house datacenters all make sense for users seeking different tradeoffs. It's difficult to give useful advice without knowing more about the tradeoffs you want to make and the relative values you place on setup time versus run time versus money.

Re:Redoing all my home networking by cweagans on Sunday December 03, 2017 @10:50PM (#15841333)
You might be better off using EC2 or similar. Do the work to configure everything once and package it as an AMI, then when you need your swarm, spin up as many instance as you need, render whatever needs rendered, and then turn off the instances. Should be reasonably cheap, and it'll let you decide on the speed vs cost tradeoff (i.e. more servers cost more, but will get your work done faster; fewer servers cost less, but will take longer). I don't think a rack full of hardware is a very good choice for this.
Re:Redoing all my home networking by mentos on Sunday December 03, 2017 @11:26PM (#15841432)
I will probably try my luck with one of these X1 instances https://aws.amazon.com/ec2/instance-types/x1/

Only fear is that the time/cost it takes to spool up/down would not let us iterate as fast as ideally possible if we owned the hardware and had it running 24/7 for dedicated just in time use.

So rather than rent an X1 instance indefinitely we might just pony up the $ upfront and buy our own.

"Each X1 instance is powered by four Intel® Xeon® E7 8880 v3 (codenamed Haswell) processors and offers up to 128 vCPUs."

Anyone know if the X1 instance is 1 motherboard with 4 sockets or 2 separate mobos with 2?

Re:Redoing all my home networking by cweagans on Monday December 04, 2017 @11:17AM (#15844967)
The time to spin up is on the order of seconds.

IMO, if your core competency isn't sourcing and buying hardware in order to build render farms, you shouldn't do it. It's just not worth it. AWS brings a lot of value to the table, not the least of which is the fact that you won't have to maintain/fix hardware when you're just trying to render something. If the instance is broken, you can just kill it and spin up a new one.

Seriously, this will make your life so much easier.

> Anyone know if the X1 instance is 1 motherboard with 4 sockets or 2 separate mobos with 2?

This shouldn't matter. It's abstracted away from you by design.

Re:Redoing all my home networking by insanebits on Monday December 04, 2017 @03:22AM (#15842435)
You can have a look at xeon e5-2670 v2 cpus, they're relatively cheap, around $200 and have 8c/16t so in a double cpu configuration you could have 64 cores with only 4 machines.
Re:Redoing all my home networking by chrissnell on Sunday December 03, 2017 @07:41PM (#15840615)
I'm not a fan of running a hardware firewall that's long since passed its end-of-life date. There have been a number of critical vulnerabilities on these ASAs in the past. I'd rather stick with software that's kept up to date, like PFSense or OpenBSD.

Definitely agree on using other cast-off hardware, though.

Re:Redoing all my home networking by z3t4 on Sunday December 03, 2017 @06:35PM (#15840296)
Or you build a super quite desktop server, that will both have more CPU and use less power. The trick is to use large heat sinks with large silent fans. When the CPU's crank up to 100% all you'll hear is a subtle wind breeze.
Re:Redoing all my home networking by Ascetik on Sunday December 03, 2017 @09:14PM (#15841007)
Sys Admin/Hardware dork here.

Absolutely this. No need to run enterprise level hardware at home. You can easily run a huge environment with 128gb of ram and an 8-core Ryzen processor. If you want to get crazy you could even get an Epyc CPU with a SuperMicro motherboard, put it into a eATX whitebox case and run silent fans with a huge heatsink and still barely use any power. You can even run all the networking OS's in VM's, either with PFSense or some other vm template or use GNS3. No need to purchase a ton of hardware.

It really makes no sense to me why people drop all this money on R710's and make their house sound like the Hartsfield-Jackson Delta Terminal.

Re:Redoing all my home networking by rashkov on Sunday December 03, 2017 @11:55PM (#15841536)
Do you have more details on this? I've been looking for a way to build a fanless server but I've been having a difficult time finding info on it. The only stuff I can find is one or two commercial sites that sell custom builds, and so I wonder if it's difficult to do on one's own, or if the fanless tech is still very niche and undeveloped
Re:Redoing all my home networking by z3t4 on Monday December 04, 2017 @02:19AM (#15842117)
You could go with an intel Nuc, though they use mobile CPU's you can still get a lot of performance out of it, at least considering the form factor. If you want more power you need fans though, try the Noctua fans, they are close to quite. There's really no point going fanless.
Re:Redoing all my home networking by mafro on Monday December 04, 2017 @02:29AM (#15842165)
Check out the builds here [1]. I used this mobo for a NAS build recently since it has ECC support.

1: https://pcpartpicker.com/builds/by_part/fk98TW

Re:Redoing all my home networking by HankB99 on Sunday December 03, 2017 @09:46PM (#15841125)
What about the benefit of using ECC RAM for a file server? I don't see that available in consumer grade H/W. (As far as network H/W I'm in full agreement.)
Re:Redoing all my home networking by namibj on Sunday December 03, 2017 @10:08PM (#15841208)
AMD h/w pretty much always supports ECC DIMMs, it just depends on whether the MoBo propagates that to the sockets, as it takes more traces.
Re:Redoing all my home networking by san-fran on Monday December 04, 2017 @12:16AM (#15841611)
If you plan on running a ZFS file server at home, which enthusiasts prefer because it's one of the only filesystems that supports full file checksumming to prevent bit rot.

Anyway, the ZFS approach is to inherently trust the data that's in RAM over the data on the drive if there's a discrepancy, so ECC RAM is required to maintain the integrity of data on RAM.

Re:Redoing all my home networking by DCKing on Monday December 04, 2017 @01:18AM (#15841841)
All filesystems inherently trust the data that is in RAM. There's nothing any piece of software can reasonably do about memory it can't trust. It's therefore a good idea to run ECC RAM, regardless of which filesystem you run. You can be running your filesystem on Ext4 or the legacy BSD Unix File System, and it would still be a good idea to use ECC.

That ZFS needs ECC moreso than other filesystems is an often repeated misconception. It's just that ZFS (and e.g. Btrfs) is often run on file servers, on which ECC is recommended.

Re:Redoing all my home networking by empthought on Monday December 04, 2017 @03:43PM (#15847297)
> the ZFS approach is to inherently trust the data that's in RAM over the data on the drive if there's a discrepancy, so ECC RAM is required to maintain the integrity of data on RAM.

This is absurd. Please read up (and understand) this before talking about it again. The benefits of ZFS and the benefits of ECC are orthogonal, though if your fileserver has both, it's a pretty darn robust system with respect to integrity -- assuming that (for example) the disk controllers don't lie about something being synced.

Re:Redoing all my home networking by zippergz on Sunday December 03, 2017 @07:17PM (#15840496)
I used to have an ASA has my home firewall, but the problem was that its ethernet ports were 100 megabit, which meant that it eventually became the bottleneck in my internet connection. I ended up replacing it with a Ubiquiti router, which was faster, far cheaper (than even a used market ASA), still getting security updates (without a support contract), and a lot easier to manage (and I say this as someone with extensive experience with Cisco gear, dating back to the 90s). The Ubiquiti might not look as nice or be as cool of hardware, but in practice it's better in every way that matters to me.
Re:Redoing all my home networking by rayiner on Sunday December 03, 2017 @08:33PM (#15840812)
You can get some good deals here too: https://forums.servethehome.com/index.php?forums/for-sale-fo....

I've been wiring up 10G throughout the house, and managed to get a ER-8-XG (beta version) on this site for under $500. (8 10-gig SFP+). Wife made me get rid of it because it was too loud. :( Got a Dell T1700 (they're like giving them away on EBay). Quad core Xeon E3, plenty of headroom to do firewall/routing in software.

Re:Redoing all my home networking by amenghra on Sunday December 03, 2017 @05:29PM (#15840024)
That’s really cool! I have a bunch of old hardware, unfortunately I can only bring them online when I want to play with them and don’t have any permanent space.

Do you know how many watts you consume? Are alternatives to old enterprise gear from eBay going to be cheaper from an electricity point of view?

Re:Redoing all my home networking by ThrowawayR2 on Sunday December 03, 2017 @06:59PM (#15840405)
I'm kind of guessing here but:

-Cisco ASA 5510 firewall - 100W

-Dell R610 running ESXi - 400W

-Old Dell 2950 full of 2TB drives running FreeNAS - 400W

-Cisco 'Small Business' 50 port managed switch - 100W

-A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted -servers - 2 x 300W

So, maybe 1600W under medium load? A 15A circuit can deliver 1800W max., so it would be occupying one whole circuit.

Hardly worth it, considering the noise, heat, space, and power. Old enterprise hardware gets given away for a reason.

Re:Redoing all my home networking by akeating on Monday December 04, 2017 @12:29AM (#15841657)
yeah, I considered holding onto a pair of old 2950s but ditched them considering my peak electricity rate is 43c kw/h. This setup makes me shudder.
Re:Redoing all my home networking by gravypod on Sunday December 03, 2017 @05:38PM (#15840072)
How do you find data centers that trash gear? I want a setup like this eventually. There's a lot of cool things (in my opinion) I'd want to do if I had serious compute power.
Re:Redoing all my home networking by zhengyi13 on Sunday December 03, 2017 @05:57PM (#15840147)
Coming from an IT/Ops background, often the answer is, know the Ops/sysad folk in a small shop, and be around when they're upgrading.

Apart from that, places like www.weirdstuff.com might exist in your area, and have stuff like this laying around for pennies on the dollar.

Re:Redoing all my home networking by Spooky23 on Sunday December 03, 2017 @06:00PM (#15840159)
Look for government auctions. I bought a pallet of LCD screens for my son’s school for like $75.

A few years ago, they sold 4 pallets of Extreme 10/100 switches for $20! The catch is they sell lots and you have to pick them up at one time.

Re:Redoing all my home networking by loeg on Sunday December 03, 2017 @06:56PM (#15840394)
And even a few years ago, 100 mbit switches were obsolete.
Re:Redoing all my home networking by jdironman on Sunday December 03, 2017 @08:33PM (#15840816)
Can you tell me more about how you did the Dell 2950? I'm looking at NAS server options and would love to roll my own. I have 2-3 of these sitting around.
Re:Redoing all my home networking by timc3 on Sunday December 03, 2017 @06:03PM (#15840177)
This was more like the setup I was expecting when I clicked on the link, not some buying of UniFi and setting up a few VLANs and SSH Keys.

I agree with you that they are great value but I find the noise and power consumption of that generation a bit much so just built my own boxes