I have a setup similar to the OP and there's a really good reason for it: cost. For $2k in hardware (one time) and a VPS + electricity at $20/mo, I can run a 64 core, 192Gb, 24TB server. Let's call that $4400 over the course of 10 years. You would burn through that budget in roughly 1 month to get the same specs on AWS.
Obviously I'm neglecting the impact and cost of network, if it's a hobby you can just reuse your existing connection but you can always buy a dedicated line (and adjust the cost calculation accordingly)
In terms of security, you can mitigate the surface area by running a reverse proxy on the VPS. I've got nginx on the front line which does TLS then proxy_passes to my basement server at its wireguard IP address. So it's strictly limited to http, no direct database or ssh access.
I don't know if I'd ever run a "real" website this way but it's great for hobbies and side projects.
I wonder how you build such a hardware with just $2k (3990x alone seems to cost $3.5k), but with €94/mon you can get 16C32T, 128Gb, 8TB server from Hetzner[1]. It has Zen 3 5950X which has better single core performance than your 64 core assuming that you refer to 3990X, and often single core perf is more important than total perf because you don't use all cores all the time.
I estimate such a configuration to be $2.5k. With $2.5k you can keep the server for 2 years, with network and electricity provided by Hetzner. Also you can terminate it and rent another, better server at any time.
I thank the death of Moore's Law. Compute power obsoletes very slowly these days. Back in the 90s I was buying new equipment every year or two.
My primary hobby server is now 12 years old and going strong. My youngest server is 8 years old.
Every now and then I start shopping for an upgrade but eventually decide I'll spend the money later. It's still good enough for my needs and works well.
I do not regret one bit going for this set-up, although my next step toward scaling this up is indeed going to be some sort of hosted solution, the primary reason being the BTUs and decibels involved in running an actual server from my apartment.
Running a search engine or not, anyone with a serious interest in programming should have a server. I really can't recommend it enough. The ability to run large jobs that chew through hundreds of gigabytes of data for weeks if need be, without considering the cost, that dramatically increases the level of ambition you can have in your projects.
* I had an idea for a re-skinned wikipedia. So I processed all 40 gigabytes of wikipedia articles from an OpenZIM file. Took a week. No big deal. Later I used the same corpus to calculate language data models. That's another several days.
* I had an idea a while back. It required me to screengrab 500,000 websites. So I wrote a python script. Takes anywhere between 0.5-120 seconds per site. It's still not finished by any stretch, but I'm up to about 322,000 screenshots now.
It's just great.
That 64 core server is probably sitting nearly idle most of the time. You don’t just spin up big systems on AWS and leave them idle. The whole point of cloud computing is the on-demand scalability.
You can basically spend nothing until a request comes in, and most hobby projects are low traffic.
For example, I found a project that runs plex media server encoding jobs as Kubernetes pods: https://github.com/munnerz/kube-plex
When you’re not encoding anything, you’re not running much compute.
I think I’ve heard of game servers that wait for players to attempt to connect before starting the instance. If all you’re doing is playing a multiplayer game with friends that instance is going to be off 20+ hours a day.
Still, I haven’t done any napkin math on what applications represent cost savings.
True, my system is fairly static. But it scales when I need it. As mentioned, it's for my personal use. "on-demand scalability" is a bullshit buzzword in this context. My goal was to buy a machine capable of several specific tasks.
> That 64 core server is probably sitting nearly idle most of the time > ... plex media server > ... encoding > ... game servers > ... that instance is going to be off 20+ hours a day.
Not those tasks :-) I have no trouble maxxing out my disk IO and memory, slightly more challenging to keep my CPU busy but it's keeping an average load of 4-8.
If you can't keep a machine busy, just turn it off. Simple as that. I would not have bought such a beast if I didn't have a plan keep it busy!
I got 48 vCPUs and 96GB DDR3 for <$300 in 2016 off eBay. It isn't the most powerful, but it was cheap.
Together with a full standing rack on wheels, I bought them for $600 cash (plus U-Haul rental to get it home).
My trick is craigslist in a city with datacenters (PHX). There are resellers here who buy used hardware from local datacenters and resell them to smaller businesses. I've had it running 24x7 for almost 3 years now and it is still going strong.
https://www.reddit.com/r/homelab/ has some good references for acquiring hardware for a home lab setup.
The closest I can get a disposed server with those specs is at least 8k
At least that was the case when I set up my home server rack ~2 years ago. (Though 24TB might be too much since server grade SAS storage is very expensive even when it's used. So what I did was to buy servers with as little storage as possible and then just put SATA HDDs that I can get for dirt cheap.)
Is the RAM ECC ? Are the drives in a RAID configuration? Is it kept in a server rack? What router, switch, and firewall are your using?
I think in you case colocation data center would be alternative solution.
I do this for my personal web server, but also set up the network rule so that its SSH port is only reachable from my VPN.
IMO, this is super convenient. I can keep my public servers out of my home network (so clear separation of private/public networks), but still a VPN connection is required to log into any of my servers.
Here is exactly what I would do:
1. Sign in to AWS console (with my Yubikey).
2. Click "Lightsail > instances > <my-server>"
3. Click "Networking > Allow Lightsail browser SSH/RDP"
4. Click "Connect using SSH".
5. Do debugging!
In short, you can use cloud providers' web interface as an escape hatch. This just works as long as you manage the firewall using your cloud provider's network filter, such as Security Group, rather than, say, iptables.
Works even if you mess up the boot process somehow
Disclaimer: I work here
https://www.ripe.net/publications/docs/ripe-690#4-2-3--prefi...
It's a coin toss on if BT will properly allow ipv6 traffic each time the router reboots.
I've had to make provisions to force ipv4 on my machines because although composer et al. runs fine on my vps and elsewhere, if it tries to access it via ipv6 at home it more often than not hangs indefinitely.
shakes head sadly in Brexit
You forget about the opportunity cost that comes from our broken Internet where you cannot assume end-to-end network connectivity because you might have to deal with NAT and especially CGNAT ! (How many protocols end up overcomplicated (=expensive) and dead in the water because of that ?)
Most people want ipv6 so they can get more ips / static ips, but from a legislation perspective its more about future proofing and staying relevant.
My statement was clearly the use case of ipv6 a direct answer to your question. Why? as in why would people want ipv6.
Anyway have fun playing dumb.
Most of the home networks I use are still behind a NAT use a single ipv6 to interface the world and within the network each computer has an internally assigned ipv4 just in case some operating system may not be compatible with ipv4.
Still are many old routers using ipv4 only out there as not many people care to renew their hardware as long as their WhatsApp and Netflix works.
they would first ban the selling of new devices not compatible with IPv6, then forbid ISPs to advertise IPv4-only connections as "Internet", then ban the selling of new IPv4-compatible devices.
(In a few years we're going to reach anyway the point when IPv6-only connections (mostly in Asia) outnumber IPv4-only devices (mostly in Africa, which sucks because they're the least able to afford an upgrade, but then being forced behind CGNAT sucks too).)
Now, IPv6 (when properly implemented, which is another failure mode) comes with much better safety out of the box (like not being able to scan all the suffixes in a reasonable amount of time to find computers on the local network to target), but I'm still impressed that now we seemingly have hundreds of millions of personal computers "directly" connected to the Internet with at best only the OS firewall as protection (when one exists), and it hasn't resulted in major hacking issues ! (yet.)
There never really was any such thing as a safe network, but it used to be acceptable to assume LAN traffic was safe. Now we know you might be on airport wifi or a large corporate network with compromised systems and the OS and systems software needs to handle that.
I guess the hackpocalypse will have to wait for when low-end computers (like those in tvs, cameras, personal assistants, connected doorbells, home automation, refrigerators) - you know, "Internet of Shit" that you can't really expect to feature its own firewall (?) finally get upgraded to IPv6 ? (How does it look like today ?)
There's this goofy band of people that know just enough to bring their own router but don't understand why a misconfigured AAAA record can mess up the happy eyeballs algorithm and are convinced Comcast is "censoring" small business. It's funny and kind of sad to read comments from people who are certain their misconfigured networks are the result of some "international conspiracy" against them. That's probably why Comcast has it turned off on the routers they loan to customers.
You must not have met many ISPs.
https://old.reddit.com/r/tmobile/comments/97ifsv/issues_with...
I have the same network ISP for my apartment and they give me only IPv6.
I hate it.
Edit: Which I could do as my ISP and mobile network both support ip6. Y'all need better ISPs :D
If only there was one around here, but alas, they're all basically the same 2-3 ISPs - a couple of main ISPs, then smaller ones who are just using their network and renting bandwidth.
you do need to look for an ISP that gives you a static ip which isnt even possible where i live
First, network overlays are not easier to setup than VPNs. Installing and configuring a network overlay client on every device is much more work than setting up a single VPN tunnel for every network you want to access. Overlay networks are just easier to plan because there is no planning. But they're not easier to implement.
Second, and far more important, meshing all your devices into a single flat network is dangerous. There is a reason why networks are designed with isolation strategies. Introducing an overlay into your networks breaks down these barriers for you, but also for an attacker.
The only overlay network that has built in firewall capabilities is Nebula. When I started configuring its firewall rules I found myself just recreating my existing segmented networks, but in a much more obtuse way. Instead of configuring a central firewall, I was configuring firewall rules on each device.
After all my research, I'm still running the same segmented network I was running before my overlay experiments. But I would like to give some praise to both Nebula and Yggdrasil. IMHO, these are the two most existing projects coming out of this space right now.
What prevents you from meshing the individual services on a per-need basis? The fact that they're overlays makes them even more convenient for such isolation.
> The only overlay network that has built in firewall capabilities is Nebula.
Are those built in firewall capabilities really missing from the other networks?
As far as I know, ZeroTier is more of a SDN that an simple overlay (That's what made me interested in ZeroTier in the first place). If conventional "L4 transport-layer" firewall/routing capabilites are enough for real networks, then ZeroTier SDN capabilities are probably enough for it's virtual networks. Granted, it doesn't have some nice built-in high-level generic "L7 application-level" firewall capabilities, but I don't trust those in the first place.
* Exposing a server running on the home network (behind NAT on a dynamic IP) to the internet.
* Doing so by renting a cheap VPS and using wireguard to forward traffic to the server at home.
I love wireguard and use it continuously. My phone has always on wireguard to my home network so all my phone traffic goes through my home router/dns, I can access the various private servers I have at home, get dns based ad blocking etc. I use an ISP that give me a static IP so it was easy to set up. It works like a dream.
That said when I want to run a public server I just rent a VPS and run it on that. I don't want anything I don't own initiating connections to anything on my home network in any way.