IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.
Yes and no, really depends for what workloads. But freaking reliability that's for sure.
That's once again an oversimplification, performance is excellent for the intended workload.
> It's 2022 and we're about to rediscover something we know for 40 years already: mainframes are freaking expensive.
My interpretation: AWS is freaking expensive, AWS is in the center performing all computations; AWS is the mainframe we can well do without.
API access for managing configuration, version updates/rollbacks, and ACL.
A solution for unlimited scheduled snapshots without affecting performance.
Close to immediate replacement of identical setup within seconds of failure.
API-managed VPC/VPN built in.
No underlying OS management.
(Probably forgot a few...) I get that going bare metal is a good solution for some, but comparing costs this way without a lot of caveats is meaningless.
[bunch of stuff I don't need]
Exactly. Imagine paying for all that when all you need is bare metal.
Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do.
Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy.
Buy what you need.
Having all that "best practice" service is great if it works well, but when it becomes a checkbox on the purchase order then it can cause far more problems than it solves.
I have found that a lot of the push to outsource hosting is simply an attempt to deflect responsibility for problems rather than in expectation of actually providing more reliability.
For every company or startup that thinks they need 100% uptime the reality is that they can not only get away with much less but in practice will end up with much less anyway because the effort and moving parts (load balancing, distributed database, etc) will typically result in something failing anyway even if the underlying hardware is indeed 100% up, and somewhat surprisingly to quite a bit of people here, manage to survive and thrive despite that (the recent AWS outages took out a lot of services and products and they still seem to be around somehow).
Ugh. I hate that phrase. The translation into plain English is almost always "What I read in some blog" or "Because I want to" or "It's what our sales rep told us." Even from C-levels who should know better.
Yes.
The opposite is also true though: Imagine not wanting to pay for that and needing it!
There's a reason why most homes connect to the power utility companies. Yes, we can run generators ourselves. Does it make sense to do that? Not usually.
Same thing with this server. If it makes sense for your use-case, outstanding. In many cases, people are better off offloading this to another company and focusing on their strengths.
The minute gens/solar/wind/batteries combo becomes less expensive than public utility I'll switch. For now it makes no financial sense.
With the clouds it is the other way around. My dedicated servers running my software kick the shit out of AWS performance wise for a fraction of the price. And no I do not spend my days "managing" it. I can order new dedicated server and the same shell script will reinstall all prerequisites, restore data from backup and run it in few minutes add (however long it takes to import database from backup). Where needed I also have standby up to date servers.
Other than running this script to test restoration once a month my management overhead is zero.
See... anyone can pick <random thing>, and describe how people don't do it. Of course, you cite a generator, others use solar cells, right?
Looks like sensible/nonsensical based on who you are and what are your needs. Not everybody needs cloud, not everybody can self host.
And that's fine. People should make choices like that. I was objecting to saying people paid X for RDS and others paid Y for one bare metal server. Those are not in the same category, there's no point comparing those prices without a lot more information.
I don't care where those snapshots are stored and how much space those take. In case I need to restore my IaaS provider gives me 2-click option to restore - one to click restore and 2nd to confirm. I sit and watch progress. I also don't care about hardware replacement and anything that connects to that. I have to do VPS OS updates but that is it.
I do my own data backups on different VPS of course just in case my provider has some issue, but from convenience perspective that IaaS solution is delivering more than I would ask for.
But of course I am running build tools and bunch of other dev tools locally that are not even installed on our servers.
But in all reality, laptops are really, really powerful these days.
My laptop at the moment is a HPE something ... HP 250 G6 Notebook PC according to dmidecode. I used to have a five year old Dell 17" i7 based beast but it ... broke. I whipped out the SSD, shrank the root fs a bit (with gparted) and used a clonezilla disc and an external USB link to get my system onto a Samsung EVO M.2 thingie from the SSD. This laptop was an employee cast off/
As is probably apparent from the above, I use Linux. I have some decent apps at my disposal but in general I don't need much hardware. Decent: RAM >= 8GB and SSD storage are key. I don't play games much. I've always specified 17" screens in the past for my laptops but now I have to use a 15" jobbie, that's becoming less of a hard requirement.
I am a Managing Director (small business - IT consultancy) but I do spend rather a lot of my time doing sysadmin and network admin stuff. I also do business apps. I once wrote a Finite Capacity Plan for a factory in Excel with rather a lot of VBA. Before you take the piss, bear in mind I used the term finite and not infinite. That meant that quite a few people had employment in Plymouth (Devon not MA) in the 1990s. I won't bore you with my more modern failures 8)
Anyway as you say, modern laptops are phenomenally powerful. Mine have wobbly windows 8) I can't be arsed with MS Windows anymore for my own gear - it gets in the way. Me and the wife said goodbye at Windows 7.
They tend to be the things that you don't need until you absolutely need them right now (or yesterday).
It’s easy to trivialize $20-30k a month when it’s someone else’s money and it’s less work for you.
People on my time would rather be developing software than doing sys admin stuff
I use AWS, DO and Hetzner (bare metal) for (different) cases where each makes sense.
In the past few years, maintenance burden and uptime between AWS and Hetzner hosted stuff was comparable, with the cost being an order of magnitude less for Hetzner (as an added benefit, that machine is more beefy but I wasn't even comparing that).
> People on my time would rather be developing software than doing sys admin stuff.
I am not surprised software developers would prefer doing software development :-)
No, they often don’t understand the alternatives available because the wolves are guarding the henhouse (software devs who don’t want to work with the pace of on-prem IT).
> People on my time would rather be developing software than doing sys admin stuff
No shit, that’s why software devs aren’t sysadmins.
What I have a problem with is:
- the premium over bare metal is just silly
- maximum vertical scaling being a rather small fraction of what you could get with bare metal
- when you pay for a hot standby you can't use it as a read only replica (true for AWS and GCP, idk about Azure and others)
Though it seems to require you to have 3 instances rather than just letting you read from the standby... I don't quite get the rationale for that.
I'm not sure what you mean here. At least for MySQL you can have an instance configured as replica + read-only and used for reads. Aurora makes that automatic / transparent too with a separate read endpoint.
The fact that a hot standby is usually in some sort of read-replica state prior to failing over is a technical detail that AWS sort of tries to abstract away I think.
https://aws.amazon.com/blogs/database/readable-standby-insta...
>> 1. A solution for unlimited scheduled snapshots without affecting performance.
You can very comfortably have instant and virtually unlimited snapshots with zfs/jails (only occupying space when files change). Very easy to automate with cron and a shellscript.
>> 2. API access for managing configuration, version updates/rollbacks, and ACL.
>> 3. Close to immediate replacement of identical setup within seconds of failure.
There is a lot of choices for configuration management (saltstack, chef, ansible, ..). I run a shellscript in a cron job that takes temporary snapshots of the jail's filesystem, copies to a directory, and makes an off-site backup. A rollback is as simple as stopping the server, renaming a directory, and restarting it. It's probably more than a couple of seconds, but not by much. I think I'm uncomfortable exposing an API with root access to my systems to the internet, but I'm not sure how these systems work. I don't think it would be hard to set it up with flask if you wanted it though.
>> 4. No underlying OS management.
I don't know what this is, but I'm curious and looking it up :D.
In most of the posts I'm reading here, people have really beefy rigs. But you could do this on the cheap with a 2000s era laptop if you wanted (that was my first server).
Yes, but that both requires manual scripting it and remains local to the server. Compare to scheduled RDS backups which go to S3 with all its consistency guarantees.
> There is a lot of choices for configuration management (saltstack, chef, ansible, ..)
Sure, those are an improvement over doing things manually. But for the recovery they can do only so much. Basically think how fast can you restore service if your rack goes up in flames.
> I don't know what this is, but I'm curious and looking it up :D
It means - who deals with kernel, SSL, storage, etc. updates, who updates the firmware, who monitors SMART alerts. How much time do you spend on that machine which is not 100% related to the database behaviour.
I wasn't recommending everyone use RDS. If your use case is ok with a laptop-level reliability, go for it! You simply can't compare the cost of RDS to a monthly cost of a colo server - they're massively different things.
EDIT: I do use cloud services for some stuff. My point isn’t being anti-cloud, just that nothing is perfect.
For instance, there are LVM2 snapshots. Maybe those do affect performance. If the cost difference is big enough though, couldn't you just account for that in the budget?
I agree that literal "bare metal" sucks, but self-hosted with cloud characteristics (containers, virtualization) is not totally obsolete.
When you're using the cloud you're paying someone else a very good margin to do all of those things for you.
If you do them yourself you can save quite a lot. Hardware is surprisingly cheap these days, even with the chip shortage factored in, if compared to cloud offerings.
No access to the underlying OS can actually be a problem. I had a situation where following a DB running out of disk space it ended up stuck in “modifying” for 12 hours, presumably until an AWS operator manually fixed it. Being able to SSH in to fix it ourselves would’ve been much quicker.
"Identical setup within seconds of failure" being the exception as you need to deploy from backup which can take minutes/hours (depending on backup size) even if you have the spare hardware. Fortunately, that's the least-needed feature in your list.
It's called software, you put that on top of a server, it's not some kind of magic.
Sure, you're paying the cost of some DBAs and SREs with that price. Still, seems too way above what it costs.
Factor them all in, and "the cloud" is 1000s to millions of times more expensive than bare metal. I've seen people pay $100k/month, for services which could run on an $140/month bare metal server failover pair.
Meanwhile, people still spend enormous resources managing "the cloud", writing code to deploy to the cloud, dealing with edge cases in the cloud.
There are no savings, time wise, management wise, or money wise, with the cloud.
You are paying for ignorance.
The costs were very clearly spelled out and always lower that SaaS.
Why do people here think that cloud companies are charities? The cost of their hardware leasing, personnel, electricity, building management, insurance... it is all paid by the customer.
Plus a beefy profit margin. All paid by you.
Your AWS bill is directly proportional to the number of developers, because they all spin up cloud toys they are fond of.
Idle power consumption of an R730 with no drives is typically about 85W, though higher-end CPUs drive that up a bit. The older R720s are a bit higher in the 100W range. When the extended warranties expire, high-spec 2nd hand equipment floods the market.
20% load (9 VMs with 11GHz usage), 196W.
Historical trends for Last Week:
Avg: 203 W | 693 BTU/hr
Max: 279 W | 952 BTU/hr
Min: 150 W | 512 BTU/hr
Historical Peaks:
465 W
Cumulative Reading:
Since Mon Jul 13 11:32:33 2020 2723.893 kWh
UK <Apr2022 = £50.96 ($68 USD), £0.2022 price cap rate
UK >=Apr2022 = £76.16 ($102 USD) £0.3022 likely new price cap rate
Thus the reason r/homelab users in Europe tend not to have Craigslist blade servers of any sort!
That's the line the clouds, MSPs, and service contract providers sell you on, but it's never true. When you need help you're getting some minimum wage slave who hired on 6 months ago, and is struggling to juggle the dozens of other clients competing with you for his time, too. When you don't need help, they get to take your money for doing nothing.
Anything your provider can do for you, you can do at less cost. Cut out the middle man and you're far better off.
All that knowledge is now in-house, and you have access to it at a moment’s notice.
I think it's also worth considering that many outfits wouldn't get good value from the 24/365 coverage you propose and don't care to pay for it.
To pay them around 70K, you need to budget for 120K. It will go into social security, taxes, training, overheard of management, HR team, other costs centers like a margin to account for sickness, absenteeism, people who are trying to "find themselves", personal crisis etc...
If some cloud setup can halve your sysadmin labor, then a company with 20 sysadmins can go to 10 sysadmins, but a company with 1, 2 or 3 sysadmins still needs all of them.
____________
That's how I've seen it working in datacenter:
cheapest junior admins (450$/month) were having night shifts
and if something broke, then they were calling an engineer
For 1 bare metal server? You do know these things run on electricity, right?
The main difference between what we've got and a proper setup is on the order of about €5k of fixed costs, all of which have a lifespan of ~decades. The savings from owning your own hardware might shock you.
Also, if you think that the "professional" hosts do any better, consider that that Hetzner box you linked is a desktop machine shoved onto a rack shelf.
This.
People think "professional" and act like it hides the fact that it's all the same people running stuff on a shoestring budget.
That $50/mo boxes in Hetzner tend to stay up for years it a testament to that there isn't any black magic or huge costs to keep them running.
My wife is quite concerned about her being left with a web of home automation, hosted emails etc. I understand her and I am trying to find a way out.
My current idea is to document how to de-automatize the home and how to deal with emails and fiber access (the main things to worry about).
Any ideas are very much welcome
Technically option 3 should be the best since it also engineers around generations (options 1 and 2 would roughly be locked to a narrower age group). But it also can be a double edge sword, what if they don’t like tech? Or if you overdo it with trying to make them interested in tech and self hosting and it backfires?
So yeah, no real solution yet. But I’d subscribe to that newsletter if there was one
3 could have been a solution but my kids are not interested in development (if they would, it would have been my first solution)
Maybe some kind of script(s) that could be run that just do all the de-automation?
People who are experts in cars can own very expensive cars and tools to tune them.
People who have been working in music can have very expensive instruments and expensive headphones, microphones, sequencers, etc.
We seem to be looking down on experienced "computer experts" and wanting to take their tools away. It's been grinding my gears lately.
Server hardware is fun but it's not trivial to manage, buy or run.
So when someone talks about how they've managed servers for 2 decades, own a house where they can install a 42 rack and how much better it is than a hosted solution. A lot of people rightly point out that this is hardly feasible for most people
Also I said _not_ buy a rack, just put it on a shelf.
> You pretty much get all the pros and cons of home ownership.
And that's the heart of the matter. Everyone here is arguing in circles based on their feelings about cloud vs their feelings on bare metal (and from what I can tell, pride in their own abilities), but at the end of the day it's a cost-benefit tradeoff. Everyone picks that tradeoff for themselves. As my immigrant parents are getting older, I'm thinking of getting them a business internet line and a SIP phone in their house so that if they need me in an emergency (health or otherwise) they can reach me quickly/reliably. It's something I'm weighing based on the cost of the service/infrastructure/maintenance against my parents' technical and social capabilities (which are limited as older immigrants from a non-Western country.)
It's all about tradeoffs. As computing professionals we have the knowledge and the skills to uniquely take advantage of these tradeoffs in our personal lives. Much like I know quite a few cabinetmakers who do actually make furniture for their house with their skills, their tools, and the shop they work in. That doesn't stop most people from buying furniture from Ikea or most businesses from using cloud hosting.
> <erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
Heaps of Pis in rafters will quickly turn into a cable spaghetti hell tied into ugly knots.
Introducing the... "RafterRack™" ;)
My closet of RPi's are quiet.
Your main point still stands though, not everyone can or wants to do that.
Then there's a set of about six or seven $4 per month KVM VM at geographically distributed offsite 3rd party hosting companies for things like secondary authortative nameservers and such.
And an offsite backup system that mirrors everything on the system in colocation to a medium sized disk array that lives in the corner of a closet in a family member's basement.
I don't think systems administration is a lost art, people just wanted to do things easier for less money and staff, and no one can blame them or the industry as a whole. The art is still right there, linux hasn't changed that much in 20+ years, I still have my first discs. It also takes some experience to "feel" your way through systems administration problems, working from the bottom up of course, feeling like its a network problem still has a high rate of success when troubleshooting for me!
I try everyday to teach some of these skills to people I work with, I just call it devops, or SRE, or some cyber cloud support position someone makes up, they are all systems admins/engineers to me still. I enjoy watching people learn and apply that knowledge to future problems, getting to the root cause or close to it, and the satisfaction that comes from fixing the issue from start to finish on your own, looking things up is not cheating in systems administration!
[0] https://raspberrypi.stackexchange.com/questions/135610/conne...
[ 150.076220] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:468 dev_watchdog+0x308/0x30c
[ 150.076255] NETDEV WATCHDOG: eth0 (bcmgenet): transmit queue 1 timed out
I run my stuff from home too, though it is smaller scale than yours currently. Off-site & soft-offline backups are on encrypted volumes on servers & VMs elsewhere.
was that intentional? my current understanding of MIT license is a "do anything, i don't care, just don't blame me".
if i were you and wanted to use "dont use our resources" part, i would set it as AGPL or even SSPL if you are brave enough.
I get my static IP by using the smallest VPS from vultr with a wireguard tunnel forwarding http/s traffic to a docker container running nginx proxy manager.
For those wanting to learn, I highly recommend joining r/homelab and r/selfhosted. Those communities have a lot in common and you can learn a lot.
Most modern routers have an option to integrate with dynamic dns providers.
Knowing how to keep your server running and understanding internals is a great skill, but that doesn't mean that progress should stop.
Standalone servers are great, but this greatness comes at a price. It takes time to maintain server, it takes time to configure additional services. But at the same time they bring you joy (and frustrations) and much more knowledge and deeper understanding of what goes under the hood.
I'm gonna run my private cloud merging 3 different un-backed up physical computers, and migrate services off Google.
That's my second free 42U track, but the other was mostly used as shelf space. I've got a third rack rusting in my backyard which I bought for locally for $200, originally intended to run my former employer test infra, that I brought back home after they laid us off.
I have venting into my ceiling which connects outside but I am mostly using consumer gear like Raspberry pis and Intel NUCs and the rack fully fills the doorway so volume and heat is not a real issue.
As far as you know?
Your data is exposed to The Internet so someone could be accessing it.
Maybe I am naive, but I think the chances of this happening on a cloud based service like Dropbox are going to be a lot lower.
https://infosecwriteups.com/how-i-discovered-thousands-of-op...
Who will keep maintaining your infra when you die, say you get hit by the infamous bus later today?
I however mostly detest how almost every third party SaaS sells out our metrics and data for profit, long term consequences be damned. The less I empower profit maximizing machine learning to manipulate me and my family the better.
I do not even need to trust my friend as duplicity encrypts all data against a yubikey held pgp keychain before it leaves.
The backup NAS could phone home from anywhere with internet access.
When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber.
I know where our data lives and no one can access it without a warrant and my explicit knowledge. I also save myself several hundred a month in third party cloud provider fees to host the same services and can reboot upgrade or repair anything whenever I want, but in general no more maintenance than cloud servers . I also never end up with exciting bills when experiments are forgotten about.
You pretty much get all the pros and cons of home ownership. For me it is mostly pros. Also keeps me dogfooding all the same practices I recommend to my clients.