Is it? It has clean and logical abstractions, and consistency. Services depending in each other isn‘t complex or difficult to understand.
I suspect that a nice GUI would make systemd quite usable for non-expert users.
BTW: It‘s called ”systemd“:
> Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. [0]
[0]: https://www.freedesktop.org/wiki/Software/systemd/#spelling
For a technologist or engineer, yes. For a layperson, no. The average consumer who desires privacy is probably neither a technologist or engineer, so the longterm target is something that just works.
Laypeople also aren't going to entertain the kind of pedantry that is systemd vs systemD vs System D vs SystemD so making systems that abstract further away from those communities is beneficial.
Edit: Thank you for your correction, as a systems engineer, but I couldn't help but highlight this is a big hurdle even in the Linux communities that I've been a part of as desktop Linux as gained wider adoption by laypeople.
Lay people work in a factory or shoe store or accounting firm. They have 1 or more kids. They hear what their friends are doing, and as long as it only requires signing up on a web page, they will consider it. They will use the same password as they do for their bank.
And that’s FINE! There is life outside technology, and those lay people are busy living it.
I wouldn't call them sophisticated admins, and I wouldn't trust them with anything mission critical, but the servers often work.
[1]: http://stackoverflow.com/questions/44586441/ddg#44586701
* No previous tech knowledge
* A full-time job
* No external help
learn systemd, or how to setup and maintain a web server, or whatever. Individuals may still self-host of course, but I'm skeptical people will do it en masse anytime soon. After all, if the entry barrier weren't so high, we wouldn't have an entire service industry that does this for you (SquareSpace, Wix, Substack...)
Would be nice if there were more little appliances that handle stuff like this so someone could just buy it at best buy or whatever and plug it in next to the router.
You ... might be surprised.
But there is a wide range of knowledge in this space. A 'lay person' is kind of hard to get down to a spot without potentially lumping people together improperly.
Take my customer for example. She's not stupid. She knows she could probably fix the computer I am fixing for her, herself. But she also knows there are things she just doesn't know and probably should get someone else who does know those things to do it instead. Is she the lay person? From talking to her, I am pretty sure with a couple quick hours of reading some stuff, she could probably set up a basic Nginx server or something very easily for herself. Etc, etc. Yet, by others standards, she is a lay person because she can't utilize Apache or AWS.
So, yeah... that's my take on it.
But the fact remains that I have a A+ book from Comptia left unread that I found at a 2nd hand store, a book on Bash that I found at a bar also unread, and a book on Python from O'reily via Amazon; yes, unread. All of them unread.
Why? I dunno. Can't be damned to? I taught myself pretty much everything I know through trial and error. If I want something in a program to do something else, I alter the code and reverse engineer it until it does what I want. This is how I make mods for games usually, when I tinker at all. How do I know what to tinker with? I read, ironically. Yeah, I know. Go figure, right?
But still, those books lay unread in a desk next to me as I type this.
So what am I reading? Well, it's quite simple. If I ever get to a point where my own intuition or knowledge isn't good enough, and I don't see the answer staring me in the face via some comments in the code, or some error code thing, etc, etc...
I google it.
Unit files → fairly simple text files that are used to define such stuff for systemd (usually in /etc/systemd/system and /usr/lib/systemd/system).
Package manager → essentially the same as an app store on a phone. It's how you install and update packages (packages could be a fully fledged graphical app or just a terminal command).
Webserver → what allows you to "run websites" on your own machine. For the simplest example, if you run `python -m http.server` in a terminal on a Unix-like system, congrats, you can now browse your files in a browser (127.0.0.1:8000). If you know your local IP, you can also open it from a phone and download stuff from your desktop, zero apps or cables or FTPs necessary.
So, to decipher that comment above: you usually don't have to worry about how things run automatically because that's usually pre-configured when you install them. In most simple self-hosting scenarios you just "install a website" on your spare laptop or whatever and you're good to go. That website usually serves traffic via a webserver on some port, and you access it via local network IP and a port (example: 192.168.1.100:8000).
Now if you go deeper and want to run multiple things simultaneously, each accessible via a domain instead of a port, accessible from outside of your home, properly backed up and with a valid HTTPS connection, and then you hear about this thing called Docker... well, from my experience, you're gonna wake up on a day like this and go to work as a sysadmin with 3 years of experience, basically writing YAML for a living.
So in conclusion, apart from some outliers like Plex.tv, I wouldn't call the process layperson-friendly, but hey, it might make your tinkering into a career.
I have friends that selfhost stuff on mac, windows, and linux. They are probably above average as far as tech goes but they are all 30 somethings in non-tech jobs, real-estate, finance, advertising, sales.
I field questions from time to time, but it's pretty rare. Never once got a question about an init system. On any os or distro.
These guys aren't writing init scripts, they are going through the apps documentation and typing 'systemctl start appName.service' and then moving on.
or starting it with docker or even one of the clever little web UIs that you can install that helps you install selfhosted apps.
I thing there's a part of us that wants to believe this shit matters to normal users but they just don't care. And if the service doesn't start, they file a bug upstream and the maintainer usually takes care of it.
- Some people want to talk about how difficult things are, and come up with reasons to make things sound difficult,
- Some people have an axe to grind, regarding systemd.
The original poster brought up "workload scheduling" and while I've done plenty of that at work, I'm at a complete loss trying to guess why you would need workload scheduling for any self-hosting project.
Ah.. I was there when it was all SystemV vs Upstart vs Systemd... the golden days...
Let's start treating tech as the world treats everything else: Ignorance is not a justification
Sure, there's a midpoint where we would ideally want as many people to be able to use a technology as possible, which means making things easier, but we underestimate the capability of most users and dumb things down so far that those users not only believe that X technology is too hard but believe themselves to be too stupid to do anything that requires even the faintest amount of know-how.
I do think we've gone way too far in the direction of acting like everyone, except we archmages on HN, have the intelligence of toddlers and can't figure out anything for themselves. Everything becomes so easy, but at the tradeoff of everyone being dependent on the centralization.
The incentive has to be there, though. Convincing people to self-host is like telling them to eat more broccoli. Just because it's good for them doesn't mean they're going to do it, and companies of nearly any size certainly won't choose to make things hard on their users.
We would be much worse off. In order to survive, everyone would have to become a dabbler in everything and many would be unable to keep up at all. We would lose all of the efficiency gains of specialization.
So assuming we do not want everyone to have to learn a bit about medicine and law and engineering etc, why is computer knowledge different? Is a computer not merely a tool to accomplish a task? We don't expect people to learn to become a mechanic, let alone an automotive engineer, to be able to drive a car. We expect cars to be reasonably easy to drive and low maintenance, with occasional help from mechanics. Shouldn't we expect the same from computers?
We can be realistic about making hosting accessible, or you can bury your head in the sand with enthusiasts as AWS becomes the entire internet. It’s already pretty obvious which approach is winning.
People working in other sectors, maybe with a family when they come home, do not have that skill or luxury.
And speaking from experience, documentation is often greatly lacking. For example just today I had to thumb down a couple of google docs because it was riddled with inconsistencies and lacking crucial information. And that’s a company with near infinite money. And its like that for most software, with great docs an exception rather than a rule.
The number of people who can learn all of the skills that someone in the world considers essential is small. The number of people who can do all those things while having a job, kids, and a working partner is smaller still. The rest of us have to focus on a proper subset.
For me, every minute I waste learning how to configure my server is a minute I could have been playing Legos with my kids or riding my bike or sleeping.
But eventually they will be old enough to require an email address, so should I just let them use Gmail and stop worrying, or try to have something up and running until then? At least protect them from the data kraken until they can make their own informed decisions about that. But then I'm not just responsible for my email, but my whole family's email. What about 30 years down the road, when they long moved out? Am I still gonna sysadmin my family's self-hosting galore when they finally have kids on their own? Is it even still possible then? At least for mail it looks like when it's growing over my head I can just sign up with some mail provider (that hopefully values privacy) and let them handle that whole BS.
It might not even be worth it anymore at some point though, if basically everyone else you communicate with via mail is on Gmail. Then Google already has all my email as well.
An awareness campaign to go the extra mile in terms accessibility would be great (and that doesn’t mean dumbing down, the two seems to be equated sometimes)
But also at past companies there was never the idea that internal tools had to support accessibility features, as we give for granted that we would never hire people with some sort of disability and they are just users that stay at home, it's sad and upsetting, as a person who doesn't need any accessibility feature to operate now, I really feel sorry for others who do, and I hope that I will never need them, because like your situation with products at employers like control panels and similar is very sad
The best I managed to do is that my family's email addresses use a custom domain I manage so I am not locked into a single webmail provider. Occasionally I have to deal with some issue or delivery delays (especially for authentication emails) but it's orders of magnitude less effort than when I made the mistake of trying to run my own mailserver.
Being self-sufficient is a noble goal, but I think it's also helpful to be realistic about the costs in order to make the proper cost/benefit decisions.
For the layperson that's a bus driver by day, not so much, most definitely, but I think no matter how simple you make setting this up, just having to maintain this and having to do something when the setup eventually has some unexpected bug a few years down the line is just too much.
There's this strange idea that laymen self-hosters are doing all this stuff and the choice of init system matters in the lease. They were going to run the command to start the service blindly regardless of which init system it was and if it breaks, they aren't going to really troubleshoot either one.
In comparison to system V initd startup files, systemd unit files are, arguably, less complicated.
I'd say the "complexity" of systemd unit files is mostly irrelevant to end users.
For a relatively non-technical user, implementing whatever application/service one might want to use should be as simple as installing the relevant package(s) and dependency(ies) via existing, well managed package management systems.
That said, too many developers encourage self-hosting, but don't provide appropriate packages and defaults for most popular distributions.
If developers spent just a little more time creating buildable packages (supporting the creation of binary and source .rpm, .deb, etc. packages) with sane defaults/startup files could make the inclusion of such apps into the standard/extras repositories of a broad swathe of Linux distributions much simpler and, for the non-technical user, easy to install and configure.
Matrix Synapse[0] and Diaspora[1] both come to mind in this respect. Installation and configuration of these platforms requires the installation of several software development frameworks and separate (from the standard system package managers, e.g., DNF, apt, dpkg, etc.) package management tools for the language dependencies.
Requiring installation of software dev environments and building the software/databases/admin tools for such "self-hosted" solutions just confuses non-technical users.
As a professional with decades of Unix/Linux implementation and management experience, I find implementing such platforms simple enough. Just read the docs, install the dependencies and compile/install/configure the software.
For a non-technical person, that's likely a non-starter unless there's a UI that will do so automagically.
Fortunately, there is such a UI for most Linux/Unix distributions -- it's called the system package manager.
Unless and until developers provide distribution developers/maintainers with appropriate packageable sources (or even separate repositories with binaries!) to be added to the default repositories, self hosting many apps will only be the purview of technical users.
This annoys me. A lot. Not because I, personally, mind a complicated set up process for such applications, but because it limits the ability of both Linux/Unix distributions and self-hosted applications/platforms to be used more broadly by non-technical users.
Especially with tools like Diaspora, Matrix/Synapse and others which have the potential to overturn centralized hell holes like Twitter, Facebook, Instagram, WhatsApp, etc.
It's been at least five years since I first installed a Diaspora pod and a year since I installed Synapse and a STUN server. In both cases, had I not been a long-time user/manager/implementor of Unix/Linux and associated sw dev environments, the install would have been nightmarish.
For both platforms, installation pretty much requires knowledge of software development tools and practices, as well as more than a passing familiarity with Unix/Linux shells and environments.
I can't imagine my 64 year-old sister in-law (a reasonably well educated and smart cookie with decent problem-solving skills) taking the time to learn how to use git, clang/gxx or even docker to install this "self hostable" stuff.
That should be the target audience for such self hosted tools, not devs and other technical people.
Taking the time to make one's application/platform easily installable/configurable (and building from git repos and/or Docker-compose aren't "easy" for non-technical folks) by non-technical end users could make a huge difference in this space.
Further more, they should require absolultely zero configuration to get up and running. We've had SQLite for many years now, so it's easy to make a webserver that does not require a separate sql server to be installed and configured in order to get going. (There are also non-sql embedded data storage engines, many of them written in pure Go, eliminating the CGO problem).
Ideally the user will download a program for their own operating system (say, windows). They will run this program, provide it with the ip and root password of their linux server which they rented from some provider, and this program will then upload the actual server program to the host machine and launch it.
It would have been much harder to pull this off if these services required large interpreters, adjacent daemons running, and complex orchestration. Given the amount of time I have, I might have just skipped the SELinux step.
I want to edit a service to harden it for example. Oh, wait I shouldn't edit it directly with vi? Because it gets overwritten by package updates. Okay, makes sense, I need to use systemctl edit instead. But that opens a file that has everything commented out. Do I uncomment the [Unit] heading? What do I need to keep and where do I add my additions? I recall there being a comment at the start of this file, but unless I'm misremembering it doesn't answer that.
All I ask of it to do one thing - start something.service after other.service. yet it just refuses to order them this way. Why? I have no idea. I also have no idea where to start debugging a problem like this. There's a billion ways to try and do this after all: do I add Before=something to other.service? Do I add After=other to something.service? Both? Wants=something?
I don't think you're supposed to edit anything inside /usr, except perhaps /usr/local, but even that has a better alternative in the form of $HOME/.local, which is a well defined standard at this point.
Maybe I'm unaware or mistaken but if you're editing anything inside /usr as a normal user, you're either using Linux wrong or doing something unexpected or unusual. This is why requests to make GUI file explorers have root escalation capabilities sound absurd to me. I can't think of a reason why one would need root access when using a file manager, especially a GUI file manager.
I'm not doing any of this as a normal user. I'm doing all of this as root
Anything inside /usr should only ever be modified by the package manager, not by the root user, or any other user for that matter.
If you want to make system wide changes, make them in /etc. If you want to make user specific changes, make them in $HOME/.config, $HOME/.local. Your package manager should never overwrite anything in /etc or $HOME. If it does, it's a bug.
I'll admit something; I forgot where the service files were (I don't use systemd often). So, in my unix wisdom I used `find` to find the files ending with 'service'. The only ones that came up were in /usr/lib
You're right, I'd typically never edit files in /usr. But I wasn't sure how else to do it at the time.
I wish package managers would make patching packages easy, this kind of thing is so much more manageable on Nix.
This doesn't happen. The package manager installs the new configuration under a different name so that you do not lose your changes and can merge them easily.
It's like saying a car is too complicated because I can't just swap the engine out without any prior knowledge. It's not designed for the average guy to be able to do that.
systemd is not some product meant for the average Joe, it's an integral part of the system for managing services and other things. To run a web server at least somewhat reliably you don't even need a lot of systemd knowledge but you still need to know some networking, firewalls, DNS, in general how the internet works, how to configure a web server and other services, some basic security. If you don't want to learn these things then there are managed services that you pay to do those things for you or you hire someone to run it for you. Just like you take your car to a mechanic when you're not interested in figuring out how to reassemble the engine.
Yeah, things can always be improved and made simpler but to create something fool-proof for the average person would take a huge amount of work and there would need to be a business opportunity there for someone to invest in that or there would need to be some passionate generous soul that would invest their time in a project like that.
And like I already mentioned, there are already solutions to the "it's too complicated" problem: 1) companies offering managed services, 2) companies/individuals for hire to do it for you.
I have lost track how often that happened with sysvinit, because the "logic" how to treat customisations was usually handled by the package manager and they messed it up regularly.
systemd has a standard way to handle customisations. As long as you put everything you do in /etc/systemd/system, everything is fine. It's simple and works across distributions.
Traditionally, init scripts were installed into /etc but package managers (or at least some of them?) took/take care to not overwrite files under /etc but instead let you merge in the new changes.
The better question is why should a layperson ever need to know about systemd in the first place.
I'm a technical person and I don't know about systemd, nor have I ever needed it.
When designing products for end users who are not technical, the overriding design goal should be easiness and reliability.
Easy: does not require special knowledge or expertise to operate.
For someone who just wants to self host, having to learn about systemd, by definition makes the product/system not easy.
Reliability: does the system implode in unexpected way based on the slight variations in the environment? If it implodes as such then it is fragile. If it does not, then it is robust and reliable.
For the end user, what matters is that the system is easy and reliable.
If the system is easy but not reliable, the user is effectively forced to become an expert in order to fix the system when it breaks. Thus, if a system is not reliable, then it doesn't matter that it's "easy".
Simple/Complicated only concerns the developers. It's important to keep systems as simple as possible under the hood, because a system that is simple is easier to make reliable than a system that is complicated. But for the end user it does not matter directly. Simplicity is about how many parts the system is composed of and how these parts interact. If there are many many parts that all have to interact with each other, then the system is complex.
Maybe once someone learns about systemd they can find it conceptually simple. But that's a moot point. The point is: they should not even have to learn about its existence.
A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise.
Imagine yourself a windows user. You edit a text file. Restart the machine, and now it doesn't boot, or boots in a special text-only mode. (This is not unheard of on linux based systems).
Sometimes HN threads seem to self-sustain forever starting from a pointless argument and discussing it like it was the truth.
Learning systemd is not necessary for anything in day to day life, not in desktop use, not in self hosting, default configuration from package managers work fine out of the box. At most you need to learn how to start, stop, disable and query service status.
I've been using linux since decades, way before systemd, never had to learn systemd until I needed to ship my own custom services on my own custom devices. But at the point you're way far from being a layperson.
> A system where editing text files can make or break the system, is not a reliable system. It's easy to make mistakes in text files. Specially for users without expertise.
I don’t think that’s a salvageable notion of “reliability”. I could bork Mac OS or Windows easily enough by editing text files, if I really wanted to.
Editing text file to make advanced configuration may be acceptable if the casual end user never needs to do it. When I say never, I mean never ever ever. If you need to edit it once every few months, you better have really really good "linting" facilities to tell the user right away whether their edits are going to be accepted or going to cause the system to break.
Now granted, systemd is a system utility, not an end user application.
If the system overall is built such that the end user never ever ever needs to know about systemd, then it's sort of ok.
Now, when it comes to self-hosted, the line between "system utility" vs "end user application" may be blurred.
If you think a web server is a system utility, then by all means, go ahead and make it fragile and complicated. That's how everyone is doing it.
But if you think - and at least I do - that it's important for casual users to be able to self-host their own websites with ease and reliability and without having any system level expertise, then the points I mentioned about the system not imploding due to the content of text files is extremely important.
I absolutely have problems with configuration via text files.
A lot of things on linux desktops that a typical user wants to do require editing text files to work. For example, try searching for information about how to enable Japanese input in Gnome or XFCE desktops. Invariably, the steps involve installing and configuring some specific packages, and editing several text files to let the system know about the fact that you want to use an IME for text input.
This is unacceptable.
Well, that’s fine. We’re mostly talking about self-hosting web servers, and you don’t need to know anything about systemd in order to run a web server on Linux, any more than you need to know about fuel injection computers because you drive a car.
The reason I’m familiar with systemd is for other reasons besides hosting a web site.
I don’t know how problems with IMEs or desktop Linux got brought up, I’m not sure I’m following the conversation.
> You take something that's […] easy to understand for yourself, and somehow generalize it to everyone. "If it's easy for me, it is easy for everyone"
No. I am taking something that‘s easy for me, and suggest making it easy for everybody.
> The better question is why should a layperson ever need to know about systemd in the first place.
They shouldn‘t, and I did not say so. That‘s the strawman.
Does an average Windows user know about the NT kernel? No, and he does not need to.
> When designing products for end users who are not technical, the overriding design goal should be easiness and reliability.
I agree.
> having to learn about systemd, by definition makes the product/system not easy.
You would not have to learn it though. You could just flip a switch in the GUI.
> they should not even have to learn about its existence.
And they would not have to.
I don‘t know how my car works, and I don‘t need to. I can use the simple interface (steering wheel, pedals, shift) which is exposed to me.
> A system where editing text files can make or break the system
> You edit a text file.
Again, complete strawman. You would not edit text files.
(This can, of course, happen if you put a technologist outside their element, too)
It is for me with Systemd - I had to spend hours (on two different occasions, if I remember correctly on Debian & Linux Mint) trying to understand how to set a dependency against an NFS filesystem mount so that a DB would not be started before that, and to make that work reliably => Systemd's docs & behaviour (& special distro settings related to systemD?) weren't that great for me.
Sorry, but an actual layperson would already be lost upon reaching the word "abstraction".
I think the abstractions are good, but there's a lot of terms that are needed just for the basic "Spin up this process on boot" (one might say that explicit is better than implicit, but I dunno).
I am not an expert here, but to the best of my knowledge systemd has no concept of inter-host dependencies, making it effectively useless for anything distributed. For example classic lb+app+db cannot be directly controlled by systemd.
But why stop at self-hosting? Beyond self-hosting, it could be extended to local-first paradigm meaning that there's a choice to have a scalable on demand auxiliary cloud based for handling bursty access demands if you need it. In addition, you can have extra backup for the peace of mind.
I'm currently working on realiable solutions (physical wireless and application layers) to extend the local-first system to be automatically secured even you have intermittent Internet outage unlike TailScale and ZeroTier [2]. This system will be invaluable where Internet connection is not reliable due to weather, harsh environment, war, unreliable power provider or lousy ISPs [3].
[1] Local-First Software: You Own Your Data, in spite of the Cloud:
https://martin.kleppmann.com/papers/local-first.pdf
[2] Internet outage:
https://en.wikipedia.org/wiki/Internet_outage
[3] NIST Helps Next-Generation Cell Technology See Past the Greenery:
https://www.nist.gov/news-events/news/2022/01/nist-helps-nex...
Sorry but i don't understand what this means. Internet outage should not affect your LAN services, no matter what selfhosting distro you use.
Also, what prevents my TLS/SSH/whatever certs from working in an offline setup? i do that all the time without downsides, so i'm probably missing your point. if you have a page explaining your project and usecase feel free to link
Self-hosting means:
- Needing to know how to configure your linux host across firewalls, upgrades, backups.
- Negotiating contracts with network service providers. While verifying that you have the right kind of optic on the network line drop.
- Thinking through the order of operations on every remote hands request, and idiot proofing them so that no one accidentally unplugs your DB.
- Making sure that you have sufficient cold spares that a server loss doesn't nuke your business for 6-12 weeks depending on how the hardware manufacturers view your business.
- Building your own monitoring, notifications, and deployment tools using both open source and in-house tools.
- Building expertise in all of your custom tools.
- A 6-20 week lead time to provision a build server.
- Paying for all of your hardware for 3-5 years, regardless of whether you will actually need it.
- Over-provisioning memory or CPU to make up for the fact that you can't get hardware fast enough.
- Getting paged in the middle of the night because the hardware is over-provisioned and something gets overwhelmed or a physical machine died.
- Dealing with the fact that an overworked systems engineer or developer is never making any component the best. And everything you touch will just passably work.
- Everyone will have their own opinions on how something should be done, and every decision will have long term consequences. Get ready for physical vs virtual debates till the heat death of the universe.
And yes, having a web hosting company manage your server isn't "real" self hosting, for all the reasons you described, but it's a far cry from dumping all your data with a big data-guzzling giant. Your data isn't calling home, isn't being used by a company, it's sitting on your server under your control, and only you manage it.
I think that's their main gripe.
The privacy angle of self-hosting generally ends as soon as you are renting a physical server with internet access. Someone else has access to the hard drives, the network traffic, the hardware, and via hardware remote access the OS/Data. At this point you would need to trust the legal bindings on what the hosting company can and can't do to your physical machine.
One really exciting angle on end2end encryption is that your provider may be technically incapable of any malpractice (for better or worse, I may wish my provider could have a sys eng bail me out every now and then).
For example, an OS that has a nextcloud-like suite of services, and a very easy to use GUI to enable a VPN / mesh network for all of your devices pretty much removes much of the concerns you mentioned regarding networking/firewalls/etc.
Except that:
Build servers took a day or so depending on the approval chain.
Hardware could be leased, or the capital expendite written off over 3 years, plus it came with a 4-hr onsite service warranty (if you worked with the right partners), and it being a capital, rather than operational, cost had bottom-line benefits.
24/7 service coverage for major incidents was very doable if you planned correctly, plus you owned the kit and so could control how you brought in resources to support incident recovery, rather thank waiting for your service provider to put the green dot back on their service status page.
//CSB The max total outage time for one corporate where I ran the IT dept was 12 minutes in 3 years, while we swapped out an HP battery-backed disk cache controller that took itself offline due to a confidence check failure.
> Build servers took a day or so depending on the approval chain.
This would only be true in a large shop with cold spares or virtualization. Server hardware generally has a 6-12 week lead time. The exception being if you are paying out the nose to a reseller who could do faster delivery.
Just imagine the time it took to setup Nagios or Zabbix for monitoring. In a small shop you are probably talking about at least 1-3 days of work + calendar time for hardware. Add to that some time for dealing with scale of metrics storage etc. depending on the shop.
today i accidentally nuked everything by not flushing the disk before rebooting and then naively letting fsck try to ‘fix’ it (which just makes things worse since it unlinks every inode it thinks is wrong instead of helping you recover data). now i’m manually dumping blocks and re-linking them in, supplementing whatever’s not recoverable with a 3-day old backup. that’s probably gonna take an entire day to fix up.
after this i have to figure out a better backup solution, because it costs me $5 of API requests every time i rclone the system to Backblaze, making frequent backups too expensive.
after that, i have to figure out the email part of things. AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting. you have to at least proxy it through a VPS, or something.
and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible because of the negative caching common in DNS.
no, this stuff is just way too involved for anyone who hasn’t already spent a decade in the CLI. i’m only doing this because i’m a nerd with time on his hands between jobs. self-hosting isn’t gonna catch on this decade. but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good.
also, i don’t think privacy is the right angle for promoting self-hosting. a good deal of the things that people self-host have a public-facing component (websites; public chatrooms; etc). if privacy is what you seek, then you should strive to live life offline. the larger differentiator for self-housing is control.
Its abstractions are still a bit shaky and a lot can be improved so it's far from ready for the general public, but i still consider it a great step forward because for the supported configurations, all i have to do to setup a new server is:
- edit config.yml
- run roles/deploy.sh
- enjoy
I'm happy to answer any questions on why (politics) and how (technics) i'm building this, and i'd be more than thrilled to receive feedback and contribution. In the past week i started working on a test suite so it's easier to contribute.
What do you mean? Did you interrupt the reboot process (eg. repetitive ^C)? Otherwise the OS should flush everything properly.
> after this i have to figure out a better backup solution
If you have other friends doing selfhosting, giving them a HDD to put in their server so you can rsync your data is a good solution. Also, keeping another local backup is a good solution. Doing both is even better.
> AFAICT it’s pretty much impossible to 100% self-host email because of blacklisting
It depends. It's almost impossible with some bad-faith providers (google/microsoft) otherwise everything works well. And even with those bad-faith providers, residential IPs usually have better reputation (if only because your ISP likely blocks outgoing TCP 25 by default and you have to ask them to unfilter it) than VPS IPs which all have a long history of abuse.
> and in between that i may spin up a DNS server to overcome the part where it takes 60min for any new service to be accessible
If you didn't previously query that domain name on your resolver, it will not be cached and will resolve quasi-instantly. The question is how long does your authoritative name server take to apply your changes to the config: if it takes any longer than 10s you should consider switching providers.
> but maybe we can federate, so that you just need to have one friend who cares about this stuff manage the infra and provide it to their peers as a social good.
Very good point! That's the entire premise behind hosting coops like disroot.org or various members of the libreho.st/chatons.org federations.
here’s my best guess: in my setup i have a host running a qemu vm and most of the interesting stuff happens inside the vm. originally that vm image was just 8 GB, but then i got a HDD to dedicate to it. with VM powered off, i partitioned the HDD and then dd’d the VM image onto it. then i booted the VM via KVM passthrough of /dev/sdb…
it booted fine; i ran ‘df’ and noticed that i forgot to resize the fs to the HDD, so i ran resize2fs. 3 days later, i `shutdown` the VM and then `reboot`d the host. the host didn’t actually come back: the power light and activity lights were off. after 5 minutes of this i power-cycled at the wall socket.
host came back up. vm wouldn’t boot. ran fsck from the rescue shell. now it booted, but no services were operational. since i couldn’t login to the vm (ssh broken + password logins had long been deactivated), i shutdown the vm and mounted its fs on the host. ‘df’ showed that the host thought the fs was only 8 GB in capacity.
i don’t think it was outright disk corruption, because the poweroff wasn’t that messy (but i come from btrfs, which has handled like 20 power faults on me w/ zero issue: idk how solid EXT4 is to these things). my best guess is that somewhere along the way, the changes from resize2fs didn’t actually make it to disk, or were overrided with stale in-memory values. maybe when i updated the guest’s kernel some post-upgrade script did something to push the old fs size to disk somewhere. or maybe the host had the old 8 GB fs size cached and flushed that during shutdown/start. unfortunately i’m not sure i’ll ever know.
Host a static page on github pages makes all that a ton easier and also free.
That's a trite example sure. But when I was at a previous company who did almost everything on premises I couldn't help but think if we had an internal portal/system a la GCP's console or that of Amazon's that could be a way for devs to spin up resources and have it all managed and even be a bit programmatic (no no K8s doesn't solve all of this it's its own bag of crazy) then we'd not need cloud much since we'd not need the almost infinite scale that cloud offers.
If Mozilla still had FlyWeb things could be plug and play.
I have a set of proposals here to bring some of that back: https://github.com/WICG/proposals/issues/43
And some are considering the Tox protocol, but in general, we have not solved the most basic issue of self hosting. How do I connect to my device in a way that just works, LAN or WAN, without manually serting up the client or registering for a service?
The linux model of "first learn how to use a package manager, edit configuration files by hand, and configure init scripts" is never going to be something that I can comfortably explain to computer users like my parents...
Good for them that it is seeing traction :)
I'm pretty sure that's exactly what we did and ended up where we are today. Any sufficiently-advanced self-hosting is indistinguishable from AWS?
I'm not sure how joking I am.
Which is exactly why OP pointed out this is where we should be headed if we care about privacy.
openziti is strong for app-centric use cases - put the (programmable, zero trust) network into your self-hosted app (via SDKs for various languages), rather than putting the app on the network.
https://openziti.github.io/ (quick starts) https://github.com/openziti
disclosure: founder of company selling saas on top of openziti
https://ziti.dev/blog/bootstrapping-trust-part-5-bootstrappi...
If you truly believe that a layperson can "easily follow" a technical guide or that a guide with the most common cases is sufficient to maintain a webserver... then the only thing I can say is that your experience with both laypeople and webservers is worlds apart from mine.
There's some projects trying to tackle the workload orchestration piece; CasaOS (https://www.casaos.io/) being one of my favorites but there's also Portainer (https://portainer.io). TailScale and and ZeroTier are great for Mesh VPN networking, where you may need to run some workloads in the cloud but want them networked with your home applications (or just to keep them offline). They also allow you to access applications running on a home server that doesn't have a static IP. Cloudflare Access is okay; I haven't tried it because it deviates from the mesh VPN model significantly.