I don't use GitHub for work, but it's always amusing to see the cries of people who do, when it does down from time to time. The ability of many companies around the world to continue operating depends on just one company continuously making the right calls.
In the future, when Ubisoft's servers go down, I won't be able to play Assassin's Creed games anymore, even though the servers don't really provide any value to me as a player.
On Spotify I keep downloading the same songs over and over again. Poisoning the environment more than if I got a CD. These days I try to buy songs to download once and then play in my favourite offline music player.
And of course SSH, the "secure shell", application which doesn't really optimize for shells. Responsiveness of typing things in the shell prompt relies on the quality of the Internet connection and the CPU load on the server machine - a process during which the server doesn't really have anything interesting to say.
I'm working on the side on creating a company and those opinions lead me to choose the harder path of desktop applications which don't need Internet connection for anything that doesn't require it by definition. They're more resilient and comfortable to use. I'll probably fail, but I really can't make any other choice at the moment, I need to at least try and see for myself.
One of my favorite games, if not overall favorite. It's impossible to find a new copy now. Any legal copy you find is likely only a CD-key from some reseller site, or a physical disc for PS3/Xbox 360.
It's not available from Ubisoft's store, nor Steam's store (where I purchased my copy).
And uhm.. eternal terminal? https://eternalterminal.dev/ (discussed in 2019 https://news.ycombinator.com/item?id=21640200)
And, perhaps others? There's this article https://console.dev/articles/ssh-alternatives-for-mobile-low... (edit: I just submitted it to HN https://news.ycombinator.com/item?id=29081008)
Very refreshing, I really appreciate that choice. As humans in tech I'm sure it's frustrating that everything requires accounts and constantly online services, since we can imagine the software working perfectly fine without it. Games are especially nefarious with account requirements, but even different work software will fail without an account. Using a modern 3D art authoring pipeline can easily have you logged in to two or three services while you work.
I buy all of my music through bandcamp, download the FLAC files to my home server and play those. I prefer to pay an artist for their work directly or a label who might offer a physical or shirt/patch/etc. I also like the concept of buying an album because often times the album is a complete work, not a loose collection.
I tried spotify for a few days and it was a poor experience as the suggestions were terrible, the selection weak, plus they barley pay small artists. No thanks.
> I'm working on the side on creating a company and those opinions lead me to choose the harder path of desktop applications which don't need Internet connection for anything that doesn't require it by definition. They're more resilient and comfortable to use. I'll probably fail, but I really can't make any other choice at the moment, I need to at least try and see for myself.
Is that in line with the offline-first movement ? I can only cheer you on this journey, because constant connectivity eats our focus and our planet. I'd love to see what you come up with :)
As pertains to lowtechmagazine - Love these guys, love the idea, love the minimalist culture about it. Fully support it!
This is not really the case on Xbox.
Your general point is taken, but, yet, in this particular case, git is the (or "a") perfect answer to the host not being available. Other remotes could be used to route around the failure.
In the companies I've worked for so far, issue tracker and CI have been self-hosted along with the rest of company infra. One fewer point of failure.
You can download your checkout, but the app requires an internet connection. No signal, no audiobook.
100% reliability is a terrible, dangerous lie.
People make mistakes.This is well and good when we are talking about entertainment and Instagram.
However we are now adding fragile Internet-connected code in critical infrastructure: cashless society means that if your bank is down, you can't get food.
All cycling stands in London need to talk to their server, and when it's down, you lost ability to get transport.
Same for public transport and Uber - if they fail in during a winter night, someone somewhere will freeze to death.
The door dialing in my house needs the internet to do IP-calling to your phone. They were digging up my street and cut the cable, and I was stuck in the cold for an hour.
This is going to spread - imagine a failure of the system managing running water, or god forbid the suer?
This is actually a take most SRE's would / should believe. Every added 9 to the reliability increases the price exponentially. Finding the correct level of reliability is something most companies should focus more on, because sometimes a single physical machine that could go down once a year for a few hours is perfectly capable of providing all the resources a medium seized business could need. Proper backups, monitoring and recovery runbooks can even decrease the downtime of such a simple system to minutes, while easily saving you maybe thousands per month.
Slowing your young company down in order to turn 0.9995 to 0.9998 is almost always a terrible trade. Even turning 0.995 to 0.999 is hard to justify in most places. (That improvement saves about 35 hours of downtime per year.)
We did do some "analysis", meaning that we made some underlying guesses and multiplied them together, but the real value is in getting people to think that 1.000 is not the actual goal-line, then tracking and doing RCA on all the outages, bucketing them into categories so you know whether to invest more in diverse networking, software testing, HA for DB servers, failover sites, zero downtime releases, etc.
Many times, you can avoid entire massive projects (“we need to be hosted in 2 geographically diverse data centers for availability” “uh, no we don’t; we have a budget of 262 minutes of downtime per year and that project will save us less than 60 minutes per year on average, using the best case assumption that our own changes to implement it cause no downtime”)
So we needed a web server, a database, a queue system to run these heuristics and we needed to host/distribute ~100GB worth of content, most of it video.
We were bootstrapping, so I was trying to (1) save as much as possible on operational costs and (2) punt on all the "scaling issues" that would require more of my devops time that would be better spent developing and adding more features. I deployed the whole system on a single server from Hetzner: Django app, Postgresql, Redis for caching and session management, RabbitMQ for celery. All in one machine with 32GB of RAM and a RAID system with enough capacity to hold the data. I think it was costing us less than 50€/month. That is all we needed to (easily) serve ~800 students and the staff who would author new content.
In the end we delivered everything we promised to our first customer, but we were not able to grow our revenue as much as we expected, so by end of 2013 we just put the whole company on the backburner, got a small maintenance contract with the main customer and went on to find another jobs.
From end-2013 until 2018, I needed only to make sure that our domains and SSL certificates were up-to-date every six months, upgrade django packages in case of security issues and deal with ONE incident (in 2016 IIRC) where a disk failure put the array in degraded mode, which I solved by getting a new server at Hetzner (better specs and cheaper, after all those years), warning the customer that the service would be taken offline for a couple of hours later in the day, rsyncing the content, restoring the database and redeploying the application with the fabric script.
This is one the projects that I am most proud of what was accomplished given all the constraints and made me realize the difference between a Software Developer and an Engineer. Yet, it translates to a very poor entry on an CV. We are too used to ask on interviews what people have done and what technologies they have used, but we rarely ask about the moments where it was best to avoid doing something.
The speed is incredible if compared to ec2 or root server performance from other vendors. Even if they've dedicated resources.
Why anyone would run their pointer chasing code in a heavy cache eviction environment is beyond me. The code is slow to start with, and then you make sure that none of your data is in the cache. Why you'd pay 10x for slower hardware makes no sense.
What people should be doing is running on bare metal and turning off all the garbage meltdown protections that kill performance. If you're not a cloud provider and you're allowing people to execute arbitrary code on your hardware, you've got much bigger problems than meltdown.
If you compare on demand lrice for cloud, sure. Reserved and spot instances change the balance significantly. If you're running a handful of servers, sure it's a no brainer. But when you start dealing with any sort of human cost (operations, it) the savings you get are dwarfed by the human costs because that's what you're paying for with aws and azure. And, when you're at mega scale you're negotiating separate deals anyway.
That's also not considering the value of the combined offerings. On aws for example, I can spin up a kubernetes cluster with rolling updates pushed by GitHub actions in less time than it took me to write this comment, and it will be usable and modifiable by anyone who has experience with aws or k8s. the cost savings of running my own infrastructure and managing all the moving pieces is dwarfed by the fact that the service provided is widely used and well known.
That does sound like a really good deal!
Until now i've only been using VPSes (apart from homelab servers as CI nodes etc.) because they're cheaper for the smaller sizes, but for comparison's sake, the cheapest VPS provider's (that i know of and trust) offering with 64 GB of RAM and 640 GB of storage would cost ~260 euros a month: https://www.time4vps.com/?affid=5294
Well, i guess there's also other VPS providers out there that can nearly match the price, like Contabo, though they do have mixed reviews: https://contabo.com/en/ (personally i just found their UI to be extremely dated and there are setup fees, but otherwise they were decent), though even then they'd cost anywhere from 30 - 90 euros a month.
I was using a Netcup root server with 2 dedicated cores/8gb ram before i switched to my current hetzner baremetal server. It only cost ~7€ per month, so much better value if i you don't mind that everything just takes a little longer.
i dont think i'll ever go back though. even using the shell on the baremetal server is so much more responsive vs the vps.
but for what its worth: you can get a VPS with similar resources (16 cores, 64gb ram, 2tb ssd) for 40€ with netcup.
- AMD Ryzen 5 3600 6-Core Processor (Cores 12)
- 64GB
- 2 x 2TB HDDs
We really underestimate the costs of running in the cloud.
A great idea to be honest, the market willing to overpay for server will probably be able to pay more to you.
On small platforms we are still stuck into the 1990's approach of having one reliable system.
We need distributed[1] systems and protocols even in small applications. Easy to use and self-healing.
[1] No, I'm not talking about blockchains
Three nines also means you can't afford to intentionally take a system down to work on it, or you'll burn all your "oopsie" downtime. That means a ton more work in infrastructure and deployment processes, than two nines.
There are things which are actually life critical that you want to be up and then there’s everything else, many people have really strange opinions about downtime and treat it like some kind of immorality.
I don’t really buy the power arguments about extra capacity but that’s a big thing.
https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1553-2712....
And it's definitely complicated. Just because work necessary to preserve human life is allowed doesn't mean anything done in a hospital is therefore permitted -- a sizeable fraction of the work in a hospital is about bookkeeping, billing, insurance, scheduling followups, paperwork, etc which can wait a day or two without anyone dying.
It does not really matter for the Low Technology website of course, but if you get into a serious car accident and the surgeon is like "sorry but our solar powered operating theater is down right now, that's just life sometimes" that would be a very unwelcome thing to hear.
If we focus the same resources on providing reliability where it is needed most, we could actually improve reliability of outcomes. That is, for the same cost of providing a few hours of battery backup for the entire grid we could provide weeks of battery backup for hospitals and data centers.
Suppose twitter/reddit/HN shut down for 10 hours a day.
10 hours was just an arbitrary number, but there are plenty of more creative solutions. Shut down every March, or Weekday mornings, or every other hour.
That's up to the customer to decide. Insofar as we live in a world where customers decide the fate of businesses in a do or die way, I feel the conclusion is obvious. Either the customer shares your philosophy that your downtime is part of an enjoyable rhythm of life, or you die.
That's why guaranteed overnight delivery generally costs more than waiting a week or two for your package.
I work at a company that provides business services, and some customers are fine dealing with irregular availability of resources if it means the service cost an order of magnitude less.
It's not common in power yet, but in the future a hospital might pay $1/kWh for high-uptime power whereas someone else might pay $0.01/kWh for excess capacity provided as available to charge their car.
If there are any issues with the grocery store or the technologies in it, then perhaps it's possible to pay in cash. If there are larger problems with the grocery store's supply chain, i can just go outside to my greenhouse and get some fresh veggies. If i don't have a greenhouse, i have a bunch of beans and other canned goods in my cellar, as well as crackers and so on. If i don't have my cellar, i have a bunch of frozen meat in the fridge. If i don't have said meat, i have a rifle and a forest with animals for that. And if i don't have any of that, i can always just go to my neighbours and ask whether they have anything to share (or offer the same sort of help, should they be dealing with the same hypothetical situation, instead of me; things would only break down when everyone would be starving).
If there are electricity interruptions, i have a few UPSes in place. If the UPSes run out of power, then i have a diesel generator in the cellar (with the appropriate ventilation system in place). If i don't have any diesel, my house has a stove that burns wood and has central heating with radiators, as well as candles or a petroleum lamp, or even a flashlight with a generator for lights. And if HN isn't online, i'll just read a book, since there is a shelf with some lovely literature to be enjoyed, or will simply write something in my diary, as people do.
I'm saying this not to discredit the fact that many people choose to live in the larger cities, since there are definitely different advantages to that choice, however at the same time you have to consider the overall trends in society, when you take things as electricity or a stable food supply for granted, which may not always be the case.
That said, there are relatively few things that are truly time sensitive, such as having a supply of important life sustaining medication. I wouldn't expect my grocery store to have a 99.99% uptime, nor would i expect that even out of my electricity supply, hence the UPSes and generator, and HN going down wouldn't be the end of the world either.
And when we are talking about the greater supply chains that we don't see in our daily lives, honestly there always should be a business continuity plan in place - if your computer system goes down because of Y2K, a solar flare or the data center burning down, then you should be able to keep on trucking with calculators and some paper. Anything less is not all that responsible, if we're really talking about important domains.
Not necessarily. Similar to software, each nine costs (in money, pollution, effort, etc) at least an order of magnitude more than the last, and at some point it's more efficient to focus on resilience elsewhere in the system.
Wind and solar are great but they're far too irregular to rely on so in places without hydro storage you're often left with either chemical batteries, which would be prohibitively expensive to support the entire grid for a few days, or fossil fuel peaker plants, which will ultimately result in coastal flooding.
One alternative has been to use demand response to pay people to cut back and dynamic (wholesale) prices to encourage people to conserve when supplies are low. Perhaps at some point electricity providers could even institute different prices for different levels of service. That is, one would pay $1/kWh for any capacity that requires less than an hour of downtime per year like servers and medical equipment, but $0.01/kWh for excess capacity used for vehicle charging that is irregularly available day-to-day. The "smart grid" and meters necessary to support this kind of thing don't exist, but may in the future, and adapting to this kind of irregularity would allow us to have a much lower-cost electricity grid.
Reliability often comes at the expense of making some people's lives much less pleasant (especially in situations of underfunding or not having the corporation back any of your SRE initiatives), or, alternatively, with exponentially increasing costs, since all of the sudden everything needs to be high availability and have redundancy in place.
Consider the recent Roblox outage. No one died because of it, yet countless engineers were working around the clock to ensure the profitable operation of the platform.
When the Latvian Electronic Tax Declaration system goes down on the first day of the year when you can declare your taxes (basically every year), the ministry representatives shrug their shoulders and essentially say: "Nah, we can't make it cope with the load. Just wait a day." And you know, maybe they can't, but at the same time even so everyone still manages to submit their taxes eventually.
Thus, perhaps it's worth it to consider when and where we need the scalability and high availability, e.g. government centralized auth services under Latvija.lv which also went down because of the load and as a consequence people had to wait longer to sign up for COVID vaccines and couldn't use other systems until it was resolved: https://blog.kronis.dev/articles/manavakcina-lv-and-apturico...
Then, we should focus on those more important cases. Now, I'm not trying to suggest that the incompetence or dismissive attitudes are (always) okay, instead, I'm merely suggesting that we should consider our priorities and wonder what a more laid back environment would look like.
I still recall once sending a hotfix for a prod issue at 3 AM. I wonder what things would look like if instead of worrying whether a certain business process can take place at a certain date, the attitude would have been more like: "We'll just tell everyone to wait a day or two. It's not like our product is time sensitive."
In places with competition the customer is the one who decides whether you are screwed. Is downtime from social media or news rather trivial? The business deadliness of downtime is different from the societal deadliness of downtime, but what significance this takes on is up to the customer to say, and the customer develops expectations based on the competition.
Also, I got a vaccine almost a month later because our vaccine sign up page also had technical difficulties. I sat around in queue for 90 minutes to do an online sign up and all the times were already gone. The next time I tried and saw the queue I just closed the browser.
Also, these are things that you can go to prison for if you mismanage it. People are much more likely to put up with these screw ups, because they have to.
Heck, even just charging a battery with the extra solar power would provide better reliability.
I totally agree on saving power for users given that's a energy cost you pay N times - albeit I'd prefer some better defaults (in terms of colours and fonts).
I guess we need a better, lightweight browser.
I wonder if whoever wrote this has actually done the calculations. These days it's not really difficult to provide fairly reliable solar power by overgenerating while still beating fossil fuels - perhaps not beating them as much as with zero overgeneration, but beating them anyway. So I'd take the quoted text with a considerably large grain of salt.
Our banks have those, and it's a really shitty experience.
Yeah, sure, e-bank works, you pay, but if the person receiving the money is at another bank, that my bank has no extra contract with, I have to wait until the next workday for the funds to be transfered. If I want to buy something with a bank transfer on a friday night, the funds won't get transfered until monday morning.
If both sender and recipient use a mobile phone service like Vipps then the recipient also receives the money immediately (within seconds) regardless of whether they use the same bank. I think similar services exist in other countries too.
The difference is that here, the downtime saves energy. "Normal" downtime is unintentional. Servers are still running, just in an error state. They may be constantly attempting to restart themselves at some level (looking at you, k8s). Users are still trying to hit them and their requests may be partially handled. Traffic may be pushed over to newly spun up instances.
We can accept that things happen and running large internet services is hard, and SREs and developers everywhere will rejoice, but it won't save energy.
Because of day/night cycle these working hours would usually overlap. This would put people who can’t afford to do their administrative business on the internet during work hours at a disadvantage.
My bank's website has been down for maintenance for 24h this week-end. And French government website for declaring VAT will be down for a few hours tomorrow.
Many people still consider downtime as normal. It's not trendy to talk about it, but downtime isn't such a big deal. (Going down because you cannot handle your peak load is very bad though)
If it's a planned downtime with stuff, people can plan around (eg pay your bill before the weekend), that's bearable.
But if the government wants you to do something (eg. declare VAT), but then the page isn't available, and they still expect you to do it, and you could get fined if you don't... that's a really shitty thing to happen.
In the twenties century, trains in Spain where completely unreliable: they could leave one hour early, or 3 hours late, yet people would get fired if they couldn't get to work in time. But it still worked: people just didn't run into the train station 5 minutes before departure.
There is a UK Gov website that only works during certain times (I think it's something to do with DVLA) - although I assume that is a technical thing, rather than it being powered by solar!
B&H Photo famously goes offline every Shabbat and I think it is great. It goes to show that you can have an sustainable and profitable business without sacrificing on principles.
It is much easier to have it from renewable sources as your DC doesn't move and the CO2 to social benefit is so huge.
Imagine giving up all mass CO2 producers, alone transport of people and goods but still being connected to everyone and sharing knowledge and learning etc.
I would like to see addicting things like reddit having office hours but being able to shop at night for something I need (like a new pair of pants after the ol d one ripped) is saving tons of CO2.
Global replication could also give us more edge computing, which puts less load on global infrastructure (assuming good weather to use the nearest server), which in turn should be able to reduce resources consumption a bit more.
Give a check to NNCP:
While I know everyone likes to hate on Lotus Notes here, once of it's key architecture features was that it assumed a seldom-connected and replicated data model - the clients and servers would contact each other at intervals to get updates, and otherwise everything was local.
Decades later, I still miss this functionality a lot - just force and update/replicate before you go somewhere (either office-home or a trip across the world), and everything is the same. No worries about connectivity quality; if it was choppy, just give it more time to handle the retries/ecc, and get the good updates. (of course, many system managers/admins/devs didn't really consider that much of a feature and treated it like any other network app, so not taking advantage of it, but those who understood it's power...)
I really wish others would think in such a seldom-connected model, as the network is becoming increasingly brittle, and working in that model was such a joy.
Also, the old NNTP and email lists are still alive.
I use SLRN on batch mode on Usenet and tildex (Unix based micronetworks), and it rocks. In batch mode, I pull down all the comments/articles and I push down the answers and written articles. No need to be online 24h/7d.
For email, i use Getmail against an IMAP server.
Thus, I can fetch all the data offline at once with a 'do_the_internet.sh' script in a breeze.
I would love to use HN over NNTP.
How would you represent upvotes and ranking over NNTP, or would you just drop that for comments altogether?
(I'm asking because I've been thinking of implementing something exactly like this, with NNCP also maybe, but the ranking is what I'm not sure on.)
> tildex (Unix based micronetworks)
Ooh what are these tildexes?
>Ooh what are these tildexes?
Notably wrt uptime:
> Uptime
> The solar powered website goes off-line when the weather is bad – but how often does that happen? For a period of about one year (351 days, from 12 December 2018 to 28 November 2019), we achieved an uptime of 95.26%. This means that we were off-line due to bad weather for 399 hours.
> If we ignore the last two months, our uptime was 98.2%, with a downtime of only 152 hours. Uptime plummeted to 80% during the last two months, when a software upgrade increased the energy use of the server. This knocked the website off-line for at least a few hours every night.
And while they found inefficiencies in their energy conversion system,
> One kilowatt-hour of solar generated electricity can thus serve almost 50,000 unique visitors, and one watt-hour of electricity can serve roughly 50 unique visitors.
You may be surprised to hear that some websites _do_! I was trying to search for the page but can't find it at the moment, but I've experienced this with a Dept. of Homeland Security page that would only be available during certain hours. I believe it was due to some batch-processing related task but can't recall exactly now.
I think we'd have to have a pretty different internet in this hypothetical world.
And we were not so much worse than today (I'd say even better, but that may be loosely coupled to 100% reliability - although FOMO also has a taste of it).
B&H Photo is offline on Saturdays to keep the Sabbath.
Less than 100% reliability is going to be mandatory as part of the degrowth we'll need if we want to seriously tackle climate change. Less reliability, all the way towards offline-first, is the most sensible way to go
As long as downtime is expected and bounded, as long as the uptime planned period is actually up, you're at 100% reliability (but not 100% uptime).
Now, maybe you're right and we should be using another word, and aim towards less uptime ?
So yes, it's worth it to be specific over the difference between being available and being reliable (== meeting expectations, including being unavailable at times).
If I claim a 99.5% uptime SLA guarantee for my webservice and that it does, it's reliable. However, 0.5% of the time, it may be unavailable _and that's not an issue_.
> In contrast, this website runs on an off-the-grid solar power system with its own energy storage, and will go off-line during longer periods of cloudy weather. Less than 100% reliability is essential for the sustainability of an off-the-grid solar system, because above a certain threshold the fossil fuel energy used for producing and replacing the batteries is higher than the fossil fuel energy saved by the solar panels.
This, in particular, is an amazing take:
> Less than 100% reliability is essential for the sustainability of an off-the-grid solar system
Saying that sacrificing uptime is a good thing for achieving a particular goal is a take that's so wildly out of line with what most of the industry believes is refreshing, because it really makes you think. What would the world look like, if instead of being obsessed with SRE we simply assumed or accepted that there will be downtime, and treated it like something perfectly normal. Maybe a nice message at ingress/load balancer level, telling everyone that the site will be back in a bit.
I have no illusions that any company would want something like that, or that it's even feasible for many industries, e.g. healthcare, though the Latvian "e-health" system is an accidental experiment of this at the cost of around 15 million euros of taxpayer money: https://www-lsm-lv.translate.goog/raksts/zinas/latvija/par-e... .
However, if companies have working hours, why couldn't websites? What a funny thought.
Edit: on an unrelated note, it's wild how different setups can be.
For example:
> The web server uses between 1 and 2.5 watts of power...
While at the same time, i have two servers with 200GEs (essentially consumer hardware) running in my homelab, with a TDP of 35W each, though with the UPS and its inefficiencies, as well as the 4G router, the total energy consumption that i have to deal with is around 100W. Granted, i also use them for backups and as CI nodes, but there's still a difference of 20 - 50 times more power usage between one of my servers and one of theirs. I guess one can definitely talk about the differences between x86 and ARM there as well.