I almost have the regex "fatal|invalid|unknown|error|except|critical|cannot" in muscle memory many years after having last had to type it - must have typed it thousands of times tailing and grepping logs :- )
It takes a bit to work this into the code and infrastructure everywhere but it’s worth it.
%w[fatal invalid unknown error except critical cannot].any? {|word| 'My critical error message'.include? word}
At fail-over my local machine will also switch from the dev database to prod (connecting to the RDS instance). It'll only has read-only access to relevant data, but it'll keep the site 90% functional.
Is it perfect? Hell no. Pretty hacky, but it's kept my application running twice after a failure in the past two years.
This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst.
You are married to your mobile & laptop.
I also shows there is no free lunch: you need to invest in redundancy.
I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.
This is not a long term solution though. You need to add people and take a break at some stage.
Very clean and simple, but not trivial or bare. Really good job, jmstfv.
I've been experimenting with functional (atomic) CSS lately and built a small framework that ended up using in Hexadecimal (if you peek at the source code, you can see weird classes like m-0, p-0, etc..). I liked it, but one should be careful in not overusing it, otherwise, it is an unmaintainable mess, yet again.
Below, I linked my Netlify config [0]. My static files live in the src/ folder, so you should probably change that if you have them in a different folder. If you don't know what Content-Security-Policy (CSP) or Strict-Transport-Security (HSTS) headers do, leave them out. It is easy to shoot yourself in the foot if you don't know what you're doing.
[0] https://privatebin.net/?c794ba90771f6e10#GSnJJ3dnd5r3ZDGgi9R...
Have you considered a trial extending to their first outage notification, rather than 14 days?
I would imagine if they don't see an outage in 14 days they're not seeing the value.
However if their trial expired following their first 'site is down' reminder - that's where your value comes in.
Just my $2c
I'll investigate that. Appreciate the suggestion.
It is highly unlikely that all of them will be knocked down at the same time.
If workers suspect that there is something wrong with your website, they will check it from all locations at once. Website is only marked as down if more than half of them can agree on that. That alone knocks down most of the false positives: https://tryhexadecimal.com/docs/uptime#sequence-and-frequenc...
* Account deletion (DELETE FROM \"accounts\")
* New successful sign up (INSERT INTO "accounts")
* New signup attempt (Started POST "/signup") [0]
* Fatal Exception Error ("fatal" OR "exception" OR "error")
* Warning ("warn")
I do share my costs and tools publicly, in case if you're interested: https://tryhexadecimal.com/costs
[0] I got hit by a botnet attack couple weeks ago, so I keep a tab on their activity (https://news.ycombinator.com/item?id=21327416)
https://devcenter.heroku.com/articles/background-jobs-queuei...
* Redundancy. If you process background jobs, have multiple workers listening on the same queues (preferably in different regions or availability zones). Run multiple web servers and put them behind a load balancer. If you use AWS RDS or Heroku Postgres, use Multi-AZ deployment. Be mindful of your costs though, because they can skyrocket fast.
* Minimize moving parts (e.g. databases, servers, etc..). If possible, separate your marketing site from your web app. Prefer static sites over dynamic ones.
* Don't deploy at least 2 hours before you go to sleep (or leave your desk). 2 hours is usually enough to spot botched deploys.
* Try to use managed services as much as possible. As a solo founder, you probably have better things to focus on. As I mentioned before, keep an eye on your costs.
* Write unit/integration/system tests. Have a good coverage, but don't beat yourself up for not having 100%.
* Monitor your infrastructure and set up alerts. Whenever my logs match a predefined regex pattern (e.g "fatal" OR "exception" OR "error"), I get notified immediately. To be sure that alerts reach you, route them to multiple channels (e.g. email, SMS, Slack, etc..). Obviously, I'm biased here.
I'm not gonna lie, these things make me anxious, even to this day (it used to be worse). I take my laptop everywhere I go and make sure that my phone is always charged.
[0] https://tryhexadecimal.com