LaserToy
LaserToy on Hacker News
100%, and we have all of those things. Canary acts as the last line of defence, and honestly, when Canary detects and rolls back, it is already an incident that is being auto-mitigated with a limited blast radius.
To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.
Posted by
LaserToy on Sunday November 09, 2025 @11:21AM
from the techblog.cloudkitchens.com dept.
Azure portal still insists the issue is jsut with Console.
We had to bypass the Frontdoor
TLDR:
A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.
Posted by
LaserToy on Tuesday October 21, 2025 @01:01PM
from the techblog.cloudkitchens.com dept.
The architecture of our in-house Rust based logging engine
I love actors as a concept and I heard some large companies (Expedia) implemented large parts using them.
But I also saw how hard it is to understand a large system that built using actors. It is just hard to comprehend all the communication pathways and what happens in the system.
Can anyone talk about a better model?
Some things to consider:
1) The company needs a way to weed out folks who are net negative. In general, if someone is not playing their part, there should be a mechanism to evict if up-leveling fails.
2) The company needs a way to distribute incentives (bonuses) as fairly as possible.
Exactly. So, we need to make those companies more successful:)
Was trying to gauge whether you are underpaid because you only accept remote work.
Economy will be the one that will determines the ultimate outcome, especially for tech companies, as they compete globally
How much are you paid? Total comp?
Companies need to make money. High performers will go to companies that make tons of money, even if relo to Sahara is needed
It's a cool way to avoid painful database-related outages.
I've never heard of DataLog before, and it seems nice.
Also, City Storage Systems is known as CloudKitchens — a new startup of ex-Uber CEO Travis Kalanick.
Posted by
LaserToy on Thursday October 10, 2024 @08:55AM
from the techblog.citystoragesystems.com dept.
Try Azure. I thought AWS was bad till I faced GCP support and some weirdness. I thought GCP was not great till I faced Azure.
That is very interesting. As CRDB user, I priced Spanner (had to do some estimates during load testing), and Spanner came 3 times more expensive includign our eng salary to run CRDB
Fascinating.
How does streets align on the Chinese maps though? If random offsets are added, nothing should align. Or 500 meters is not large enough offset to create a mess?
What I mean, arrears that are parallel in reality, may not be parallel on a map.
I did some math. Assuming a human male needs around 3000 KCal per day, it sums up to around 1,095,000 KCal per year. Potato is one of the most calory dense, as 1 acre produces 17.8 M KCal (found online).
So, one person will need 1 Acre (43560 sq feet) / (17.8 / 1.095) = 2679.6741573 sq feet of potatoes. Which will be 248.8872 sq meters (10 by 20 meters field).
It is either my math is wrong, or article is wron by order of magnitude
I’m no farmer, but 200 square feet feels way too small. Really?
Good OLAP needs very similar properties as OLTP, they just don’t as it is extra hard
I love the topic. My view on the companies: they are similar to countries in terms of organizational complexity. And the way companies are run are similar to the way countries are govern. Mostly due to the fact that there are not that many ways to manage groups of people or individuals.
So, I’m not surprised the observation about startups also applies
Having that many KPI makes them useless. Also, lines of code written?
One day, put all of it in PCA or tree model, and compare to your own observations on who is the most valuable eng in the org
Managers are responsible for project failures, that is true.
And they have a mechanism of achieving it: hiring and firing.
The author is right, managers are not responsible for individual’s success, their ability to persevere, time manage and so on. We work with adults, they have their parts to do.
Looks like you are in us. Drag them into small court by claiming $9999 in damages. They will have to send a human or pay. I guess if they just pay, you can do it per product.
Class action might be possible as well
In US, it is relatively easy to commit fraud. The question is: what happens if you get caught. Is it a slap on your wrist or a catastrophically failure (termination, prison time)
You can, for example, get employed at 10 companies and then outsource all the work (someone got caught doing it and it was in the news).
If you want to cheat, you can, but you probably should not be discussing it on hacker news.
Every control/alignment system has overhead. That is exactly why distributes systems are both hard and slow, just look and Spanner/Cockroach and compare them so single node PSQL.
To get rid of managers, you will need to solve comms problem.