WookieRushing

These show up once you have a certain scale where it is either cost inefficient or the hot spots are very dynamic. They also try to avoid latency by being eventually consistent sidecars instead of proxies.

I’ve seen them used for traffic routing, storage system metadata systems, distributed cache etc

Yup! This is the reason why its so cheap for them. Other companies in similar positions have cache nodes in the ISPs and this dramatically lowers the cost

Why do you think its easy?

There's a lot of systems where you can easily take down some hosts, but taking down more than N% at a time causes issues. If your fleet is large enough then you are limited by the largest set of hosts where you can only take N% down at a time. Now you could say keep the sets of hosts small or N% large. But that can cause other issues as you typically lose efficiency or zonal outage protection.

A solution to this could be VM live migration or something similar. This breaks down for storage systems where you can't just migrate those disks virtually since they're physical disks or places that don't use VMs.

My bet is on cache servers having a bad release

It’s millions a year and ms teams is near free. In a cost cutting environment this is near perfect.

Why is there input lag?

I’ve used something identical and mosh makes this just work. Most devs at that company swear by remote builds and hate laptop builds

They can get it from your android phone or from chrome if you’re running it

No, cryptocurrency cannot be used as a normal currency. Nothing as easy as handing cash to someone else exists in cryptocurrency.

There's one giant benefit of using cryptocurrency and a few smaller niches. It is a currency of last resort to guard against hyperinflation or currency controls or for drugs. You should really only be using it if you can't turn your money into dollars but can earn cryptocurrency another way.

Check out an example like https://mission.org/hidden-in-plain-sight/bitcoin-a-lifeline... . In this case its easier to earn crypto as payment for doing work online than it is to be paid in USD.

The smaller niches include things like flash loans

Flash loans are actually new thing in the world but still pretty dangerous. You can get a massive massive loan (Think $1 billion) without putting anything down as long as you pay it back in the same transaction you get it. Ethereum can guarantee the loan and repayment succeeds so the borrower really can loan $1 billion. Of course this only works as long as whatever tool the loan contract is using to check that the borrower can repay $1 billion works. If not then the money is going to be stolen...

As for goal, there isn't one. Its whatever folks want it to be and a large number think ponzis are the goal...

As for normal currency, it does not need to be stable. See Venezuela as recent example. It typically can't follow a real currency like USD exactly. In fact trying to keep an exchange rate between a strong currency and something else the same almost always ends horrifically like https://en.wikipedia.org/wiki/Black_Wednesday

Its like Fish shell vs Bash shell.

Bash has weird defaults so you end up googling for everything. In fish, it just works and you barely need to search for anything.

Sane defaults matter. With hg, I don't need to struggle to get it to do what I want, it just gets out of the way. With git, sure it works but like you said it has a bunch of ducktaped tools together that change the defaults or just generally make things easier.

Now hg is half the pattern here. The other half is stacked commits. Each commit should build and get reviewed separately. There isn't any waiting for reviews on each commit, they all get reviewed over time and you rebase any changes that are requested. With git this is amazingly painful and half my zshrc is about making this simple. With hg, it just works. Take a look at hg absorb or hg split, theyre features built on top that yeah can replicated in zsh scripts but its kind of nice when you can assume they just work. It means junior engineers don't spend hours trying to fight git with stacked diffs.

Sapling is trying to fight the network effect here by doing the classic built a compatible but legitly better front end. Compatible with github but sane defaults is a BIG thing.

When risk to society becomes high enough then it becomes necessary.

See paying taxes, conscription, jury duty

Though judging the level of risk is very hard and there is often not a clear line of when force is needed.

It wasn’t when I last saw it. Rocksandra ended up being a stepping stone to fbs most common distributed db, zippydb https://engineering.fb.com/2021/08/06/core-data/zippydb/

Zippydb is honestly one of the best parts of fb infra. It let you select levels of consistency vs latency

What psaux mentioned makes more sense. A node == one Cassandra agent instead of a server.

Past 100k servers you start needing really intense automation just to keep the fleet up with enough spares.

If you’ve got say 10k servers it’s much more manageable

The fun thing is Cassandra was born at FB but they don’t run any Cassandra clusters there anymore. You can use lots of cheap boxes but at some point the failure rate of using soo many boxes ends up killing the savings and the teams.

I was also surprised by this. 300K nodes for a distributed DB is kind of crazy. I’ve worked with similar systems but they stored much more than 100 PB with 10x less nodes

Apple is using less than one TB per server…

But when you see the 1000s of clusters it starts to make sense. They probably have a Cassandra cluster as their default storage for any use case and each one probably requires at least 3 nodes. They’re keeping the blast radius small of any issue while being super redundant. It probably grew organically instead of any central capacity management

What happens is you work somewhere that has stacked diffs and suddenly you learn how to shape your diffs to make them easy to review. Thinking of how folks will review your code in chunks while writing it makes it cleaner. Having small but easy to read diffs makes reviews faster and helps junior devs learn how to review.

Sometimes this doesn’t happen in which case you end up need to split your commit at the end. This is where git utterly fails. You end up needing git split and git absorb to make this productive.

Git split let’s you select which chunks in a commit should belong to it and then splits that into a commit and then you do it again and again until you have lots of commits. You’ll still need to probably test each one but the majority of the work is done

Git absorb takes changes on the top of your stack and magically finds which commit in your stack the each chunk should belong to and amends it to the right commit

You also need git branchless https://github.com/arxanas/git-branchless as it lets you move up and down the stack without needing to remember so much git arcana.

Yup, the core part of Tor is written in C so it can't get much speedier. Rewriting it in rust is almost is entirely to reduce the footguns you get in C

Yup, stacks are wonderful for new features. Typically I run a stack of 5 for new features.

First one is the API/interface with null implementations. Then subsequent ones each implement a method with associated tests

Mercurial solves this 90% of the way with hg absorb: https://gregoryszorc.com/blog/2018/11/05/absorbing-commit-ch...

Its extremely close to magic. You make all the requested changes on the PR4 and type hg absorb and it'll figure out which commit in your stack each change belongs to.

I've used this to work on lots of stacks 20 commits deep in mercurial.

While isolation is important for managing state, the other side effect of isolation is allowing separate scaling of resources.

If you can scale up your number of workers for a particular emergency then things get easier to handle.

Nope, looks legit: https://www.google.com/search?q=livermorons+site:www.reddit....

Atomic rename is useful, but one of the points of not having it is that it makes these giant distributed file systems much much easier.

Building DBs on top of these systems is done, but they always try to be shard aware to avoid cross shard interactions. Atomic rename across shards tends to suck. These interactions tend to be where lots of not fun problems with latency and errors happen.

It’s now EEA 2

I’m not so sure. Moxies reasons about how federation leads to protocol development slowing and then freezing are solid.

It’s why we re not using smtp for chat. SMTP can’t be extended enough so replacements are built instead. Similarly if signal federated, eventually it would freeze and a few years later users would move to wherever they could get new features.

Federation is a good thing but only when the protocol is finished or if there is a forcing mechanism to allow updates to the protocol. ethereum/Bitcoin are good examples as they have flag days that force the value of currency to be in the balance to keep the protocol moving forward.

This only works for stateless services. If you’ve got frontends that take longer than 10 mins to serve traffic then you have a problem.

But if you’re running a DB or a storage system, 10 mins is a blink of an eye. Storage systems in particular can run a few hundred TB per node and moving that data to another node can take over an hour.

In this case, the frontends have a shard map which is definitely not stateless. This is typically okay if you have a fast load operation which blocks other traffic until shard map is fully loaded

This is really common and works pretty awesomely. MySQL is extremely battle tested and there’s lots of experts out there for it.

FB built a similar system to maintain their graph: https://blog.yugabyte.com/facebooks-user-db-is-it-sql-or-nos...

It’s a ton of tiny DBs that look like one massive eventually consistent DB

I highly recommend A Fire Upon the Deep. Its a rare mix of really interesting hard scifi with an actually good story and characters. Hard scifi often has very flat characters but this is not a book which suffers from it.

It has a very very cool twist to explain the Fermi Paradox and is a really good example of a universe with one modified rule.

Yeah I totally agree, especially with the blocking entry nodes part.

There are many other ways to detect TOR connections or nodes and block them. Theres enough that there are a whole set of ways of obfuscating traffic called pluggable transports: https://trac.torproject.org/projects/tor/wiki/doc/AChildsGar...