Re:A 1.5GB String
by userbinator on Saturday April 08, 2023 @11:05PM
(#35499737)
You'd have same problem if you tried to serialize <format> into a field in <format>
Length-delimited formats don't need (exponentially increasing) escaping.
But why do you bring XML? The parent is against human-readable formats, and XML is one of them.
> And the problem here wasn't JSON in the first place anyway
Yes it was. They stored state in a huge JSON dump which ended up working like a zip bomb by the way they had to escape a string. This alone resulted in an exponential increase in memory that quickly became a problem.
Think about it for a second: would they experience this issue if they stored an in-memory stack per user? They wouldn't.
Heck, it seems that they could live with their JSON choice if they replaced '/' with any other inescapable character and converted it back to '/' when reading.
XML would also have a problem with this: you'd replace < with <, then with &lt;, then with &amp;lt;, … It's not exponential, but still inefficient.
Inefficient is not the same as problematic. They turned what otherwise would be a linear increase in memory usage into an exponential increase, which expectedly resulted in memory exhaustion problems.
There are plenty of problems to go around this approach and overall system design, but the weakest link is undoubtedly how JSON was used.
Can you imagine arguing in a tech meeting that a reason not to use JSON is that it doesn't support unbounded recursive levels of escaping very well?
And the problem here wasn't JSON in the first place anyway, it was
> The “back” stack is unlimited, meaning we’re saving more and more state until we explode
And storing previous state in serialized string in current state, instead of just having it stored natively. You'd have same problem if you tried to serialize <format> into a field in <format>