News for hackers. Stuff that matters.
Re:The $440M software error at Knight Capital (2019) by bob1029 on Monday May 02, 2022 @05:52PM (#31241440)
Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team.

Rolling back code is another thing I have no tolerance for anymore. The only option we entertain these days is a roll-forward. If your software takes so long to iterate/build that you need to go back to and old version in an emergency, you need to review your languages/tools/frameworks/processes. We maintain a contractual obligation to our customers for same-day code updates (in cases of production/regulatory emergencies) because we have enough confidence in our processes.

Re:The $440M software error at Knight Capital (2019) by sokoloff on Monday May 02, 2022 @07:32PM (#31242505)
You’re likely using human-readable names for the flags and shipping a multi-MB payload of JS and JSON. An HFT firm is likely bit-packing flags so they can send an 8 byte payload rather than 10 and might be using an FPGA hanging right off the PHY to figure out “is this message even interesting to me?”

Your feature flag might be “SHOW_STRIKETHROUGH_PRICING_ON_CROSSSELL_OFFERS”; theirs is a bit mask macro to pick off the 5th bit from the 7th byte. (Why do they care? Because if they allow themselves to get fat and slow, a competitor will take the money.)

Roll-forward only, same day SLA is probably right for your business, but isn’t for a company that could have their systems dusting off $1M every handful of seconds that bad code is running.

Different business problems call for different technical approaches. You should no more adopt theirs than they should yours.

Re:The $440M software error at Knight Capital (2019) by hn_go_brrrrr on Monday May 02, 2022 @07:54PM (#31242633)
Rolling back isn't about avoiding rebuilds, it's about restoring to a known-good state. Making an emergency patch is typically far riskier than going back to last week's build. We always favor rollbacks, unless there were critical fixes in the current release we absolutely cannot afford to lose.
Re:The $440M software error at Knight Capital (2019) by jll29 on Monday May 02, 2022 @08:11PM (#31242752)
The only thing worse that comes to mind than repurposing flags is re-using UUIDs, which I have seen some production DB do (no, not mine, thank you very much!).

Reusing the RAM occupied by flags might be do-able in a clean way using guards like the structured enum in Rust, which permits unions (objects that occupy the same space) that always have the right type (i.e., the compiler has knowledge what is in there at each point in time). This mechanism could in theory be extended beyond type-safety to accommodate other contexts in systems programming use cases where memory usage is extremely important.