Goodhart's law in action. If "neatly composed and verbose documents" is the metric higher ups use to judge product managers, then documents will become neater and more verbose to the detriment of everything else.
In the end, the alignment problems that plague AI safety research are all playing out in company structures every day. It's hard to give a metric that measures your intended outcome, and any proxy leads to unintended consequences once there's an easier way to reach that proxy.
I've actually pointed this out as a fallacy of the premise of current alignment theory. That being we simply need to apply humanities values for the AI to be properly aligned. However, every alignment problem can also be seen as an existing human problem for which we have never resolved.
Oil, plastic, electricity, natural gas, mines, wood, poultry, meat & fish, transportation etc. They're nice and super useful and when used in moderation or when there aren't that many humans to begin with you can easily absorb the consequences. But if you multiply the number of people while at the same time you increase the consumption/production on some or even all of these in tandem the effects are massively compounding.
We want AI to reason like humans and have humanities values. According alignment theory. Yet humans will also cheat to get a result. Our values also are what put us in conflict with each other.
FYI, I've elaborated in much more detail here on such issues with containment/alignment - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
Yes yes yes holy crap yes.
Corporations are AI, and we are already in the AI apocalypse. Corporations exhibit emergent behavior that any individual human involved with them is powerless to stop, and they self-optimize towards inhuman goals. Coca Cola inc is a paperclip maximizer, except it maximizes the consumption of flavored sugar water. We are completely failing as a society to adequately rein them in.
It's scary how many of the things that are in that movie as caricatures you can now find IRL.
I've tried to watch it twice and I can't get more than 20 minutes in because it's my life. I've been living in the Idiocracy since I was eight years old.
Everyone seems so slow. Their problems are trivial. They only reason they don't live well is because they just aren't interested.
- - - - -
I have high hopes that the talking computers will make good rulers...
> The problem isn't thinking machines, it's letting machines do the thinking for you
...and then I like to point out that many of our social structures (including corporations) are, essentially, machines. They're just machines running on meat instead of silicon.
I'd be worried about false accusations of this. Some people are much better at communicating in writing, when they have time to think about how to word things, and not so good when put on the spot in a meeting. Others might be better at communicating verbally, and perhaps too verbose in written communication.
However, when the junior product managers are proposing things with terms they don't even understand then something is clearly wrong. Imagine joining a meeting where a product manager is trying to explain their own document but can't understand what "they" wrote when they get to various bullet points. Or the document is correct, but their understanding of their own document is incorrect.
Have you ever seen a student caught cheating? Someone who has the right answers on the paper, but can't explain why or remember why they wrote it down? It's exactly that.
It's obvious because their writings in Confluence and late night e-mails suddenly became very verbose and used the sometimes obvious ChatGPT voice. There's a stark contrast between content they write/share on the spot or discuss in meetings.
Then you get them into a meeting to explain further and they some times can't even remember what they were thinking when they wrote various bullet points.
It's eye opening to see how much the ChatGPT produced content can fool managers at the top who catch glimpses of neatly composed and verbose documents. Then you get the people into meetings and they don't really know where they were going with their "own" documents.
It wouldn't be a problem if they could take the documents as some sort of next step for their learning or research, but it's really becoming useless to have all of this ChatGPT-created content out there where none of them can really execute on it without going back to ChatGPT over and over again to come up with more content.