If I'm responsible for project A, I should be able to make 85% of the decisions for that project. Too often, recently, development organizations make every decision from above, then break that up into miniscule tasks that require no thought from developers.
Hackathons, IMO, are just gimmicks to gloss over a deficient product management group or software architecture. If your organization can't be convinced of the need to spend time on something outside of a gimmicky "hackathon", then something needs fixed.
I agree that organizations that don't trust their engineers with any design or implementation decisions are wasting their talent and probably creating a lot of bored and burned out engineers. If an engineer wants to use a fancy new piece of tech and they can justify it, let them do it.
Currently my manager gives me tasks like "so and so needs something, you should talk to them." That eventually becomes a bunch of tiny tasks, but I don't receive the work pre-chewed.
Hackathons are usually what management comes up with when they think about this problem. However, I guarantee you that this conversation regularly happens among your developers at Friday happy hours: - So how's that project Z going? - You know, same old. Looks easy in theory, and every little task is a slog. I spent this whole week fighting the Blorgifier again. - Wait, that abomination is still in our codebase? That was a hack we came up with at 3am during one of those death marches last year when we needed to ship something for the Initech demo. - Yeah, you know what they say. Nothing more permanent than a temporary bandaid. I wish I could have a week of my time back. - Yeah... cheers!
There will always be very important seeming bugs that have a great definition and visible customer impact. Unfortunately, obscure poorly-defined tasks like "rewrite the Blorgifier" immediately trigger a response of "that's just developers wanting to have fun" in managers. And they end up adding a tax on every single one of those important fixes until eventually the whole thing collapses under its own weight.
How to solve this? Ask the developers what's important to fix? Nah, they don't know how to plan a project and don't understand the customer.
Does this mean that developer pet projects need to get 100% time allocation? No. But in most companies their time is undersized by a factor of 1000 or more.
The tiny tasks model also lets you ship updates to the end user much faster and also ensures the product is never in a half-broken state. It's also more robust when you don't have a team of all-star programmers.
I agree that building big projects on your own (or with a tiny team of motivated folks and no mandated meetings) is way more fun and sometimes far more productive. But that's why I think it's important to incorporate things like hackathons and prototype phases into the engineering department. Each software development model has its strengths and should be used when appropriate.