We build tools that read and write Excel files (open source library: https://github.com/sheetjs/js-xlsx) There are plenty of very difficult problems involving ill-specified aspects of the various file formats and errors in specifications, but it is largely a matter of grinding and finding files in the wild that capture the behavior you want to understand. Those are "difficult" in the sense that people still get these things wrong (related: recently a bug in the Oracle SmartView corrupted US Census XLS exports, which boiled down to an issue in calculating string lengths with special characters) but they don't feel difficult since most of the work didn't involve any really clever insights.
IMHO the hardest problem is now fairly straightforward: How do you enable people to test against confidential files? The solution involves running the entire process in the web browser using FileReader API: https://developer.mozilla.org/en-US/docs/Web/API/FileReader , and that is an obvious technical solution in 2017 but few thought it was even possible when we started.
I can imagine you get some pretty warty excel files. I mostly get PDF for my sins. I'm sure, like me, you've spent hours taking bits out of files until they work as expected and then figuring out what the difference is :-)
> It's all relative. Sometimes solutions require insights. Other times solutions involve a ton of grinding.
> Many people have a tendency to overemphasize the first and greatly under emphasize the second, even though the grinding may actually be harder than devising clever solutions.
> No individual problem is hard. It's arguably hard to do tedious work day in and day out for months at a time, but I don't think people would call that "technically hard".
Not being able to answer this question is just a telling as an answer itself.
Last year, I came up with a solution to a problem that we'd been solving sub-optimally for years. My solution is arguably optimal (given a certain set of assumptions) and requires multiple orders of magnitude less code than the previous solution. The solution is the core part of a paper that was recently accepted to a top conference in its field. That sounds like it might be good evidence the problem was a hard problem, but in fact the solution just involved writing down a formula that anyone who was exposed to probability in high school could have written down, if it had occurred to them that the problem could be phrased as a probability problem (that is, the solution involved multiplying a few probabilities and then putting that in a loop). When I described the idea to a co-worker (that you could calculate the exact probability of something, given some mildly unrealistic but not completely bogus assumptions), he immediately worked out the exact same solution. It wasn't an objectively hard problem because it's a problem many people could solve if you posed the problem to them. The hardest part was probably having the time to look at it instead of looking at some other part of the system, which isn't a hard "technical problem".
Another kind of problem I've solved is doing months of boring work in order to produce something that's (IMO) pretty useful. No individual problem is hard. It's arguably hard to do tedious work day in and day out for months at a time, but I don't think people would call that "technically hard".
I'm with you. Maybe I never work on "REAL hard things"? How would I tell?