If your boss at the workplace you've been working probably "within the month" or a little longer tells you to write a program to do X, you'll maybe pulling all your knowledge out to write a program to do X as if it were the most important program in the world.
I don't necessarily do BDUF or anything, but I generally do more "engineering" than is often directly necessary. Partially, it's my personality. But part of it is that in my experience, prototypes have a way of becoming production systems.
Last time I had to do a prototype for work, I split the difference: working quick and dirty, just meeting the minimal requirements, but also documenting what we would need to do if we decided to move forward with it. That felt like a good balance.
I use it until I realize it’s not quick after all. Now I default to writing one off scripts.
For something like this, the first thing I'd do is extract the first row manually in Vim, recording my actions as a macro, then use that macro to extract the rest. That would take a few minutes tops, depending on the structure of the data it might take seconds or there might be a bit more massaging necessary, but no more than about 5 minutes for this step. I have a keybinding that maps execution of the macro in the q register to the spacebar to streamline this process; I double-tap q to start recording, press q again to stop recording, and press/hold the spacebar to execute that q macro. In my experience, 2 or 3 passes are usually enough to cleanly extract a table of data from some random HTML page.
If the job was a once-and-done, then I'm already done. If I am meant to be creating something that can be re-used later, then I still do the above anyway and now I have the result with which to test whatever more permanent solution I'm writing.
Perhaps the dev thought it should be a tool that you will be using every day, or a thing to maintain in the future.
For example, in what you wrote, I really understood the thing you wanted to do by reading what you've finally did. Perhaps the input to the dev should have been: "Hey we have to do this, and we need this mock data from this HTML to start developing. First we take the data out. Don't worry much about how you do it, even if it's nasty. We don't need a full-fledged production-grade "HTML data extractor" to do this. This is data that we would be importing twice at max. So even if you lose 15 minutes copypasting it into a CSV file, it's fine".