The euphet treadmill took us from programmer to developer to software "engineer" to product "engineer".
CEO: I want X....and Y, Z, and enhance A-F, all yesterday
SE: you outsourced me last month!?
PE: I can’t do anything without more specs
Or
CEO: I want X
SE, PE: 3 months later deliver X
Sales: Customer doesn’t want X, wants Y and Big Deal deadline is tomorrow
CEO: drop everything, R.I.P. out X and build Y and ship it tonight!
Sometimes the old school methods are better than whatever modern crap has the CEO making feature requests to engineering.
CEO: "We need a search box"
Product Owner: "Okay, let me do some research and build out a design. Here are all the metrics we'll track and quantify the success of the project on"
Engineer: "Okay I've implemented your design, and put in all the essential logging to power those dashboards"
Product owner: "Great, after our search box, sales went up 7% for these items. This means $75,000 a year. Success"
CEO: "K."
This one confused me. Shouldn't you be logging as much as possible anyway? There's going to be so many things you can't predict ahead of time with this kind of stuff.
Having an access log and error log is sufficient, until you know what you want to query. Writing a script to do some custom query is a few minutes of work and can run over all the files created...ever. Then the person who wanted the query will just as likely realize the question they had is impossible to answer right now or doesn't want to expose the answer, or any other of a myriad of results that aren't "this adds business value".
Stop building more than you need. Get back to the work.
Logging, auditing, anything that isnt immediately available to solve the next problem is often disregarded until "wow it would have been nice to know..."
In my experience companies are experiencing data overload and data paralysis because they have tons of data that is absolutely useless now and will be even more useless tomorrow. And they keep adding data about the data because that's the current koolaid everyone is drinking.
The best answer is somewhere between these two extremes, and I like your approach of starting simple and clear - a file based approach also easily lends itself to centralizing or bootstrapping to another medium - the bigtable approach calls for a migration and potentially even a change in the guarantees made by the underlying system because someone started to repurpose it as not just an audit trail because "of course we can just query it." (ugghgghghgh)
That's the extra complexity. If dumps into data lake are that important, then it becomes its own special beast that needs to be fed. If the dumps into the data lake aren't important, then the question to ask "Do I need the data lake at all or am I doing it because everyone else is doing it?" If the answer is "yes" then absolutely build that functionality. If the answer is "Eh" the question becomes much murkier.
Your examples essentially differentiate between 1) a person who accomplishes an assigned task because it was assigned to them and 2) a person who knows that there is more to completing a task than meeting the defined requirements.
Company 1: First question was to do a merge sort on the board. Second question was to do some sql statements.
Company 2: “We are starting a new greenfield project and all we have are two developers who have been writing PowerBuilder code, stored procedures, and they are just learning C#. What would be your 90 day plan to start the project”. The correct answer started with budget requirements, training requirements, hiring contractors, etc.
Guess which job I took?
Two years later my complete interview process around the job I took was around making processes more “cloud native” and more highly available as opposed to one I didn’t take where they asked me to write code on a board to generate a Fibonacci sequence.
Short version: if they ask you to do coding whiteboard interviews instead of architectural interviews, you’re being hired as a software engineer.
Also, IMO the term "Product Engineer" does not denote that it has to be software-related at all. It could be someone who designs shoes or training seminars or whatever (see https://en.wikipedia.org/wiki/Product_engineering).
But, I made it clear at the interview that my first steps would be to build up the department, develop processes and training. They already had a barely working 20 year old PowerBuilder implementation where the various offices were forced to use Citrix terminals to access the system. They needed some tighter web and mobile integrations they couldn’t currently do.
I found out the real money was in cloud consulting. They hired a few consultants to “move us to the cloud” (a bunch of no nothing “lift and shifters”). I found out that’s where the real money was.
Left there after a year and half with a better resume, self demoted (responsibility not pay) to my current job and I’m getting a chance to fill in some resume gaps to become a more well rounded (and paid) consultant in a couple of years.
CEO: We need to add a product search to our website so customers can easier find the products which they like.
SE: Ok - and walks off and builds a search box into website.
PE: Ok - walks off and builds a search box into the website. Also logs all search queries with zero results into a big table to give the startup a better insight into products which customers want, but couldn't find.
Another example...
CEO: We need some real time messaging into our communication channel (Slack) when a customer order couldn't be fullfilled during packaging, so customer support can pro-actively reach out to the customer and notify them about whatever the problem was.
SE: Goes away for a few days and builds a complex real time notifications API which needs to hold state about which notifications it has already sent and which not. Uses latest fanciest tech for that.
PE: Starts asking critical questions. Why does it have to be real time? What time are most orders being packed? What time does customer support staff start actually working? Turns out after 10 minutes of asking the right questions that a real time messaging service is overkill if most CS staff only starts working after all the packaging is already done. A simple stateless daily report being sent once to the Slack channel is completely sufficient and does the job just as good if not better than a real time solution. PE walks off and implements this in a couple hours and has the majority of the week still free to do more impactful work.
EDIT: Product engineers with a real "get shit done" mindset are invaluable not only to startups but even well established companies. They are the ones who really drive success through good communication, great understanding of the business, all the different parties and the tech which they have to their availability to find the best solution for any given problem.