The value the LLM adds is interpreting/processing data without having to tailor input streams. Imagine if formats change, fields get renamed, and so on. The maintenance would be a headache if this was done on a per-service level. I think the reduction in fidelity seems like a reasonable tradeoff, but that's for the user to decide of course along with local/cloud processing and proprietary/open source software.
Even things like invoices from the same service change format over time.
A prompt every few minutes that would ask "What are you doing now?" would be interesting to me, as a professional procrastinator. Maybe an even better one would be one that says something like "In the last 10 minutes, you spent 90% of it on Hacker News".
And, as long as there's an API, I am fairly certain that maintaining a compat layer is a lot less work than retuning the LLM when the images change. (And you'll want to adjust your tuning, at least with current SOTA, or your error rate will reach Unpleasantville fairly quickly)
Yes, it seems easier on the face of it - but the reality of building an LLM pipeline will quickly point out a lot of edge cases.
Recall has lost a ton of useful metadata you already have - both URL visits and streaming are clearly discernible actions, both at the network stack level, and from your browser history. Throwing that away to trust an LLM to re-infer the same data is both reducing data fidelity and significantly increasing processing cost.
If you want to see this done reasonably well, I'd suggest looking at e.g https://beepb00p.xyz/promnesia.html (which not surprisingly bears a strong similarity to what the article discusses)
LLMs don't add much value here, outside of tightly locked down systems where screenshots are the only way of exporting.