Nice, i can apply to be one for the 11 months this position shall exist

and be rejected for not having 8+ years experience in it.

It's frustrating how the data discipline is split up into so many different job titles. I regularly have to work with data engineers, data analysts, analytics engineers, data scientists, machine learning scientists, and machine learning engineers. Each view their role really narrowly, so you need to pull in a lot of people to get something done. It's often easier to just do the work yourself.

I get that some specialisation is useful, but I would like to see more T-shaped data folks.

It used to be the case that a "data engineer" was just a software engineer that was good at data-related things (maybe they were good at Kafka, airflow, DBT, knew why you'd pick postgres for one job and Clickhouse for another, etc).

Then the BI people (who had never written a line of code except maybe some SQL) got jealous, started calling themselves "data engineers", broke everything ("what do you mean 'we need tests???'"), and ruined the title.

Once upon a time they were called DBAs.

I have a different take on this. I have a friend who used to work at a bank as an analyst, and he had a hardcore finance/math background, and was a self-taught R/Python/SQL coder before this whole 'big data' thing became popular a decade ago.

Back then his bank started building a data science/engineering dept, and he jumped ship, and essentially did the same work for 2x the salary. Let me tell you, understanding Black-Scholes and what the data actually means and how to source it, as well as what finance people actually are looking for is a skill that's 1000x as difficult to get as learning R/Python/Pandas etc.

Yeah, as a "T-shaded data folk" (have worked as a data analyst, data scientist, data engineer, and machine learning engineer, so I think I'm close to a full bingo care) in data, I agree!

I think the specialisms are really helpful for hiring - data engineering and data science involve a lot of different things so it's a lot easier to hire someone if you don't require skill in both.

That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.

The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.

> Yeah, as a "T-shaded data folk" (have worked as a data analyst, data scientist, data engineer, and machine learning engineer, so I think I'm close to a full bingo care) in data, I agree!

Me too. There were no data engineers when I was a baby data scientist (15 years ago now).

I think that this was a terrible idea (the splitting out of roles). I remember once being asked in an interview if I was able to pull my own SQL or would I need an analyst. That kind of split always struck me as absurd, given that the only way to build a decent model is to know the data very well first..

> That said, I think we have a culture problem in defining our roles narrowly, I've seen data scientists look down at data-pipeline parts of their job, and ignoring model latency and deployment; data engineers be deeply uncurious around what happens to the data after they put it in a warehouse; etc.

I honestly feel like this is driven by the titles. I will note that most of the unsuccessful/bad data scientists I've met were the ones who only cared about the models. As I always said to people, I can run sklearn in a loop over model families if that's all you bring.

> The best people I've worked with, within data and elsewhere, are the ones who are curious about every aspect of the stack, even if they mostly focus on one side.

This is probably just a good signal in tech in general, some curiosity is really required to be successful in this industry.

I’ve come to see it as a consequence of the industry’s pivot toward every firm of every size becoming more enterprisey. The enterprise habit of using 3rd-party vendors in particular.

15, 20 years ago, me and my T-shaped skills could get by just fine with a database server, a NAS, and some frank conversations about priorities.

Nowadays it’s not enough to to know how to work with databases and files. There’s also all these platforms: databricks, snowflake, new relic, sentry, datadog, GitHub actions, prefect, dbt, kubernetes, cassandra, elastic, and on and on and on. Each brings a new set of idiosyncrasies, its own terms for things, its own enterprise integration headaches. It’s just a hell of a lot of knowledge to as one person to cram into their head on top of whatever skills and domain expertise form the actual core of their job.

That makes sense for an enterprise. Enterprises have a lot of bodies to throw at the problem, so they can have an expert for each tool who’s available to support everyone else. And they like to have support contracts and SLAs, which you can get from à B2B vendor but not from the KISS principle. And they don’t mind great heaping piles of unnecessary and mutually conflicting abstractions because that sort of thing is just SOP for a large bureaucracy.

But if you want - need - a system that one person with T-shaped skills can reasonably fit into their head, this is not the way.

Keep in mind that the exact same arguments were being made 15, 20 years ago about why specialization was important. And yet, we look back at that era being the halcyon days of when specialization wasn't necessary.

Every era views itself as an intricate snowflake, but they rarely are.

I think that might be the goomba fallacy? There is no universal “we”. Some people think specialization is important. Others think generalists are important. Others talk about T-shaped skills. These are potentially all different groups of people that ideally realize that there are tradeoffs to each option and the best compromise is context dependent.

And I do think it’s more about how you build things than it is about the calendar. The calendar is just a proxy for what’s fashionable. For example, the startup I currently work at is in the process of incrementally simplifying our stack. Because we agreed that it was getting difficult to understand and expensive to operate. We don’t view this as turning the clock back to 2006; the discussions have framed it as a judicious application of the KISS principle. And yes, it has helped us break down knowledge silos and enable team members - especially data scientists - to be more self-sufficient.

My favorite title is 'digital prostitute' - we do anything if you pay us money :D

Agreed, after decades writing scala, go, rust and python and building high throughput analysis systems, I'm now not allowed to touch any of our infrastructure, or write any airflow jobs because our teams got partitioned into data engineering and data analysts, and I was given the analyst team.

30 min livebug fix and review for me on a job I specced out? Ticket to another team to review and schedule for next sprint and thus 2 weeks of metadata likely lost or malformed.

These setups look great (but full of invisible friction) when things are going well but shit the bed as soon as you come across the simplest of human errors that needs a solution.

This sounds more like a culture problem than a title problem though. Are you saying you can't even open PRs with a pipeline proposal? Your data engineering team is playing emperor and wants to protect their kingdom at all cost

Weird. I feel the other way. I do too many things from all areas that none of the titles fit me. I build pipelines, analyze data, engineer the infrastructure around data orchestration, etc.

You get a new title, analyst engineer!

senior analyst engineer!

Is the era of adding more together already over? Like Developer & Ops -> DevOps; & Security -> DevSecOps; & Finance -> FinDevSecOps

Me too. We could do with a few more people but I would not want them to be overly specialised.

> It's often easier to just do the work yourself.

And yet if a data person were to try the same on your specialization, surely this would also be well and good.

Could you describe a typical day-to-day thing that each of those roles does? (I see the terms often, but I couldn't say what each of them actually, ya know, does.) Thanks!

I don't know what they call them now but there's typically a role where one is essentially a fancy DB admin that extracts and centralizes all data into db tables as per how the analysts want. Sometimes this is local and sometimes a data pool of s3 buckets. This is harder than it sounds where depending on the business and how fast it needs to action data you are likely skilled on low latency networking and fast algorithms to transform and load data from all kinds of hard to extract places like other proprietary software. They also have you adding desired analytics like PostGIS for location-based aggregations, designing all the backups, and some companies have 100+ pipelines as many proprietary data sellers exist that give an end of day dump or real-time fire hose you have figure out how efficiently capture and store.

An analyst then loads this up with whatever visualization software or 3rd party BI tools and interprets the findings and I've found this varies from giving a simple presentation after AI tools do everything for them to doing full on R Studio analytics such as these models https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/

If the analyst then selects a feature set from the data and performs machine learning I've found they are called 'Data Scientists' otherwise they are analysts but these titles mean something different everywhere.

How scary deep end to end ownership of some kind of a data analytics pipeline solution is, depends on a lot on organizational maturity which based on what I've seen is not high when it comes to data. In-house developers are avoiding these kinds of responsibilities not due to a lack of skill or ambition but because your job just becomes "churn dashboard e2e pipeline expert". You somehow can not be a T shaped developer and use your broad expertise in hopping in and out of projects and domains.

Stakeholders prefer a single point of dev contact so you can never get rid of the role, and you can only repeat that for so many domains or solutions before you run out of hours in the day. Trying to spread this ownership across a team results into either horizontal or vertical specialization which are the two dysfunctions. I'm not sure why this is like this in data.

The best quote I've seen about AI and work: "If your job can be done from a laptop, you should be seriously considering what you'll be doing in 5 years".

Isn't that the same message we've been hearing ad-nauseam by now?

What makes that quote so special that it is worth pointing out as "the best" in 11/2026?

It's not new, but especially after the release of Opus 5.5 it's been sticking in my mind lately.

You haven't answered my question.

Since the release of Opus 5.5 I have more work than ever and it enables me to move out from the coding I never really enjoyed to a full stack product engineering.

I'm also working on transitioning from a classic SWE to a small independent team doing our own products.

Sure I won't make as much money for doing bullshit changes in some old legacy system but I will actually deliver things people will like using, and I'll be in touch with other people on a daily basis.

Nice, now the less-than-14d old accounts are teaming up to push the narrative for the upcoming IPO

I have zero stock or interest in Anthropic IPO, moreover I really hope open weights catch up price-wise to what Claude Max provides (tokens there are heavily subsidized right now), so that I can make a switch and not feed the beast.

I'd like to also use a model on my owm hardware, but alas, I'm relatively young in the industry and my wallet is not fat enough for a local AI stack.

I thought so too. After a few weeks of semi-auto mode, I can clearly see how my codebase deteriorates. Humanity is not over yet.

Related: https://news.ycombinator.com/item?id=50438318

My view, as someone who was flat out "AI is a hype scam" up until around Jan this year. It was hype for a while, and forms of it are still very much hype, but the reality is catching up with the hype.

I.e the reality has so far been ~2 years behind the hype.

I'm not saying I like these companies, or that I think the economics of any of them is sane. Just saying LLM's are capable.

Sure, but these are still astroturfing bots, and treating the messages as anything else but that helps them achieve their goals.

Interesting that you think I'm a bot

Well that's one option. It could also be someone that is paid. It could be someone who drank the Kool aid. It could be someone financially invested

Or it could be a kid that watched too many tech YouTubers.

But, fundamentally, that doesn't matter, because functionally it's as described.

Look, I'm a new account, a dev and I'm a human last I checked.

I am not alone in dropping a positive AI message once in a while without having "an account" which is an outdated, useless notion anyway.

I find the greybeards unbearably stuck in anti-AI psychosis. People like myself have been saying LLMs are capable for years and were nothing but stigmatized and ignored. But now ya'll are seeing the light? A little tardy to the party. Again. Always late, but with a big mouth.

Cut a long story short: I don't gaf about Anthropic or any American company for that matter but I also don't gaf what happens to shortsighted big mouth developers that need literal years to see what was obvious on day one. No longer do I need to bow down to these types and it's extraordinarily liberating. In my company the dickhead sysadmins are falling one by one and it's gorgeous. Good luck memorizing manpages and obscure pf settings, because that's what we really need now. Humans memorizing things and calculating because that's what we're really great at. Great long-term strategizing there Newton.

We're now at three fresh accounts and one additional greybearded anti AI strawman.

I was out for 2 weeks, and our resident meat proxies on my team managed to half-destroy the project by doing everything the way AI wanted to, instead of how the company systems work, introduced like half a dozen cross-connections across barriers that should never be crossed, introduced a parallel artifact repo, permissions system, and different build tooling to what the rest use (including what the current project was built with), and managed to create a lot of drama with external teams who refused to work with these new systems.

AI still needs lots of guidance, and might seem like a genius that does everything on its own, until you give it to people who can't guide it and things begin spiraling out of control.

The same could easily have happened pre LLMs (just slower). Its why things like ArchUnit existed, or style guides, or annoying architects who stopped engineers from using latest JS framework pre LLMs, to define high level requirements for software.

I try to avoid justifying my own existence, but its a reason why I think skilled engineers are way more productive using LLM's than person off the street. Because they can confine them. Though there is no technical reason that ability can't be codified.

I don't think it'd have been likely. First, pre-LLM, people read code 90% of the time and wrote it 10% of the time - so the flow would've been to first understand the exsiting code and figure out how to interact with it, and/or asking others when stuck, rather than just plowing ahead with the favorite set of tools that particular LLM has

> The same could easily have happened pre LLMs (just slower).

"Slower" is doing a lot of work there. It's the difference between an engineer or two wrecking havoc doing a major refactor and convincing a manager that it's ok while you're away on vacation, and the same engineers failing to convince the manager because it will take forever, and quietly dropping the project after hitting the first roadblock.

"How long is this going to take?": "a week" vs "a quarter" makes a huge difference.

Extreme speed is a form of quality; the fear is that it enables bad things to happen.

I've actually been thinking about this issues for a while, that agents are way more successful if they have a decent architect guiding them vs. full vibe.

I've been solo prenuering for the last year so didn't have a lot of data from others but your comment convinced me its a problem worth solving, so I put this together https://vitruvius.8by3.com/

Its hard wip, pushed seconds ago, paint still wet. But the idea is to "enforce" good architectural practices for your agents. Can store constrains and ADRs in git so everyone in a team (everyones agents) can read from the same page.

Depending on the field, five years is generous. 6-12 months (if not already gone) for many things.

I made my 20 year career as a person who could do stuff with computers, software engineer etc. Every time I hear someone say, "my non tech friend who vibe coded an app got a bug he couldn't fix" etc, my response is they are using the wrong model and give it a few months. And I say this with great pain, I am in no way an AI booster, just a realist.

Every task that would be a multi-week engineering effort, I defer for a few months, until inevitably a model that trivializes it appears, repeat ad-nauseum.

I am wondering whether us folks in the IT space are missing things about how the real world out there functions with those predictions.

As long as we do not hand over our collective decision making power to the machine god, which I don't think is a good idea and am not sure is even possible, the world out there is at this moment still shaped by humans, with other humans, for humans.

I have this feeling that there is more about society and economics than efficiency. I have right now people I know that work in jobs that could have been automated for many years to a very high degree. That is even without AI.

> As long as we do not hand over our collective decision making power to the machine god, which I don't think is a good idea and am not sure is even possible,

Counterpoint: This is not only possible but has effectively already happened. From the outside, the current trajectory is indistinguishable from humanity serving the AI, pulling all possible resources out of literally everything else in order to burn them on AI R&D, training and inference.

[delayed]

It will probably take decades before the world adapts to AI. Even some of the most modern societies still haven’t fully digitilized yet. There will be plenty of time to ride the wave of this adaptation, and still get payed for software work. There’s not enough mechanisms in place required for a fully automated adaptation. Human expertise will remain the missing piece for doing this shift efficiently.

I'm less convinced. I think as engineers/software-folks we tend to look at "something being built" as the end-result (since that's what we work towards).

Big companies though, are more concerned with value and risk. Say some data model provides $5 million in value, hiring a data person to check "everything is kosha", even if in theory someone could vibe code it by prompting an LLM, is a pretty good investment if it derisks the process.

To your point, I wonder if organizations will trend towards hiring 3rd party companies to check work so they have someone to blame and can "focus on innovation". It's already a popular tactic to hire consultants for risky business decisions. Perhaps this is a new prediction: companies get leaner while outsourcing the checks to the same people they fired.

If I am still on this realm, it will be getting closer to 60, thus getting some early retirement most likely.

So my job should require a desktop computer?

I'm yet to see an AI that can take responsibility for me, and could go to jail if it does a poor job. I'd be delighted to pass it to an AI.

PS: Yes I mostly work from a laptop.

Green account with three-hundred karma says that you should be afraid. More at your regularly scheduled programming.

They asked ChatGPT to invent a new job title for marketing, genius. At a previous company our marketing person created some new term for a segment so we could say we are the only ones doing it. I think there was a term for this grift but I don't remember.

This fake job title sounds like what BI analysts or analytics engineers should be doing as part of their job, not a new job. These jobs are the ones that are usually working on the modeling, answering business questions, and making data accessible and would be creating the context, prompts, and evals to make sure it's working as intended. I think really, analytics engineering and BI should be combined.

I think there are big risks with just labelling it as "wild west" and let users do whatever they want there.

It's always been possible to massage your data to fit whatwver case you want to make. Previously you had to go through a data analyst and convince them to let go of their principles. Now you can just let an agent figure it out.

So how do you make sure insights are actually shareable (and trustable) with other parts of the company?

I think just exposing the metrics layer is probably the answer here.

It's kind of funny that the industry has tried to democratize data for so long, but now we scramble to restrict access to protect users from making incorrect decisions.

> I think just exposing the metrics layer is probably the answer here.

This is the short answer. Metadata has taken on a whole new level of importance and if your model isn't tight you're going to have a bad time in a lot of new and exciting ways that weren't exposed in a more traditional data stack.

> A model doesn’t know that when the sales team talks about revenue they mean commissionable revenue, and the finance team uses something different.

Umm, the model knows the same way humans know: either by looking into slack threads/confluence pages/emails… If your model has access to those things, then it can know. And I foresee that in the near future the model can simply have an incarnation as a Slack member, and when context is not enough, they will simply ask people around (and update docs accordingly)

True, except when it’s not. The correct definition depends on the situation. For example if the if a sales director is pulling numbers to give to a finance director, then the normal sales definition does not apply. How would the LLM know that?

This underestimate tribal knowledge, wrong assumptions shared across stakeholders, same name ideas with different domains, similarities of words that can mean different things (e.g: revenue) which misleads LLM.

People have a judgment call to say “no this is impossible, when they know something”, but an llm when reads a meeting description of someone that is effusive and truly believes that can be wrong.

As someone that works on data and having seem llm full of context getting wrong assumptions, I’m still looking for the ideal thing.

everything old is new again. it used to be the venerable ETL guy who would be the pivotal cog in the machine learning gear. someone who knows where the data is and what it means is always important.

That's just data engineer enhanced with agents.

On a more serious note, I want my next title to be "Super High Intelligence Trainer"

Like a PT for AI's ... Getting them into shape, and fighting strong.

While in general I agree with most of what's said, some of these feel like "Horseless Carriages".

Like for example

>It doesn't know how your company counts revenue, or which of several similar-looking tables is the real one (it is revenue_cpx_new_revised_v2_use_this_one )

This assumption mostly only holds for non-clean data ( which is mostly a result of humans being lazy and backlog being clogged up ). As AI starts building more of the pipelines upstream - this becomes a non-issue.

Even for existing pipelines, I'd expect fixes to move upstream: when an agent encounters ambiguity at query time, the useful output isn't an edit of the context,but a PR that fixes it.

You can't adequately clean some data though. That your psp had a hickup and only processed 50% of orders on Christmas Eve can't be modeled away. That's where you'd need a context layer to explain why it happened

Yeah fair point tbh. But even that doesn't require an human in the loop no ?

I guess as long as it gets into the context layer it's all good! I'm excited, but skeptical, for those company OS agents that will be able to figure this out

We're all figuring out as we go i guess :)

Can we move this whole business outside of CS please?

I’m here using Opus 5.5 and Fable 5.2 and I wouldn’t say the code is generally very good.

so now you want to be a data intelligence engineer (die) ? wait die ? like my career ? oh yeah!

Such blatant LLMisms ... "blast radius," concepts living in other concepts, someone "living with the churn number" (!?), etc.

Lots of people used military-esque terms before LLMs and I assumed that was the training data that gives rise to LLMisms.

The job you will DIE for.

People don't don't make decisions by looking at dashboards and visualizations. They do while looking at emails and listening to others in calls.

Yes and no. Most dashboards and visualizations are shit and, yes, people don't generally make decisions from this. Effective visuals, like effective discussions and meetings, help to drive ideas and decisions.

I've never had business users as excited about data and they're spewing out visuals on their own with Claude and an MCP connection. They're sharing them with one another. It's still new and shiny and it's created challenges in terms of curation and we have a lot of "source of truth" discussions when folks wander outside the data model, but this is a genuine revolution in the discipline. There's a lot of truth in the article.

This is not true. An example is that good B2C company will use events data to track feature performance and “guide” the development.

Bank app - get’s engagement on security features marketing, so it’s a hot topic Language app - get’s engagement on ai posts from blogs, invest on AI Seller on Amazon - analyzes trends and see there is a whole on specific product, or analyzes competitors prices and see it can get’s.

Data is everywhere, the thing is not everybody uses and not every product can.

The correct nomenclature is now “Super Intelligence Engineer”, right? I think it’s required if you want to seek any federal government contracts for the next two years, at least.