News for hackers. Stuff that matters.
Re:Goodbye, Data Science by samvher on Tuesday November 29, 2022 @09:04AM (#33787689)
> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case.

In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle.

A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data "leaked" from the future), or to rely on in-sample results too much in other ways.

Because these issues are subtle, management will often not pick up on them or not be aware that this kind of thing can go wrong. With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.

Re:Goodbye, Data Science by CapmCrackaWaka on Tuesday November 29, 2022 @09:43AM (#33788178)
I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies.

I had one candidate who was in charge of a multi-armed-bandit project at their current company. I asked them how it worked, and how they settled on that. Their response was "you know, I'm not really sure, the code was set up when I got there". He had been there for over a year, and could tell me nothing!

> A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data "leaked" from the future), or to rely on in-sample results too much in other ways.

It's funny you mention this, we have a direct competitor who does this and advertises flawed metrics to clients. Often times our clients will come back to us saying "XYZ says they can get better performance", the performance in this case being something which is simply impossible without data leakage or some flawed validation strategy.

Re:Goodbye, Data Science by adamsmith143 on Tuesday November 29, 2022 @10:17AM (#33788568)
Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.
Re:Goodbye, Data Science by semi-extrinsic on Tuesday November 29, 2022 @10:25AM (#33788688)
These days if you have a company selling cat food or rivets for aerospace or providing taxi swrvice to a random city, or whatever, they might have a few data scientists helping them make "optimized" business choices. Obviously they won't have a very adcanced recruiting process for that.
Re:Goodbye, Data Science by quickthrower2 on Tuesday November 29, 2022 @12:37PM (#33790614)
Re:Goodbye, Data Science by noelsusman on Tuesday November 29, 2022 @10:46AM (#33788970)
It's different at a lot of non-tech companies. I'm in the nonprofit world and my interview barely had any technical component at all.
Re:Goodbye, Data Science by bo1024 on Tuesday November 29, 2022 @12:54PM (#33790810)
To be able to tell whether a candidate is good, the hiring team has to be expert! No chicken, no egg.
Re:Goodbye, Data Science by bootsmann on Tuesday November 29, 2022 @11:14AM (#33789401)
I think the issue here is that "data science" encompasses two very distinct branches of work. One answers to business needs and the other produces data based solutions for the product itself i.e you might have a data scientist who A/B tests your website design so you minimize your churn rate and the other is the team at uber eats who maintains the recommendation engine. While the distinction might not always be as sharp, the former makes up the bulk of data scientists in the market (and I suspect the OP is in that boat) with comparably simple interviews while the rest is the 5 step interview process with hackerrank test you are more familiar with.
Re:Goodbye, Data Science by CapmCrackaWaka on Tuesday November 29, 2022 @12:54PM (#33790802)
Yes we definitely fall into more traditional "predictive modeling" data science than deep learning / recommendation algo roles.
Re:Goodbye, Data Science by wirrbel on Tuesday November 29, 2022 @01:16PM (#33791101)
I think the distinction is not so much on the domain/application. Rather it’s just that many Organisations decided to jump on the data-science wagon and don’t quite know yet for what qualities to look out for during hiring. And in second order as long as the predictive model is not included in a business process the over fitting is not as easily visible to the layperson stakeholders (and junior data scientists).
Re:Goodbye, Data Science by random314 on Tuesday November 29, 2022 @02:26PM (#33791990)
The ML interviews at FAANG are absurdly simple. Design YouTube recommendations for which canned answers are readily available.

A simple stats question. If I double the number of samples, how much will the confidence interval change? Most FAANG ML engineers can't answer this question.

Re:Goodbye, Data Science by adamsmith143 on Tuesday November 29, 2022 @03:25PM (#33792775)
And then reverse a binary tree?
Re:Goodbye, Data Science by random314 on Wednesday November 30, 2022 @12:49AM (#33797434)
What's your point? The question about sample sizes is arcane trivia?
Re:Goodbye, Data Science by adamsmith143 on Wednesday November 30, 2022 @10:16AM (#33801857)
For someone on an ML team? Yes. You could spend years building computer vision models and not once think about sample size.
Re:Goodbye, Data Science by random314 on Wednesday December 07, 2022 @12:59AM (#33891116)
The Dunning-Kruger effect is strong here. "What I know is what makes me the expert. What I don't know is irrelevant".

The definition of Standard deviation is in chapter 1 of Stats 101. https://www.google.com/search?q=standard+deviation&tbm=isch Apparently, asking Stats 101 chapter 1 question of a so called "Data Scientist" is too much of an irrelevant question!

> expect people to have very high Stats skills

Or as you have made apparent, expect people to have ZERO stats skills!

Some of the innumerate activities I have observed in "expert" data scientists and ML engineers who have years of experience without once thinking about sample sizes

1. Using A/B tests to accept the Null hypothesis instead of rejecting it

2. Squandering away 30M $ in annual revenue because they wanted to avoid a situation/meeting in which they might look like they don't understand statistics. This is hilarious because they simply nodded their head as if they understand all the calculations and then simply dropped any other meetings or followups and left 30M $ on the table

3. Not refreshing a key revenue generating model for 18 months because the were "trying to figure out" why the AUC was improving when the performance on "golden set data" was dropping

4. Using thresholding and aggregation to produce poor quality distorted training data of rich perfectly sampled data

5. Trying to use A/B tests to estimate impact even when the control and variant are not independent

All of the above at FAANGS! My coworkers in a non FAANG company were much more sophisticated. These are the kind of candidates a "build recommendations for youtube" interview selects. Template appliers.

The list of stupidities goes on and on! But yeah, none of them think that a basic understanding of statistics is necessary for work. The good thing about Javascript engineers is that they don't have an understanding of Statistics and are aware of it. However the DS/MLEs are unskilled and unaware of it.

Re:Goodbye, Data Science by Radim on Tuesday November 29, 2022 @11:00AM (#33789199)
> clients will come back to us saying "XYZ says they can get better performance"

Oh yes, good old marketing.

Along with buying off "Industry Awards" – hey, we're objectively the "Best cybersecurity company of 2022!" With a matching "platinum/gold badge" to go on our website! Or buying a place in the "10 Best Products for X" and "Independent X-vs-Y Comparison", another classic.

Because it works. Are your customers not sophisticated? Are they unable (or unwilling) to follow up on defects and outright lies? Or reality simply doesn't matter all that much to them? Humans LOVE a good story more than reality, after all.

Then your contribution as an engineer to your company's success, and hence its longevity and your job security, is strictly inferior to that of marketing. Not everything is the work of evil marketers – a lot of the supplied BS is in response to an existing demand for BS.

Re:Goodbye, Data Science by CapmCrackaWaka on Tuesday November 29, 2022 @11:08AM (#33789324)
> Are your customers not sophisticated? Are they unable (or unwilling) to follow up on defects and outright lies?

You would probably be depressed if you knew who our customers were, and how technologically unsophisticated they are.

Re:Goodbye, Data Science by Foobar8568 on Tuesday November 29, 2022 @11:37AM (#33789727)
I manage at a client an application which is the actual leader (most top right and by far) in Gartner magic quadrant for its category, and for years, I have never seen a product this bad, where the implementors and supports are clueless of their own product. And obviously it's buggy as hell.

Lies and deceptions.

Re:Goodbye, Data Science by jacobyoder on Tuesday November 29, 2022 @11:48AM (#33789908)
The people who make the decisions don't use the product. That's almost always the root cause of this stuff. I worked on a system for my state - another vendor came in and 'took over' all the functionality my system handled. Supposedly. 7 years later, my system powers the exception to the mandate to 'use system X', because... they refuse to provide the functionality that they sold the state. Contractually, "we provide feature ABC", but the reality is.. they don't. I even provided them our code to use - it was paid for with public money, they should just integrate it and then sell it to other people to make their product better. They can't even be bothered to take the code and integrate it... they prefer to continually lie and say "we provide feature ABC" when... they don't. It's beyond insane. A large majority of the people on the ground know it's bad/lacking/broken, but ... they have 0 voice in the matter.
Re:Goodbye, Data Science by ambicapter on Tuesday November 29, 2022 @12:13PM (#33790260)
Gartner is it’s own confidence trick. They don’t rate you unless you pay them to rate you. It’s manufactured reputation extortion by another name.
Re:Goodbye, Data Science by Radim on Tuesday November 29, 2022 @06:15PM (#33794743)
With a handle like Foobar8568, why not be more specific? It sounds like you might have a good story.
Re:Goodbye, Data Science by f1shy on Tuesday November 29, 2022 @02:26PM (#33791994)
Is this the US? I'm concern about the extremely low level or the bar to get a PhD in Europe... and I'm wondering if that is a global problem, or only Europe.
Re:Goodbye, Data Science by DenisM on Tuesday November 29, 2022 @02:39PM (#33792192)
> XYZ says they can get better performance

Can you do your analysis both ways? Give your customers both, then tell them you method is more modern, but if they want outdated methods you have those too.

Re:Goodbye, Data Science by 6gvONxR4sf7o on Tuesday November 29, 2022 @09:54AM (#33788303)
This is the defining pain point for data science, in my experience. There’s no simple ground truth to test competence against.

If someone tells you that the data says their work is good, the only real way to know if they’re right or wrong is to look at what the data says yourself. If 99% of the work is building and 1% is checking something like latency, then you’re likely to have more than one set of eyeballs on that 1%. But if 99% of the work is putting the data together and doing the analysis, then you’re unlikely to have more than one person ever look at that part.

So incompetence goes unchecked (or worse, it is rewarded).

Re:Goodbye, Data Science by vsareto on Tuesday November 29, 2022 @10:42AM (#33788908)
That's the same for many tech jobs. Competence is often only a local thing, subject to politics, reputation, and appearances. There's also no ground truth because the ground changes so fast. No one knows if the technologies mentioned in the OP will be popular 5-10 years from now.
Re:Goodbye, Data Science by jimbokun on Tuesday November 29, 2022 @10:58AM (#33789167)
But if the page loads slowly or the UI is unresponsive, people notice.

The output of Data Science is harder for non-specialists to evaluate.

Re:Goodbye, Data Science by vsareto on Tuesday November 29, 2022 @11:30AM (#33789641)
For establishing competence, you still have to dig in to see what caused the slowness. A regular user can't tell you that.
Re:Goodbye, Data Science by pessimizer on Tuesday November 29, 2022 @12:21PM (#33790357)
> For establishing competence, you still have to dig in to see what caused the slowness.

Not as management. You just have to see that other people's similar sites are not slow with the same resources, therefore it is possible for your site not to be slow. You don't have to know why you're failing to know that the totality of the people you hired were not as good as the people those others hired.

This is of course barring management failure; but if you're failing at management, that's about the same as saying that your engineers were under-resourced.

Engineering competence is largely composed of the skills to figure out what is causing problems e.g. slowness. If you can't figure out what is causing the slowness, your engineers aren't good enough to figure out what is causing the slowness, qed.

That's different than data science.

Re:Goodbye, Data Science by quickthrower2 on Tuesday November 29, 2022 @12:48PM (#33790743)
Software Engineering is one of the few knowledge working areas where you can actually test the result in various ways as a layman. You can flush the toilet before paying the plumber to a large extent and hire another counter-team called QA. QA themselves are tested by future production bugs.

In other disciplines it is way more fuzzy. If you are in the conclusion business and there isn’t a clear path to test your conclusion in the short term you can bullshit away!

Re:Goodbye, Data Science by icedchai on Tuesday November 29, 2022 @06:22PM (#33794810)
Unfortunately, I haven't worked at a company with dedicated QA in the past 5+ years, maybe longer. QA is often seen as a side job for engineers and product teams.
Re:Goodbye, Data Science by quickthrower2 on Tuesday November 29, 2022 @09:55PM (#33796453)
Oh! Dedicated QA makes a big difference, especially when they take leadership and are willing to get involved in lets call it qa-ops: improving automated testing and such like.
Re:Goodbye, Data Science by chasely on Tuesday November 29, 2022 @09:57AM (#33788348)
I've been pitched by many "data-driven" vendors offering predictions. They often have very impressive accuracy metrics (RMSE, R2, etc). When I dive into the details these metrics are often reported using in-sample predictions.

I see this pointing to any of the following:

a) DS teams overpromising the accuracy of their approaches b) marketing driving the narrative and DS getting pulled along c) incompetence from the DS team

Re:Goodbye, Data Science by alexpetralia on Tuesday November 29, 2022 @12:51PM (#33790762)
Are these inferential statistics not designed to be in-sample?

I would imagine predictive statistics use more out-of-sample metrics like precision and recall.

Re:Goodbye, Data Science by antipaul on Tuesday November 29, 2022 @02:19PM (#33791893)
If you do it right…

That’s the problem: these metrics often come from overfitted or in-sample data, and are completely unrealistic when it comes to expected generalization performance.

I’m at the point where I never trust performance metrics anymore. Or rather, the worse they are, the more I trust them!

Re:Goodbye, Data Science by ZephyrBlu on Tuesday November 29, 2022 @08:42PM (#33795862)
I feel like you might be conflating a couple of things, though I'm not a DS so could be off base here.

My reading of the OP's description is that the vendors were offering interpolative predictions, but did not use a test/train split of data. This is in contrast to extrapolative predictions which I would call out-of-sample.

Thus due to not using a test/train split, they achieved extremely good accuracy because they were testing on the same data they trained on. Even though this is "in-sample", you can't use the same data for testing and training.

Re:Goodbye, Data Science by whatever1 on Tuesday November 29, 2022 @10:04AM (#33788420)
Blame statistics for that. Wrong outcome? Well you were unlucky you fell into the 1% error range.

Correct outcome? You totally predicted it correctly.

There is literally no way you can screw up something in statistics and not being able to make up a story to defend your approach.

Re:Goodbye, Data Science by lagt_t on Tuesday November 29, 2022 @10:07AM (#33788453)
You don't look at single outcomes with statistics.
Re:Goodbye, Data Science by whatever1 on Tuesday November 29, 2022 @10:08AM (#33788462)
See? “Better luck next time”.

Not being mean to you, just showing how typically the goal posts are moved.

To give you an example from physics, if you find just one experiment that goes against your model, you immediately invalidate the model. You don’t just make grand claims that the model in general works.

Re:Goodbye, Data Science by cjglo on Tuesday November 29, 2022 @10:50AM (#33789025)
Could be wrong here, but in physics and most natural sciences, you don’t throw away your model if you have one experiment against it.

Usually isn’t it looking for an experiment that proves it and is repeatable?

If I discover a new element in one experiment, the results are published.

After publication, many labs will try to repeat and its not taken away if one can’t do it. Only if all can’t and it casts doubt on whether I did it in the first place.

Re:Goodbye, Data Science by Jensson on Tuesday November 29, 2022 @06:24PM (#33794827)
Scientific method? Models are disproved, not proved. Do data scientists not know about science? People usually understand how science works here on HN, but not in this thread.

Example of a test that invalidated our old theory of gravity and validated Einsteins claims:

https://en.wikipedia.org/wiki/Eddington_experiment

This is how science is done. But apparently not data science.

Re:Goodbye, Data Science by caddemon on Tuesday November 29, 2022 @08:45PM (#33795883)
Models can also be knowingly/intentionally incomplete, which necessarily introduces noise that you are not controlling for. Meaning you have to use statistics, and there isn't really a concept of prove or disprove in the true sense of those words.

Maybe very far in the future there will be models of human biology that are as robust as classical physics, but right now there is such a large amount that is not understood, it's simply not feasible. A drug could work for one person and not another for reasons beyond the realistic scope of the original development hypothesis. It requires a probabilistic view to make any sort of statement about the efficacy then.

I suppose you could argue these models are just wrong and thus trivially disproven, but I don't think that's a productive framing. I doubt any biologist or doctor would claim they have anywhere near a complete model of how their specialty works. That doesn't mean a particular model isn't useful or isn't the best we currently have to work with.

Plus maybe the third best model will actually turn out to explain a separate puzzle piece in an eventual better model. Mechanistic models in biology aren't always well done in practice, but it's certainly not binary either.

Re:Goodbye, Data Science by md224 on Tuesday November 29, 2022 @11:46AM (#33789874)
> if you find just one experiment that goes against your model, you immediately invalidate the model

Pierre Duhem would like to have a word with you:

https://plato.stanford.edu/entries/scientific-underdetermina...

> Holist underdetermination ensures, Duhem argues, that there cannot be any such thing as a “crucial experiment”: a single experiment whose outcome is predicted differently by two competing theories and which therefore serves to definitively confirm one and refute the other.

Re:Goodbye, Data Science by lagt_t on Tuesday November 29, 2022 @11:55AM (#33790019)
Quantum mechanics says otherwise
Re:Goodbye, Data Science by bart_spoon on Wednesday November 30, 2022 @10:11AM (#33801779)
The goal posts are only moved if they were in the incorrect position in the first place. Statistics isn’t the science of prediction, it’s the science of uncertainty management. There is always uncertainty, and how well you’ve measured uncertainty only can be accurately assessed over a large enough time frame over a large enough number of events.

It’s like when people got upset about Trump winning when 538 only gave him a 30% chance. That one event tells us nothing. But if all predictions 538 says have a 30% chance of occurring happen 30% of the time, then they are spot on. That’s not apparent with a single event though.

The problem is that most managers, companies, and people (including yourself, apparently) are statistically illiterate enough to not understand this, and jump head first into data science initiatives expecting immediate results, which is usually doomed to fail, at which point they blame others and not their poorly formed expectations.

There’s plenty of bad data science out there, but most failed data science initiatives are doomed before anyone every builds a model or analyzes any data.

Re:Goodbye, Data Science by whatever1 on Wednesday November 30, 2022 @12:47PM (#33804498)
Whoa, slow down with your elitism.

Who do you think you are?

Re:Goodbye, Data Science by bart_spoon on Saturday December 03, 2022 @10:07AM (#33844000)
What elitism?
Re:Goodbye, Data Science by etempleton on Tuesday November 29, 2022 @10:28AM (#33788719)
I become wary any time someone utters the phrase, "show me the data" or any variation there of. There is a specific type of leader who thinks that within the data lurks a magical solution just waiting to be discovered. There is also the leader who uses data as a trump card to win arguments and these folks are perhaps even worse. This is not new. The origination of the phrase, "lies, damned lies, and statistics," can be traced to the 1800s. I propose the following update:

There are three kinds of lies: Lies, damned lies, and data

I am being glib, I of course do not think all data is inconsequential, rather it is more often used from a place of ignorance or a place of ill intent it is rendered, on the whole, useless.

Re:Goodbye, Data Science by lotsofpulp on Tuesday November 29, 2022 @11:06AM (#33789283)
I have only heard “show me the data” when someone wants someone else to support a claim. I do not see why this would necessarily be a bad thing.
Re:Goodbye, Data Science by fn-mote on Tuesday November 29, 2022 @11:19AM (#33789479)
It's BS because the people asking for the data do not have the sophistication to actually do a reasonable _analysis_ of the data. Or criticize an existing analysis.

Unfortunately, as many posters here are pointing out, there's plenty of ways to do a correct-looking analysis of the data to get evidence to support your agenda.

Maybe your agenda is right and maybe it's not, but I'd love to hear a story of someone standing up and saying "your consultant submitted a report with glaring flaws, they should not be paid and you should reconsider X." It's more likely the little company just goes out of business or the big company buries the failure.

Re:Goodbye, Data Science by ifyoubuildit on Tuesday November 29, 2022 @11:38AM (#33789753)
So whats the alternative? "Just trust me"?
Re:Goodbye, Data Science by yunwal on Tuesday November 29, 2022 @12:02PM (#33790120)
The alternative is to trust in most circumstances (you did hire me after all), and do thorough analysis once in a while as a gut check.
Re:Goodbye, Data Science by ifyoubuildit on Tuesday November 29, 2022 @12:54PM (#33790807)
And how do you do that thorough analysis without asking to see the data?
Re:Goodbye, Data Science by yunwal on Tuesday November 29, 2022 @03:14PM (#33792644)
The person above wasn’t complaining about data science as a whole, they were complaining about data science theater. The scenario where as long as you put some numbers in your bosses face, they could care less what the real implications are. In cases where you’re doing thorough analysis, you should look for the data yourself, rather than ask someone to market it to you
Re:Goodbye, Data Science by nindalf on Tuesday November 29, 2022 @11:22AM (#33789522)
GP was asked to support their claims with data and didn’t like it.
Re:Goodbye, Data Science by giaour on Tuesday November 29, 2022 @12:26PM (#33790444)
The issue is more that the kind of executive who says "show me the data" is often not numerate enough to understand the limitations of the data set in front of them. (Maybe the solution they see in the data has a tiny effect size or is too likely to be statistical noise; maybe the intuitive conclusion is demonstrably wrong once you apply a Bayesian approach.)

A typical VP will have an MBA and maybe took statistics in high school.

Re:Goodbye, Data Science by thinkharderdev on Tuesday November 29, 2022 @12:33PM (#33790559)
> I have only heard “show me the data” when someone wants someone else to support a claim.

I've heard it a lot in situations where somebody is demanding a level of rigor that they themselves do not live up to. This is usually soon after they have framed the conversation around a solution that they want to pursue that also lacks any supporting data. That is to say, being data driven is on net good but it can also just be a thinly veiled appeal to status quo bias (which is itself not a terrible heuristic) or "highest-paid-person-in-the-room" bias.

Re:Goodbye, Data Science by etempleton on Tuesday November 29, 2022 @12:53PM (#33790800)
This is exactly what I am referring to.

I am not talking about in the instance of claim verification. I have seen a number of instances where a leader just wants to see data. Not any specific data, just all of the data. There is a belief that data can solve problems if only they had enough of it.

Re:Goodbye, Data Science by scottLobster on Tuesday November 29, 2022 @11:22AM (#33789523)
Yep, these are the same people who backtest their portfolio and go "see, if you'd held this exact portfolio I put together through trial and error, you'd have turned one dollar into a million without any additional contributions!"

Not a data scientist, but it seems like a lot of people in business refuse to accept the fact that reality is generally boring, best practices are often "best" for a reason, and meaningful progress is hard. Of course it is possible to be too conservative, but 95% of ideas to improve a business or product are ego-stroking bullshit. Everyone wants the V10 engine to go down the highway at 65 mph, while towing a trailer, and there's only budget for an oil change every 15000 miles; don't look at the transmission fluid, just don't look.

Re:Goodbye, Data Science by naijaboiler on Tuesday November 29, 2022 @02:54PM (#33792381)
if you torture data long enough, it will confess to anything
Re:Goodbye, Data Science by apohn on Tuesday November 29, 2022 @10:43AM (#33788922)
>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case.

One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use case turns out to be wrong in the long run.

>With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.

Forget Data Science, you see this in KPIs as well. Say a crappy metric has to be moved by Q2 next year and people will destroy the company to move it.

I feel like Data Science is just one of those areas where you are exposed to a wider range of people and get to feel the full crapola of the insanity of working in a corporation. For lots of roles (e.g. Engineering) you get to hide in a hole behind layers of people and not see some of this insanity.

Re:Goodbye, Data Science by scottLobster on Tuesday November 29, 2022 @11:53AM (#33789987)
Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response in the media/government, to petty crime being outright tolerated, to in my own career I've at times burned through enough money badly enough (albeit with good intentions) that I thought I was going to be fired, only to be told in a performance review I was doing a good job (grateful to stay employed but WTF, I would have fired or at least demoted me). Importantly, the motivation for this lack of consequence doesn't seem to stem from a desire for forgiveness or positive reinforcement or any mechanism that might make things better.

It seems like there's a general apathy/nihilism that's growing in society, whereas by contrast my entire education from childhood up I was held to strict standards and reliably punished when I failed to meet them, and this was in US public schools (albeit a highly ranked school district) and a public university. That or I was just raised in a bubble, and the historical examples I referenced growing up and reference to this day are just a case of survivorship bias, and all the bullshit that was alongside them back in the day has simply been forgotten. I'm not sure, but it is disappointing how little people at large seem to give a shit. Maybe it's a side-effect of the obesity epidemic and people just have less energy or something

Re:Goodbye, Data Science by nostrademons on Tuesday November 29, 2022 @12:26PM (#33790441)
Parenting & the public education system is a very artificially constructed bubble designed to reinforce and reward "good" behavior, where "good" is usually defined as "that which makes life easier for my caregivers". That gives kids a falsely inflated sense of how much everything matters: your caregivers want you to mind your behavior, because then they don't have to, even if you would've been perfectly fine playing with mud or swearing in school or watching TV all day.

In real life there's basically one absolute goal, and that's survival. And that's largely assured in developed western countries these days, unless you do something really stupid. Everything else is socially constructed, and pretty arbitrary. There are some decisions that are fairly consequential for what your life will look like (where & whether to go to college, what field to go in, what metro area to move to, which employers to work for, who to marry, whether & when & with whom to have kids), but you will still have a life regardless, it just might be a slightly smaller house or a spouse that you click with worse or less disposable income for travel.

That's also instructive for what decisions actually do matter. Don't do drugs. Wear your seatbelt. Don't get pregnant unless you mean to. Don't play with loaded guns. If you're staying away from major causes of death you're generally doing pretty well.

Re:Goodbye, Data Science by tasuki on Tuesday November 29, 2022 @04:19PM (#33793448)
> In real life there's basically one absolute goal, and that's survival. And that's largely assured in developed western countries these days, unless you do something really stupid.

Or just get unlucky: no need to do anything stupid. One can easily die of cancer at 30 and leave a toddler behind.

Re:Goodbye, Data Science by JoeAltmaier on Tuesday November 29, 2022 @04:26PM (#33793545)
Well, not so easily. The chances per-capita are in the sub-percent range at 30.
Re:Goodbye, Data Science by tasuki on Tuesday November 29, 2022 @07:08PM (#33795167)
Yes, easily. My partner died of cancer at 30 despite exercising, avoiding alcohol, going for the screening, and generally trying her best.

Chances are it won't happen to you and your close ones. Perhaps try being grateful rather than dismissive?

[Edit: perhaps we have a misunderstanding as to the word "easily". I'm not saying it's likely, I'm saying it can and does happen without any warning signs and no amount of planning/preparation can save you.]

Re:Goodbye, Data Science by EdwardDiego on Tuesday November 29, 2022 @07:20PM (#33795266)
Unless you're exposed to a carcinogenic chemical...
Re:Goodbye, Data Science by angstrom on Tuesday November 29, 2022 @09:18PM (#33796134)
Nah, not even. I have a mutation called CDH1 that happens to be pathogenic and predisposes me to a greater than 40% chance of stomach cancer. It's a dominant gene which means it has a 50% chance I've passed it onto my daughter as well.

That cancer is what's known as a Hereditary Diffuse Gastric Cancer gene (HDGC). It just so happens that the E-cadherin control that suppresses those cancer cells is not processed properly. The diffuse part is what makes it particularly tricky. It's on the surface of the stomach epithelial cells and progresses from there. The only solution is a total gastrectomy (prophylactic if you do it early). No carcinogen necessary. It's found in populations all over the world and pathogenic lines don't even have to be related. The mutation can occur independently in the germline and is passed on. As long as you reproduce before it kills you nature really doesn't care.

Fun side fact. It also predisposes carriers to 70% chance of breast cancer. As a result many of those diagnosed are women who then find out they need to also have their stomachs removed.

Re:Goodbye, Data Science by eecc on Wednesday November 30, 2022 @03:58AM (#33798437)
Ouch, that’s a raw deal. Very very very sorry about you. If you don’t mind me asking, what are the consequences and mitigations necessary to live with a total stomach removal?
Re:Goodbye, Data Science by xg15 on Tuesday November 29, 2022 @05:43PM (#33794434)
True until you have kids yourself or in other ways become responsible for people you care about.
Re:Goodbye, Data Science by PragmaticPulp on Tuesday November 29, 2022 @06:12PM (#33794712)
This is the kind of mindset that felt obvious to me when I was young and resented anyone else trying to influence what I chose to do for myself.

But after growing up and having kids of my own as well as watching others' kids grow up with varying degrees of parental involvement, I have a whole new appreciation for adult caregivers who get involved and help shape healthy behaviors and habits in kids.

> your caregivers want you to mind your behavior, because then they don't have to, even if you would've been perfectly fine playing with mud or swearing in school or watching TV all day.

You've got it backwards. The easy way of caregiving is to just not care. Let kids watch TV all day, swear in inappropriate social situations, and whatever else they feel like doing. You don't have to get involved if you just don't care what they're doing.

But anyone who has worked with kids in an education setting can tell you that this doesn't actually produce good outcomes for the kids. There are occasional exception stories where students with minimal parental involvement lean heavily into becoming successful in life, but the more common outcome is that hands-off or absentee parenting styles lead to poor outcomes for the children, including social and personal issues. It's not just about getting good grades just because. It's about learning how to operate and function within a civilized society, as well as how to balance your own emotions, impulses, desires, and other behaviors they need to learn as they grow up.

Re:Goodbye, Data Science by EdwardDiego on Tuesday November 29, 2022 @07:27PM (#33795317)
It so happens that "good" behaviour that we seek to embed in our children is generally the same as behaviour that is good for society.

Broadly the values I've tried to instil in my children break down as

- Take responsibilities seriously

- Apply your best efforts

- Be considerate

- Cultivate empathy

- Value yourself

- Don't be a dick

- When you screw up, admit it, and make amends

These, when applied, lead to the behaviours that make parenting easy, and I'm hoping will make them good members of society in general.

Re:Goodbye, Data Science by logicchains on Tuesday November 29, 2022 @11:54PM (#33797152)
> It so happens that "good" behaviour that we seek to embed in our children is generally the same as behaviour that is good for society.

The behaviour that most of the school system seeks to embed in children is primarily "obey and do as you're told, don't question", which is far from good.

Re:Goodbye, Data Science by EdwardDiego on Wednesday November 30, 2022 @02:54PM (#33806656)
I disagree, but our school systems probably differ.
Re:Goodbye, Data Science by koide on Wednesday November 30, 2022 @01:11AM (#33797549)
I would say that there is indeed a problem with modern tech companies where things matter even less than they should. It's not a problem of being raised in a context where everything matters, but that many companies, especially tech are totally care free with their vc money, and we can see that changing in the last months of downturn.

Raising children to care is good and takes lots of effort.

Raising children to leave parents alone usually means the children end up not caring or worse.

Re:Goodbye, Data Science by PragmaticPulp on Tuesday November 29, 2022 @12:37PM (#33790612)
Consequences often catch up slowly. It took years for Elizabeth Holmes to be sentenced because it takes time to collect evidence, build an airtight case, and give people their due process.

As I get older, I'm actually noticing more and more consequences catching up with people, albeit slowly. The people I knew who drank heavily through their 20s and 30s are in much worse shape than basically anyone who made an effort to stay healthy. People with poor diets and low physical activity are visibly worse off than others who paid attention to their inputs. I knew several people who got into recreational drugs in their 20s thinking they were safe because they educated themselves before hand, yet who ended up losing jobs, relationships, wealth, and a few who even lost their lives.

I've also noticed more peoples' career reputations catching up with them. It's not uncommon to interview someone only to later discover that they left a very negative reputation at a previous company where I happen to know someone.

I was very jealous of one of my peers who job-hopped his way up the salary ladder, joining companies and then immediately focusing on nothing other than interviewing at his next salary increase. He rotated through several of the big companies here until his reputation for demanding high salaries and then delivering nothing at all finally locked him out of any company with well-networked people who knew about him. He literally had to leave the state and go somewhere new to escape his past network and get new jobs after 10 years of this.

Consequences do catch up to people most times, but it's not immediately obvious. If you expect immediate justice or for people like SBF to go straight to jail the moment the headlines break, you're only seeing the beginning of the story.

Re:Goodbye, Data Science by daydream on Tuesday November 29, 2022 @12:44PM (#33790684)
> Consequences often catch up slowly.

Agreed. Few people defy gravity; in the end most hit the ground.

The phrase “slowly, then suddenly” comes to mind.

Re:Goodbye, Data Science by ghostbrainalpha on Tuesday November 29, 2022 @03:42PM (#33793006)
Should term consequences rarely align with expectations.

But Long term consequences always do.

Re:Goodbye, Data Science by ChainOfFools on Tuesday November 29, 2022 @05:34PM (#33794351)
The universe is eventually consistent
Re:Goodbye, Data Science by satvikpendem on Tuesday November 29, 2022 @04:16PM (#33793420)
I favorited this comment for showing me a new analogy in your second sentence. I'll have to use that one in the future.
Re:Goodbye, Data Science by hnhg on Tuesday November 29, 2022 @04:07PM (#33793324)
I can't match up your anecdata with mine. I can think of numerous people who have done all the things you have mentioned and have no suffered no ill-effects. In fact, many have prospered from lying or cheating the system. From substance abuse to habitual lying, there were no consequences and actually in some cases great wealth was accrued. A great deal of awful people have a very fine life out of it, and there is no greater cosmic justice to address this.

Also, one could argue another interpretation of what you are advising is never take a risk, because it will have consequences. Well, in real life, it doesn't always. You can get away with a lot, and people do.

Re:Goodbye, Data Science by PragmaticPulp on Tuesday November 29, 2022 @06:04PM (#33794643)
> Also, one could argue another interpretation of what you are advising is never take a risk, because it will have consequences.

That's not at all what I was saying. I was referring to predictable negative consequences of unhealthy behaviors.

There are many risks that don't involve gambling away your health, your reputation, your credibility, etc.

Re:Goodbye, Data Science by icelancer on Tuesday November 29, 2022 @06:28PM (#33794862)
I think both you and the GP are correct. I do think that consequences are being detached from actions, at least in the last decade of free money and rapid growth. But I also agree with you on the slow burning nature of small bad decisions made many times over years on end.

With the economy contracting and inflation skyrocketing, consequences should be back in fashion relatively soon. We're already seeing it in mass layoffs and other areas of business.

Re:Goodbye, Data Science by ZephyrBlu on Tuesday November 29, 2022 @08:13PM (#33795659)
The problem is that time value is extremely relevant. If it takes 10-20 years for consequences to catch up, the person is likely to have already built up an unassailable lead that the consequence barely dents.

> He literally had to leave the state and go somewhere new to escape his past network and get new jobs after 10 years of this.

That's not even that bad of a consequence. It sounds like his strategy was worth it tbh.

Personally I hate this kind of behaviour, but from a maximization POV (Especially in regards to career) it seems like the best move. There is likely some risk of ruin, but the upside appears to be much greater.

Re:Goodbye, Data Science by insightcheck on Tuesday November 29, 2022 @11:32PM (#33797056)
> The problem is that time value is extremely relevant. If it takes 10-20 years for consequences to catch up, the person is likely to have already built up an unassailable lead that the consequence barely dents.

I largely agree with you: the big names attached to the resume, the pay, and the effort spent on interviewing skills likely offset the negatives of the reputation (though I also intuitively don't like it because the strategy is rather self-centred).

However, the consequence is rather significant if he has roots. It's harder to pack up and move if one has a romantic partner who is settled into a job at a particular place, and you could also possibly be leaving family and friends. Sometimes one has to move, but typically one has the option to come back, which wouldn't be practical for the person in question. It's still plausibly worth it for the person if he didn't have roots and collected a lot of compensation, but especially when one is older (the commenter mentioned 10 years of workin experience), moves can be tougher.

Re:Goodbye, Data Science by insightcheck on Tuesday November 29, 2022 @11:19PM (#33796989)
To add another piece of evidence, while previous poster noted that the initial response by police officers on January 6 seemed less violent than they could have been (though even then, one person was shot and killed), the US Department of Justice is continuing to publish press releases about charges of people involved in the January 6 Capitol attack (at https://www.justice.gov/news , with full records with dates at https://www.justice.gov/usao-dc/capitol-breach-cases). The charges for many of the people involved caught up eventually, though it took time.

Separately, to put a positive spin on this, it often takes time for positive habits to pay off. When picking up a positive habit (e.g. exercise and especially learning a new technical skill such as a language), oftentimes much of the reward doesn't come until far later. This is important to keep in mind, especially if one has self-doubts or even a lack of encouragement for trying to adopt a new positive habit in one's life.

Re:Goodbye, Data Science by sharadov on Tuesday November 29, 2022 @12:30PM (#33790504)
This is a more recent phenomenon - I would say the last 10 years or so - money became cheap, tech saw an infusion of billions of dollars. I would also say "tech folk" are the biggest beneficiaries of this largesse. My friends who are lawyers and doctors, not so much, they bust their ass for a lot longer and for a lot less. I agree with you though - there is a malaise in society, folks in power don't even get a slap on the wrist, working hard and being sincere does not get you anywhere, deceit and fraud are the currency of our times.
Re:Goodbye, Data Science by aaaronic on Tuesday November 29, 2022 @12:07PM (#33790172)
I think you hit the nail on the head there with the survivorship bias and the raised in a bubble comments. Most people are raised in a bubble because children generally can't cope with how messy and complicated the world is. And systems and companies that last a long time can point to how successful they were because of their good decisions while ignoring their equally bad decisions that really should have undone them had they not been lucky.

The older I get, the more I realize how fragile a lot of human systems really are, but I suspect it has always been this way and it won't change significantly any time in my lifetime.

Your comment itself sound somewhat nihilistic, so I hope you're doing well mentally!

Re:Goodbye, Data Science by sangnoir on Tuesday November 29, 2022 @12:32PM (#33790529)
>The older I get, the more I realize how fragile a lot of human systems really are, but I suspect it has always been this way and it won't change significantly any time in my lifetime

I agree that human systems have always been fragile, but have long been papered-over by things like "decency", "tradition" and "doing the right thing" and in extreme cases, mobs with pitch-forks.

I disagree that it won't change in our lifetime(s) - the extreme polarization and tribal politics will get worse and people will let systems break - or intentionally break systems just so that their team will gain a short-term win. I have no idea what new horror it will take to remind people to be decent to each other again, but looking back at how divisive COVID-19 was, I'm not hopeful.

Re:Goodbye, Data Science by saltcured on Tuesday November 29, 2022 @01:05PM (#33790960)
I took the prior post as in, "the fact that they are fragile won't change", not that the systems themselves won't change. And I would agree with that---I see it as yet another expression of the human condition. We may try to build order over chaos to make society, but we also keep loopholes and wiggle room for our psyches. I think the fragility of human systems emerges from that contradiction.

Students of history and the arts can get an earlier exposure to this worldview. I think we engineering types can get too focused on technology and imagine everything is innovation and progress. You have to work uphill against your default interests to expose yourself to a longer view and consider that fundamentally modern people with modern minds lived for (many) thousands of years doing almost all the same cognitive things as us, just with different physical props.

Our lungs are constantly in flux as we breathe. But at the same time, we're just breathing and that doesn't really change until our end. I'd say human social systems are much like that.

Re:Goodbye, Data Science by aaaronic on Tuesday November 29, 2022 @05:41PM (#33794422)
Yeah, I was really going for something like "the more things change, the more they stay the same."
Re:Goodbye, Data Science by scottLobster on Tuesday November 29, 2022 @01:12PM (#33791044)
Thanks for the concern, but I'm all right, I have the privilege of living near the top of Maslow's hierarchy and actually pondering these questions. :) If I'm a nihilist I'm at the "creating your own value system" part. The world is generally a giant blob of apathetic flavorless jello, I can at least inject some sugar and food coloring wherever I'm at. There's also some freedom in that, when people don't care they also tend to give way pretty easily. It's just disappointing, except for when you encounter that rare person that also gives a shit. Part of the reason I spend a lot more time on HN than reddit. :)
Re:Goodbye, Data Science by aaaronic on Tuesday November 29, 2022 @05:40PM (#33794407)
Good to hear you're doing relatively well, and I agree that injecting positivity where you can is probably one of the sanest ways to live.
Re:Goodbye, Data Science by kjkjadksj on Tuesday November 29, 2022 @11:58AM (#33790069)
Part of it too is the stressful expectations school puts on you that as you’ve found, don’t actually exist in the real world.
Re:Goodbye, Data Science by thewarrior on Tuesday November 29, 2022 @12:01PM (#33790098)
Enforcing consequences is difficult as laws and bureaucracies become ever more complex.

This gives plenty of space for opportunists and tricksters to hide.

You don’t ever have to fear being beheaded by the people whose life savings you stole and you don’t have to face consequences if you have a good lawyer.

To do well in todays world learn all the rules and where the loop holes lie. Violating the spirit of the law is fine as long as you can lawyer around the letter of it.

Re:Goodbye, Data Science by AppleBananaPie on Tuesday November 29, 2022 @12:18PM (#33790325)
I think this hits the nail on the head. People are learning the meritocracy they were taught growing up isn't real so why would they work 80 hour weeks for a 25% bonus instead of a 10-15% bonus. The calculus gets even worse when the bonus people get is insignificant to that of switching careers often.

For what you specifically experienced, my opinion, the bigger the organization the more inevitable this seems to become. To make things worse the size of the organization isn't limited to just a company or non-profit but to the size of all groups involved, i.e. a small charity or non-profit that's part of a huge government program is similar to a small engineering team in a huge tech company. They could do huge things or be completely worthless and so long as they pass along positive messages up the chain and the org or company as a whole is doing well then yay no consequences.

We're (hopefully) at the beginning of a cycle where companies realize they are causing apathy amongst the majority of the employed and hopefully experiment (and succeed) in providing meaningful pay raises to the lower echelons which will come at the short term costs of profits but are justified for long term productivity. Or we'll just keep divolving into a dystopia

Re:Goodbye, Data Science by lrc on Tuesday November 29, 2022 @12:59PM (#33790874)
My own thought (I know there is a great deal of room for disagreement) is that the J6 crowd saw no consequences for the attempts to obstruct the Brett Kavanaugh confirmation and thought the rules had changed. One of them was shot in the neck, many others are still incarcerated two years later despite a clear constitutional right to a speedy trial. I'd rather the Capitol Police had just cracked heads at this point. You might think one protest was more justified than another, but the differential in response works to dissolve confidence in the fair application of the law. At any rate, the participants in J6 have been broken, so you're not likely to seen that again...yet I feel we could get another riot season provoked by police brutality at any time.

There is garbage and tent encampments thoughout much of my city, and I am told that nothing can be done about it. I've been invited to engrave something on my catalytic converter. I wonder what good that would do.

Re:Goodbye, Data Science by eigen on Tuesday November 29, 2022 @10:58PM (#33796861)
> You might think one protest was more justified than another, but the differential in response works to dissolve confidence in the fair application of the law.

in 2018, the capitol was open to the public, no one broke in. 78 were arrested in the Capitol on Oct 5, 2018 and charged with Crowding, Obstructing, or Incommoding [1].

in 2022, the Capitol was closed to the public and people broke in. 12 people were arrested on Jan 6, 2021 and charged with Unlawful Entry or Assaulting a Police Officer [2].

Assaulting a Police Office is a felony; Crowding, Obstructing, or Incommoding is a misdemeanor. Seems there was a differential in severity of breaking the law as well.

[1] https://www.uscp.gov/media-center/press-releases/us-capitol-...

[2] https://www.uscp.gov/media-center/press-releases/us-capitol-...

Re:Goodbye, Data Science by gammarator on Wednesday November 30, 2022 @12:15AM (#33797244)
I must have missed the part where the Kavanaugh protesters showed up in body armor with zip ties and plans to hold members of the Senate Judiciary Committee hostage.
Re:Goodbye, Data Science by bart_spoon on Wednesday November 30, 2022 @09:36AM (#33801305)
The J6 crowd broke through windows and members of congress were literally barricading themselves into rooms for protection. Some brought zip ties for the purposes of apprehending individuals. They beat a capital police officer to death with a fire extinguisher.

I’m not sure how once can fail to see a difference between that group and the one protesting Brett Kavenaugh’s confirmation.

Re:Goodbye, Data Science by chrisweekly on Tuesday November 29, 2022 @01:11PM (#33791032)
I feel your pain. It often seems to me like Quality is on the decline, on many different fronts. Hard to say if it's just my perception. It does make me more fully appreciate it when I do encounter true craftsmanship or excellence -- which though it might be increasingly rare, is still relatively easily found.
Re:Goodbye, Data Science by panax on Tuesday November 29, 2022 @01:42PM (#33791440)
It might seem that way but it is hard to say with certainty but there is also probably sampling bias or declinism in that you are more likely to hear about negative events while normal or positive events are filtered out. And like PragmaticPulp said it can take time for things to catch up but they often do and people often eventually get what was coming to them.
Re:Goodbye, Data Science by antipaul on Tuesday November 29, 2022 @02:05PM (#33791712)
Enlightening sociological reflection!

It seems that individuals often bemoan such a lack of consequences, but for some reason they are still quite prevalent in our “systems.”

I wonder how to harness the good intentions of individuals…

Re:Goodbye, Data Science by aschearer on Tuesday November 29, 2022 @02:33PM (#33792105)
While what you're saying appeals to my biases I think it's a somewhat ahistorical. Not long ago we had Nixon. We had JFK's, MLK's, and RFK's assinations. Plus Reagan's attempted assination. We had the Vietnam war. And so forth. If I were an adult during that era, I imagine it would have felt like consequences were slow to come.
Re:Goodbye, Data Science by civilized on Tuesday November 29, 2022 @02:53PM (#33792365)
> as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect.

I wouldn't say fewer uniformly, but certainly very noisy. Some have their lives destroyed for minor or non-existent misdeeds, others get away with egregious crimes.

Re:Goodbye, Data Science by ericmcer on Tuesday November 29, 2022 @04:21PM (#33793475)
Your post mentions how you are surprised you can skate by at work without facing huge consequences for your actions. Most everyone is like you. We are self-centered and worried about our own security, over-analyzing our own problems and barely being aware of others. I don't know if this is a new problem or one as old as humanity.

The Jan 6 riots are possible because again, the Capitol Police weren't ready to lay their careers and lives on the line "cracking skulls" to defend an old building. Most of them probably were taking in the spectacle and thinking about how exciting it will be to recount with their friends/family later.

Re:Goodbye, Data Science by coupdejarnac on Tuesday November 29, 2022 @04:26PM (#33793547)
The owner at a boutique engineering firm I worked for told me that in a large corporation the best thing you can do is massively fuck up at the beginning. Everybody would learn about you and then eventually forget what you did wrong. The extra bit of notoriety would help with name recognition and people would think you're a "good guy".
Re:Goodbye, Data Science by fatneckbeardz on Tuesday November 29, 2022 @05:04PM (#33794028)
no, warehouse workers, nurses, other people with extreme work hours requirements and tons of metrics, are constantly being fired for failing to meet quota or whatever.

its the "the higher the pay the easier the job" paradox.

Re:Goodbye, Data Science by snapplebobapple on Tuesday November 29, 2022 @05:55PM (#33794545)
If you are only 35 you aren't old enough to remember when interest rates were correctly pricing risk. You should see a lot more consequences show up (probably painfully for all involved) as the low interest BS makes way to people actually having to have a high likelihood of generating high positive returns to get funding and those companies that are otherwise profitable retrenching to pay off the overspend from the zero interest years.

I think you are also seeing the effect of the oligopolization of the world stemming from the bad rework of the antitrust laws relaxing antitrust enforcement significantly from the 1970's through now.Any sort of market power is really bad for this kind of behavior because almost noone wants to rock the boat if they don't have to and when you have an oligopoly/monopoly you can abuse you often can hide this stuff in slightly lower but still excessive profits.

Re:Goodbye, Data Science by johnthescott on Tuesday November 29, 2022 @07:19PM (#33795250)
stay in your lane, go the speed limit and ALWAYS look for an escape route.
Re:Goodbye, Data Science by layman51 on Tuesday November 29, 2022 @09:15PM (#33796108)
After reading your comment, I think you have captured some of my own thoughts about consequences and deserts (i.e., worthiness or entitlement to reward or punishment). I agree with the other comment that replied to you that says that thinking like this is a product of being raised in a bubble.

I am not sure if apathy/nihilism is growing in the larger society. I think that things have always been like this because people have always struggled to find meaning in life. After taking an intro psychology class, I was exposed to the idea that society wants an individual to police him/herself. The "super-ego" that makes one feel guilty for breaking rules and want to aim for perfection.

Re:Goodbye, Data Science by heurist on Wednesday November 30, 2022 @02:09PM (#33805939)
My parents raised me with the message that there are no second chances. I've only found that to be true in a few particular situations in my life.
Re:Goodbye, Data Science by ozim on Wednesday November 30, 2022 @03:03PM (#33806798)
I would not agree with general apathy. I get it more as reality. Strict standards are BS. If you get up sober in the morning and go to work it is like 80% of what is expected from an adult.
Re:Goodbye, Data Science by giaour on Tuesday November 29, 2022 @12:21PM (#33790362)
> One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use case turns out to be wrong in the long run.

This is purely anecdata, but I have found that this is more pronounced in a data science context. Managers and executives are (in my experience) more willing to admit they don't understand engineering work product and seek input from technical advisors, and executives and managers deal with decision making on a daily basis and understand that it can be nuanced. But since almost everyone reads financial reports or has to make a chart in Excel every now and then, they know enough to read someone else's analysis but not enough to recognize their knowledge gaps (particularly wrt advanced statistics).

Re:Goodbye, Data Science by apohn on Tuesday November 29, 2022 @01:15PM (#33791080)
IMO the reason behind this is that a lot of "data science" driven decisions are short term decisions. So you can look at something on a PowerPoint, not really care if it's wrong unless you personally will get fired if it turns out to be wrong, and back out of it a quarter later when it turns out to be wrong. IME there's no shortage of justifications or pivoting when it comes to a decision you made a quarter ago. The consequences are relatively small, so the caring is only bravado, not really caring.

When it comes to disastrous long term decisions, there's plenty of time to get input from multiple stakeholders. I always remember the armies of companies who went chasing after Hadoop because Big Data was going to transform something or the other. All the stakeholders were on board, from the CEO and CTO to IT and Engineering management. How much money and time got flushed down the toilet trying to implement and extract value from data with Hadoop. They only people who paid the consequences were the employees at Hadoop companies who thought their stock options would be worth something.

Re:Goodbye, Data Science by icedchai on Tuesday November 29, 2022 @03:27PM (#33792809)
About 10 years ago, I worked at a company that really wanted to use Hadoop for some reason, so I was forced to use it for a project. The amount of data we were processing was minuscule (a few hundred megabytes per run) It could've been done with a simple script on a single EC2 instance for the entire duration of the project without any scalability issues. Instead, I had to provision Hadoop clusters (dev, staging, production), fit the script into the map-reduce paradigm, write another script to kick off the job and process the results, etc. At least we were using Hadoop.
Re:Goodbye, Data Science by remram on Tuesday November 29, 2022 @02:29PM (#33792053)
This reminds me of the seminal article by The Correspondent about online advertising: https://thecorrespondent.com/100/the-new-dot-com-bubble-is-h...

Relying on your data science or marketing department to tell you how good your data science or marketing department is doing, with their own metrics and their own evaluation methods that you don't understand, can only really lead to one outcome.

Re:Goodbye, Data Science by 2devnull on Tuesday November 29, 2022 @10:55AM (#33789125)
This gave me a chuckle. If you read the feature article you understand that this is also because management wants “decision driven data.” They have an idea and use ds to provide charts and tables to support their idea. The harder the idea is to support, the greater value data science is able to provide.

I guess data science is inferior to research in this way. People care about research methods, rigor, etc… Maybe data scientists should adopt stricter standards, like actual scientists.

Re:Goodbye, Data Science by samvher on Tuesday November 29, 2022 @11:15AM (#33789427)
I did read the article - some of the problems with judgements of work quality also come up with (hypothetical) well-intentioned truth-seeking non-political long-term-optimizing managers who just don't happen to be stats experts.
Re:Goodbye, Data Science by 2devnull on Tuesday November 29, 2022 @11:43AM (#33789815)
Sorry, wasn’t trying to imply you didn’t and I fully agree. Even managers that know stats can be busy or but into hype about ml or other shiny new things that they don’t have time or resources to deconstruct. This is another big problem with data science, “black box” systems and cargo cults. It’s easy to think “LLMs will change the world! We should use them, the competition will.”
Re:Goodbye, Data Science by dqpb on Tuesday November 29, 2022 @11:12AM (#33789371)
> Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results.

This is to be expected from an information theory point of view. It's why "fake news" will always be a thing.

Re:Goodbye, Data Science by jmount on Tuesday November 29, 2022 @11:59AM (#33790081)
Super point. I can't resist repeating it back. When incorrect work outperforms correct work in superficial evaluation, it is then selected for.
Re:Goodbye, Data Science by mmcnl on Tuesday November 29, 2022 @12:17PM (#33790315)
This is somewhat captured in the article as well.

"Managers will say they want to make data-driven decisions, but they really want decision-driven data. If you strayed from this role– e.g. by warning people not to pursue stupid ideas– your reward was their disdain, then they’d do it anyway, then it wouldn’t work (what a shocker). The only way to win is to become a stooge."

In science, a good scientific result can be bad for business. There is often little appreciation for the "science" in data science.

Re:Goodbye, Data Science by throwingit0 on Wednesday November 30, 2022 @12:41AM (#33797390)
>There is often little appreciation for the "science" in data science.

It feels like even Google falls prey to this at times: they keep redoing the same A/B test until it comes up in favor of the change (or the designer whose pet project it is runs out of political capital, presumably).

Re:Goodbye, Data Science by whiplash451 on Tuesday November 29, 2022 @12:20PM (#33790354)
You should pay the price for data leakage very quickly in production.

Does management look at slides or AB test dashboards?

Re:Goodbye, Data Science by disgruntledphd2 on Tuesday November 29, 2022 @12:51PM (#33790767)
People don't pay attention to production metrics, and a noisy problem (like marketing or whatnot) can often be pretty bad for a looooonnnnnggg time before anyone notices.
Re:Goodbye, Data Science by jrochkind1 on Tuesday November 29, 2022 @12:22PM (#33790371)
> Approaches that are wrong from a statistics point of view

When OP talked about "the main bottleneck to my work" in terms of areas he would need to learn more about -- I was expecting him to talk about facility with statistical methods and using them appropriately!

I'm not sure what to take from the fact that he never did! I would like to ask him what he thinks about that!

Re:Goodbye, Data Science by mumblemumble on Tuesday November 29, 2022 @12:22PM (#33790384)
The problem is that nobody actually wants data science. They want data pseudoscience.

And for the same reason that people tend to want pseudoscience instead of science in any other domain, too. Science is slow, tentative, and messy, and usually responds to questions with even more questions rather than with answers.

Pseudoscience tends to be much more concerned with exuding confidence and providing clean-cut answers. It's what happens when a desire for science meets a need for instant gratification. Along the way, things like blinding and controls and watching for bias and validating assumptions tend to get dropped when they're inconvenient or difficult to explain. And they're always inconvenient and difficult to explain.

Re:Goodbye, Data Science by onlyrealcuzzo on Tuesday November 29, 2022 @12:43PM (#33790675)
> The problem is that nobody actually wants data science. They want data pseudoscience.

Technically, I think investors & owners would want the company to use real data science to improve products & maximize profits.

Everybody in the middle just wants to use data to lie to get promoted faster - because you don't get promoted for actually doing a good job - you get promoted for convincing people you did a good job, and lying is a VERY useful / effective tool.

Re:Goodbye, Data Science by deckard1 on Tuesday November 29, 2022 @02:42PM (#33792237)
Welcome to the world of OKRs/KPIs/Scrum, where everything's made up and the points don't matter.
Re:Goodbye, Data Science by bart_spoon on Wednesday November 30, 2022 @09:58AM (#33801608)
> I think investors & owners would want the company to use real data science to improve products & maximize profits.

This is based on the assumption that companies are focused on long term profits and stability, and I’m not sure why anyone believes that to be the case anymore. The vast majority of companies are run based on next quarter’s stock price or growth metrics.

I worked on a newly formed data science team coming out of grad school that was tasked with taking some predictive initiatives that the company had relied on external consultants to produce, and implementing them in-house. The external team’s results always looked exactly like what the business wanted to hear, but they rarely played out in practice. This was in part because the underlying data quality was terrible, and the company wasn’t executing in a way that allowed anyone to actually answer the questions being asked. The consultants would just torture the data until they could come up with a report that would ensure the company would come back the following year. So we spent a lot of time trying pouring cold water into the business groups who saw data science as a magic wand that would conjure up more money at no cost. But we never were able to convince them to invest in anything that would take longer than a year. Anything that would require a change in their marketing or strategy executions that wouldn’t immediately deliver increased results was just a non-starter. But actual data science requires that kind of investment for long-term layoffs. So the data science team became figure-heads, never given the buy-in to actually make impact on business, but kept around so teams and leaders could tout being “data-driven” and throw “AI” and “machine-learning” into PR and marketing materials.

You aren’t wrong about middle management is looking to get promoted faster. But every single individual from the employee looking for a promotion to the executive suite to the investors are addicted to incentive windows no longer than 6-12 months.

Re:Goodbye, Data Science by urthor on Tuesday November 29, 2022 @05:43PM (#33794436)
In my experience, a little more nuanced.

LARGE INTERNET DATA COMPANIES. They want the real data science.

For them, data science actually allows them to perform a core business function (target their customers) in a profitable way (one way, asynchronous relationship. Note the complete lack of any "talking to a human being" in your relationship with big tech).

For everyone who isn't a large internet data company with an asynchronous relationship with their customers... what's the point?

Usually, they have only a handful of technical projects that benefit from data science.

In my experience, my multi-billion dollar organization got by with a shockingly small number of "real" data scientists.

Re:Goodbye, Data Science by rgrieselhuber on Tuesday November 29, 2022 @12:23PM (#33790392)
I’ve always disliked how data science was positioned within companies as well, it’s outside the critical path of product and engineering, which means it becomes a mere abstraction to management (e.g. “throw that problem to the data science team and see what they come up with”), resulting in very vague and abstract requirements and, hence, deliverables. I think there is huge value in the discipline and technologies, but it unfairly gets relegated when not integrated to the whole product / engineering process. Hence, the title / concept of Data Engineer seems like a much better fit for this role within many companies.
Re:Goodbye, Data Science by rgavuliak on Tuesday November 29, 2022 @04:27PM (#33793558)
Yeah, as a Data Science manager I've experienced this pain a lot (not part of the critical path). I am now an Engineering Manager that works with a cross-functional team including FE/BE/DS/Devops and it's the most power I ever had to put Data Science in front of our clients in a meaningful way.
Re:Goodbye, Data Science by hcks on Tuesday November 29, 2022 @07:44PM (#33795430)
> Because these issues are subtle, management will often not pick up on them or not be aware that this kind of thing can go wrong.

Management is not your teacher at school, it is not there to check up your results make sense.

Management mostly assumes you’re competent at your job.