marbletimes

They are not slightly different, they measure something (totally) different.

Confidence intervals are relative to E[y|x], Predictive intervals are relative to y. Sometimes, for example there is not much variation in y|x, the two intervals may be similar, but that is due to the nature of the data, not because they are one just "a bit larger than the other" (or, otherwise, think about (1) the uncertainty around the mean of an empirical symmetric distribution with a very small standard deviation--we are 95% confident the true mean is between z and k--and (2) the 2.5%-97.5% intervals of the raw data distribution. Numbers can look similar, but they are representing different measures).

I paste an example here below that I had made in a following comment:

--- In the vast majority of the cases, what we want it the range for y (prediction interval), that is, given x = 3, what is the expected distribution of y?. For example, say we train a model to estimate how the 100-m dash time varies with age. The uncertainty we want is, "at age 48, 90% of Master Athletes run the 100-m dash between 10.2 and 12.4 seconds" (here there would be another difference to point out between Frequentist and Bayesian intervals, but let's make things simple).

We are generally not interested in, given x = 3, what is the uncertainty of the expected value of y (that is, the confidence interval)? In this case, the uncertainty we get (we might want it, but often we do not), is, "at age 48, we are 90% confident that the expected time to complete the 100-m dash for Master Athletes is between 11.2 and 11.6 seconds".

----

The two intervals can be similar according to some metrics ("ah, come on, 11s or 12s who cares"), but they are measuring/estimating something very different and in many cases, they would matter a lot.

Why do I say they "would" and not they "do"? Because many, and the vast majority I'd say, of decisions in industry settings (outside some niches) that are taken even when ML or statistical models are included in the process, are using point estimate (so, not even uncertainty intervals) only as one of the many input in the decision-making process.

Let me give you an example. I was years ago developing models for estimating ROI relative to certain (very popular) products. The calculations made previously were absurdly wrong, there were log-transformations involved and guess what, they were using confidence intervals ("the uncertainty around the expected ROI for a similar class of products is") instead of predictive intervals ("the ROI for this class of products is expected to be between w and j").

I provided the correct intervals (i.e., predictive), but in the end the decisions changed little, because those making decisions they were not even considering uncertainty in any way in the decision-making process. That's why, in general, I don't worry too much about uncertainty on the rare occasions these days when I develop models.

I mean, who outside of academia (and even there...) measures the accuracy of a predictive model taking also into account the predictive intervals, for example adding to a metric like mean absolute error over test data also the proportion of test data that falls within the uncertainty intervals that were estimated for the model given the training data? The answer is "very few".

I agree that statisticians would better than CS people appreciate the importance of uncertainty intervals--it is mostly cultural--but that "In reality there are very few ML applications that don't need confidence estimation and estimation of monetary costs" is empirically false.

If ML application require uncertainty attached to point estimate, we would see plenty more uncertainty intervals attached to point estimates, but in industry, outside of niches (e.g., banking, bio, actuary to name a few), very few bother dealing with them.

I am currently part of a large team (we are talking hundreds) of ML specialists, and I have yet to see a single presentation in which a point estimate was associate with some uncertainty interval. And in my previous company it was the same and when I interview candidates (dozens? hundreds?) I never get a satisfactory answer to the confidence interval vs predictive interval question I ask about.

From a good SO answer, see https://stats.stackexchange.com/questions/16493/difference-b...

"A confidence interval gives a range for E[y∣x], as you say. A prediction interval gives a range for y itself.".

In the vast majority of the cases, what we want it the range for y (prediction interval), that is, given x = 3, what is the expected distribution of y?. For example, say we train a model to estimate how the 100-m dash time varies with age. The uncertainty we want is, "at age 48, 90% of Master Athletes run the 100-m dash between 10.2 and 12.4 seconds" (here there would be another difference to point out between Frequentist and Bayesian intervals, but let's make things simple).

We are generally not interested in, given x = 3, what is the uncertainty of the expected value of y (that is, the confidence interval)? In this case, the uncertainty we get (we might want it, but often we do not), is, "at age 48, we are 90% confident that the expected time to complete the 100-m dash for Master Athletes is between 11.2 and 11.6 seconds".

Correct, quantile regression is an option. Another is "pure" bootstrapping (you can see by googling something like uncertainty + machine learning + bootstrapping that this is a very active area of current research).

The major problem with bootstrapping is the computational time for big models, since many models need to be fit to obtain a representative distribution of predictions.

Now, if you want more "rigorous" quantification of uncertainty, one option is to go Bayesian using probabilistic programming (PyMC, Stan, TMB), but computational time for large models can be prohibitive. Another option is to "scale down" the complexity to models that might be (on average) a bit less accurate, but provide rigorous uncertainty intervals and good interpretability of results, for example Generalized Additive Models.

A note here is that I saw certain quantification of uncertainty by people who were considered very capable in the ML community that gave me goosebumps, for example since the lower bound of the interval was a negative number and the response variable modeled could not be negative, the uncertainty interval was "cut" at zero (one easy way to deal with it, although it depends on the variable modeled and the model itself, is log-transforming the response—but pay attention to intervals when exp(log(y)) to get back to the natural scale. Another useful interview question.)

When I was in academia, I used to fit highly sophisticated models (think many-parameters, multi-level non-linear mixed effect models) who were giving not only point estimate but also confidence and predictive intervals ("please explain to me the difference between the two" is one of my favorite interview questions and I still have not heard a correct answer).

When I tried to bring an "uncertainty mindset" over when I moved to industry, I found that (1) most DS/ML scientists use ML models that typically don't provide an easy way to estimate uncertainty intervals, (2) in the industry I was in (media) people who make decisions and use model prediction as one of the input for their decision-making are typically not very quantitative and an uncertainty interval, rather than give strength to their process, would confuse them more than anything else: they want a "more or less" estimate, more than a "more or less plus something more and something less" estimate. (3) When services are customer-facing (see ride-sharing) providing an uncertainty interval (your car will arrive between 9 and 15 minutes) would anchor the customer to the lower estimate (they do for the price of rides book in advance, and they need to do it, but they are often way off).

So for many ML applications, an uncertainty interval that nobody internally or externally would base their decision upon is just a nuisance.

But if you know that the current company cannot pay like the other company you are interviewing for, or that in this new company the future salary outlook is much better, why bother asking if they can pay you more?

In the work life, one needs to be strategic; it appears to me that most people on here who are commenting somewhat negatively about people giving 2-week notice and nothing else have never been laid off or fired or somewhat "mistreated" by a company.

The way to be an excellent employee is to always deliver work to the best of one's capabilities and give 2-week notice before leaving. In one company I worked for, somebody gave a 2-week notice and was let go immediately. It is not unheard of, and most legacy companies could not care less about a possible PR problem. Those legacy companies live in a galaxy that is far away from the one inhabited by start-ups and similar good PR-dependent companies, they are populated mostly by old dinosaurs that recruit heavily among the not-too-brilliant technical people.

The assumption here is that the employee leaving the company was due to some problems that could not be solved after careful discussion with the manager, skip-level etc. But leaving a company can also happen because another company offers you 50% more money or a different role, or you just feel like changing.

For any employee, including managers, directors and up—there is only one thing to remember: when the company intends to let you go, they don't ask for your opinion. Two-week notice is professional and more than enough.

Jiu-Jitsu 3 classes a week for 1.5 hours each. Weight training, 3 times a week, 1 hour each time. Running, 2 times a week, 45 minutes each run. That's on average, total 9 hours a week. Sometimes I do more, when on vacation I don't stress too much, if there is gym, fine, otherwise it could be rest week or 200 push up and 400 squats.

I enjoy every second of being physically capable without being obsessive about fitness.

One of the few advices that I give to people I care about: never give up moving your body. I am a bit older than 40 years old, and that's the time when you see people either be fabulous or going downhill with the wheels off. Apart from disabilities and pathologies, there is not a single reason for anyone not to exercise at least a few hours a week. It makes you feel better, be better, look better. I see friends getting married and having kids and work 5 jobs like someone says in the US especially etc. and they stop exercising, and they suddenly look older, talk older, and often going downhill real fast. That's not a good example for kids, it is a good prompt for your partner to look elsewhere.

Be bold, be in shape, be beautiful.

"My theory is that he’s too valuable where he is, and that they don’t want to move him elsewhere."

Assuming the "they are too valuable here to be promoted elsewhere" hypothesis is true, and it may be true in some cases (and not true in the majority of cases), there is always an increase in compensation in the form of increased base salary, bonus, or equity, depending on the company, available. A promotion is both a reward and a belief that the person would have a greater impact in another position. More money is also a reward.

The hypothesis here is that there is a trade-off between making bank and living a pleasant life, whatever attributes one wants to associate with "pleasant", maybe low professional stress, maybe time spent with loved ones, maybe waking up when the world is already spinning at full speed.

There might be a weak positive correlation at the population level between (broadly speaking) "being miserable/not having a good time/give up life for money" and "income", but at the individual level what we can say is that there are millions of people in the world making good to very good to great money while having the time of their lives day after day.

I might be one of those, for now. In the future, who knows.

Agreed. That's why I wrote,"but I consider that money far from ----financially---- making someone "a very successful 40-year-old"."

As someone living in the United States and earning much more than I would in my home country, even though my professional contribution would be the same if the same opportunities existed in my home country (I took my PhD in my home country), there is no comparison between the quality of life I have in the country I currently live in and my home country given the expected local salary - setting aside loved ones, friends, and memories.

If a two-bedroom apartment goes for 200k and after taxes what I have is 25k or so, my financial situation, considering kids, car(s), aging parents and what we call life, would certainly not be one of unquestionable success.

"Sailing/horse riding/cars/light aircraft" would certainly be a stretch.

It depends on where.

In a European country, think Spain, Italy, France, "get good grades, study a valuable major, don't eat garbage, don't marry crazy people/idiots, stay off dangerous drugs, and mainly show up to work, be clean and nice" and you are likely to end up with job paying 40 to 80k (before taxes) a year, which is not bad, but I consider that money far from financially making someone "a very successful 40-year-old".

With my STEM major and a PhD, I would have made between 40 and 50k per year before taxes.

What makes the real difference is curating followers and avoiding mindless "reel type" scrolling. As always, it's being intentional rather than setting hard limits (e.g., being friends only in real life, limiting the number of follows after some weak sociological studies) that makes the difference in the quality of the experience.

I've learned a lot from Tiktok, Reddit, Twitter, certain corners of Facebook, etc., and met a lot of interesting people on dating apps. And I hope people have learned from me. Sure, I've wasted time too, and waited too long before I muted people who made me angry, and it took me some unnecessary time before I realized they weren't talking to me, but my world has gotten a lot bigger and more interesting and more adventurous than before.

Feynman certainly cared a great deal about what others thought of him, and he has been known to embellish many stories to appear as the common genius in the situation. In short, he was a genius, but he also wanted it to be known very well by others.

Let's not forget the lies that are broadcasted left and right on social media, in which thousands become millions, and trite tropes are sold like congress-worthy insights.

You are right, it takes a certain type of people to succeed on Twitter and, jealous as I am of my private life for example and unwilling to interact or share much online, I am definitely not among those.

It is a tricky subject. When I referred to a model I had published elsewhere, I was asked to write about the model again in the current paper, since readers may not be able to access or want to access the other paper. But more recently, I was asked to do the opposite, since the model I was writing about in the Methods section was the same I had published elsewhere.

I see no reason to change terms and structure in scientific publications so as not to plagiarize myself.

When I was in academia--before I moved into technology and began applying statistical modeling and machine learning to a variety of tech problems--my research in mathematical biology was primarily on understanding what happens to populations and life histories of individuals (i.e., the timing of key events in an organism's life and the trade-offs between life history traits, e.g., growth, survival, movement, fecundity) that are affected by catastrophic events, such as floods, storms, and fires that cause mass mortality.

I summarized and then tested using freshwater fish populations as model systems some hypotheses about what happens to populations and individuals after catastrophic events in the paper "Genetic and life-history consequences of extreme climate events" published a few years ago in PRSB (http://simonevincenzi.com/Publications/Vincenzi%20et%20al.%2...) and in some subsequent work that can be found at http://simonevincenzi.com/publications/. I combined genetics, life-history theory, field studies, and simulations, and had a lot of fun doing it. Later, I applied many of the tools, models, and ideas I developed and learned when studying extremes to problems I faced when working in technology (http://simonevincenzi.com/Publications/biology_and_movies_v2...).

A major difficulty in generalizing the consequences of disasters is that the effects of extreme events are largely context-specific. For example, the emergence of adaptations may depend on both the life histories of species and the recurrence, intensity, and nature of extreme events. Next, the demographic and genetic effects of extremes are often the result of chance and thus are not easily predicted or generalized across species or habitats.

Luck also plays a role in determining whether a population will recover after a population crash. For example, I found (combining field studies and novel genetics tools and findings, such as how to assign sex to salmonids and how to infer trios mother-father-offspring in highly inbred populations) that the nearly complete recovery of a fish population that had been reduced to a handful of individuals after a flash flood was due to the large production of offspring from a single pair. Given the high variance in adult reproductive success in most animal species (including humans)--which is at least in part due to differences in individual "quality" (a tremendously important, but I believe understudied, trait)--if the vigorous mating pair had been killed or moved during the flood, population recovery would have become much less likely.

This is one of those witty sentences that sound good (the balance of life, laughs here, tears there, if you want to enjoy living you need to accept dying, everything happens for a reason), but they are just biblical nonsense. There are plenty of very accomplished, successful (internally and externally) people who feel much more joy than sorrow, and plenty of evil people who have an internal life that is no worse than much more saintly people, but according to The Prophet they all should cry more. I remember I went on a date, and they said, "when a relationship is ending, I really want to feel the pain, as it makes the relationship something of value". I thought it was bananas, there is very little to be gained by pain and spiraling introspection after a break-up. But the other side of the coin says, should I keep my mouth shut during cringy conversation, so I can then have more enlightened, or presumably enlightened, conversations with someone else?

It is one of those common tropes, "interviewing is hard", that seems to justify the inevitability of poor attitude, wasting time of interviewees (remember the interviewers are paid for those hours) and a painful process for basically anyone. I often interview for my company/group and yes, it is challenging in one hour to find out "life, death, and miracles" of some guy who is being interviewed, but it is especially challenging when the interviewer is in a bad mood, does not want to be there, or sees interviewing as a nuisance, and he is looking for a reason to make someone fail. I worked at a FAANG and I got offers from FAANGs-adjacent companies, so I am not one of those you-cannot-win-at-this-game-so-you-complain people. I saw: someone stopping the interview for unexplained reasons after the first 45 minutes, an interviewer at a blue company looking at my private parts for a good part of 45 minutes, multiple people interviewing while doing something else in person or virtually, recruiters (those were the fronts, the hiring managers were responsible) dragging the process forever and then say no with zero explanation. Is it really "hard" doing better than that?