Re:Don't Overreact to Weak Signals
by MrGilbert on Tuesday October 25, 2022 @07:49AM
(#33329255)
My initial reaction when reading the title was like: "Yeah, that's good advice when going on a date!"
After reading the first few lines, I realize that the topic is completely different, yet somehow the same.
Re:Don't Overreact to Weak Signals
by dalbasal on Tuesday October 25, 2022 @08:57AM
(#33329855)
Evaluating the strength of evidence is one thing. Another is correctly what the evidence supports, regardless of strength.
Another common family of mistakes are those that overgeneralize or overapply. Taking evidence to.mean something deeper or broader than it has any business commenting on.
Re:Don't Overreact to Weak Signals
by rrwo on Tuesday October 25, 2022 @08:46AM
(#33329744)
> Considering one instance of user feedback to represent a pattern
Sometimes a user makes good suggestion.
> An instance where a feature is abused by disabling it entirely
If the abuse may be dangerous, then the safest thing to do is to disable it until the feature can be fixed.
If there is no business case for keeping the feature (e.g. it isn't important, doesn't contribute to income or productivity, or it's too expensive to fix), then it's not worth re-enabling it in the future.
> Attempting to copy what your immediate competitors are doing
Sometimes they have good ideas.
> Doing things that aren’t directly related to the objective
Re:Don't Overreact to Weak Signals
by dereg on Tuesday October 25, 2022 @09:09AM
(#33329970)
This is generally good advice. Poor management tends to overreact to these sorts of signals which helps them create an illusion of busy-ness but creates havoc within their team.
I’ve seen teams spend days consumed by their overreaction, make unnecessary changes to product, and then shrug a week or month later when everyone collectively agrees when that change should be undone. Bizarre.
Re:Don't Overreact to Weak Signals
by helen___keller on Tuesday October 25, 2022 @09:26AM
(#33330171)
I’ve seen this especially with teams looking for product market fit
You have a handful of customers using a product. One point of data comes in. The team, hungry to build a product customers want, jumps on that one point of data and builds a whole feature around it. Nobody uses the feature.
Re:Don't Overreact to Weak Signals
by nerdponx on Tuesday October 25, 2022 @09:52AM
(#33330465)
Or takes forever building the feature and/or takes forever getting it right, and the one customer who wanted it gives up and leaves in the meantime. Or even worse they hang on with some tiny engagement that turns out to not actually bring in a lot of money, but still sucks up 15% of developer time (not to mention customer support reps & business development people constantly having to keep them happy) and adds extra value for no other customers.
Re:Don't Overreact to Weak Signals
by treeman79 on Tuesday October 25, 2022 @10:16AM
(#33330735)
Having team use their own product for same reasons as customers is a great investment to reduce these issues. Ideally from launching as a new customer to daily tasks.
I try and do this at everyplace I work. Usability issues become completely obvious very fast.
Idea struggles if customers are wanting to adapt your product to them. Best if Customer is able to adapt business to product.
Re:Don't Overreact to Weak Signals
by rognjen on Tuesday October 25, 2022 @12:35PM
(#33332892)
In general, this is excellent advice. But we were running marketplace for food ingredients. And while my co-founder had lots of experience in the industry, there was no realistic way to dogfood it.
Re:Don't Overreact to Weak Signals
by nerdponx on Tuesday October 25, 2022 @09:50AM
(#33330437)
Classic HN off-topic post, sorry.
Was a particular CMS or static site generator used for this page? It reminds me a lot of software documentation pages, and I never considered before that this kind of site layout and design would be good for a blog.
Re:Don't Overreact to Weak Signals
by jerf on Tuesday October 25, 2022 @09:53AM
(#33330471)
This is good advice.
Unfortunately, it's really hard, because an opposite of the statement is also good advice: Don't underreact to weak signals. I honestly couldn't tell you which I see more of. Overreactions are intrinsically easier to notice, since they're big and showy, but I kinda suspect there may be more "underreaction" in the world.
Unfortunately, having the precisely correct level of reaction is really hard and I don't have any solid snappy advice for it, except that's one of the things that is probably irreducibly related to having experience and wisdom and not just raw intelligence.
Re:Don't Overreact to Weak Signals
by Mathnerd314 on Tuesday October 25, 2022 @10:05AM
(#33330608)
Well usually there are lots of signals. Overreacting means you might do something that's misguided and expensive. Underreacting means you miss a signal or two and delay a big risk until the information is certain. The pay-off is asymmetric.
If you have a process for spending money in place and people with too much free time on their hands then overreacting might be OK. Too much research never hurt anyone so "overreacting" by investigating thoroughly is probably not wasting your time. But if your reaction process is throwing money at ideas you'll probably just get scammed.
Re:Don't Overreact to Weak Signals
by jdaw0 on Tuesday October 25, 2022 @10:45AM
(#33331147)
I certainly don't think this is bad advice, but without some discussion of how to assess the strength of a signal or the appropriate reaction size, I'm not sure it's useful advice. Knowing not to overreact to weak signals seems like the easy part. The hard part is knowing the difference.
Re:Don't Overreact to Weak Signals
by jdaw0 on Tuesday October 25, 2022 @12:52PM
(#33333135)
Thanks. I spend a lot of time working around pretty thin data, so I'm always interested to hear how people think about extracting insights when you don't have the luxury of doing proper, robust experimental design or sophisticated statistical analysis. I'll keep an eye on the page for updates.
Re:Don't Overreact to Weak Signals
by tchock23 on Tuesday October 25, 2022 @01:08PM
(#33333343)
Perhaps more dangerous is over-reacting to data that was recently gathered (recency/availability bias).
I’ve seen founders come out of discovery interviews wanting to pivot right away. In this case it wasn’t necessarily the strength of the signal; rather, it was how recently that data point was gathered.
Re:Don't Overreact to Weak Signals
by jkaptur on Tuesday October 25, 2022 @01:41PM
(#33333779)
It's interesting to consider this together with Jeff Bezos's famous quote: "The thing I have noticed is that when the anecdotes and the data disagree, the anecdotes are usually right. There is something wrong with the way that you are measuring it." And my understanding is that he frequently forwards customer feedback to product teams - is this not "Considering one instance of user feedback to represent a pattern"?
Perhaps the synthesis of these ideas is not to overreact to weak signals, but to be sure you're aware of them.
Re:Don't Overreact to Weak Signals
by Mathnerd314 on Tuesday October 25, 2022 @05:25PM
(#33336376)
I think that's just a story about how Amazon is always trying to quantify performance and getting it wrong. Seriously, Google quantifies a lot of things too but they don't screw up. If Bezos still hand picks examples and that's the only way real change can happen, that's probably why Amazon has a reputation as a meat grinder.
Another common family of mistakes are those that overgeneralize or overapply. Taking evidence to.mean something deeper or broader than it has any business commenting on.
Sometimes a user makes good suggestion.
> An instance where a feature is abused by disabling it entirely
If the abuse may be dangerous, then the safest thing to do is to disable it until the feature can be fixed.
If there is no business case for keeping the feature (e.g. it isn't important, doesn't contribute to income or productivity, or it's too expensive to fix), then it's not worth re-enabling it in the future.
> Attempting to copy what your immediate competitors are doing
Sometimes they have good ideas.
> Doing things that aren’t directly related to the objective
What does that have to do with weak signals?
I’ve seen teams spend days consumed by their overreaction, make unnecessary changes to product, and then shrug a week or month later when everyone collectively agrees when that change should be undone. Bizarre.
You have a handful of customers using a product. One point of data comes in. The team, hungry to build a product customers want, jumps on that one point of data and builds a whole feature around it. Nobody uses the feature.
I try and do this at everyplace I work. Usability issues become completely obvious very fast.
Idea struggles if customers are wanting to adapt your product to them. Best if Customer is able to adapt business to product.
Was a particular CMS or static site generator used for this page? It reminds me a lot of software documentation pages, and I never considered before that this kind of site layout and design would be good for a blog.
[1] https://jekyll-themes.com/just-the-docs/
If you mean the mailto link, that shouldn't be a bug, but should pre-populate some content. Lemme know if it doesn't.
Unfortunately, it's really hard, because an opposite of the statement is also good advice: Don't underreact to weak signals. I honestly couldn't tell you which I see more of. Overreactions are intrinsically easier to notice, since they're big and showy, but I kinda suspect there may be more "underreaction" in the world.
Unfortunately, having the precisely correct level of reaction is really hard and I don't have any solid snappy advice for it, except that's one of the things that is probably irreducibly related to having experience and wisdom and not just raw intelligence.
If you have a process for spending money in place and people with too much free time on their hands then overreacting might be OK. Too much research never hurt anyone so "overreacting" by investigating thoroughly is probably not wasting your time. But if your reaction process is throwing money at ideas you'll probably just get scammed.
Overreacting to signals in a bull market was probably the right move with the abundance of opportunities.
The opposite is of course true now.
[catchy title]
[completely anodyne observation]
[completely anodyne observation]
[common knowledge]
Here at throbbing rooster dot io we’ve really learned this lesson thanks for reading.
The way this advice is presented it reads rather abstract.
PS: I have read the examples. They don’t make it better. There is also no real story behind them
I’ve seen founders come out of discovery interviews wanting to pivot right away. In this case it wasn’t necessarily the strength of the signal; rather, it was how recently that data point was gathered.
Perhaps the synthesis of these ideas is not to overreact to weak signals, but to be sure you're aware of them.