News for hackers. Stuff that matters.
Re:I wasted $40k on a fantastic startup idea by Gatsky on Monday January 18, 2021 @09:27PM (#25829290)
This article was posted before several years ago. The whole premise is bumptious - "I can copy data out of a bunch of papers [which I am in no position to screen for quality or relevance], run a canned 'gold standard' analysis in R [the idea that there is one true way to generate valid data is ridiculous], and then go tell the professionals what they are doing wrong." He even brags that his meta-analysis for depression had more papers than the published one, as if this was a valid metric. The Cipriani meta-analysis he cites was publised in February 2018. His meta-analysis was done in July 2018, and had 324 more papers - what explains this difference, other than obviously sloppy methodology. A proper meta-analysis is a lot of work, researchers spend years on one meta-analysis. The whole concept is ill conceived, and the author is too caught up in themselves to even realise why.

Meta-analyses are a good idea, but the mere presence of a meta-analysis does not denote a useful undertaking. The literature is polluted with thousands of meta-analyses. As far as I can see this is mainly because there is software available which lets almost anyone do it, and once someone else has done a meta-analysis it is much easier to do another one because they have already found all the papers for you. The publication rate of meta-analyses far outstrips the publication rate of all papers, and shows some unusual geographic variation (Fig 2) [1].

[1] https://systematicreviewsjournal.biomedcentral.com/articles/...

Re:I wasted $40k on a fantastic startup idea by sillysaurusx on Tuesday January 19, 2021 @01:20AM (#25830586)
With all the negative pushback this is getting, it’s making me think he was onto something. The exact same criticisms would apply to Airbnb, for example. “They have not the slightest idea how the hotel industry works. This is a very professional industry with a lot of legal hurdles...”
Re:I wasted $40k on a fantastic startup idea by hn_throwaway_99 on Tuesday January 19, 2021 @09:14AM (#25833546)
Just because people think an idea is bad is doesn't mean it's a billion dollar startup idea. Indeed, most ideas people think are bad are actually bad - it's only the few outliers that are actually successful. Even then I think there is a ton of mythologizing around this idea that the founders were able to see something nobody else did, usually to make the founders look like some sort of diamond-in-the-rough geniuses, when in reality what they built was just a natural evolution of tech that existed at the time (successful founders usually just execute better and faster than others).

I mean, despite all stuff I've heard on HN about how a lot of big VCs passed on AirBnB, when I first heard of it it seemed like a very natural evolution from sites like Couch Surfing and VRBO that had existed for years.

Re:I wasted $40k on a fantastic startup idea by f6v on Tuesday January 19, 2021 @03:00AM (#25831000)
Well, Airbnb and Uber aren’t the best examples, are they? Their growth and “success” is fueled by either operating in a legal gray zone, or defying the local regulations all together. Many people all over the world think their lives were made much worse since Airbnb is negatively affecting the long-term rental market.

Point is, the effect of the company on the society can’t just be measured by market cap.

Back to the original article, the author was using statistical analysis to provide medical advice. Now, it’s incredibly easy to arrive to false conclusions with statistics. That’s why there’s regulations, peer reviews etc. What if the “Egyptian contractors” screwed the data up. Was the founder qualified to spot an issue?

Re:I wasted $40k on a fantastic startup idea by jacobion on Tuesday January 19, 2021 @07:02AM (#25832597)
Arguably one reason those two businesses were successful in areas with entrenched players and business practices was because they handle the money. If AirBnB was asking either travelers or hosts to pay $X to be on a recommendation site, probably very few people would. There's always a cheaper competitor when you're selling information. Because you book through AirBnB, for a service which is relatively expensive, they can skim off quite a lot of money in an opaque way.
Re:I wasted $40k on a fantastic startup idea by sillysaurusx on Tuesday January 19, 2021 @03:10AM (#25831056)
I think they’re ideal examples. Market cap is pretty much everything. It affects the world more than morals do.

HN has drifted further and further from reality, which has been very strange to watch. The classic example was someone dismissing Dropbox when they first launched, but now it’s turned into dismissing billion dollar companies after they’ve clearly won.

Re:I wasted $40k on a fantastic startup idea by coldtea on Tuesday January 19, 2021 @03:43AM (#25831254)
Only the parent doesn't dismiss "billion dollar companies", they dismiss comparing them to shut down, non winning, companies like in TFA.
Re:I wasted $40k on a fantastic startup idea by sillysaurusx on Tuesday January 19, 2021 @03:45AM (#25831270)
Ah yes, one failure = the idea is horrible. Another classic trope.
Re:I wasted $40k on a fantastic startup idea by f6v on Tuesday January 19, 2021 @04:39AM (#25831615)
You’re constantly steering the conversation somewhere else, aren’t you?

The meta-analysis idea wasn’t terrible. It’s just that there’re many assumptions in a statistical sense, the founder might not be the right person to implement it and he might have targeted the wrong market. Some people are under the impression that everything can be solved just by build an app. However, some fields are much more complicated than your gig economy food delivery.

Re:I wasted $40k on a fantastic startup idea by sillysaurusx on Tuesday January 19, 2021 @05:10AM (#25831808)
Also known as “having a conversation.” I’m not sure why you don’t see the bad faith in your words, but I have no interest in talking more. Goodnight.
Re:I wasted $40k on a fantastic startup idea by coldtea on Tuesday January 19, 2021 @03:49AM (#25831289)
Well, the idea was rejected by patients, advertising revenue, and doctors...

I also note the "weasel word" idea. This wasn't just an idea, but an implementation.

The same thing might make sense as a value-added feature in a more comprehensive health service (so the "idea" might be good when put in that use).

But as an idea for a service based entirely on it, it failed hard. What exactly twist do you have in mind to save it? Or are you just saying "we'll never be sure" with more words?

Re:I wasted $40k on a fantastic startup idea by sillysaurusx on Tuesday January 19, 2021 @04:02AM (#25831362)
If you study a lot of history, you start to notice that old ideas are bad until they’re suddenly very good. Cannons sucked for a long time, till Napoleon showed they weren’t so bad.

I think posting haughty words is a lot easier than trying to make something work.

Re:I wasted $40k on a fantastic startup idea by coldtea on Tuesday January 19, 2021 @04:17AM (#25831460)
>If you study a lot of history, you start to notice that old ideas are bad until they’re suddenly very good.

Also the opposite: dumb ideas are tried again and again to no avail ever.

Re:I wasted $40k on a fantastic startup idea by a1o on Tuesday January 19, 2021 @05:54AM (#25832129)
Adding as example of dumb ideas, people constantly trying to patent perpetual motion machines.
Re:I wasted $40k on a fantastic startup idea by hef19898 on Tuesday January 19, 2021 @11:00AM (#25834722)
Nitpic on Napoleon, cannons were used centuries before him in sieges. People knew they were good all along.
Re:I wasted $40k on a fantastic startup idea by jacobion on Tuesday January 19, 2021 @07:04AM (#25832612)
Wasn't the someone who dismissed Dropbox when they first launched Steve Jobs?
Re:I wasted $40k on a fantastic startup idea by RobertKerans on Tuesday January 19, 2021 @09:45AM (#25833830)
How is that a "classic example" of drifting away from reality? "I don't think this startup business that doesn't seem likely to succeed will succeed" is a comment that looks funny in hindsight, that's all
Re:I wasted $40k on a fantastic startup idea by aaron695 on Tuesday January 19, 2021 @08:51AM (#25833383)
> Their growth and “success” is fueled by either operating in a legal gray zone, or defying the local regulations all together.

We just got an impossible vaccine in under a year, I'm happy for all of medicine to spend a bit of time in a "legal gray zone" to see what might happen.

> Many people all over the world think their lives were made much worse since Airbnb is negatively affecting the long-term rental market.

Absolutely, just like the Luddites (Although they would be well off Luddites) their world is worse. But humanity has been made far better.

Re:I wasted $40k on a fantastic startup idea by wartijn_ on Tuesday January 19, 2021 @03:16AM (#25831090)
Aren't you falling for the survivorship bias trap? Sure, people have said that about Airbnb. But I think there are loads of that startups that failed because they didn't understand the industry they were in, or because of legal hurdles.
Re:I wasted $40k on a fantastic startup idea by spzb on Tuesday January 19, 2021 @06:13AM (#25832263)
I think there's probably a lot more to be learned about business success by studying business failures. There's certainly a load more source material.
Re:I wasted $40k on a fantastic startup idea by hef19898 on Tuesday January 19, 2021 @10:56AM (#25834687)
At least the last part seems to be right, doesn't it? As a user I love Airbnb, way cheaper and better than hotels, epsecially with a family.
Re:I wasted $40k on a fantastic startup idea by MrDresden on Tuesday January 19, 2021 @07:18AM (#25832722)
Saying that there was a nuanced difference in what these two companies tried to accomplish would be a gross understatement.
Re:I wasted $40k on a fantastic startup idea by sova on Monday January 18, 2021 @10:03PM (#25829520)
Statistically speaking, isn't it sound to throw all the papers into the mulcher and see what comes out the other end? We do use the term "outliers" a lot in statistics, do we not? I understand that the quality might not be up to snuff for some, but won't the law of averages take care of that?
Re:I wasted $40k on a fantastic startup idea by hn_throwaway_99 on Monday January 18, 2021 @10:22PM (#25829647)
Have you ever used a mulcher to chop up some yard waste, only to accidentally put in some dog shit, and then the whole thing stinks to high heaven?

In all seriousness, with meta-analyses it's still "garbage in, garbage out". It only takes one or a few egregiously bad studies to throw off your results if that study has a large sample size but something fundamentally wrong with its methodology or implementation.

Re:I wasted $40k on a fantastic startup idea by ufmace on Monday January 18, 2021 @11:31PM (#25830011)
I've dealt with enough types of data that I feel super skeptical that you can just dump numbers from hundreds of studies into some data store programmatically, do statistical calculations, and get valid results. It's very difficult to believe that there aren't a ton of variations in how the data is gathered, filtered, and presented that need to be accounted for before any comparisons can be made. I'm not going to trust the law of averages to negate the effect of completely out of whack data when peoples' health is on the line.
Re:I wasted $40k on a fantastic startup idea by haser_au on Monday January 18, 2021 @11:20PM (#25829958)
This assumes all papers are of equal quality, peer-review and accuracy of results. Which we know they are not. Some studies should have more weight than others. Which has been mentioned in a previous comment; there is no 'right' answer, just a variety of ways to allocate different weights to papers based on various metrics.
Re:I wasted $40k on a fantastic startup idea by brabel on Tuesday January 19, 2021 @03:04AM (#25831016)
You misinterpret the law of large numbers. What the law says is that if you have a large amount of samples, and assuming there's no pervasive bias in the samples, then any large enough sample (and often that's much smaller than you think - the classic example being election voters, with a group of only a few thousand representative voters being enough to predict the outcome of an election over a large country with millions of voters) will look identical to any other... that is, over a large enough sample, in the case of this article, the conclusion of many papers should converge to the same answer, with outliers being marked out as likely "bad" papers.

The only assumption you may reject here is that there's no systematic bias in the papers. Perhaps there is... or perhaps most papers are just very unreliable, in which case there should also be no convergence... but if you find convergence, there's a good chance the result is "real".

Re:I wasted $40k on a fantastic startup idea by cycomanic on Tuesday January 19, 2021 @05:46AM (#25832055)
But the crucial bit here is the "large" in "large numbers". I expect that even for quite popular drugs the number of studies are maybe in the hundreds, which depending on statistics could well be quite a way from large enough. In particular if a significant fraction are crap studies.
Re:I wasted $40k on a fantastic startup idea by Karrot_Kream on Tuesday January 19, 2021 @03:13AM (#25831075)
You mean the Law of Large Numbers (LLN), not the Law of Averages, right? Both the Weak LLN and the Strong LLN presume all samples are independent and identically distributed. If we make a hierarchical model on the data of each paper, we can bind all the data into a single distribution, but assuming that each of these studies is independent is a _long_ shot. WLLN and SLLN _only_ apply to, roughly, sampling from the same process. Its scope is more applicable to things like sensor readings.
Re:I wasted $40k on a fantastic startup idea by jacobion on Tuesday January 19, 2021 @06:59AM (#25832569)
The Law of Large Numbers is an actual math theorem. The Law of Averages is a non-technical name for various informal reasoning strategies, some fallacious (like the gamblers fallacy), but mostly just types of estimation that are justified by more formal probability theory.
Re:I wasted $40k on a fantastic startup idea by pfdietz on Tuesday January 19, 2021 @08:29AM (#25833222)
More generally, see "concentration of measure".

https://en.wikipedia.org/wiki/Concentration_of_measure

Re:I wasted $40k on a fantastic startup idea by flatline on Tuesday January 19, 2021 @01:43AM (#25830684)
You get some numbers, they look good - fine, but at best it’s grounds for a proper study, at worst wildly misleading. You can easily fool yourself with statistics, and other people too.

For a good read about studies with solid statistics and bogus results, see [0].

[0] https://slatestarcodex.com/2014/04/28/the-control-group-is-o...