But anyway,
I thought about taking a random sample of pages who returns a "200". Let's say 150, and manually tagging them to find if they're "dead" or not.
And then reuse the "dead or alive but a 200" ratio for all the pages but I was afraid that I'd need to tag much more than 150 pages to have a significant statistical result.
It’s obviously blog content designed to promote your product, hosted on the company’s product website. I don’t see how the FYI is unfair.
I added it because the content was valuable but HN can be finicky about blog posts from companies advertising their own products. Trying to get ahead of indignant dismissals.
There's so many blog posts posted here that could fall under "content marketing" umbrella if you want to be strict. I feel like there's no problem with that if the content is valuable and people like/upvote it. After all this is a platform that is doing marketing for YC where YC companies are supposed to post their content too.
That "warning" also stuck out to me as a bit unfair as I was even looking for how it hooks into ScrapingBee (as I was curious how these scraping-aaS platforms interface with custom code) and couldn't find anything.
from scipy.stats import binomtest
chance_of_dead_page = binomtest(landing_page_counter["dead"], landing_page_counter["total"]).proportion_ci(confidence_level=0.90)
print(f'Chance of a dead but existing landing page (90% Confidence Interval):{chance_of_dead_page.low * 100:.2f}% to {chance_of_dead_page.high * 100:.2f}%')I've worked in or adjacent to the content marketing world long enough to know that a CTA is not necessary for the post to be marketing/advertising. One of the major goals of content marketing it to establish the authority of the brand. You are well aware that the raison d'etre of that post is to spread awareness of and establish the authority of ScrapingBee.
It doesn't mean the post is not interesting, useful or valuable. But that post exists fundamentally for marketing/brand purposes.
Parents warning is completely fair, especially since they immediately point out the value of the post.
If you're paying attention you know it's content marketing. If you're oblivious, the marketing probably isn't working.
Either way, you probably don't need a warning.
Is that a lot? I would have been less surprised if it were 1/3rd of links still live.
Sure, but it's no different than any other blog post from a company. And framing it that way is quite disingenuous since the post pretty much only sticks to the topic and doesn't overtly promote their product.
Yeah we have all seen it is on scrapingBee, no need for a warning.
This would very well fit a "fail fast" attitude with testing MVPs, wouldn't it? At least that's what I would guess. Got a great start with PH but didn't move on from there, so the domain was not renewed...
Granted there are cases where that market IS aligned with your product, eg if you built a low cost site-builder or low cost social media publishing platform
The most interesting chart is one of the last: Proportion of Failures over time. As expected, more recent product links are less likely to 404 or 5xx.
Going back to 2014, almost 1/3 of the featured links give a 4xx or a 5xx response. That’s a lot!
More surprising, links as recent as 2020 show a 1/4 failure rate. Those projects basically launched on PH, then shut down shortly afterward.
Moreover, this analysis can’t actually account for products that have been shuttered but still have landing pages online. It’s ultra cheap to keep a placeholder “Sorry we’re closed” page online, so I imagine a lot of these projects are shutdown but counted as “success”.
Subjectively, this matches what I’ve gathered from watching PH. Getting a PH featured product listing seems to be a badge of honor, but PH users aren’t really interested in using 99% of the products and the submitters aren’t actually interested in building them past proof of concept. Recently, the bulk of postings seem to be advertisements for paid information products or pay-to-join communities.