News for hackers. Stuff that matters.

Show HN: A bookmarklet to remove clickgates on New York Times, Medium, etc.

Recently i stumbled on too many clickgates on the Medium blog Towards Data science. Considering that most people publish to share knowledge on Medium and are driven into putting their content behind a paywall, without actually getting paid for it, including myself. I felt like Medium is running the academic publishing scheme. Get free content and get paid for it. So I decided to create a small script to bypass the paywall on Medium, it turns out it also works on other newssites. Heres the website: https://sugoidesune.github.io/readium/ For the curious I will explain the technical aspects in a comment.
( 49 comments | 71 points | on HN )
Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @03:34PM (#21756565)
Here is a gist of the commented code. It works by fetching the HTML content of the website, anonymously with no cookies. Using the fetch API. In a second step the HTML is preprocessed, removing javascript, inserting elements like images that might be done through javascript etc. The third step is to rerplace the current windows HTML with the clean-preprocessed HTML with the article.

https://gist.github.com/sugoidesune/884bfdf8a975920e98e7307e...

Re:Show HN: A bookmarklet to remove clickgates on New York T... by wiseleo on Tuesday December 10, 2019 @04:17PM (#21756954)
Nice...
Re:Show HN: A bookmarklet to remove clickgates on New York T... by hprotagonist on Tuesday December 10, 2019 @03:47PM (#21756683)
Re:Show HN: A bookmarklet to remove clickgates on New York T... by lewiscollard on Tuesday December 10, 2019 @03:56PM (#21756763)
Good stuff timar - this definitely beats my trick of hitting the Esc key at just the right moment to stop JS from loading. I've gotten very good at that, but also life is very short and I shouldn't need to! So your bookmarklet will come to the rescue :)

Could this be integrated with Firefox's Reader mode somehow?

Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @04:09PM (#21756879)
Not very familiar with firefox or reader mode. But if it's something firefox does with the html of the website youre currently on. Than I see no reason why it couldn't work together. First getting the article with Readium, secondly starting Reader mode.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by barretts on Tuesday December 10, 2019 @04:13PM (#21756917)
Do you think the 1,600 journalists at The New York Times, many of whom work in difficult environments (including the White House, lol), should work for free? Quite the Scrooge move.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by wolco on Tuesday December 10, 2019 @04:54PM (#21757267)
Calling someone a scrooge or cheap for finding a hole in a technical limitation device that is sloppy in its implementation as it is click-bait in it's dark patterns of pretending the page is an open page/free page so it can compete in the search engines but in reality it presents a paid-gate that only appears after leading content is very unhackernews like.

NYT one of the largest most profitable companies pretends to show you an article baits you and demands money and you are a scrooge for not paying? In that context they are the rich entity and should give the content away (the way Scrooge should have). No where in the story does it mention Tiny Tim finding coal in Scroogle's garbage and someone stopping him and calling him cheap for not paying full price because Scrooge employes so many people.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by baddox on Tuesday December 10, 2019 @05:04PM (#21757363)
> it is click-bait in it's dark patterns of pretending the page is an open page/free page so it can compete in the search engines but in reality it presents a paid-gate that only appears after leading content

This is the most important point here. If your content is not actually public, you really shouldn't get the benefits of search engine exposure, HN exposure, or even the distribution from sharing what appears to be a URL to a hypertext document.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by godshatter on Tuesday December 10, 2019 @06:08PM (#21757855)
Whenever I see a New York Times link somewhere, I just move on now since I know I can't read it. I'm not going to pay a subscription for every web site I might want to read an article on. Their business model is not my concern; if they want my business they'll have to try something else.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by sjdegraeve on Tuesday December 10, 2019 @04:31PM (#21757069)
For those who want a daily concise version of the New York Times, check out https://timesdigest.com/.

It's not free, but I've found it to be a great way to read the NYTimes.

Example of the 10-page daily PDF: https://timesdigest.com/samples/timesdigest

Re:Show HN: A bookmarklet to remove clickgates on New York T... by cmdshiftf4 on Tuesday December 10, 2019 @07:42PM (#21758404)
>Quite the Scrooge move.

Seconded. There's something very tasteless about a bunch of very well paid tech employees who'll readily decry adtech, tracking and data brokerage (despite many of them working on it themselves), who'll proclaim "there has to be another way!", who'll also lambast modern journalism for "falling quality", circumventing content subscriptions and depriving publishers of revenue simply because of a built-up sense of entitlement to everything being free.

If the content is worth reading, i.e. it's from a publisher you believe to be of sufficient quality to try and find methods to work around their subscriptions, then it's worth paying for.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by antisthenes on Tuesday December 10, 2019 @07:21PM (#21758274)
Reading many opinion pieces from NYT, I wouldn't cry if the vast majority of those journalists became unemployed. It's outrage journalism and thus not worthy of my time or attention.

NYT does do good journalism, but, even for a company with their credentials and resources, its share seems to be ever dwindling as time goes on.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @04:31PM (#21757073)
I feel like, if PewDiePie can make 1 Million a month in youtube ad-revenue and 8 million total. A Newssite like NYT ought to be able to support their staff, thanks to a global reach and 240Million views per month.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by sharkmerry on Tuesday December 10, 2019 @04:35PM (#21757112)
8 million dollars over 1600 journalists is $5000 each. Pewdipie doesnt have the costs of running a news company and hes not in the news industry
Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @04:38PM (#21757149)
Yeah, thats 8 million a month. i feel like 5k a month is pretty good pay for a journalist.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by pvg on Tuesday December 10, 2019 @04:48PM (#21757222)
NYT should adopt the business model of a youtube edgelord and then pay its journalists submarket wages does not sound like a very realistic approach to running a solvent news organization.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by baddox on Tuesday December 10, 2019 @04:55PM (#21757280)
I don't think they should work for free. I'm not under the impression that they do work for free, or that blocking ads or paywalls will cause them to work for free.
[dead]
Re:Show HN: A bookmarklet to remove clickgates on New York T... by eindiran on Tuesday December 10, 2019 @05:11PM (#21757433)
From the HN Guidelines:

> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

https://news.ycombinator.com/newsguidelines.html

Re:Show HN: A bookmarklet to remove clickgates on New York T... by ailideex on Tuesday December 10, 2019 @05:02PM (#21757335)
I don't subscribe to the idea that you can steal something which is non-rivalrous and non-excludable.

To steal is "to take illegally, or without the owner's permission, something owned by someone else" - can't really take the bits from NYT - after someone reads their article without paying them NYT has no less bits than they had before.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by knolax on Tuesday December 10, 2019 @04:20PM (#21756990)
They should pay me reparations for reading some of the nonsense they publish.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by barretts on Tuesday December 10, 2019 @04:24PM (#21757010)
I'm sorry for your indentured servitude.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by dang on Tuesday December 10, 2019 @05:39PM (#21757657)
Please don't feed egregious comments by replying to them. That's in the site guidelines: https://news.ycombinator.com/newsguidelines.html
Re:Show HN: A bookmarklet to remove clickgates on New York T... by knolax on Tuesday December 10, 2019 @09:44PM (#21759097)
Thank you.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by neeksHN on Tuesday December 10, 2019 @04:54PM (#21757272)
Do you assume all HN users would reach the paywalls' read-limits through organic everyday use?

I felt forced to disable cookies and javascript on the following sites just after exclusively clicking on links submitted to HN: NYT, Medium, Washington Post, Bloomberg, MIT Technology Review and Harvard Business Review.

I'd prefer if submitters choose more user-respecting sources, but until then I refuse to feel guilty over stopping these intrusive sites and their dark patterns.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by larnmar on Tuesday December 10, 2019 @04:34PM (#21757101)
No, I feel that they should all go and get other jobs.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by kerkeslager on Tuesday December 10, 2019 @04:40PM (#21757163)
Do you think that if the New York Times sends you a copy of their article, you're obligated to pay for it, even if you didn't agree to pay for it?

I'm happy to pay for content. I donate $10/mo to both Mother Jones and ProPublica. That's more for each than the cost of a New York Times subscription. Neither of these has a paywall. I'd prefer that they didn't have popovers either, but that's about as good as I can expect in the modern anti-privacy, anti-attention culture of the internet. I just don't feel obligated to pay for content that is sent to me without me ever agreeing to pay for it.

I also think search engines shouldn't index paywalled content, and that newsfeeds like HN or Reddit shouldn't allow links to paywalled content, and I consistently downvote paywalled content.

EDIT: I'm also going to start requesting that people not link to paywalled content when I see it, I think.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by chiefalchemist on Tuesday December 10, 2019 @04:17PM (#21756955)
Sidebar: working for and writing at tbe NYT - or any "news" publisher - doesn't make you a journalist. Journalist is a process and a result. We're too quick to use the word journalist. The low bar should not be so normalized.
[dead]
Re:Show HN: A bookmarklet to remove clickgates on New York T... by chiefalchemist on Tuesday December 10, 2019 @05:02PM (#21757338)
Ya think? If you read the NYT in the 70's and 80's like I did - and then compare it to today's - then you'd realize how hard I'm laughing right now.

You reflecting their approach (i.e.,unable to be objective) is nothing short of vindication (for me).

Re:Show HN: A bookmarklet to remove clickgates on New York T... by fidla on Tuesday December 10, 2019 @04:46PM (#21757207)
Thanks! I've been looking for something like this for ages. I use Brave browser on my android and pc. This will be super handy!
Re:Show HN: A bookmarklet to remove clickgates on New York T... by qnsi on Tuesday December 10, 2019 @03:39PM (#21756610)
I though people are paid for posting on medium?
Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @03:54PM (#21756745)
They do, but you have to apply and be selected. Compared to all users, its a fraction. And everyone else gets pushed into the paywall, because medium basically says: "If you don't we will not promote it and your post will die in obscurity"
Re:Show HN: A bookmarklet to remove clickgates on New York T... by sharkmerry on Tuesday December 10, 2019 @04:36PM (#21757127)
so medium is running a business? Did you apply and get turned away?
Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @04:42PM (#21757172)
No, I don't aspire to be a writer for a living. If I publish something I do it to share knowledge or for personal branding.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by timar on Tuesday December 10, 2019 @04:47PM (#21757214)
Ofc they're a business. But in youtube terms. As a creator you either put your video behind a paywall,and not necessarily get paid, or your video will never show up in any recommendations. So unless somebody searches for it or has the link. Nobody is going to see it. And every viewer on youtube gets 5 videos a month.

Has a bit of a bad taste for me.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by Eduard on Tuesday December 10, 2019 @04:39PM (#21757153)
How to use it with mobile Chrome on Android?
Re:Show HN: A bookmarklet to remove clickgates on New York T... by wiseleo on Tuesday December 10, 2019 @04:18PM (#21756960)
Thank you. Both for the code and for giving me an inspiration. :)
Re:Show HN: A bookmarklet to remove clickgates on New York T... by mchintu on Thursday December 12, 2019 @08:07PM (#21778232)
This is fantastic
Re:Show HN: A bookmarklet to remove clickgates on New York T... by selfishgene on Tuesday December 10, 2019 @03:56PM (#21756769)
This one is also quite helpful:

https://github.com/iamadamdev/bypass-paywalls-firefox

... for all of the folks out there who don't have a university picking up the tab for the articles that they want to read.

It's important to remember that paywalls disproportionately impact those who are not "working" in an academic setting.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by atombender on Tuesday December 10, 2019 @04:15PM (#21756939)
For Chrome and Brave: https://github.com/iamadamdev/bypass-paywalls-chrome. Seems to work well.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by barretts on Tuesday December 10, 2019 @04:16PM (#21756948)
And evading paywalls disproportionately impacts the journalists creating the stories you seem to value, though for some reason refuse to pay for.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by baddox on Tuesday December 10, 2019 @05:06PM (#21757387)
Do you think that people reading newspapers for free at the library negatively impacts journalists?
Re:Show HN: A bookmarklet to remove clickgates on New York T... by Red_Leaves_Flyy on Wednesday December 11, 2019 @03:17AM (#21760563)
The average value I glean from an average article is less than if I had stared at a wall instead. I'll gladly pay a modest sum for a consistently high quality long form news service. However, everyone I've found is asking hundreds- thousands/yr which is neither modest nor affordable for most people, myself included.

NYT, wapo, BBC, fox, CNN, etc all push out tonnes of clickbait garbage every day. If that's not making them enough money then maybe they should get back to investigative journalism. I want long, painstakingly neutral, well researched, and cited articles on topics that concern me. E.g. healthcare, economics, politics. I find more value from the participants in discussion boards I frequent than just about every article I read. Sure, there's a few that provide more value than the participants like the work out of the ICIJ, but they don't hide their work behind paywalls, have ads, or run clickbait, and yet somehow they remain solvent... Hmm...

Re:Show HN: A bookmarklet to remove clickgates on New York T... by LoSboccacc on Tuesday December 10, 2019 @04:38PM (#21757145)
this it's literally facilitating copyright infringement

if you don't want to pay for them is fine, just don't read them

Re:Show HN: A bookmarklet to remove clickgates on New York T... by baddox on Tuesday December 10, 2019 @05:08PM (#21757399)
It's certainly not copyright infringement to download a hypertext document from a company's public web servers.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by LoSboccacc on Wednesday December 11, 2019 @01:42AM (#21760146)
did I say it's copyright infringement?

no.

"facilitating" is not just there for shit and giggle.

https://www.law.cornell.edu/uscode/text/17/1201

Re:Show HN: A bookmarklet to remove clickgates on New York T... by baddox on Wednesday December 11, 2019 @02:11PM (#21765713)
What I meant is that the thing it's facilitating is not copyright infringement. It's not copyright infringement to download a document that was intentionally provided by a public server. The fact that they display paywalls to some visitors under some circumstances is irrelevant. Remember, you can also "circumvent" these paywalls by reading a newspaper at a library or office lobby.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by Red_Leaves_Flyy on Wednesday December 11, 2019 @03:29AM (#21760620)
That's stretching the definition. By your argument any parsing of a payload containing copywritten work is copy right infringement.
Re:Show HN: A bookmarklet to remove clickgates on New York T... by LoSboccacc on Wednesday December 11, 2019 @03:47AM (#21760696)
no? section 3a circumvention definition is exceptionally (and intentionally) wide in the and section 3b does the same for what is considered a restrictive measure.

> any parsing of a payload

again, no? authorized parsing does not.

Re:Show HN: A bookmarklet to remove clickgates on New York T... by Red_Leaves_Flyy on Thursday December 12, 2019 @08:17AM (#21771555)
As far as I'm aware most websites don't have (enforceable) eulas that state what parsers can and can't be used. If I want to read the binary stream from the wire that's my prerogative.

If you have a website that shows a blank page, to everyone that doesn't send a specific token to your webserver, and I discover that I can view said page by manipulating the dom with any old browsers console I am not violating your copyright. You willingly sent that information.

To bring this home, if I can read an article visually obscured by code that has otherwise been given to me in an unencrypted form I am not violating copyright. If a company has a problem with people seeing everything said company sends to them then they're woefully ignorant of how browsers work. The solution is to stop including the article. Rightfully, this kills seo rankings.