News for hackers. Stuff that matters.
Re:Please Add RSS Support to Your Site by djhworld on Tuesday August 27, 2019 @03:29PM (#20813555)
I tend to read my RSS feeds on the London Underground, where signal is non existent in the tunnels, and patchy in the stations. My app of choice (newsblur) lets you sync your feeds for offline browsing so I do this before heading home.

Unfortunately a lot of feeds I subscribe to tend to truncate their articles with a summary or maybe just the first paragraph, with a link to continue reading the full article.

I get why they do this, getting people onto the site gives a greater opportunity to show ads etc, sites gotta be funded somehow.

A few years ago I wrote a tool that ingests these truncated RSS feeds, visits each new link in the feed, and then uses a text extraction library to pull out the full text. These full text articles are then supplied as a brand new RSS feed for my client to subscribe to.

I felt really guilty about it though and wasn't sure about the legal implications of it

Re:Please Add RSS Support to Your Site by danShumway on Tuesday August 27, 2019 @03:40PM (#20813663)
As someone who hand-codes their RSS feeds and doesn't include article content at all (just a title/link/description), what's the recommended setup for this?

Can I just spit the raw HTML at you? Do I need to inline images? I'm assuming that someone who's consuming a full feed isn't going to want any CSS included.

I'm not against this, but I'm also hand-coding all of my blogs, and this seems like a problem that would be more easily and more consistently solved by a user-run service like the one you've made. I don't know what readers currently support, and I'm looking around quickly on DuckDuckGo and not seeing many resources about generating full-text feeds that don't boil down to, "click the button in Wordpress."

Again, I have no problem supporting this, but if I'm going to put the time in to change my blogs, I kind of need more prodding than just one or two people saying it annoys them. I don't have any specific reason I wouldn't want to accommodate people like you -- I'm not advertising on my blogs and I don't have any analytics, so I don't care where you read. But I don't want to invest a lot of time into building a completely new content pipeline.

Re:Please Add RSS Support to Your Site by novok on Tuesday August 27, 2019 @03:46PM (#20813731)
You can just put raw html and normal img tags. Just think of it as an unstyled version of your normal website, just rendering the article portion.

Also look at other pages’ rss feeds that include full articles.

Re:Please Add RSS Support to Your Site by djhworld on Tuesday August 27, 2019 @03:48PM (#20813746)
Definitely understand the hand crafting etc has value if your post is complex and has interactive elements or lots of graphic design, and would be a shame to go to waste.

I guess the main thing you need to provide really is the text and images, <p>, <strong>, <img> tags etc. If you look at your site in Reader View in Firefox or Safari and it looks OK - that should be good enough

Re:Please Add RSS Support to Your Site by AndrewStephens on Tuesday August 27, 2019 @03:48PM (#20813734)
> I'm not advertising on my blogs

Perhaps you should, I enjoyed finding stuff like this : https://danshumway.com/blog/gamasutra-vulnerabilities/

Re:Please Add RSS Support to Your Site by danShumway on Tuesday August 27, 2019 @07:17PM (#20815366)
I'm really glad you enjoyed the article :)

I realize that there are some ways that people serve ads where they vet them and make sure they're not tracking readers. But even if it was easy to set something like that up, I wouldn't feel good running sponsored content or ads on my platforms -- it's not the relationship I want to have with my readers.

I take a much more negative view of ads than most other people I know. I don't just dislike ads that track users or that are poorly vetted. I dislike the psychological effect that ads have on people, I dislike the attention that they take, and I am very cautious of monetization schemes that are too closely integrated into the consumption of content. I can't think of a way I could run ads that wouldn't make reading my blog a worse experience.

I know that's a somewhat extreme position to take. But it's not that advertising is inherently evil, it's just that it has more consequences and costs than most people realize. It's not only about privacy, security, or money.

Given how large the current advertising industry is, where possible I try to block it and harm it indiscriminately. I encourage people to block ads even on websites that they would otherwise support. I think it would be good for society if the advertising industry got a lot smaller, even if that makes funding online content harder.

Re:Please Add RSS Support to Your Site by saagarjha on Tuesday August 27, 2019 @04:33PM (#20814152)
Perhaps they enjoy not having their website marred by online advertisements?
Re:Please Add RSS Support to Your Site by sduff on Tuesday August 27, 2019 @09:06PM (#20815985)
Agree that that is an awesome article.

Suggest you include the posting date on that page, as reading it, I wasn't sure how recent the info was. I was then shocked to find that these vulnerabilities were identified late last year. Some of these are security 101!

Re:Please Add RSS Support to Your Site by danShumway on Tuesday August 27, 2019 @10:06PM (#20816241)
I didn't realize I wasn't attaching dates to posts, I will add those under the page headers.

This is off-topic at this point, but I still stand by the ending to that post -- Gamasutra did reach out to me and did work with me for a little while to fix some of the issues, but eventually communication fell off.

I didn't re-check to validate that the issues were fixed because I felt like I had done everything that could be expected of me at that point and because, honestly, the whole thing was really stressful. Even when companies are being nice (and UBM was really nice about the whole thing), public disclosure is still scary. You just hear horror stories.

I still recommend that people be cautious using Gamasutra. I only wrote up specific vulnerabilities that I found accidentally, I didn't pen-test the site. My point with this article was that the site needs to be pen-tested. As far as I know, that's never happened.

Re:Please Add RSS Support to Your Site by rchaud on Tuesday August 27, 2019 @03:42PM (#20813680)
Sounds like an automated version of Pocket, which has been around for ~10 years and is now owned by Mozilla; so the legality aspect of it shouldn't be an issue. I use it daily, although recent versions of the app send you to the original article page, ads and all, should you dare to open it with data/wifi active. Airplane mode still pulls up a text-and-image only version of the article.
Re:Please Add RSS Support to Your Site by Wowfunhappy on Tuesday August 27, 2019 @03:50PM (#20813762)
FiveFilters.org does this: https://fivefilters.org/content-only/

The free version has some limitations, like only three articles per rss feed. They offer a self-hosted version for a one-time 35euro fee (although software updates only for one year), which I really appreciate.

Re:Please Add RSS Support to Your Site by AndrewStephens on Tuesday August 27, 2019 @03:36PM (#20813630)
Sorry, I do this (provide only truncated text) in my RSS feed because I couldn't figure out a way of keeping the special formatting my site is designed around.

Its a cop out that I still feel bad about. Maybe I will fix it sometime.

If it makes you feel better, you can run your extractor against sheep.horse as much as you want.

Re:Please Add RSS Support to Your Site by jeena on Tuesday August 27, 2019 @04:23PM (#20814068)
I also like reading in my feed reader, which is basically like a automatic offline "reader view" with automatic "read this article" marks. Because of that I quite early decided to never subscribe to a feed which doesn't give me the full article. So I don't ever have this problem. I basically feel that there is so much good content to read out there which I can get delivered to my rss reader directly that I don't really need to jump though the hoops and clicking on 'read more' links and adapting to badly designed websites with really small fonts and gray on white text colors.
Re:Please Add RSS Support to Your Site by kickscondor on Tuesday August 27, 2019 @04:51PM (#20814306)
Rather than extracting the text, could you just render a cached copy of the full web page? I don't see how anyone would have a problem with that. Many times you will end up missing parts of the article anyway if you try to pare it down.
Re:Please Add RSS Support to Your Site by about_help on Tuesday August 27, 2019 @03:38PM (#20813652)
Could you use a full web engine to scrape the info which would also load the ads?
Re:Please Add RSS Support to Your Site by MapleWalnut on Tuesday August 27, 2019 @03:32PM (#20813588)
That's a great idea. I would definitely use a service like that.
Re:Please Add RSS Support to Your Site by k1m on Tuesday August 27, 2019 @03:36PM (#20813631)
Posted this in another reply, it's a project I've worked on for a while, works essentially the way djhworld describes: https://fivefilters.org/content-only/

Source code of a slightly older version available at: https://bitbucket.org/fivefilters/full-text-rss/src/master/

Re:Please Add RSS Support to Your Site by djhworld on Tuesday August 27, 2019 @03:35PM (#20813628)
I think other people in this thread are running services that do a similar thing.

My crisis of conscience was around storing the full text of the articles on my own server and then presenting them as a new feed on my domain, e.g. https://mydomain.com/feeds/eurogamer.xml

Everything in the feed is that gets generated is exactly the same as the original feed (canonical link to the actual article, publsh date, author etc) the only difference was the text in the body of each entry.

Re:Please Add RSS Support to Your Site by WiseWeasel on Tuesday August 27, 2019 @08:40PM (#20815822)
It's likely that unless you can claim status as a library or some other exception to copyright protections, as a commercial entity providing that service without compensating publishers, such a straightforward violation would get a bunch of takedown requests and eventual lawsuits for failure to respond to those requests, along with more suits if publishers have no easy way of opting out.

As a personal tool you built for yourself and your friend(s) nerdy enough to understand it, I wouldn’t worry too much about it, as long as no search engines index your site and no one finds out about it.

Re:Please Add RSS Support to Your Site by pvorb on Tuesday August 27, 2019 @04:41PM (#20814218)
And others would definitely sue a service like that. (I'd use it, too, but I understand why it won't happen.)
Re:Please Add RSS Support to Your Site by robk on Tuesday August 27, 2019 @04:38PM (#20814189)
I do this with huginn