News for hackers. Stuff that matters.
Re:RSS-Bridge – The RSS feed for websites missing it by 1vuio0pswjnm7 on Monday January 03, 2022 @01:44AM (#29778050)
"... Facebook has made their newer pages fiendishly difficult to scrape."

This has not been my experience. It is still as easy as ever using the mobile sites.

However I suspect one person's notion of "scraping" is not always the same as another's. I prefer to work on the command line, in textmode. Thus when I check Facebook I want text-only, no graphics. First I extract the all the story.php URLs and sort them by date. Then I retrieve the contents stored at these URLS and dump into a text-only format so I can read through comments. If there is something that looks interesting I can save the story URL and check out the photos later on a computer that has graphics layer loaded.

TBH, with Facebook I mainly just check messages and notifications. Saving story URLs in chronological order is useful for me because it makes it easy to go back and find items from the past. But if I were really serious about monitoring a "feed" I can create one myself, better than Facebook's, by retrieving the profile page of each friend and extracting story URLs from their source instead of relying on Facebook's manipulative algorithms that deliberately hide stories and re-order what they do show, non-chronologically, in a way that suits Facebook's interests over the user's.

I do not use any fancy software, just a local TLS proxy, netcat (or equivalent) and ubiquitous base userland UNIX text processing utilities. I use a text-only browser (not lynx) for reading HTML and a pager (less) for reading formatted text.

I use the m.facebook.com or mbasic.facebook.com sites, not www.facebook.com because the site on the www subdomain uses GraphQL instead of hyperlinks. Not to mention the mbasic subdomain has no ads.

The way websites use GraphQL instead of simple hyperlinks is unnecessarily complex, reminiscient of a Rube Goldberg cartoon.

Re:RSS-Bridge – The RSS feed for websites missing it by sitta on Monday January 03, 2022 @07:48AM (#29780229)
Interesting. I'm surprised to hear that you've been so successful with that technique. My statement was slightly inaccurate. I have no idea how difficult it is to extract the content from the page once you're able to get it. What is more immediately challenging was to get FB to serve you the page at all. See https://github.com/RSS-Bridge/rss-bridge/issues/2047

There were a handful of pages on Facebook that I used to keep tabs on that would have had to have been public as I do not have an account, but now I can't get FB to let me see some of them without a login (for example https://www.facebook.com/MadisonAudubon). I have no idea if my IP as been flagged, which they appear to do aggressively, or the pages were made private or FB introduced some addition settings and the pages are now configured or what.

Re:RSS-Bridge – The RSS feed for websites missing it by 1vuio0pswjnm7 on Monday January 03, 2022 @04:05PM (#29786880)
I use a login. (Because I am primarily checking messages and notifications.) I cannot access that MadisonAudubon page at all without signing up or logging in. Apologies as I should have realised that the discussion was about retrieving "public" profiles/pages without being logged in. I seem to recall a time when that was universally possible but at some point, quite some time ago if I remember correctly, it started to become limited. I would imagine the reason public pages are created on Facebook is to gain access to Facebook users, who are more heavily surveilled than non-users. Public websites like https://madisonaudubon.org could mirror the content they post on Facebook. In practice, this never happens. Calling Facebook a "walled garden" is misleading. It may be "walled" but there's nothing garden-like about it.
Re:RSS-Bridge – The RSS feed for websites missing it by RobSm on Monday January 03, 2022 @03:02AM (#29778424)
What's the point of TLS proxy?
Re:RSS-Bridge – The RSS feed for websites missing it by 1vuio0pswjnm7 on Monday January 03, 2022 @04:24AM (#29778878)
In this instance it is to enable use of any TCP client, namely ones that do not support TLS, e.g., original netcat, tcpclient or an early version of a text-only browser. I use a variety of clients and with only one exception I only trust the proxy to make remote TLS connections. If one uses a client with acceptable TLS support, then there is no need for a TLS proxy.
Re:RSS-Bridge – The RSS feed for websites missing it by RobSm on Monday January 03, 2022 @04:29AM (#29778910)
How one goes about using TLS proxy? Are there any services? You have any links? Thanks.
Re:RSS-Bridge – The RSS feed for websites missing it by 1vuio0pswjnm7 on Monday January 03, 2022 @03:13PM (#29786057)
"How one goes about using TLS proxy?"

Note I am referring to a local proxy, listening on the loopback device and under the control of the user, not one listening on a network interface connected to the internet and certainly not one operated by a third party as a "service".

https://www.stunnel.org

Good luck.

Re:RSS-Bridge – The RSS feed for websites missing it by bandie91 on Monday January 03, 2022 @11:40AM (#29782624)
shameless plug: see autossl.so[1]. it is not a proxy per se, but an LD_PRELOAD-able lib which upgrades plain socket connections to TLS for applicaitions which does not support TLS themself (or using too old SSL versions).

[1] https://github.com/bAndie91/libyazzy-preload/blob/master/src...