News for hackers. Stuff that matters.
Re:The Evolution of Marginalia's Crawling by mariusor on Tuesday August 23, 2022 @11:20AM (#32566205)
I'd be curious if the new architecture can be adapted to use different content types, other than HTML. I think the Fediverse (even if it can be crawled as the normal web pages) it would benefit for a custom crawler that can jump from inbox to inbox and understands (at least some) ActivityPub jsonld.
Re:The Evolution of Marginalia's Crawling by marginalia_nu on Tuesday August 23, 2022 @11:39AM (#32566522)
Yeah, that should be fairly doable.

The beauty of the design, since it's mediated by a portable language of JSON-objects, is that you can in principle replace or extend any of the steps, including support for other protocols (like gemini:// ?) or content-types.

Re:The Evolution of Marginalia's Crawling by sfsdfsdfasfasd on Tuesday August 23, 2022 @11:46AM (#32566643)
It's very simple to write a basic ActivityPub crawler (I have done this), but you'd go from outbox to outbox, not inbox to inbox. Regardless, it's all just JSON, so it is more straightforward to crawl than the web.

I tried writing a crawler to map the Fediverse (to discover homeservers) but I discovered quickly that most of the content in the Fediverse is extremist content, both left-wing and right-wing -- it's basically all tankies and Nazis, with a few techies in a bubble using mastodon.social, who think Fediblock is a solution to this problem.

Oh right, and there are a ton of pedo instances too, mostly based in Japan where "loli" is legal. Due to the way ActivityPub publishes federated content, I didn't feel comfortable running a homeserver, because a pedo instance could federate illegal content to my homeserver and get me arrested and charged for possession of content I did not request and do not want to store on my computers, and if I'm not monitoring what content is getting federated to my timeline, I might not even know it's there. Too risky.

Demoralized, I abandoned the project, and I don't think much about ActivityPub anymore.

Re:The Evolution of Marginalia's Crawling by mariusor on Tuesday August 23, 2022 @12:06PM (#32566989)
> because a pedo instance could federate illegal content to my homeserver and get me arrested and charged for possession of content I did not request and do not want to store on my computers, and if I'm not monitoring what content is getting federated to my timeline, I might not even know it's there. Too risky.

I also work in the activitypub space and my solution to avoiding unlawful content is to make the "federating" relationship between two servers a mutual one. One asks to follow, the other replies favourably (or not). This puts a damper on the "wild west" of everyone federating with everyone, but ensures content is curated at a basic level by a SysOP so there are no surprises.