News for hackers. Stuff that matters.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @09:46AM (#39679417)
Not sure if DMCA is invoked to take them down, but a friend of mine recently challenged me to scrape a video file from one of these shady free streaming sites. I promise this has something to do with GitHub.

Gamely, I accepted the challenge. Browser dev mode to sniff things out was very difficult - they put a lot of work into jamming that up. Turns out it is trivially easy with Fiddler.

What I learned in the process was interesting, trying on a few video files. What these sites do is slice up their video files into thousands of chunks, and then rename the files as innocuous file extensions (like *.jpg, *.html, *.csv, etc.) and stick them on various hosting services where you can do a GET on the file for free with no authentication. They spread those thousands of files for each video file all over the internet. Usually across multiple accounts / repos on a service like GitHub but not just GitHub - any place they can manage. A lot I looked at while testing had files on GitHub though.

Then, once the thousands of files are distributed, they stitch them together via an *.m3u8 playlist file which the in-browser player can handle buffering to make it all seamless.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by tamimio on Tuesday March 12, 2024 @12:27PM (#39681406)
> Then, once the thousands of files are distributed, they stitch them together via an *.m3u8 playlist file which the in-browser player can handle buffering

I did try streaming one time with that method, besides the high latency (~5seconds), it was CPU intensive for both the server and the client(browser), so not so sure how viable this would be.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by johndough on Tuesday March 12, 2024 @12:09PM (#39681199)
I noticed that many of those websites abuse the debugger functionality, freezing the browser whenever someone tries to open the developer console. Did you find a way around that?
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by password4321 on Tuesday March 12, 2024 @12:43PM (#39681644)
The post mentioned using Fiddler instead. https://docs.telerik.com/fiddler/configure-fiddler/tasks/ins...

If read-only access is all that is needed, SSLKEYLOGFILE might help. https://my.f5.com/manage/s/article/K50557518

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @12:55PM (#39681839)
Yes. You can disable breaking in dev mode, which gets you a bit further.

But they have some script in there as well that checks for if you're in dev mode somehow, and as well on top of that misbehaves if you're in private mode.

At that point I got tired of fighting it, and realized none of this Javascript nonsense was going to work against Fiddler or other proxy-based solution.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Invertigrate on Tuesday March 12, 2024 @10:08PM (#39687355)
You don't even need to do that, just get the video to start playing and then press F12 to open the dev tools as the video will keep streaming and you'll still be able to see the network requests despite the messages.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Wednesday March 13, 2024 @12:35PM (#39693694)
That approach will miss the transmission of the playlist file. Chasing the individual fragments of video is not the right way to go about it.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by IncRnd on Tuesday March 12, 2024 @12:07PM (#39681178)
That's like RAID0 for streaming - or in a different way like people who backup their drive onto gmail messages, where each file block fits into a gmail message.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @12:14PM (#39681251)
Heh. That's how the warez scene used to operate back in the AOL chatroom days. You would request a bot to send you a file, and it'd forward the whole series of messages to you, which you'd add to your download manager, and then un-rar once done.

I've heard.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Grimblewald on Tuesday March 12, 2024 @09:52AM (#39679513)
Thanks for sharing, that was enormously intriguing. Do you have a bigger write-up on this? I'd love to read it.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by EMIRELADERO on Tuesday March 12, 2024 @10:22AM (#39679868)
https://github.com/movie-web covers this quite well
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by cosmojg on Tuesday March 12, 2024 @01:26PM (#39682258)
Oh wow, this is quite practical! Thanks for sharing! In the past, I've typically relied on a great deal of google-fu and considerable time investment (~10 minutes! *gasp*) whenever I couldn't find a show on one of the big streaming services, and it seems like the backend sources made available by the movie-web extension are a superset of those used by the various shady and ephemeral middleman-of-the-month streaming websites I would typically land on.

It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by EMIRELADERO on Tuesday March 12, 2024 @01:53PM (#39682594)
> It's also quite nice how they've neatly open-sourced and atomized all the little tools that go into making the service work such that one could, in theory, build other services on top of these tools. The quality of the documentation is outstanding, too.

Astonishing is the right word. It's like "neat sparkling new JS-based startup" meets "mid-2000s underground piracy", and I can't get enough of it.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Retr0id on Tuesday March 12, 2024 @10:43AM (#39680143)
I wrote a twitter thread about doing a similar trick with twitter's image CDN: https://twitter.com/David3141593/status/1587978423120666625

Unfortunately I'm unable to access twitter threads now so ymmv - I think it still works.

The TL;DR is that the playlist metadata file (mpd or m3u8) references multiple fragments, and arbitrary file offsets within those fragments. Slice your video into fragments (using standard video processing tools), embed them into PNGs using standard twitter-compatible file format polyglot tricks, and upload them to twitter.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by treyd on Tuesday March 12, 2024 @10:20AM (#39679846)
Why wouldn't they do some basic erasure coding (or even just xor with a repeating pattern) to get around the trivial DMCA takedowns that's inviting?
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Retr0id on Tuesday March 12, 2024 @11:08AM (#39680420)
The main "crime" they're committing here is ToS violations for the hosting backends (they're basically using them as a free CDN). They're probably playing takedown whackamole here already, and if I'm right then most of those takedowns aren't DMCA, so they're not bothered.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by zaik on Tuesday March 12, 2024 @10:43AM (#39680136)
Probably browsers don't support any extra decoding steps when playing a .m3u8 file.
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Retr0id on Tuesday March 12, 2024 @11:09AM (#39680435)
Browsers can't play m3u8's natively in the first place, client-side JS is responsible for piping the fragments into the video element. As sibling comments point out, I think it's more a case of "can't be bothered", than "can't".
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @10:25AM (#39679915)
If they were that sophisticated, they probably wouldn't be involved in movie piracy.

I'm sure there's a certain amount of laziness that also comes with how keen each site is on enforcement. The most recent one I looked at, all the files were on the same site, in the same folder.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by xp84 on Tuesday March 12, 2024 @12:23PM (#39681361)
> If they were that sophisticated, they probably wouldn't be involved

That's an odd assertion. Some of the smartest and most sophisticated people I've ever interacted with were very actively involved in the piracy scene. These types love a challenge, and that's why they get involved in piracy. In some cases they aren't even all that into the content itself.

I don't unilaterally hate all IP (other than patents) btw, so this is not an endorsement of piracy itself, just a frank acknowledgement that a lot of these people are very smart.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @02:15PM (#39682880)
Let me clarify what I'm getting at:

These streaming sites are laden with ads, and some even have offer subscriptions, so there's a profit motivation for the people that run them.

I'm not talking about the people who upload torrents (there's not much in a profit motive there, since that stuff is hard to monetize), or people who crack software. That sort of stuff. Interviews digging into their motivations have been going around for a long time now.

If you're hacking on running a pirate streaming site to make money off it, if you have skills enough to stay ahead of the people constantly trying to take your content down, at some point you're going to realize you can make a lot more money using those skills working for a legit company.

My take on the profit motive for malware authors is not the same; a malware dev can make a lot of money if the group manages to hit the right victims. The ones who are state sponsored probably make decent money too.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by swozey on Tuesday March 12, 2024 @11:48AM (#39680942)
I worked in this space on the CDN non-pirate side and it's basically the same thing but everyone (including the major ones you know) distribute all of their content like that over various cdns and they're all competitively tested against one another for errors/latency/etc and traffic auto-shifted based on live current cost + that magic.

If you're interested in some of the tools used that essentially re-unites from endless sources, auto determines bitrate, allows you to interject interstitial ads based on geography/user tokens etc, check out things like bitmovin. No relation, I've just used it. Then cedexis for the latency/error-rate magic that makes your cdns compete against one another. There's also jwplayer which a lot of video platforms like Univision and Fox use.

https://bitmovin.com/ https://github.com/bitmovin

Another interesting thing is how the DRM is handled, the video chunks that are all muxed/reunited/demuxed, etc on the fly- https://developer.bitmovin.com/encoding/docs/how-to-protect-...

I started out in video stuff back in RTSP days (2005ish), took a LONG break in that area (2016ish) and the systems today are so cool. IIRC back then everything was rtsp.

I actually have no idea beyond what you just mentioned how the pirate side does things so that's neat. I wonder what tools they're using.

https://jwplayer.com/ https://www.cedexis.com/ .. which I guess was bought by citrix so I have no idea whats up with that now.

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by mdrzn on Tuesday March 12, 2024 @10:33AM (#39680022)
Very interesting, would love to know more about it!
Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by RajT88 on Tuesday March 12, 2024 @01:17PM (#39682130)
Wow, this comment got a lot of attention!

If anyone is wondering the other piece of this, which is re-unifying all the thousands of video fragments, the game is over once you have the URI to the playlist file. You feed that URI into VLC's "Convert/Save" functionality and it handles it like a champ. No need to script out anything to grab the pieces and work FFMPEG magic.

At the end of the day, it is a 2 step process I could document for my friend who is more or less a layman. A skilled user (decent with Excel), but not a power user (can't script).

Re:DMCA Notices Took Down 20,517 GitHub Projects Last Year by Invertigrate on Tuesday March 12, 2024 @10:06PM (#39687343)
Just looked this up for myself and it's indeed partially correct (mostly webp, jpg, png, etc though).

You left out an important part which is that they're proxying these requests through their own servers to avoid the source being found.

This way they could be placing the files on servers that actually require some sort of authentication such as a login page and once that's done the sky is their limit as you could even be hosting files on Discord and it would be very difficult to find the source.

User -> Pirate proxy server -> Github (or some other site)