News for hackers. Stuff that matters.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by jordan801 on Friday February 08, 2019 @11:53AM (#19116099)
Anyone who has written a few scrappers knows how brutally ineffective this is. Yelp tried to pull the same thing and it took me about 3 minutes to rectify my "for fun" scraper. It's also really not that difficult to write a smart scraper that you say, "Look for these things in this post. However you find them, replicate it for the others". Which is ultimately what I made my Yelp scraper do.

If there's a pattern, I will find it, and I will exploit it. <3

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by singron on Friday February 08, 2019 @01:48PM (#19117402)
It actually seems pretty effective in this case. This uBlock Origin issue has been open for half a year and none of the default lists block the ads: https://github.com/uBlockOrigin/uAssets/issues/3367
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by robryan on Friday February 08, 2019 @02:58PM (#19118064)
Extensions like ublock having public block lists makes it even easier for Facebook to write something that breaks the filter.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by IshKebab on Friday February 08, 2019 @03:35PM (#19118386)
Yeah and Facebook has far more resources than uBlock to fight the cat-and-mouse. So it is effective.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by AndrewKemendo on Friday February 08, 2019 @12:43PM (#19116724)
That works for a single iteration, but if there are multiple implementations that are randomly chosen when rendered it's a lot harder.

Pretty easy to build a randomizing span algo that you can't hardcode.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by osrec on Friday February 08, 2019 @12:51PM (#19116788)
Not really. We have scraped many sites successfully that try this randomisation logic. There is always a pattern, which often can be determined via heuristics. It does make things trickier, but not impossible or especially difficult.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by Novashi on Friday February 08, 2019 @01:21PM (#19117111)
It feels like the point is just to raise the difficulty for script kiddies.

After all, there’s always headless browsers and OCR

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by osrec on Friday February 08, 2019 @01:24PM (#19117138)
Yeah, it's a bit pointless really. If you're going to put data on the open web, you should be prepared for it to be copied.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by lazopm on Friday February 08, 2019 @01:25PM (#19117148)
I think you can just iterate over the text nodes and see if you stumble upon every letter you're looking for in the right order, it would work for any kind of randomly added text.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by folkhack on Friday February 08, 2019 @12:38PM (#19116656)
100% true. Have written PLENTY of scrapers and methods like this are ultimately ineffective.

Even if you absolutely mangled the HTML/selectors/DOM/etc. I feel you could always have it process screenshots of the interfaces to rip text/figure out how to interact etc. If it's human readable, it's bot readable imo. (but in years of botting it's never came to this - I've always been able to figure out how to use the existing DOM/selectors to do my work even with anti-bot measures)

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by chucksmash on Friday February 08, 2019 @12:54PM (#19116828)
+1. At a previous employer we fed images of interest from the web into Google's OCR API to see what we could see. In addition to scene descriptions, the API will transcribe any text it detects.

With all the easy to use tools available to programmers today, it would not be terribly hard to use OCR on a screenshot to find the text of interest and derive the scraping code by searching for the OCR'd text in the markup.

If none of your extant parsers can extract the info you want from the page, send it to OCR pipeline (or, hell, Mechanical Turk) and generate a new one.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by folkhack on Friday February 08, 2019 @02:32PM (#19117833)
Yep yep - if the text isn't distorted I can rip it from an image within minutes using pre-built OCR libraries. If the text is distorted there's full-blown API-driven services for solving CAPTCHAs and the like.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by TimothyBJacobs on Friday February 08, 2019 @05:56PM (#19119468)
It seems like a time span of minutes wouldn't be fast enough for on-the-fly blocking of sponsored posts?
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by folkhack on Friday February 08, 2019 @06:03PM (#19119497)
Oh yea - I guess I had a specific use case in mind when I said that =)

What I meant is that I can hammer out some Node/Python that will grab an image w/text and put it through OCR for character extraction. "Programming" it would take me a handful of minutes.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by TimothyBJacobs on Saturday February 09, 2019 @01:05PM (#19123648)
Ahh, that makes sense!
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by eeeeeeeeeeeee on Friday February 08, 2019 @11:56AM (#19116147)
Yep, seems like a total waste of time. The people scraping will spend the necessary time to get around this (and then distribute that knowledge to the masses) so it seems like a pointless arms race. Facebook employees could better use their time on developing actual features that bring value.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by SketchySeaBeast on Friday February 08, 2019 @12:18PM (#19116413)
> Facebook employees could better use their time on developing actual features that bring value.

The problem here of course is that circumventing the ad blocking is the most direct way Facebook can find value.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by matz1 on Friday February 08, 2019 @12:04PM (#19116250)
How is this waste of time / pointless ? They do bring value for fb, they reduce the loss of ad money.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by eeeeeeeeeeeee on Friday February 08, 2019 @12:17PM (#19116407)
Because they are fighting the people that absolutely do not want to see advertisements whatsoever (most normal people I know do not have ad-blockers), which already makes them the least likely to be attracted by advertisements. If anything, those brands are souring their reputation anytime they are seen by those kind of people in a medium that those consumers find obnoxious.

You can make a similar argument that the RIAA/MPAA going after piracy is a waste of time. Again, focus on delivering value to actual customers.

If Facebook spent more time on making a friendly ecosystem / community I would be more open to signing back up again. Instead, it seems they are hyper-focused on advertisements at the expense of everything else.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by rchaud on Friday February 08, 2019 @01:13PM (#19117030)
"Delivering value to customers"? It's a website-cum-message board. There is no further value to deliver via software features; all of that work is going towards meeting the needs of advertisers.

And advertisers are asking them to show what they are doing to combat adblocking. FB isn't doing this to target customers least likely to convert, they're doing it to check a box for their ad sales team.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by otabdeveloper2 on Friday February 08, 2019 @02:21PM (#19117717)
Newsflash: Facebook's customers are advertisers, not the honeypot victims stuck to their site.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by tomatocracy on Friday February 08, 2019 @03:22PM (#19118286)
But there’s a trade off. Fewer ads better placed means fewer people likely to block them and more people likely to actually look at them instead of skim past. It’s also likely that the people who were blocking ads are less likely to click on them and thus (for some advertisers) lower value impressions anyway.

Revenue is (simplistically) the product of impressions and value per impression. It’s not therefore immediately obvious that moves like this actually do increase revenue for them, especially over the long term since one potential side effect of doing this is giving more ammunition to the ‘delete Facebook’ crowd.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by recursive on Friday February 08, 2019 @01:14PM (#19117034)
> which already makes them the least likely to be attracted by advertisements. If anything, those brands are souring their reputation anytime they are seen by those kind of people in a medium that those consumers find obnoxious.

Facebook thinks otherwise. Between you and them, I suspect they are more likely to have some evidence or trials to support their position.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by pastor_elm on Friday February 08, 2019 @11:58AM (#19116174)
Value doesn't keep the lights on.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by IntelMiner on Friday February 08, 2019 @12:01PM (#19116211)
Uh. Yes it does?

You can't make money off something without any value to people. People have to "want" the thing

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by taf2 on Friday February 08, 2019 @12:39PM (#19116676)
They could render the whole thing in canvas for example
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by nacs on Friday February 08, 2019 @01:02PM (#19116931)
So you'd block all canvas elements if ads are always a <canvas>.

If they turn all their posts into <canvas> then it'd kill any accessibility features and the ability to copy-paste text and such so I doubt they'd go that far.

Even then, a scraper could run OCR on the canvas image to get the text out of it.

Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by sqd on Friday February 08, 2019 @01:23PM (#19117134)
I don't think these html pieces is very accessibility-tool friendly..
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by taf2 on Friday February 08, 2019 @07:56PM (#19119999)
And then you’d switch to an img tag
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by danillonunes on Friday February 08, 2019 @01:25PM (#19117143)
Ad blockers are not as powerful as a scrapper, they basically a glorified CSS selector engine. This may be easy to bypass for a dedicated extension targeting Facebook, such as Social Fixer, but for a regular ad blocker it may be harder.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by cauk on Saturday February 09, 2019 @02:55AM (#19121248)
Hey there. Regarding the semi-automatic “look for these things in the post, and however you find them replicate for others”. I’m new to scrappers, do you have a good resource you could link on this? Thanks!
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by nradov on Friday February 08, 2019 @12:11PM (#19116340)
This would make an interesting blog post, if you have time to write up the design.
Re:Facebook adds 5 divs, 9 spans and 30 CSS classes to every... by MarsAscendant on Friday February 08, 2019 @01:50PM (#19117423)
> it took me about 3 minutes to rectify my "for fun" scraper.

Did you mean to say "rectify" as in "fix/adjust"? It sounds like you might have meant "reify" – as in, "create" – but I don't know whether you had the scrapper before that.