News for hackers. Stuff that matters.
Re:RoboBrowser: Your friendly neighborhood web scraper by thomasahle on Thursday May 12, 2016 @08:17AM (#11682795)
I'd like to write a small scraper for a website that uses NTLM authentication, the headers it sends are:

    HTTP/1.1 401 Unauthorized
    Server: Microsoft-IIS/8.5
    WWW-Authenticate: NTLM
    WWW-Authenticate: Negotiate
    ...
Does RoboBrowser support these kinds of protocols? I tried to get it to work with Scrapy, but it seemed non-trivial...
Re:RoboBrowser: Your friendly neighborhood web scraper by kej on Thursday May 12, 2016 @03:02PM (#11685876)
It's been years since I've used it, but I think cntlm can do this. Point your Scrapy code at the cntlm instance, and it should handle all of the NTLM headers for you.