Re:Python urllib CRLF injection vulnerability
by kccqzy on Monday March 18, 2019 @01:39PM
(#19423189)
This is far from uncommon. Back in DEFCON 2017 Orange Tsai gave a talk about inconsistencies in different URL parsing libraries in different languages. The opening example was a single URL that had a different hostname when parsed by urllib, urllib2, and requests. He also demoed examples of using unusual characters like spaces and newlines to talk to Redis or SMTP while pretending to be HTTP.
Re:Python urllib CRLF injection vulnerability
by haikuginger on Monday March 18, 2019 @02:24PM
(#19423643)
Python urllib3 maintainer here. urllib3 made a change to be more RFC-compliant in December, and which fixed this issue, but that change has not been released yet. We are in the process of looking into that.
I have verified that Requests, which uses us, appears to have its own handling, back at least to requests 2.0 (released in 2013) that prevents this when used directly as an abstraction layer on top of urllib3.
Re:Python urllib CRLF injection vulnerability
by aldoushuxley001 on Monday March 18, 2019 @02:58PM
(#19424003)
Interesting. I was recently debating whether to use Requests or just urllib3 directly. Figured I'd minimize dependencies by just using urllib3 but didn't think it might actually be more secure to use Requests. Great work btw!
Re:Python urllib CRLF injection vulnerability
by aldoushuxley001 on Monday March 18, 2019 @04:53PM
(#19425165)
I was just using urllib3 to post a form on another website and get the resulting html page, then parsed it with BeautifulSoup.
Since it was just a one off use case and ultimately very simple, I didn't see the need for any more functionality. Why bother with the extra packages? Or do you think it's still worthwhile to use Requests even still? Is it not just unnecessary bloat that might slow runtime?
Re:Python urllib CRLF injection vulnerability
by diminoten on Monday March 18, 2019 @05:06PM
(#19425315)
There's a lot to unpack in your comment, but I'll just work with the most easily verifiable thing for you; what was the response time of the resource you were querying with urllib3, and do you think using requests instead of urllib3 directly would be an order of magnitude (or two) more or less runtime?
Re:Python urllib CRLF injection vulnerability
by aldoushuxley001 on Monday March 18, 2019 @05:18PM
(#19425416)
I admittedly didn't test the response times between the two, but just felt adding additional dependencies was unnecessary. I don't realistically expect the speed to be too different between the two, but the less I have to rely on external libraries the better. If I can get the job done with urllib3 why use Requests?
Though admittedly, after reading OPs statement, I see that Requests might actually have some extra security that urllib3 alone might not have. But barring security improvements or the need for extra features that Requests has, seems like using Requests for my usecase would be adding unnecessary complexity.
Re:Python urllib CRLF injection vulnerability
by diminoten on Tuesday March 19, 2019 @12:45AM
(#19428132)
> but just felt adding additional dependencies was unnecessary
This notion, especially in Python and HTTP client programming, is wrong and will cost you many many more hours than it will save you.
Requests is an entire order of magnitude easier to use than urllib3, and while we may be dealing in minutes for this specific scenario, you will make up for any time investment you pay to learn Requests the very next time you need to do HTTP related work in the language.
It's a matter of not overreacting to a cost, and you're paying way more than you should to get a much smaller gain than you could, if you paid that cost elsewhere (by learning/using Requests and how to manage dependencies in Python, which you have to do anyway with bs4).
Re:Python urllib CRLF injection vulnerability
by aroch on Monday March 18, 2019 @03:28PM
(#19424312)
Not the OP, but deploying code on government servers that interact with the public web means that minimizing the required modules saves you piles of paperwork and meetings. I'd rather spend the extra two days writing/testing my own code then filling out paperwork and waiting weeks
Re:Python urllib CRLF injection vulnerability
by peterwwillis on Monday March 18, 2019 @01:43PM
(#19423220)
Key takeaway: don't expect a library to do the safe thing; always sanitize all your input. (If your language supports taint mode, enabling it can prevent these bugs)
Re:Python urllib CRLF injection vulnerability
by kevin_thibedeau on Monday March 18, 2019 @02:00PM
(#19423398)
Urllib2 introduced breaking changes with urllib so a new lib was added to preserve the functionality of the old one. Urllib3 also has breaking changes but it purposely doesn't live in the standard so it can be changed more readily.
Re:Python urllib CRLF injection vulnerability
by aftbit on Monday March 18, 2019 @01:57PM
(#19423367)
urllib and urllib2 are in the stdlib of Python2, and neither of them has a very friendly interface. They have been consolidated to just urllib in Python3.
urllib3 is a 3rd-party library that powers requests. It tries to offer a more powerful set of features behind a better interface.
Slides: https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20pre...
I have verified that Requests, which uses us, appears to have its own handling, back at least to requests 2.0 (released in 2013) that prevents this when used directly as an abstraction layer on top of urllib3.
Since it was just a one off use case and ultimately very simple, I didn't see the need for any more functionality. Why bother with the extra packages? Or do you think it's still worthwhile to use Requests even still? Is it not just unnecessary bloat that might slow runtime?
Though admittedly, after reading OPs statement, I see that Requests might actually have some extra security that urllib3 alone might not have. But barring security improvements or the need for extra features that Requests has, seems like using Requests for my usecase would be adding unnecessary complexity.
This notion, especially in Python and HTTP client programming, is wrong and will cost you many many more hours than it will save you.
Requests is an entire order of magnitude easier to use than urllib3, and while we may be dealing in minutes for this specific scenario, you will make up for any time investment you pay to learn Requests the very next time you need to do HTTP related work in the language.
It's a matter of not overreacting to a cost, and you're paying way more than you should to get a much smaller gain than you could, if you paid that cost elsewhere (by learning/using Requests and how to manage dependencies in Python, which you have to do anyway with bs4).
Edit: this bug sat around for almost 2 years, it will be interesting to see if it gets fixed now that it is getting attention on Hacker News
https://github.com/kennethreitz/requests/blob/75bdc998e2d430...
hannob opened an issue asking about this:
https://github.com/urllib3/urllib3/issues/1553
actual CVE entry: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9740
golang had the same bug which was fixed in this commit: https://github.com/golang/go/commit/829c5df58694b3345cb5ea41...
urllib3 is a 3rd-party library that powers requests. It tries to offer a more powerful set of features behind a better interface.