My url isn't your url

Re:My url isn't your url by thaumasiotes on Wednesday May 11, 2016 @10:47AM (#11675775)

> Query string and fragment are encoded using page encoding and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding.

Well, for links that appear in a page to be defined using page encoding doesn't sound so shocking. That doesn't explain why the path would use UTF-8 regardless of page encoding, though.

Are you sure URLs directly entered in the address bar use UTF-8 for query strings and fragments rather than system encoding? If my OS is set to use GB2312, how do URL fragments that I type into the browser get encoded?

[ Parent | on HN ]

Re:My url isn't your url by kmike84 on Wednesday May 11, 2016 @10:13PM (#11680897)

A good question; I'm not sure UTF-8 is always used because systems we tested were all using UTF-8 system encoding.

This question was not relevant for Scrapy because it makes sense to use UTF-8 as default regardless of what browsers do when page encoding is unknown; unlike encoding of URLs extracted from HTML this UTF-8 default doesn't lead to incompatible behavior.

[ Parent | on HN ]