News for hackers. Stuff that matters.

A comment on What one may find in robots.txt.

Re:What one may find in robots.txt by noobie on Monday May 18, 2015 @07:44AM (#9563843)
What's with the domain name? it says thiébaud.fr but when copied/pasted it becomes xn--thibaud-dya.fr

Edit: Thanks for the links! :)

[ Parent | on HN ]
Re:What one may find in robots.txt by mrwizrd on Monday May 18, 2015 @07:54AM (#9563868)
The domain name is being converted to PunyCode (https://en.wikipedia.org/wiki/Punycode). This is a defence developed to prevent spoofing of domain names using indistinguishable characters from other character sets than ASCII. Before this was fixed by browser developers, it was possible to execute an IDN homograph attack (https://en.wikipedia.org/wiki/IDN_homograph_attack) and spoof sites using characters that looked like e.g., google.com/paypal.com when they rendered in your browser address bar, but were actually a completely different domain as far as automated systems were concerned.

Edit: typo.

[ Parent | on HN ]
Re:What one may find in robots.txt by talideon on Monday May 18, 2015 @07:46AM (#9563847)
It's an IDN: http://en.wikipedia.org/wiki/Internationalized_domain_name
[ Parent | on HN ]
Re:What one may find in robots.txt by learnstats2 on Monday May 18, 2015 @07:46AM (#9563848)
That's how internationalized domains work.

Wikipedia: https://en.wikipedia.org/wiki/Uniform_Resource_Locator#Inter...

[ Parent | on HN ]
Re:What one may find in robots.txt by Flimm on Monday May 18, 2015 @08:52AM (#9564106)
Interestingly, Hacker News doesn't support this standard, and the link on the front page shows the unfriendly version.
[ Parent | on HN ]
Re:What one may find in robots.txt by ozh on Monday May 18, 2015 @09:57AM (#9564420)
It's because IDN domains became a standard after tables stopped to be used in page layouts </sarcasm>
[ Parent | on HN ]