News for hackers. Stuff that matters.
Re:How do search engines treat trailing slashes and capital ... by seanwilson on Monday December 30, 2019 @07:31AM (#21912888)
> There were four test URLs in total, two of which tested for how search engines deal with unique content

...

> There is one unusual thing, however – a site: search brings up the lowercase URL, but the uppercase URL is filtered out for being too similar to the other displayed URLs and isn’t shown unless the ‘repeat the search with the omitted results included’ link is clicked.

Maybe the content isn't unique enough so Google's duplicate detection algorithm marks them as duplicate?

The two examples contain similar keywords and don't seem to have any outgoing links. A human would probably flag them as spun articles so I wouldn't be surprised if Google did the same.

Maybe the results would be different if each link was a unique high quality article instead.

Re:How do search engines treat trailing slashes and capital ... by gowld on Monday December 30, 2019 @11:52AM (#21914689)
Yes, Google doesn't say the URL is similar, it says the "result" is similar.
Re:How do search engines treat trailing slashes and capital ... by joekrill on Monday December 30, 2019 @08:30AM (#21913195)
> Monitoring the server logs showed that Bingbot only crawled the lowercase version of the URL.

I've always thought it was much more common that sites written in ASP.Net have case-sensitive URLs. At least that was quite common ~5 years ago (was it a default setting or something? I haven't done .Net stuff in a while). So it's pretty crazy that Bing only crawls lowercased URLs.

Re:How do search engines treat trailing slashes and capital ... by gowld on Monday December 30, 2019 @11:50AM (#21914675)
That's overstating the "case" (pun!). Ignoring or normalizing case is different from ignoring a non-redundant URL with uppercase.

But it might cause problems if the IIS server isn't properly case-preserving to normalize back to the standard (possibly capitalized) form.

Mac filesystem was nicely case-preserving but case-agnostic in this way, going back decades.

Re:How do search engines treat trailing slashes and capital ... by saagarjha on Monday December 30, 2019 @12:03PM (#21914798)
It’s still like this by default, which can cause issues when working with folders created on other platforms.
Re:How do search engines treat trailing slashes and capital ... by lgats on Monday December 30, 2019 @01:11PM (#21915378)
I've found bing explicitly requesting lower-case versions of my urls despite mixed case being submitted in the sitemap.

Now I either 301 redirect to the proper-case url or ensure I have canonical tags set up.

Re:How do search engines treat trailing slashes and capital ... by JeanMarcS on Monday December 30, 2019 @06:38AM (#21912648)
For years (if not decades) case haven’t been important in file names nor URL in the Microsoft world (using web server on a MS computer will result as this exact experience concerning letter case. An Image.jpg or IMAGE.JPG will only show one of the two images).

Could this be an extension of that ? Looks like.

Re:How do search engines treat trailing slashes and capital ... by toolslive on Monday December 30, 2019 @08:46AM (#21913281)
BUT, I've seen case sensitive interpretation of email addresses on a microsoft mail server. (no kidding)
Re:How do search engines treat trailing slashes and capital ... by sethammons on Monday December 30, 2019 @09:00AM (#21913345)
http://www.faqs.org/rfcs/rfc2821.html

SMTP RFC 2821 says that the local part of the email address is case sensitive. Some ignore this and consider upper and lower case the same.

Section 2.4:

> The local-part of a mailbox MUST BE treated as case sensitive. Therefore, SMTP implementations MUST take care to preserve the case of mailbox local-parts. Mailbox domains are not case sensitive.

Re:How do search engines treat trailing slashes and capital ... by toolslive on Monday December 30, 2019 @10:38AM (#21913994)
same section, same RFC:

    > However, exploiting the case sensitivity of mailbox local-parts impedes interoperability and is discouraged.
Re:How do search engines treat trailing slashes and capital ... by samatman on Monday December 30, 2019 @11:14AM (#21914313)
These are of course not in conflict.

What it means is, a mailserver should resolve local part case-insensitively, so that [email protected] and [email protected] end up in the same mailbox.

But, to be spec-compliant, a mailserver MUST send on an email addressed to [email protected] to [email protected], exactly, without downcasing it to [email protected].

I'd imagine this is often honored in the breach, but there you have it.

Re:How do search engines treat trailing slashes and capital ... by gowld on Monday December 30, 2019 @11:44AM (#21914601)
"mail server" is ambiguous. SMTP Simple Mail Transfer Protocol is about transferring mail between domains, which must preserve the local part.

An MDA Mail Delivery Agent or LDA Local Delivery Agent inside a domain can choose whether to be case sensitive, but is discouraged from doing so.

The same issue exists for dots '.' in local part.

Re:How do search engines treat trailing slashes and capital ... by StuffedParrot on Monday December 30, 2019 @10:05AM (#21913739)
Email addresses are case sensitive, at least you’re supposed to assume they are.
Re:How do search engines treat trailing slashes and capital ... by lgats on Monday December 30, 2019 @01:13PM (#21915403)
Are there any widely-used, case-sensitive, public or corporate email servers?

I've certainly strtolower'ed some email inputs...

Re:How do search engines treat trailing slashes and capital ... by sethammons on Tuesday December 31, 2019 @12:34PM (#21922758)
See my link to the RFC in a sibling comment. You should not strtolower an email address unless you are the one running the mailbox itself.
Re:How do search engines treat trailing slashes and capital ... by Doctor_Fegg on Monday December 30, 2019 @10:26AM (#21913890)
The "Kihlepa" pages linked from the footer are quite something.
Re:How do search engines treat trailing slashes and capital ... by gumby on Monday December 30, 2019 @09:59AM (#21913697)
Would have been interesting to see how case differences in the host name portion of the URL were treated. Domain names are case insensitive — would google search both https://foo.example.com/bar and https://foo.Example.com/bar? It should not.
Re:How do search engines treat trailing slashes and capital ... by gowld on Monday December 30, 2019 @11:46AM (#21914628)
One case where this poses a problem is a dictionary site that supports acronyms or proper nouns.
Re:How do search engines treat trailing slashes and capital ... by wccrawford on Monday December 30, 2019 @06:02AM (#21912512)
tl;dr - Google crawls both, but site: searches hide one of them as similar results. Bing doesn't even bother crawling the other versions.