News for hackers. Stuff that matters.
Re:Apple might be running a web crawler written in Go by unwind on Thursday November 06, 2014 @10:28AM (#8567400)
Not at all sure why it's relevant which programming languages they use at Apple.

Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what?

The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah.

EDIT: Mis-typed the netmask, I meant /8 but typed /24. Fixed.

Re:Apple might be running a web crawler written in Go by mchanson on Thursday November 06, 2014 @10:38AM (#8567463)
Back around 1990 I had a class /16 and I was just a 14 year old. Before the web explosion it was a different time. Domains used to be free back then too!
Re:Apple might be running a web crawler written in Go by sspiff on Thursday November 06, 2014 @11:10AM (#8567642)
I was still playing with toy cars back then, but how did people lose those IPs? Were they revoked?
Re:Apple might be running a web crawler written in Go by icedchai on Friday November 07, 2014 @12:06AM (#8571268)
I still have a /24 from the era.
Re:Apple might be running a web crawler written in Go by fuckballs on Thursday November 06, 2014 @11:21AM (#8567722)
Eeek, the "C" in CIDR stands for classless, so it is either a Class B or a /16 not a "class /16"

Also, it is highly unlikely that you had a /16 as a PI allocation even in 1990.

Re:Apple might be running a web crawler written in Go by mturmon on Thursday November 06, 2014 @01:09PM (#8568590)
Your nick is deliberately obscene. I'm not a prude, but it doesn't fit the tone of the site.
Re:Apple might be running a web crawler written in Go by sneak on Thursday November 06, 2014 @01:19PM (#8568660)
There are no words that are obscene.
Re:Apple might be running a web crawler written in Go by mentat on Thursday November 06, 2014 @01:18PM (#8568655)
Class Bs for persons still existed in 1998.
Re:Apple might be running a web crawler written in Go by wyager on Thursday November 06, 2014 @10:32AM (#8567430)
>The takeaway for me was that Apple has a whole /24 subnet to themselves. That's just ... immense, for a single company. Gaah

(I think you mean /8.) Brace yourself: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...

Re:Apple might be running a web crawler written in Go by untog on Thursday November 06, 2014 @10:39AM (#8567474)
People are curious simply because Apple is so damn secretive. I'm not sure I'd ever want to work there because it seems I would be unable to talk to anyone about my projects, ever.
Re:Apple might be running a web crawler written in Go by manacit on Thursday November 06, 2014 @10:31AM (#8567427)
A /24 for a company of their size is nothing - they actually have the entire 17.0.0.0/8 to themselves: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...
Re:Apple might be running a web crawler written in Go by jrochkind1 on Thursday November 06, 2014 @10:39AM (#8567471)
Less about 'of their size' then about how long they've been around, since back in the days when a large company could get a /8, and just about anyone could get a /16.
Re:Apple might be running a web crawler written in Go by runeks on Thursday November 06, 2014 @11:16AM (#8567684)
Interesting. I know Google uses the IP 8.8.8.8 (and 8.8.4.4) for their public DNS servers. So it seems Level3 doesn't own the entire 8.0.0.0/8 block then.
Re:Apple might be running a web crawler written in Go by lmz on Thursday November 06, 2014 @12:01PM (#8567996)
They do, but they reassign some of them to other parties: http://whois.arin.net/rest/net/NET-8-0-0-0-1/children
Re:Apple might be running a web crawler written in Go by unwind on Thursday November 06, 2014 @11:01AM (#8567597)
Yeah, that was what I meant, brain error. Thanks, I edited my comment.
Re:Apple might be running a web crawler written in Go by NateDad on Thursday November 06, 2014 @11:29AM (#8567794)
Go is not really a "Google" technology. It's just a programming language like any other. There's no reason Apple shouldn't use it, if it's the right tool for the job.
Re:Apple might be running a web crawler written in Go by josu on Thursday November 06, 2014 @11:45AM (#8567901)
Well, it'd be pretty weird if Go-ogle started developing their applications in Swift.
Re:Apple might be running a web crawler written in Go by Shebanator on Thursday November 06, 2014 @12:02PM (#8568007)
Actually, google does a lot of iOS development and has dozens of shipping iOS apps (https://itunes.apple.com/us/artist/google-inc./id281956209), and thus has a pretty sizable community of Objective C programmers. So Google will almost certainly be shipping apps containign Swift code as soon as the tools stabilize enough for them to do so.
Re:Apple might be running a web crawler written in Go by yepguy on Thursday November 06, 2014 @02:06PM (#8568998)
I think they're mechanically translating their Java code to Objective C:

https://github.com/google/j2objc

Re:Apple might be running a web crawler written in Go by gegenschall on Thursday November 06, 2014 @12:06PM (#8568035)
Re:Apple might be running a web crawler written in Go by NateDad on Thursday November 06, 2014 @12:01PM (#8567997)
I don't think it would be weird if they developed OSX/iOS applications in Swift.
Re:Apple might be running a web crawler written in Go by a_c_s on Thursday November 06, 2014 @12:04PM (#8568014)
I'd be surprised if they didn't use Swift for their iOS apps.
Re:Apple might be running a web crawler written in Go by tricolon on Thursday November 06, 2014 @02:00PM (#8568962)
They're not using Swift at the moment. But they might at some point in the future. Source: friend who works at Google.
Re:Apple might be running a web crawler written in Go by CmonDev on Friday November 07, 2014 @06:43AM (#8572074)
Apple simply doesn't have any own solutions for server-side development. Google has own languages for both client (Dart) and server (Go).
Re:Apple might be running a web crawler written in Go by Bahamut on Thursday November 06, 2014 @11:16AM (#8567690)
The software tech Apple uses generally doesn't seem to get as publicized compared to most large tech companies - for example, I know second hand that Apple uses Angular.js, but I have never seen anything in the wild about it.

One thing to keep in mind though is that Apple is much more a hardware company than software.

Re:Apple might be running a web crawler written in Go by NateDad on Thursday November 06, 2014 @11:32AM (#8567815)
For a hardware company, the write a crapload of software... like Mac OS, iOS, and all the built-in applications. I'd say they probably do a lot more on software than hardware, actually. They only release a few pieces of hardware a year.
Re:Apple might be running a web crawler written in Go by ihsw on Thursday November 06, 2014 @11:44AM (#8567900)
It's not difficult to argue that reliable hardware is only a delivery mechanism to enable the usage of Apple's software, however I doubt users would purchase a "Samsung iPhone" or "LG iPhone" in the same manner that they do for Android devices.

The Apple hardware brand, the hardware itself, and the software all together sustain the user demographic.

Re:Apple might be running a web crawler written in Go by NateDad on Thursday November 06, 2014 @12:03PM (#8568010)
Oh sure, the product they sell is the integration between hardware and software. But saying they are "much more" a hardware company than a software company is selling them pretty short.
Re:Apple might be running a web crawler written in Go by beltex on Thursday November 06, 2014 @11:34AM (#8567835)
"Apple views itself as a software company."

- Steve Jobs

https://www.youtube.com/watch?v=dEeyaAUCyZs

Re:Apple might be running a web crawler written in Go by CmonDev on Friday November 07, 2014 @06:44AM (#8572076)
Certainly doesn't come across as one. UX/hardware comes to mind first.
Re:Apple might be running a web crawler written in Go by hueving on Thursday November 06, 2014 @10:33AM (#8567439)
A /24 is only 255 addresses...
Re:Apple might be running a web crawler written in Go by chatmasta on Thursday November 06, 2014 @10:39AM (#8567472)
You might find this guide to calculating subnets helpful. ;)

http://www.dslreports.com/faq/cisco/30.1_Quick_subnet_calcul...

Re:Apple might be running a web crawler written in Go by yebyen on Thursday November 06, 2014 @11:10AM (#8567645)
Maybe the poster expected Apple to eat their own dogfood? They have ObjC and Swift, and Google is one of their major competitors, in mobile space yes as well as pop-language space.

A /24 is also only 256 addresses, so really unremarkable. I think you are reading it backwards. MIT has an /8, that's the large one.

Re:Apple might be running a web crawler written in Go by yebyen on Thursday November 06, 2014 @11:16AM (#8567685)
Mea culpa, I see that Apple does indeed have an /8.
Re:Apple might be running a web crawler written in Go by dagw on Thursday November 06, 2014 @10:40AM (#8567477)
HP has two /8 subnets. What's even weirder is that Eli Lilly (the pharmaceutical company) and Halliburton (the oil company) also both have one.

(as others have said, you mean /8)

Re:Apple might be running a web crawler written in Go by jackhammer on Thursday November 06, 2014 @10:44AM (#8567502)
I seems like organisations that where early adopters of the internet got one.
Re:Apple might be running a web crawler written in Go by wsc981 on Thursday November 06, 2014 @12:03PM (#8568011)
Relevant XKCD comic "Map of the internet": http://xkcd.com/195/
Re:Apple might be running a web crawler written in Go by sandipc on Thursday November 06, 2014 @11:34AM (#8567837)
I agree. "Breaking news: Apple uses Google products"
Re:Apple might be running a web crawler written in Go by christop on Thursday November 06, 2014 @11:16AM (#8567688)
I noticed this as well a couple of weeks ago.

They're still actively visiting every page on the websites that are associated with our iOS apps.

Today alone (starting shortly before 8am CET) they've crawled over 8000 pages on https://trails.io and http://offmaps.com — without sending conditional caching headers (our pages don't change too often).

    $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l
    8254
Note that they don't appear to be scraping any URLs available from within the apps themselves, but rather the the company/support websites linked in our App Store listings.

I guess they're automatically scanning for objectionable content, since these websites are linked from the App Store and the iTunes website?

Re:Apple might be running a web crawler written in Go by 0x0 on Thursday November 06, 2014 @11:18AM (#8567704)
Ooh, that's interesting. They already increased the requirements for online privacy policies etc for ios8 apps, so perhaps we'll see some crackdown on appstore apps who are missing those.
Re:Apple might be running a web crawler written in Go by brycekahle on Thursday November 06, 2014 @10:51AM (#8567541)
The Go http client definitely has a bug that doesn't maintain the User-Agent across redirected requests.

https://code.google.com/p/go/source/browse/src/net/http/clie...

Re:Apple might be running a web crawler written in Go by bgentry on Thursday November 06, 2014 @11:27AM (#8567775)
Looks like issue #4800 is the place where it's tracked:

https://code.google.com/p/go/issues/detail?id=4800

Re:Apple might be running a web crawler written in Go by pkulak on Thursday November 06, 2014 @10:34AM (#8567445)
Go is a fantastic language for writing a crawler. I wrote the Showyou crawler in Go and it's both one of our highest load processes (and it's very efficient, trust me) and most stable.
Re:Apple might be running a web crawler written in Go by bkeroack on Thursday November 06, 2014 @11:00AM (#8567592)
I've seen golang mentioned in Apple job listings. I didn't think this was a secret.
Re:Apple might be running a web crawler written in Go by 13 on Thursday November 06, 2014 @10:14AM (#8567323)
I've got several thousand requests for this in my logs with an IP address in the same /24.

Two weeks ago it was using a different user agent:

    Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Fetcher/0.1)
This string is mostly from a beta build of iOS 6.0 beta 4, but with the same suffix the author discovered.
Re:Apple might be running a web crawler written in Go by pvsnp on Thursday November 06, 2014 @11:58AM (#8567970)
It looks more likely that someone's learning Go and going through the crawler example in the tutorials. Of course I may be wrong here. The trigger for me was the term "Fetcher" here.. http://tour.golang.org/#73

Fun exercise though.

Re:Apple might be running a web crawler written in Go by kngspook on Friday November 07, 2014 @12:31AM (#8571317)
As a former employee, that was my first thought when I read the post too. Every once in a while, I would stay late there and work on a little personal project for fun there. Sometimes one of my friends would be working late, and I'd just keep him/her company. After all, if I'm just going to do the same thing at home, I might as well take advantage of coding on my big monitor and comfy chair...
Re:Apple might be running a web crawler written in Go by pyman on Friday November 07, 2014 @07:59PM (#8575564)
IMHO, a big couch is better than a comfy chair.
Re:Apple might be running a web crawler written in Go by lifeforms on Thursday November 06, 2014 @03:56PM (#8569709)
If Apple coders are reading, I noticed exactly the same issue of User-Agent header not being used on redirects.

You can solve this by (ab?)using the CheckRedirect functionality of http.Client to set the User-Agent again.

Here's an example: https://github.com/lifeforms/httpcheck/commit/07440d952d1660... (The program is nothing special, just a little thing I use internally for monitoring)

Re:Apple might be running a web crawler written in Go by 0x0 on Thursday November 06, 2014 @11:16AM (#8567692)
So they are "apple-maps'ing" search as well, hilariously using one of Google's own tools in the process.

Would be interesting to know if/when any deals with bing/google/duckduckgo for search is expiring, like they were with google maps. That's probably when we will see an apple search engine.

Makes sense for them to fully control another part of the backend for spotlight and siri.

Re:Apple might be running a web crawler written in Go by yourad_io on Thursday November 06, 2014 @10:32AM (#8567435)
> So far, I have seen requests from two IPs: 17.147.18.33 (7 on 2014-10-15) and 17.147.18.33 (7 on 2014-10-15)

One of these should be .35, judging by the logs that follow ;)

Re:Apple might be running a web crawler written in Go by struct on Thursday November 06, 2014 @10:33AM (#8567438)
Perhaps a proxy designed for iOS testing?
Re:Apple might be running a web crawler written in Go by 13 on Thursday November 06, 2014 @10:40AM (#8567476)
I don't think so, it's only requesting HTML.
Re:Apple might be running a web crawler written in Go by sean9999 on Thursday November 13, 2014 @01:45PM (#8603243)
This just in: Someone at Apple was doing stuff
Re:Apple might be running a web crawler written in Go by jdalgetty on Thursday November 06, 2014 @10:53AM (#8567552)
I've been seeing this as well.
Re:Apple might be running a web crawler written in Go by jezfromfuture on Thursday November 06, 2014 @10:11AM (#8567299)
Way to get someone sacked at apple.
Re:Apple might be running a web crawler written in Go by mseepgood on Thursday November 06, 2014 @10:17AM (#8567338)
Re:Apple might be running a web crawler written in Go by andreasklinger on Thursday November 06, 2014 @10:18AM (#8567345)
Why shouldnt someone at apple use Go.

Servers/scripts/etc are not their usual focus and they dont really have an internal language that fits better. Go is pretty good at this kind of things.

Re:Apple might be running a web crawler written in Go by pjmlp on Thursday November 06, 2014 @10:36AM (#8567455)
I like Go, but fail to see how Swift and C++14 don't fit the bill as well.

EDIT: changed to a more constructive comment.

Re:Apple might be running a web crawler written in Go by astrodust on Thursday November 06, 2014 @11:23AM (#8567737)
Any Turing Complete language would "fit the bill", obviously.

What's wrong with a company not just kicking the tires on another language, but using it on a big project? That's how you design better languages of your own.

"Go was great, except for X and Y" leads to improvements on your own stuff.

Re:Apple might be running a web crawler written in Go by boomlinde on Thursday November 06, 2014 @11:05AM (#8567610)
"I like X, but i fail to see why Y and Z don't fit the bill as well."

Try repeating your point for any order of X, Y and Z, and you might realize it isn't really a valid reason for them to have used Swift or C++ instead.

Re:Apple might be running a web crawler written in Go by pjmlp on Thursday November 06, 2014 @12:01PM (#8567985)
Apple owns C++ and Swift compilers.
Re:Apple might be running a web crawler written in Go by boomlinde on Thursday November 06, 2014 @01:09PM (#8568592)
Your original argument was that C++ and Swift fit the bill as well. Now, as far as I can tell, you seem to be saying that C++ and Swift are better choices because Apple owns compilers for them. That is a different argument, which is in direct conflict with the one I originally responded to. My response was based on the assumption that you meant what you wrote.

Anyway, onto your current stance. That Apple don't use Swift or C++ for their spider seems to indicate that the benefit of using Go outweighs the benefit of using a language they own the compiler for. Go has a BSD-style license, so as far as Apple implementing their own changes and doing whatever they want with the source code and their own binaries at will goes, they might as well have owned it. Go is also seems like a no-brainer choice for a web spider since it has concurrency modeled in such a straight forward and cruft free way.

Re:Apple might be running a web crawler written in Go by aikah on Thursday November 06, 2014 @10:21AM (#8567360)
lol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.
Re:Apple might be running a web crawler written in Go by kaolinite on Thursday November 06, 2014 @11:22AM (#8567729)
The most likely explanation for not using Swift is the employees who worked on the crawler simply didn't know it existed. From what I've heard, only a select few at the company knew it was in development and a number of Apple employees were noticeably shocked during the WWDC keynote when it was announced.

Apple actually uses a bunch of 3rd party developer tech on the server-side and on the web. They use Rails quite a bit I believe and are starting to use Angular.js too.

Re:Apple might be running a web crawler written in Go by melling on Thursday November 06, 2014 @10:29AM (#8567413)
It's not as mature. Doesn't have the library and third party support. Does Apple use Mac OS for all its servers? They use Azure for some servers? That would imply they need the language to be cross-platform.
Re:Apple might be running a web crawler written in Go by ZanyProgrammer on Thursday November 06, 2014 @11:14AM (#8567670)
I can vouch for this-I've been trying to import CommonCrypto, and instead of just typing "import CommonCrypto" I have to do some sort of "bridging header" stuff with a .h file that just seems totally backwards to me (I don't have Cocoa or Obj-C experience). It feels very not quite ready for prime time.
Re:Apple might be running a web crawler written in Go by pjmlp on Thursday November 06, 2014 @10:42AM (#8567491)
Given how it interoperates with Objective-C runtime, I would say it already has lots of libraries to choose from.
Re:Apple might be running a web crawler written in Go by wyager on Thursday November 06, 2014 @10:29AM (#8567412)
Swift was designed for coding user-facing applications, not internet utilities.
Re:Apple might be running a web crawler written in Go by antimagic on Thursday November 06, 2014 @10:35AM (#8567451)
Swift was designed to do everything - one language to rule them all. To quote from Apple's Swift page: "Swift is a successor to the C and Objective-C languages."

My guess would be that the webcrawler was started before Swift was allowed to escape from the Developer Tools enclave.

Re:Apple might be running a web crawler written in Go by wyager on Thursday November 06, 2014 @11:15AM (#8567673)
Doing everything is not the same as doing everything well.

Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language?

Just to be clear, I prefer Swift to Go in general. However, between the two, I would personally prefer to use Go for the development of network utilities.

Re:Apple might be running a web crawler written in Go by antimagic on Thursday November 06, 2014 @11:18AM (#8567701)
I'm not saying it's a good idea, I was just responding to your assertion that Swift was designed only for writing user-facing applications. That is not the case - this is Apple's planned replacement for low-level code as well as apps. It is this design requirement which drove Swift's abandoning Objective-C's dynamic bindings etc.
Re:Apple might be running a web crawler written in Go by Dewie on Thursday November 06, 2014 @12:35PM (#8568329)
> Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language?

I don't think that's a given. Just because a language is more special-purpose, doesn't mean that a more general purpose language can do the same things as seamlessly as in the other language (maybe they would have to define an embedded DSL, but that might be simple). Maybe the special-purpose language decided to commit to certain things in its domain - like certain concurrency features - and make it "first class", since that was its intended domain. But then maybe that concurrency feature wasn't that useful and a more general purpose language - which had to keep its options open since it was general purpose - could express other alternative features more easily.

Re:Apple might be running a web crawler written in Go by melling on Thursday November 06, 2014 @10:32AM (#8567429)
Can you explain what that means? Which language features is Swift missing?
Re:Apple might be running a web crawler written in Go by wyager on Thursday November 06, 2014 @11:10AM (#8567647)
It's a perfectly reasonable general purpose language. However, it doesn't have a low-overhead green thread mechanism, and the standard library has no particular emphasis on network tools.

Even as an outspoken critic of go, I recognize that it has quite a strong networking toolset.

Re:Apple might be running a web crawler written in Go by pjmlp on Thursday November 06, 2014 @10:40AM (#8567481)
Swift is a general purpose language. I fail to see what it lacks.
Re:Apple might be running a web crawler written in Go by boomlinde on Thursday November 06, 2014 @11:10AM (#8567648)
So are Python and Ruby, but most people will agree that those are unsuitable for certain types of highly concurrent and performance sensitive network applications.

Using the right language for a particular domain can save you a lot of headache. Go is open source and a relatively proven choice, compared to Swift.

Re:Apple might be running a web crawler written in Go by pjmlp on Thursday November 06, 2014 @11:59AM (#8567975)
Python and Ruby aren't a suitable comparison, because they lack AOT compilers to native code as part of their canonical implementation, which is not the case for Swift.

As for access to the language implementation, Swift is Apple's creation.

Re:Apple might be running a web crawler written in Go by boomlinde on Thursday November 06, 2014 @01:40PM (#8568814)
The argument you made was that "Swift is a general purpose language.". Since both Python and Ruby are generally considered general purpose languages, they _are_ suitable for comparison to Swift in the exact sense that you chose to describe Swift.

I'm not trying to tell you that you are wrong. I'm just saying that your reasoning is flawed, and your arguments don't really support any particular conclusion. "Swift is a general purpose language." - OK? "I fail to see what it lacks." - Noted.

Let's only compare with native, canonically AOT compiled, general purpose languages, though. All of a sudden there are a lot of questions. Why didn't they write it in C? Why didn't they write it in Delphi? Fortran? Forth? The answer is of course that beyond being a native, canonically AOT compiled, general purpose language, different languages have different constructs and models of particular patterns that make them more or less useful for different kinds of tasks and development models.

Apple seemingly recognized that Swift was less useful than Go for the particular task at hand, and while we can't really know for sure if this was a matter of performance, development speed, personal preference of the developers etc, we can probably assume that the choice was based on taking several such quality dimensions into consideration.

Re:Apple might be running a web crawler written in Go by melling on Thursday November 06, 2014 @11:40AM (#8567875)
I would expect Swift to have comparable performance to Go since it's a C like compiled language. Where it doesn't, Apple should probably dog-food it so they can find any deficiencies.
Re:Apple might be running a web crawler written in Go by nkozyra on Thursday November 06, 2014 @10:53AM (#8567551)
Network latency and distribution notwithstanding, one of Go's core goals is low-overhead concurrency. I'm not sure how GCD compares in Swift because I haven't spent a ton of time in it yet, but that could be a point of contention, particularly in a web scraper.
Re:Apple might be running a web crawler written in Go by vetinari on Thursday November 06, 2014 @10:17AM (#8567336)
Why? This is exactly the situation, where the tool is a great match for a job.
Re:Apple might be running a web crawler written in Go by hartator on Thursday November 06, 2014 @10:31AM (#8567425)
I am doubtfull Apple wants to compete with Google in search.

Or maybe it's a way for Steve Jobs to get a revenge post-mortem on Google for Android!