News for hackers. Stuff that matters.
Re:Show HN: Download the first 10,002,378 HN comments/storie... by duggan on Tuesday August 04, 2015 @09:26AM (#10003213)
Somehow I can never turn down a data dump, despite never having done much with one.

Some day!

Re:Show HN: Download the first 10,002,378 HN comments/storie... by binarymax on Tuesday August 04, 2015 @12:05PM (#10004281)
Thank you for this! I'm training word2vec on it right now - will take several hours.

If anyone else is interested here is the (terrible) code to get it into a prototype format. https://gist.github.com/binarymax/d3691180e65ff7f0dec5

Re:Show HN: Download the first 10,002,378 HN comments/storie... by philth on Tuesday August 04, 2015 @01:36PM (#10004893)
Keep us posted on your discoveries. It would be interesting to see how different the embedding is to word2vec trained on a different corpus. I imagine borrowed words like "python" are clustered with programming languages rather than snakes in this case.

As a side note, not really having looked too deeply into word2vec, does word2vec capture multiple meanings? If so, how?

Re:Show HN: Download the first 10,002,378 HN comments/storie... by binarymax on Tuesday August 04, 2015 @03:31PM (#10005667)
All done, results are very promising! Examples are too long so here is one below, and this gist has more: https://gist.github.com/binarymax/6befa448df3f5fd6dba9

        Starting training using file 10m.txt
        Vocab size: 305432
        Words in train file: 565170189
        Alpha: 0.000045  Progress: 99.91%  Words/thread/sec: 107.57k  
        real   174m19.955s
        user   1315m35.661s
        sys    3m27.011s

Enter word or sentence (EXIT to break): startup

        Word: startup  Position in vocabulary: 390

                                                      Word       Cosine distance
        ------------------------------------------------------------------------
                                                  startups      0.808231
                                              bootstrapped      0.719379
                                              entrepreneur      0.707722
                                                    starup      0.698379
                                             bootstrapping      0.698216
                                                 incubator      0.683647
                                                  founders      0.664983
                                                   scrappy      0.660502
                                             entrepreneurs      0.660176
                                           entrepreneurial      0.656120
                                                        yc      0.652160
                                                 cofounder      0.651848
                                                        vc      0.650642
                                                 fledgling      0.636813
                                                cofounders      0.632761
                                                   venture      0.622636
                                                   company      0.617562
                                                incubators      0.612947
                                                    statup      0.608451
                                                   founder      0.608080
                                          entrepreneurship      0.604812
                                                        sv      0.603689
                                                     bigco      0.602171
                                               startuppers      0.592669
                                                 cofounded      0.588964
                                              entrepeneurs      0.585747
                                                      solo      0.582533
                                             entreprenuers      0.564045
                                               boostrapped      0.562884
                                              solopreneurs      0.559994
                                                cofounding      0.559840
                                                   statups      0.558347
                                                  business      0.552922
                                              bootstrapper      0.551885
                                                 techstars      0.545766
                                             bootstrappers      0.545263
                                                   fintech      0.545090
                                                  fundable      0.542542
                                                   shotput      0.541257
                                               accelerator      0.540787
Re:Show HN: Download the first 10,002,378 HN comments/storie... by tilt on Tuesday August 04, 2015 @08:43AM (#10002981)
Re:Show HN: Download the first 10,002,378 HN comments/storie... by cdman on Tuesday August 04, 2015 @08:44AM (#10002992)
Thank you.
Re:Show HN: Download the first 10,002,378 HN comments/storie... by theklub on Tuesday August 04, 2015 @10:49AM (#10003713)
Someone should map the use of tech buzzwords over the years. Would be pretty funny to look at.
Re:Show HN: Download the first 10,002,378 HN comments/storie... by ecesena on Tuesday August 04, 2015 @12:19PM (#10004367)
This could be a good starting point: http://dclure.org/essays/visualizing-the-humanist/
Re:Show HN: Download the first 10,002,378 HN comments/storie... by paulsutter on Tuesday August 04, 2015 @10:14AM (#10003480)
I wish it included upvotes/downvotes. Why are those secret? It would be fun to work on ranking algorithms, and any inc effective requires knowing who is doing the up/down voting.
Re:Show HN: Download the first 10,002,378 HN comments/storie... by ivan_ah on Tuesday August 04, 2015 @08:38AM (#10002944)
> 10,002,378

what date range does this correspond to? How big is the archive?

Re:Show HN: Download the first 10,002,378 HN comments/storie... by cdman on Tuesday August 04, 2015 @08:44AM (#10002988)
It is from story 1 to comment/story 10,002,378 :-)

The archive is 1.12GB big and contains 1 JSON document / per line. The JSON document is approximately the format returned by the official HN API (although there are some exceptions since some of the comments are not available through the official API and those had to be retrieved through the Algolia API and/or scraped from the site).

Re:Show HN: Download the first 10,002,378 HN comments/storie... by ivan_ah on Tuesday August 04, 2015 @08:57AM (#10003070)
Thx. Great job. Now I just have to dust off some LDA code and see some topics...
Re:Show HN: Download the first 10,002,378 HN comments/storie... by cdman on Tuesday August 04, 2015 @08:46AM (#10003007)
Also, about the date range: from October 10, 2006 until yesterday (when HN hit ~10m comments / stories).
Re:Show HN: Download the first 10,002,378 HN comments/storie... by orf on Tuesday August 04, 2015 @09:26AM (#10003211)
Does this include [dead] comments?
Re:Show HN: Download the first 10,002,378 HN comments/storie... by cdman on Tuesday August 04, 2015 @09:31AM (#10003233)
Yes, dead comments were fetched / scraped from the website (so it might not be perfect since it uses regex to parse HTML :p).
Re:Show HN: Download the first 10,002,378 HN comments/storie... by callum85 on Tuesday August 04, 2015 @02:02PM (#10005099)
Which pieces of data are included with each comment/link?
Re:Show HN: Download the first 10,002,378 HN comments/storie... by frou_dh on Tuesday August 04, 2015 @02:20PM (#10005214)
https://github.com/HackerNews/API#items

Also the file is ~5.3GB when decompressed, if anyone is wondering.

Re:Show HN: Download the first 10,002,378 HN comments/storie... by toomuchtodo on Tuesday August 04, 2015 @12:56PM (#10004620)
What license applies to the archive? Creative commons?
Re:Show HN: Download the first 10,002,378 HN comments/storie... by sitkack on Tuesday August 04, 2015 @11:18AM (#10003955)
meta data request: can someone scrape the tracker and provide a log of the all the IPs that participated in the swarm?