News for hackers. Stuff that matters.

A comment on Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine.

Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by aargh_aargh on Wednesday September 27, 2017 @06:39AM (#15346433) 

  $ cloc-git https://github.com/vespa-engine/vespa.git
  http://cloc.sourceforge.net v 1.60  T=64.10 s (224.5 files/s, 28276.3 lines/s)
  --------------------------------------------------------------------------------
  Language                      files          blank        comment           code
  --------------------------------------------------------------------------------
  Java                           6573         106215         102720         537097
  C++                            3209          76542          19178         504855
  C/C++ Header                   2985          42731          57087         158388
  XML                             389            705            550         139626
  Maven                           141            133            244          14096
  CMake                           450            254            560           8452
  Perl                             57           1124            762           7649
  Bourne Shell                    196           1257            734           6918
  Scala                            95           1685            617           6378
  Teamcenter def                  234           1474           3490           2468
  Lisp                              4            231            403           2118
  HTML                             16            211             29           1950
  C                                 7            288            198           1432
  Python                            6            132             66            556
  Ruby                              9             39              9            294
  Bourne Again Shell                3             35             12            182
  Pig Latin                         9             39             52             54
  make                              2             22              8             39
  Ant                               1              9             17             36
  YAML                              1              9              1             22
  DTD                               2              6              6             10
  --------------------------------------------------------------------------------
  SUM:                          14389         233141         186743        1392620
  --------------------------------------------------------------------------------
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by zimpenfish on Wednesday September 27, 2017 @07:14AM (#15346617)
As an ex-employee, there could not be a better description of Yahoo! development than this.
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by praneshp on Wednesday September 27, 2017 @12:08PM (#15349198)
Hey, there is no yinst/buildyblocks stuff here. That's kind of a must-have.
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by zimpenfish on Thursday September 28, 2017 @04:07AM (#15355568)
> yinst

[FLASHBACKS]

To be fair, yinst was the least worst part of the systems I was working on (Yahoo!Europe backend feeds stuff.)

[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by zymhan on Wednesday September 27, 2017 @12:32PM (#15349466)
Can you elaborate?
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by TallGuyShort on Wednesday September 27, 2017 @03:24PM (#15351370)
Even split between Java / C++, with at least 4 different build systems: 2 of which are effectively the next generation of the other 2. 2 different UNIX shells and 5 other dynamic programming languages. Not the end of the world, and there are probably good reasons for a lot of it (maybe there are bindings for various languages, maybe some of it is misindeitified, maybe they're harnessing 2 large bodies of existing work, 1 in Java and 1 in C++), but it may suggest a lot of people just doing things their own way trying to carve out their own niche, without a cohesive philosophy across the system, which mirrors my experience with Yahoo's engineering org.
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by RealJon on Wednesday September 27, 2017 @06:12PM (#15352802)
No, this is run by a single, very cohesive, very remote team.

There is some misidentification in your list (our .def files are nothing to do with Teamcenter). We use 2 languages because we Java and C++ have different strengths which make each suitable at different layers of the architecture. The rest is a combination of "for good reasons" and "leftover scraps" :-)

[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by TallGuyShort on Wednesday September 27, 2017 @11:26PM (#15354321)
Yeah I ran cloc on a few projects I'm very familiar with, and shell scripts that are interpreted with bash but not with a shebang because they're libraries sourced by others are called Bourne too. Just explaining what OP was getting at :)
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by tgb on Wednesday September 27, 2017 @07:20AM (#15346653)
Pig Latin?
[ Parent | on HN ]
Re:Open-Sourcing Vespa, Yahoo’s Data Processing and Serving ... by andreer on Wednesday September 27, 2017 @07:28AM (#15346695)
That would be Apache Pig, a scripting language for Hadoop. Some scripts are included to feed data to and query Vespa from Hadoop.

https://github.com/vespa-engine/vespa/tree/master/vespa-hado...

[ Parent | on HN ]