News for hackers. Stuff that matters.
Re:Hack: a new programming language for HHVM by teacup50 on Thursday March 20, 2014 @03:58PM (#7438210)
> Congratulations for admitting your ignorance, and lending open ears to experts as to why they made certain engineering decisions.

Yes, they made certain engineering decisions now because the decisions they made back then were stupid, and they have to dig themselves out.

Re:Hack: a new programming language for HHVM by dirtyaura on Thursday March 20, 2014 @04:55PM (#7438706)
What was the stupid decision "made back then"?

That Zuck wrote the first version of thefacebook.com in PHP, the language he was the most productive at?

That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase? If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be. I've personally experienced and heard only horror stories about rewrites. We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.

Re:Hack: a new programming language for HHVM by stormbrew on Friday March 21, 2014 @12:42AM (#7440701)
I once worked for a social network that was extremely popular in a particular geographic niche. It was even started before Facebook.

And it was also written in PHP. And we rewrote in Ruby (not rails, at the time it was not mature in the ways we needed it to be). We made a lot of great technical achievements in doing this and our codebase became much much better. On a technical level I don't regret us doing that one bit. We went long on schedule and we made mistakes, but some of the best work of my career went into that and I'm immensely proud of it.

But then in a matter of months of Facebook going to being open to people who weren't at a university or big organization (remember that?) our users, who got bored with our site not changing while we rebuilt all the tech, completely abandoned us and we went from profitable (having never taken any outside investment) to dead in another couple of years.

This isn't to say that this wouldn't have happened to Facebook had they done something like this, but it is always a huge risk, even if you do everything right.

Re:Hack: a new programming language for HHVM by teacup50 on Thursday March 20, 2014 @05:49PM (#7439072)
> What was the stupid decision "made back then"?

Using PHP to begin with was a bad technical decision. Failing to establish a reasonable migration strategy was a bad business decision likely rooted in bad engineering management that fell out of starting with bad technical decisions.

It's much harder to hire people that can pull you out of a mess like PHP when, at the same time, you have to hire people that can keep writing PHP for you.

> That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase?

That would have been a good time to bring on new engineering blood as part of scaling out, which would have provided opportunities to enact mitigation and transition strategies. Imagine if the massive amount of talent currently devoted to HHVM had been devoted to Facebook's actual business?

There are migration strategies other than "rewrite everything immediately", and in fact, I'd bet that's exactly what HHVM is. It's just a shame they waited so long that the most cost-effective migration strategy was to tackle an enormously difficult computer science and engineering problem that the world's biggest software companies already invest hundreds of millions of dollars on and provide to the world largely for free.

> If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be.

Yes, and I've also (repeatedly) been the team brought in to rewrite the mess of a code base that was about to torpedo the growth phase.

There's not much correlation between funding, initial success, and engineering talent. Which is why you so often wind up with a mess that has to be cleaned up once you can hire people who know what the hell they're doing, instead of the ones you happened to be stuck with because you didn't know how to grow an engineering team.

> We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.

Messiness is never there for good reasons other than that replacing it is more expensive than not touching it. You don't strive for hidden wisdom and inescapable messiness -- that's just what you get when you let engineering slip up.

Re:Hack: a new programming language for HHVM by nbm on Thursday March 20, 2014 @06:12PM (#7439230)
As much as I prefer not to engage trolls...

Which language would have been a good technical decision in 2002-2003? It needs to be fast enough in terms of iteration. It needs to not require more resources than PHP. It must be easy to onboard people who don't know it onto. It needs to be easy to operate, and not be costly to deploy on the tens, hundreds, and then thousands of servers necessary. (Spending time learning a new technology that others think is cool or which seems cool, while trying to also build a product, has sunk more than a few startups...)

What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth. What was bad with the decision to let hundreds of software engineers continue to build the PHP code-base while a much smaller group of people work on projects to improve the efficiency of both the execution environment but also the tooling and developer efficiency on that code-base? Their contribution there is multiplied out by the improved efficiency of those hundreds of developers and the code they wrote.

Seemingly Facebook survived its growth phase fairly well and didn't need a tiger team from outside to handle it - and without the reliability problems others who chose to use languages other than PHP for their front-end and decided to rewrite their much smaller surface area in other languages.

As much as people may dislike PHP (and I'm one of them), it was definitely "good enough". Many languages may not have been, even if they are nicer languages in some objective way.

(Disclaimer: I actually have to write code in Facebook's PHP code-base every once in a while. But most of the code I work with is Python, Java, or C++.)

Re:Hack: a new programming language for HHVM by teacup50 on Friday March 21, 2014 @12:27AM (#7440663)
> Which language would have been a good technical decision in 2002-2003? It must be easy to onboard people who don't know it onto. It needs to be easy to operate, and not be costly to deploy on the tens, hundreds, and then thousands of servers necessary.

You mean, like the JVM? 2003 wasn't the pliocene epoch, we had a working JVM. If you remember back to the last bubble, in the 90s, we were shipping "easy to operate, not costly to deploy" software on Java post-1998. Java 1.4 was released in 2002, and Java 1.5 -- what most people would say is modern Java -- was only 2004.

Scala 2.0 was released in 2006, Clojure in 2007 -- that's 8 and 7 years ago, respectively.

You really don't think there were alternatives during that long period?

> What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth.

In 2004? Nothing. In 2005-2008? Things should have been reassessed, especially before building out a millstone of an engineering team around PHP. Instead, Facebook doubled-down on an actively bad language with HPHP, and the results were hilarious:

   HPHPc required a very different push process,
   requiring a bigger than 1 GB binary to be compiled
   and distributed to many machines in short order.
So then in 2010, Facebook decides to embark on HHVM, and now four years later, we can run one of the most correctness-hostile programming languages around, quickly, with optional static typing.

That's a span of 6 years, and at the end of it, Facebook has functionality they could have gotten for free in 2003. On top of that, the intervening years allowed the PHP mess to become only more entrenched -- who on earth do you think the engineers are that accept a job writing PHP, for Facebook or otherwise?

If I had to hazard a guess, I'd guess that HHVM exists because of a large amount of political inertia in the organization that has everything to lose by PHP being eliminated entirely, and the lack of a strong hand by upper management.

I'd guess that lack of a strong hand by upper management came in no small part from hiring straight-out-of-college graduate Adam D'Angelo -- who had literally zero experience -- to serve as CTO from 2006-2008.

By the time FriendFeed was acquired and Bret Taylor along with it (2009), my guess would be entrenched interests made for a very difficult position for anyone wanting to change the ship's course.

Re:Hack: a new programming language for HHVM by Lennie on Friday March 21, 2014 @04:16AM (#7441198)
"the intervening years allowed the PHP mess to become only more entrenched"

They specifically choose to do it and not move to an other language. One of the reasons was: PHP programmers are cheap and plenty full and can do quick iterations.

Sounds a bit like you are complaining about other peoples choices, it really is their choice. :-)

I'm not saying it is possible to move to an other language. Just look at Paypal they moved their customer-facing code from Java to node.js and got a very large productivity increase: http://www.youtube.com/watch?v=V5yk5SZxWX4

Obviously the reason Paypal choose node.js are similar to why Facebook choose PHP: quick iterations, means more iterations, which means more experimentation and better results.

Re:Hack: a new programming language for HHVM by vidarh on Friday March 21, 2014 @06:09AM (#7441453)
> You mean, like the JVM?

In 2002-2003 I'd have quit in disgust if anyone tried to introduce the JVM anywhere I worked.... I still probably would, frankly.

Re:Hack: a new programming language for HHVM by bluesnowmonkey on Thursday March 20, 2014 @11:12PM (#7440429)
> Using PHP to begin with was a bad technical decision. Failing to establish a reasonable migration strategy was a bad business decision likely rooted in bad engineering management that fell out of starting with bad technical decisions.

Facebook is a ten year old company with a market capitalization of $170 billion, so "bad" is probably not the most accurate word to describe their technical and business decisions.

Re:Hack: a new programming language for HHVM by teacup50 on Friday March 21, 2014 @12:09AM (#7440602)
> Facebook is a ten year old company with a market capitalization of $170 billion, so "bad" is probably not the most accurate word to describe their technical and business decisions.

How does that follow, exactly? They haven't failed, so any inefficient or sub-optimal decisions were the correct decisions?

Re:Hack: a new programming language for HHVM by curiousquestion on Wednesday March 26, 2014 @09:19PM (#7477927)
I guess we should consider your decisions as the "optimal and correct decisions"? Are you a billionaire too?
Re:Hack: a new programming language for HHVM by reikonomusha on Thursday March 20, 2014 @06:56PM (#7439456)
A rewrite isn't such a ludicrous idea. Reddit is a prime example of a rewrite from Lisp to Python. I would say that's even a somewhat difficult rewrite.
Re:Hack: a new programming language for HHVM by nbm on Thursday March 20, 2014 @07:41PM (#7439694)
The Reddit front-end is a relatively simple application. It has a page with a list of stories. It has a page with one story and a bunch of comments. It has a page to add comments. It has an endpoint to vote up or down stories, and to vote up or down comments. And maybe a few other less-interesting things, like admin interfaces.

Facebook is surprisingly easy to underestimate even as someone using it a fair bit.

Just try find every single interface in the front-end. For your own profile. Feed is front-and-center, as is timeline. Then look at events, groups, &c.. Look at messaging.

Then look at the interfaces for managing your privacy and permissions. The apps that you're using and information about when last they requested information. Security like login approvals. Then think about the flows involved in reporting content as abusive or inappropriate. For verifying your identity if you've forgotten your password. For adding more security if you log in from a new computer or from a new location.

Then look at pages, including insights and scheduled posts and so forth. Then look at advertising - boosting individual posts, creating campaigns, &c.. Then look at the interfaces for developers. For translators. Interest lists.

The backends for most of this are in C++ and Java. There's a large amount of data processing happening to track hidden things like spam and scam prevention. But the front-end surface area is quite clearly an order of magnitude or maybe two larger than Reddit (at least where it was when this happened).

Re:Hack: a new programming language for HHVM by dirtyaura on Friday March 21, 2014 @09:28AM (#7442113)
Okay, Reddit is a good example of a successful rewrite during a growth phase. I had forgotten that. As I've personal knowledge of several unsuccessful rewrites, it would be great to hear more anecdotes about successful rewrites in growth phase and try to analyse what made them succeed.

Rewrite is more likely to succeed if you are not in a high growth phase, but even then it's risky.

Re:Hack: a new programming language for HHVM by gfodor on Thursday March 20, 2014 @04:11PM (#7438324)
Yes and the OP is criticizing the decisions being made now as "yak shaving." Could it perhaps be that the "yak shavers" made a conscious and well-reasoned decision to go in the direction of "extend PHP" vs "throw it all out and rewrite everything in language-of-the-month?"
Re:Hack: a new programming language for HHVM by teacup50 on Thursday March 20, 2014 @04:44PM (#7438612)
I'm sure it was conscious and well-reasoned, but the OP's point still stands.

Trying to build a reasonable forward-looking high-performance managed language runtime platform is hard.

Trying to build it atop of PHP is harder.

The only way I could see this as being a smart long-term strategy is if the eventual goal is to isolate and retire PHP projects entirely (and PHP usage itself) over time.

However, even then, with PHP gone, and Hack no longer necessary, you're still stuck maintaining your own incompatible VM / runtime. Is Facebook signing up to reproduce the CLR? Or do they have long term plans to somehow bridge the gap between HHVM and more established VMs/runtimes, where they can better share the maintenance load with the wider industry.

Re:Hack: a new programming language for HHVM by sanxiyn on Thursday March 20, 2014 @09:42PM (#7440148)
Maintaining your own VM is no big deal. Compared to Facebook, HHVM is a tiny codebase. A team made of relatively small number of people (high quality, but low quantity) can and do maintain VMs like HotSpot and V8. LuaJIT is maintained by a single person.
Re:Hack: a new programming language for HHVM by teacup50 on Friday March 21, 2014 @12:05AM (#7440587)
> Maintaining your own VM is no big deal.

As someone who has worked on a VM, I couldn't disagree more. It can take years to hash out things as simple as ideally performing primitives for a target architecture, and then things change.

Add to that the complexity of optimizing compilers, specification of byte code formats and a consistent virtual machine memory model that can be relied upon across architectures, and the art and science of highly concurrent garbage collectors, and your "no big deal" is a load of hogwash.

Hotspot alone is nearly 20 years of big deal.

> A team made of relatively small number of people (high quality, but low quantity) can and do maintain VMs ...

The number of people doesn't matter in this equation; your small team of (expensive, rare, high quality people) can't build a world-class VM in a day. Or a month. Or a year. Maybe in 5 or 10 years, just ask Microsoft.

> ... like HotSpot and V8. LuaJIT is maintained by a single person.

LuaJIT's said "single person" has been working on it for what, 10 years? It's an extremely impressive implementation and I don't want to bag on it, but even still, it lags in certain areas, eg, its GC implementation isn't up to par with the state of the art.

The author's skillset is extremely rare, and LuaJIT itself is an anomaly in the field. Using such a one-off example doesn't really hold water to prove that it's ideal for a company to internalize maintaining a VM for their own custom language built on top of PHP.

Re:Hack: a new programming language for HHVM by sanxiyn on Friday March 21, 2014 @12:56AM (#7440735)
I am not trying to belittle efforts necessary for the state of the art VM or programming language implementation. I get paid to do these stuffs, and I am on my third VM/PL project now. It is also true these things take time and not very parallelizable, so while man-month may not be that big, you can't make it faster by throwing more people.

On the other hand, I maintain it still is no big deal compared to rewriting Facebook. I also maintain while skillset is rare, Facebook apparently had no trouble so far and will have no trouble in the future finding (I remind you, small number of) people to work on VM. I also remind you Facebook has been working on alternative PHP implementation for 6 years now, 2 years in private(2008~2010) and 4 years in public(2010~2014). It has been profitable for them for 6 years, will be profitable in the future, and profitability does not need "sharing maintenance load with the wider industry". They can maintain it fine thank you very much. Because, in the end, VM is no big deal.

Re:Hack: a new programming language for HHVM by teacup50 on Friday March 21, 2014 @02:49AM (#7441000)
If they're wasting money on bad management decisions, they're wasting stockholder money.

They're also continuing to propagate an outwardly facing engineering culture that will make it even harder to hire people to help dig them out of the PHP hole -- perpetuating this further.

Your argument is simply another take on survivor bias fallacy.

> I get paid to do these stuffs, and I am on my third VM/PL project now ... Because, in the end, VM is no big deal.

You keep saying that, and yet, there keep being so few high quality VMs.

Re:Hack: a new programming language for HHVM by sanxiyn on Friday March 21, 2014 @04:22AM (#7441214)
What do you consider to be high quality VM? How many do you expect to see and how many do you find?

Adaptive JIT and generational GC would be a good baseline. Limiting myself to open source VM, I think (at least) HotSpot, Mono, V8, JavaScriptCore, PyPy, SBCL, Racket qualify. J9, CLR, Chakra, Allegro CL also qualify, but not open source. SpiderMonkey, LuaJIT, HHVM lack generational GC. All these projects are actively developed, and there are doubtlessly more, e.g. I am not faimilar with Smalltalk VM, some of which are commercial. Research VM like JikesRVM, Maxine qualify. I believe Bartok qualifies too.

I am not sure what you are arguing for. If you are arguing for Quercus route(PHP-on-JVM), I think it's unclear Quercus route is better than HHVM route. If you are arguing for not running existing PHP codebase, I think you are being unrealistic.

Re:Hack: a new programming language for HHVM by gfodor on Friday March 21, 2014 @07:25PM (#7445653)
It's not survivorship bias. Facebook is an existence proof that there is no "PHP hole" that they are in, that it's largely a myth propagated by programming language nerds who have never tried scaling a site in PHP. When was the last time you heard about a site closing up because of PHP-induced technical debt? You don't. People re-write sites because of poor architecture, not because of poor programming languages, and PHP (in general) does not prevent you from building a site with good architecture, both from a software structural standpoint and an operational standpoint.

PHP's APIs are ugly. It's language semantics are a bit hairy until you get the hang of it. But there are parts of PHP that are extremely elegant and easy to reason about. It's OOP support provides all that you need to produce re-usable and easily understood code.

Facebook's work on PHP has focused on largely two dimensions: reducing CPU cycles and increasing static/runtime type checking. The former is something that only really matters at massive scale: PHP is generally fast enough since most of the time PHP processes are I/O bound reading from a database or memcached. It's only for sites like Facebook where if you squeeze out an additional 10% TPS from your boxes that you will start seeing large absolute cost reduction that this level of optimization starts to make sense. On the type checking side, this is something you might start to want in any dynamically typed language when you have millions of lines of code and want to ensure basic guarantees that it will run, and is something that you'd probably see Facebook doing if they were a Ruby or Python shop anyway. It has nothing to do with PHP but with the classic dynamic vs static typing tradeoff.

Should you be writing your chat server in PHP? No. But 90% of the code you write for a large website is HTTP response code rendering HTML or JSON. PHP excels at this and you can pretty much hire any developer off the street to start cranking out code if you give them a solid foundation to build on.

Re:Hack: a new programming language for HHVM by vidarh on Friday March 21, 2014 @06:05AM (#7441443)
Facebook has already proven that they are able to make improvements that have substantially helped them to the point where this team is likely paying for itself many times over. It doesn't need to be perfect - it needs to offer return on investment, and it has.

It's possible that they could eventually get a total rewrite to give a better return, but frankly I don't think you have any idea of the enormity of trying to convert a multi-million line production platform from one language to another.

In any case, one does not preclude the other. Arguably, many of the changes they have made, such as gradual typing, and their ability to now slowly introduce other changes without breaking their existing codebase, means they have a platform for slowly firming up their codebase and migrate it towards a position where a full rewrite (should they decide to do one in the future) could be made substantially less painful.

Re:Hack: a new programming language for HHVM by reikonomusha on Thursday March 20, 2014 @06:40PM (#7439376)
My example, "OCaml", is not a "language of the month." Its roots are >30 years old and wasn't developed by someone in their basement over the weekend. As stated, Facebook even uses OCaml, among other languages, for good reason.
Re:Hack: a new programming language for HHVM by vidarh on Friday March 21, 2014 @05:58AM (#7441429)
You try hiring and/or retraining enough engineers to be able to make a switch to OCaml, and see how much it'll cost you.

I detest PHP, but I've still more than once made the choice to do apps in PHP motivated by developer availability alone.

It's not a great language, but with some discipline it is also not nearly as awful as some people like to think, and you can make up for a lot of awful with the difference in ability to hire experienced engineers who know PHP vs. many of the less common languages.

Re:Hack: a new programming language for HHVM by codygman on Saturday March 22, 2014 @01:34PM (#7450003)
I don't like how your comment implies that "throw it all out and rewrite everything in language-of-the-month" is the only option. You could also "throw it all out and rewrite everything in better-AND-mature-language" or even "throw the worst parts out and rewrite incrementally in better-AND-mature-language".