As for their actual infrastructure: not having a version control system and a build system (e.g.: never deploying directly from the VCS) is a must for any size team and sometimes even for single developers. Thankfully, nowadays most people do use VCS's but build and deploy systems are largely at the level of glorified shell scripts at best.
Management is not excluded from this either. They have to:
1. know, in depth, the business they are in and the tools they use (or could use) to achieve their goals
2. listen to what their team tells them and provide an environment where they can do the job right
It sounds like there was a lot of micro-management and putting out fires in this case.
For the record, I think it's irrelevant.. but absolutely I think it's true. It doesn't mean .NET isn't capable, or that lots of .NET programmers aren't capable...
Just that .NET is way more popular in the enterprise than on the web, and that the work that gets done at most of those enterprise shops wouldn't fly on large sites.
It may be popular to pretend that the .NET camp (and to a slightly lesser extent, the Java Enterprise one) doesn't bend this way statistically, but they do.
It is a fallacy to think that the web is the only large network that requires scale.
Goldman Sachs alone stores more data than the entire web
The Visa network has had 4 seconds of downtime in decades
Most airline systems took decades to engineer
Payment processing only deals with one type of data: money. This gives you as many shortcuts as the number of constraints it imposes.
Airline control and departure systems are probably as close to a typical modern web app as you'd get from your list.
My point is that while there certainly are engineers that have worked with high scalability issues without ever touching the web, they have also likely been solving slightly different problems.
P.S.: Other systems that require high availability but are not the web: telephony and cell communications, broadcasting and doomsday devices.
Edit: Oops, here's the link: http://www.forbes.com/global/2002/0916/038.html
Goldman Sachs told me when I was writing a proposal for them years ago that they have over a petabyte of data stored. The web is 80-200TB depending on who you ask. A single department there responsible for the program based trading would alone have an entire copy of the web (and parts of the deep web, and all of twitter, etc.) since they construct those whack trading apps that suck everything up and analyze it for signals. If there are any quants on here they could tell you about this more.
The airline system I was referring to is SABRE. Early IBM was built on their rollouts and we are talking about the 50s. V interesting story, lots of references from this page: http://en.wikipedia.org/wiki/Sabre_(computer_system)#History
It would be interesting to get more recent figures and compare again, because I do know that the investment banks hoard a lot of data.
In terms of just documents, they would easily store more than the web
(it was ~10x the web at the time)
Why would you believe that Goldman Sachs stores more data than can be reached by HTTP?! Only YouTube amounts to petabytes of data. http://beerpla.net/2008/08/14/how-to-find-out-the-number-of-...
the point is that it is frikkin big - and most investment firms now scan the entire web for signals
startups don't have a monopoly on working hard or working fast, and it is arrogant to generalize about both .NET and enterprise developers in that way since we are all in one way or anther standing on the shoulders of earlier enterprise work (where do you think what we call 'nosql' and think is new and grovvy, was first used?)
Second, arrogant really isn't fair because I certainly never gave an assessment of my own abilities or value.
Third, totally.. .NET stands on the shoulders of the same stuff the infrastructure that runs most of the internet stands on. It's just that .NET doesn't run most (or even lots and lots of) the internet.. so to suggest that the .NET development community is less likely on average to be ready to build Facebook doesn't seem like blasphemy.
That's very different from suggesting the .NET camp isn't full of awesome, hard-working developers... but really, I apologize if it comes off that way. Definitely don't mean to suggest it in the least.
Most of my familiarity with the .NET community comes from the fact that I live/work in San Diego, and in the Healthcare IT space... if the combination of which don't make up the largest percentage of .NET developers in a particular area and arena, it must be close :)
So most of my experience with the .NET community comes from knowing developers and working with .NET-turned-Java folks... It's true, I haven't done .NET in a very long time. That said, I made the same assertion (to a slightly lesser extent) about the Java Enterprise camp.. and I've got loads of experience with that :)
Ruby developers? (Ref: Twitter)
Java developers?
I think only HTML programmers are the only ones who are constitutionally constructed to create large scalable websites at a startup pace.
Also, it's probably fair to say that as a whole programmers are 'largely' not used to thinking or working in a way that's productive at scale. I didn't argue otherwise.
I just suggested that the percentage of developers in the .NET community who are aligned with values, knowledge, and experience essential to scaling a large site.. is smaller than many other platforms... Yes, I'd argue smaller than all of the ones you mentioned.
Exactly. This seems to follow the Kevin Rose formula: make horrible decisions (or be entirely absent from decision making) without any understanding of technology, then blame your developers and technology choice ("Digg V4 failed to due to Cassandra").
If it weren't for Facebook's success, I can bet you'd see people blamining PHP for Digg's failure: prior to Facebook, P in LAMP also stood for Perl (and occasionally Python) and LAMP wasn't "universally" considered proven (unlike J2EE + Oracle or .NET + SQL Server), nor has Facebook been even remotely close to a "vanilla" LAMP site (since at least 2005)-- with many mission critical subsystems also being built in Java and C++ .
The P in LAMP has been associated with PHP for much longer than the existence of Facebook.
http://plentyoffish.wordpress.com/2007/02/09/aspnet-and-iis-...
I think he thinks it's a strategic advantage to understate the work and resources he's invested into pof. (Makes for a good marketing story and newbies think building a pof clone is easy so they waste lots of money trying.)
Right about the time that blog post was written peer1 had a marketing video on their site where there was a guy walking about a data center being interviewed by someone off-camera. As they walked around the guy waved at a few racks of servers and said these are plenty of fish's servers, walked further, these are [some other big web site].. And from that video it was clear there was no way he was only using 1 server. There were lots of servers in use.
I find it sad that these days MS is laggin behind. It is difficult to get Cache frameworks and other infrastructure working with MS Stack if you got a high performance website.
The Microsoft stack is not always just a Microsoft stack any more. e.g. it has jQuery out of the box. Of course you mix and match when you get to the high end. The idea of the MS-only shop is not as true as it used to be, many people are more pragmatic.
I find it sad that these days MS is laggin behind
Why, because MS didn't supply every single piece of server infrastructure software that So use, just the main ones? That's an odd definition of "lagging".
MS have a distributed cache called "Windows Server AppFabric" (formerly "velocity"). I don't know that much about it.
See I prefer sticking with one flavor of tools because its easier for developers to adjust. TBH MS does supply almost everything from grounds up. I only had to look elsewhere for advanced distributed caching frameworks. In fact before switching to Amazon EC2 our old datacenter was running MS VMM and our stack still didn't have anything other than MS software.
Where the open source choice is more functional, cheaper or just more familiar, it often gets used instead. This is good.
How so?
Connecting OSS tech and MS stack is getting easier but still leaves you with a lot of uncertainty.
This is an example: http://wiki.basho.com/Client-Libraries.html These guys have Erlang client but there is no support for .Net.
Cache framework's are absolutely necessary. The whole idea is to avoid a SQL Server hit and return a cached data object in memory. I think Stackoverflow is the best case study here.
SQL Server is the third best Microsoft product. Right after their natural keyboard and their mice lineup.
Remember when people used ASP.Net web forms and it was hard to get rid of viewstate in the rendered page, and only the people who knew internals of platform well could fix it. It drove SEO's mad and they started to recommend staying away from web forms.
I agree with you on the staging and testing servers. Not having them is planning for disaster.
I find this shocking and somewhat unbelievable. My wife, who isn't a tech person at all thinks that page load speed matters (she just called so I asked her) -- and sites gmail as an example of a page that takes too long to load (I think that "loading..." indicator actually brings their load time to the forefront, although it really isn't that long).
I just have trouble seeing a .NET developer saying "for high volume sites page load speed doesn't matter", when most people who aren't in the tech industry would concede that it does.
Most .Net developers are working on intranet sites. They don't have problem with large footprint pages.
When they move to internet and public domain websites, they are newbies. It takes them a while to adjust to the way internet sites are written. SEO optimizations, CDN usage, Ajax calls etc are pretty important on internet site than on intranet site.
Imagine using a Update Panel to code for an internet website. It would create enough junk javascript to delay a page load but it works fine on intranet. To fix this jQuery or some other Javascript framework must be brought in.
Companies like 37signals are doing it successfully ;)
IMO, that's a very different statement. One is a difference in experience with a domain, the other is ideological.
I've read before that Facebook has shown that users tend to spend a fixed amount of time on their site. Once users hit that time limit, they're done. If your site exhibits similar usage patterns, the faster your pages load (even if they're already fast), the more users can get done on your site, which, depending upon your revenue model, may result in more revenue.
For example, MySpace was getting at one point 24 billion page views per month. If you could reduce each page view by 1/100th of a second you save 40 weeks per month for your users (assuming the model where they look at a fixed number of pages).
In the Facebook model, if you assume a page comes up in half a second, this delay results in a 2% decrease in page views -- which is a pretty huge deal when your business model indirectly revolves around page views.
I guess my point is that even for people who come from a background where page load didn't matter. it would take 30s to point out it does, and I don't think you'd get any pushback.
that sounds big and important, but it doesnt really mean much does it?
the difference between a 1/100th of a second and 2/100th's of a second is too small to translate to enough time to get any increase in productivity.
There must be someother reason that load time is that crucial.
You're conditioned to not care anymore, because ASP.NET Web Forms makes it so damn hard to achieve responsive web apps, and building responsive web apps almost universally means breaking away from the standard ASP.NET Web Forms style of development. You have to abandon pretty much everything that makes the platform convenient in order to get good performance.
MVC however is designed from grounds up to deliver speed and web standards compatibility. Building for web these days with Web Forms is just wrong. But yes, most to actually bend Web Forms to deliver requires wizardry.
oh and that viewstate forms thing from SEO is the SEO talking bullshit and trying to justify his job
I had the same reaction to SEO at first. See SEO guy is partially right. It involved page load times. Larger viewstate means longer load time and that gets penalized by Google.
Have you informed them that you work with a high traffic site? Most intranet sites are not high traffic, so if they're spending time pre-optimizing for speed instead of features/development time, they're actually wasting the company's money.
And what has viewstate got to do with SEO? You lost me.
For SEO friendly URLs, all you need a is URL Rewriter. http://www.google.com/search?client=opera&rls=en&q=a...
>only the people who knew internals of platform well could fix it
That's true of any platform out there.
Yes I talk to them that we have a high traffic site and we lose them because they don't know about building them. Besides you won't find many Microsoft stack sites dealing with scaling issues. I personally looked up Stackoverflow scaling case studies to design my solutions later.
Back in 2005/6 URL rewriting was still pretty difficult. I started out back then. I converted to MVC completely when 1.0 version came out specifically to address the SEO concerns.
But I guess our discussion is swinging to technical side :)
Edit: See sajidnizami's post for the relevant StackOverflow question. Though, this still doesn't make sense to me logically. Why would a search engine disregard a (reasonably) longer page?
http://googlewebmastercentral.blogspot.com/2010/04/using-sit...
And that, ladies and gentlemen, is why you don't want your software project managed by non-programmers. Any competent programmer would know not to do that.
Maybe they should have used Azure. At least that forces them into a staged deployment. (of course not that it was around when they started MySpace).
critical word there: largely. he wasn't saying all. he was making a generalization comparing large groups. individual cases may vary.
Yeah, that made me cringe too.
This is such BS, I can't even read it without physically cringing. I work for a ~400-person business that stands up .NET websites at breakneck speed, and we do it well. People who blame their problems on technical infrastructure decisions almost ALWAYS do so because it's easier than addressing the true underlying problems.
"The biggest problem was they didn't allow the developers to have staging or testing servers -- they deployed on the production servers on the first go-around... And all this with no change management or control. No versioning either."
Oh wow. Wow. I hereby revoke my previous statement. These are some God-awful infrastructure decisions. Version control and a staging server are the most basic necessities for a scalable dev project. I even set them up when I'm working on a personal project, alone.