News for hackers. Stuff that matters.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by habosa on Monday July 21, 2014 @01:20PM (#8064748)
> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune.

I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by AaronFriel on Monday July 21, 2014 @01:43PM (#8064875)
There are a few erroneous replies to your comment, re: Windows support for that much memory.

Memory support for versions of Windows:

                           	  User-mode process	   Kernel-mode
                           	    address space  	  address space
                           	  (process bit-ness)
                           	32-bit        64-bit	        all
  Server Operating Systems:	                    	
    Windows Server 2008 R2 	  4 GB*      2048 GB	    2048 GB
    Windows Server 2012    	  4 GB*      4096 GB	    4096 GB
    Windows Server 2012 R2 	  4 GB*    131072 GB	  131072 GB
  Client Operating Systems:	                    	
    Windows 7 (x86)        	  3 GB*          n/a	       2 GB
    Windows 7 (64-bit)     	  4 GB*       192 GB	     192 GB
    Windows 8 (64-bit)     	  4 GB*       512 GB	     512 GB
    Windows 8.1 (64-bit)   	  4 GB*    131072 GB	  131072 GB
  
  
  * requires IMAGE_FILE_LARGE_ADDRESS_AWARE flag, otherwise 2GB
  ** All units use binary exponents (GB = 2^30 bytes)
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by yuhong on Monday July 21, 2014 @05:14PM (#8066041)
process address space != physical memory
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by claudius on Monday July 21, 2014 @01:27PM (#8064790)
Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :)

The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc., but apart from these, mostly everything seems “normal” to me.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by tekacs on Monday July 21, 2014 @09:40PM (#8067060)
> e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc.

this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by bradyd on Monday July 21, 2014 @01:31PM (#8064812)
I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0].

[0] http://www.dell.com/us/business/p/poweredge-r920/pd

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by valarauca1 on Monday July 21, 2014 @03:51PM (#8065635)
Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by jaequery on Monday July 21, 2014 @10:52PM (#8067200)
Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by kondro on Monday July 21, 2014 @11:35PM (#8067279)
The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by valarauca1 on Tuesday July 22, 2014 @09:53AM (#8068963)
$1300/month w/ a 3 year commitment is $46.8k over the lifespan. Which is more then 4x more then the actual hardware cost. Your ROI on a physical server with the same stats at that price tag would be 8 months.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by habosa on Tuesday July 22, 2014 @02:07PM (#8070574)
Not saying its totally worth it, but the upside is that they take care of power costs, physical space costs, networking, and if the hardware fails they'll put your application on a new server. Basically you take all of the risk out the equation.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by egwor on Tuesday July 22, 2014 @03:21PM (#8071056)
Sure, but add power and cooling for these things plus physical space? (I remember when these big db's needed the floors in the data centre reinforcing since they were a > metric ton )

Still, I agree re. the difference in price. I'd buy too

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by Nick-Craver on Tuesday July 22, 2014 @06:23AM (#8068186)
Worth noting: AWS also doesn't have Enterprise SQL available, so plan on buying that yourself. That's a not-so-insignificant piece of the overall cost to many people.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by superuser2 on Tuesday July 22, 2014 @02:05AM (#8067574)
How common is it to buy one extremely powerful database server? Wouldn't you need the high availability properties of something like a Galera cluster? Or are most people accepting the single point of failure at huge database server?

(I honestly don't know, it's just surprising.)

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by emilv on Tuesday July 22, 2014 @03:00AM (#8067668)
While you do want HA you also want to keep things simple. You get great HA with just one backup machine. The times you lose two machines are so few that you might crash the site with software failures far more often anyway.

Databases have inherent locking problems that takes more resources to resolve with more machines, more so in some database systems than others. When scaling you often hit a point where you get down to macro logistics, so imagine a car highway: You can get more throughput by adding more lanes. But what that all cars are going to merge to one lane at some point in the middle of the trip (because of a tunnel, bridge or something: it's impossible to have more than one lane at that point)? Now you won't get any throughput benefits of the multiple lanes after all! Just more latency because you get queues up until the single lane and because of the queues all cars are going slow and need to accelerate on the single lane, so the average speed is low too. You are also having too much resources after the single lane because you can never fill all those lanes. It might be better to have one beefed-up single-lane road all the way that people can go fast on. Basically: Remove locks and you get better overall performance.

Yes, this is on the expense of HA. Yes, the costs of scaling up grows asymptotically faster than the costs of scaling out. So this is definitely a trade-off in some sense.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by jtseng on Friday August 01, 2014 @07:34PM (#8123402)
They do have a cluster: http://www.brentozar.com/archive/2012/09/microsoft-sql-serve...

They just run everything through one live database all the time under normal conditions. Personally, I think this is much simpler to manage than to be regularly spreading out queries over many queries.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by EvanAnderson on Monday July 21, 2014 @01:33PM (#8064819)
Windows Server handles that much RAM fine, and, so long as it fits, SQL Server will end up caching the entire database in RAM.

If I remember correctly they're using Dell R720's for their SQL Server machines. At current Dell retail pricing 384GB of RAM would be about $5,900.00.

At that price I'd vote for throwing money at raw hardware and reaping the benefits of caching the database in RAM, versus trying to pay programmers to come up with clever techniques to deal with having less RAM.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by x1798DE on Monday July 21, 2014 @01:26PM (#8064782)
It's not a server, but I've seen machines used in physics simulations have something on the order of 256 GB RAM in them. As far as I know they cost maybe $20k-$30k, so not absurdly expensive.

I don't think addressing that memory is a problem on any x64 architecture. Windows Server 2012 has a 4TB upper limit on RAM, apparently [1], and as far as I can tell there's no reason for this other than product differentiation between different "levels" of the operating system.

1. http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by mrb on Monday July 21, 2014 @01:51PM (#8064915)
$4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160.

Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months.

A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no funding, and you are good at assembling and troubleshooting computers, and are ok with mere warranty on parts as opposed to full vendor support from HP/Oracle/Dell, go for it. That's how many startups started (see Google and their makeshift half-donated half-loaned boxes in 1998).

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by ZachPruckowski on Monday July 21, 2014 @03:06PM (#8065416)
Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by adventured on Monday July 21, 2014 @07:44PM (#8066692)
AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high.

10tb = roughly $1,000 per month

They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by dekhn on Monday July 21, 2014 @09:27PM (#8067030)
You can negotiate with AWS.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by liotier on Tuesday July 22, 2014 @08:25AM (#8068535)
I don't believe that startups in their garage days have anywhere near the necessary leverage to negotiate anything with Amazon...
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by jaequery on Tuesday July 22, 2014 @09:41AM (#8068901)
Agree, not many people actually realize just how slow and expensive AWS actually is. Kudos to SO for going with their own hardware.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by dekhn on Tuesday July 22, 2014 @10:19AM (#8069094)
they negotiate, still. I know several startups which went to amazon, said, "hey, your competitor is cheaper, what is your value prop" and suddenly had 50% discounts, etc. It's amazing how much you get simply asking!
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by lnanek2 on Monday July 21, 2014 @02:12PM (#8065069)
Even before OS'es supported that much people were sticking that much RAM on expansion bus cards and plugging them in and treating them like fast disks. So it is going to happen either way. The performance boost for the DB is just too much compared to it thrashing disks constantly and there are quite a few benefits to having fewer machines compared to many DB servers as well.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by daigoba66 on Monday July 21, 2014 @01:38PM (#8064847)
384 GB is fairly easy to achieve today with 16 GB DIMMs x 24 slots (many general purpose servers have up to 24 DIMM slots). Beyond that you need to jump to 24 GB DIMMs which are fairly expensive.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by buckbova on Monday July 21, 2014 @06:23PM (#8066372)
SQL Server only reads data from memory. When something is requested it is first looked for in RAM (or virtual) and if not found is copied from disk to RAM.

The more RAM you have the better without exception here.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by snowwrestler on Monday July 21, 2014 @01:31PM (#8064813)
I believe the idea is to hold as much of the DB in memory as possible, for speed, so huge RAM totals on DB servers are not unusual. 384 GB should be well within modern OS reach. Once you're computing at 64 bits, the next theoretical limit on addresses would be somewhere around 16 million terabytes.

edit: wrong number

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by callesgg on Monday July 21, 2014 @01:36PM (#8064837)
Not that much, we have 128 gb ram on our hardware. That is pretty standard for upper midle class rackmounted servers.

64 bit computers has a memmory limit so large that it is not understandeble.

The ram is still lightning fast compared to other stuff.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by Sanddancer on Monday July 21, 2014 @01:28PM (#8064793)
384 gigs is pretty common these days. With Windows, you need at least the server edition to address that much, but if you're running a server, you're not gonna be using the desktop versions of the OS anyways. Pretty much everything else, it's just plug and play. The RAM is quite a bit if you're doing a typical cloud deployment, but SO seems to be more in the model of making sure all your eggs are in a really strong basket. Besides, at the end of the day, RAM tends to be a lot cheaper than lag.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by abritishguy on Monday July 21, 2014 @03:49PM (#8065620)
Not true. Windows 8.1 64 bit would be happy with that much.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by ddorian43 on Monday July 21, 2014 @03:45PM (#8065598)
*everything else is plug and pay
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by dekhn on Monday July 21, 2014 @09:30PM (#8067040)
The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints).

No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by octo_t on Monday July 21, 2014 @01:31PM (#8064811)
You can get servers with 384GB of RAM now from a vendor like Dell, without needing to call them for just under $14k.
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by junto on Monday July 21, 2014 @01:29PM (#8064797)
Their SQL Server is Microsoft SQL Server running on (shock horror) Windows.

Says in the article:

Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by Nick-Craver on Monday July 21, 2014 @02:07PM (#8065030)
To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not.

This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better SMB, and such...but not enough to make the upgrade worth it.

Nick Craver - Stack Exchange Sysadmin & Developer

Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by Just-A-Guest on Tuesday July 22, 2014 @05:59AM (#8068122)
Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!
Re:StackOverflow Update: 560M Pageviews a Month, 25 Servers by Nick-Craver on Tuesday July 22, 2014 @06:21AM (#8068181)
I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead!

Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.