Not sure about the exact pricing shown here, but it looks like Azure blows AWS GPU pricing out of the water.

Comparing specs:

  AWS g2.2xlarge:
  * DRAM: 15GB
  * GPU processor: 1xGK104
    * 1536 CUDA cores
    * 2.29Tflops/s peak single precision throughput
  * GPU memory: 4GB
    * 160GB/s peak bandwidth
  * Price: $0.65/hr (US East, Linux)

  Azure NC6:
  * DRAM: 56GB
  * GPU processors: 2xGK210
    * 4992 CUDA cores
    * 8.73Tflops/s peak single precision throughput
  * GPU memory: 24 GB
    * 480GB/s peak bandwidth
  * Price: $0.66/hr

I expect AWS will introduce a new GPU fleet and reduce pricing of the existing G2 fleet, in response to Azure.

Just a small correction the K80 is a dual GPU setup which means you get 2 GPU's with 2496 "CUDA cores" each and 12GB assigned to each GPU.

This isn't the same as having 1 GPU with 4992 cores and 24GB of memory.

CUDA kernel scaling is pretty decent but it's not 1:1 with additional GPU's it's also very much task specific so tasks that cannot be easily parallelized will not scale well or at all at least not without you splitting the workload manually. As for the memory scaling CUDA supports P2P memcopy so you can easily move memory from 1 GPU to another (or between hosts in a network distributed configuration) but under best case scenarios this goes through a PCIE bridge (in the K80 it's onboard) but no NVLINK support on the K80 as it's 2 gens behind Pascal, if you are using P2P memcopy you do not double the memory capacity (even lose some since memcopy requires another buffer iirc). CUDA also allows you to share the memory directly between multiple GPU's (and the CPU/Host) NVIDIA calls this Unified Addressing (OpenCL 2.0 later implemented something similar as Shared Virtual Memory) but this also comes at some cost the latency is greatly increased for memory access and you really need stellar code to make everything work properly if 1 GPU tries to dereference a pointer on another GPU while it's still is in use whelp "don't panic and move on" probably won't work :P

Thanks. I was glossing over the details. The point I was trying to emphasize is that the Azure instance gives you the whole K80 board (i.e. two GK210 processors), unlike the AWS G2 instance which effectively gives you a virtualized half of a K10 board (i.e. one GK104 processor)

Oh yeah the Kepler 2 K80's blow the K10 out of the water. I wonder just how expensive the enterprise grade Pascal boards are that even Microsoft doesn't offer them as an option.

It's really amazing just how much money NVIDIA seems to be squeezing out of the compute market (almost makes you feel bad for AMD) especially considering that Kepler is a 4 almost 5 year old architecture by now. The K80 is still going for like 4000-5000$ a pop I would love to know just how many GPU's Microsoft ended up buying.

Is there any reason cloud providers don't offer virtualised Titan X, or GTX 1080?

"Gaming" GPU's have reduced compute capabilities (pretty horrible DP/HP compute performance compared to Tesla parts), and their SKU's are not really designed for enterprise integration (power requirements, thermals etc.). As far as "virtualization" goes NVIDIA actually restricts that in the driver, including PCIE passthrough especially to not allow cheaper desktop parts being used for low end computing as the margins per GPU are considerably higher with their enterprise parts.

""Gaming" GPU's have reduced compute capabilities"

Not so for machine learning, at least in my experience.

http://pjreddie.com/darknet/hardware-guide/

Passthrough:

https://www.evonide.com/non-root-gpu-passthrough-setup/

You missed the end of the sentence: "....pretty horrible DP/HP compute performance compared to Tesla parts."

Might not be applicable in your case, but NVIDIA hasn't released a desktop part with FP64 capabilities even remotely comparable to their Tesla's since the original Titan.

Both Titan X's (Maxwell and Pascal) and the [Insert Physicist's Name Here] Quadros are pretty gimped as far as it comes to HP/DP operations.

If anything between generations DP performance across desktop parts have been severely degraded from 3:1 for Kepler to 16:1 for Maxwell to bloody 32:1 for Pascal desktop GPU's.

As for the passthrough it is possible but it requires a few hacks and is technically in violations of NVIDIA's policies, and unlike the Tesla parts there is no full virtualization once the a device has been initialized it can only be assigned to 1 host/guest and a full host reboot is required to reinitialize the device again. You also need to do a few UEFI hacks to prevent the UEFI from initializing the GPU before your host OS loads and it can be passed through to the guest.

This isn't something that cloud providers would dealing with, NVIDIA will not sell them anything but Tesla/Grid parts and buying from AIB's/OEM's and hacking your way through it isn't an option, you'll get zero support from NVIDIA, you could not thin provision your GPU's and you'll need to do a full host reboot every time you want to reassign the GPU or try to figure out if you can write a custom BIOS for your card that would somehow allow you to reinitialize it.

P.S. I've actually built a KVM multi-user gaming rig with NVIDIA cards, I would not recommend anyone to try that, AMD cards are more pleasant to work with, too bad the performance are utter garbage for high end gaming.

Agree with dharma1 that the FP32-based cloud services could be a lot useful for deep learning than the Tesla-based Clouds.

Baidu has actually built Titan X -based servers for themselves as they saw it as the most cost-efficient way to build the system: http://www.nextplatform.com/2016/04/22/baidu-eyes-deep-learn...

Same logic would probably work in "deep learning" clouds if there would not be the licensing issue.

And this conversation got immortalized to that slideshow as well for selecting hardware parts for deep learning systems: http://tinyurl.com/zzhj9vr

I wish one of the main cloud platforms would significantly cut their per/GB bandwidth costs. It feels like there has been no adjustments in years.

Also funny story, I was 14 or 15 and was trying to figure out a name for a stupid little chat client I was trying to make in C#, and somehow azure popped into my head. Not surprisingly azure.com was taken and just had a generic microsoft landing page (this was a little before it was announced in 2008).

That's a major lock-in mechanism. Data stays in for the compute. Which is a major pity for Microsoft since all the marketing measurement startups got built on top of AWS while Azure has superior analytics tools.

I mean, it doesn't necessarily solve anything if you're using it for public facing apps, but if you're doing a hybrid solution with Azure you can buy unmetered ExpressRoute ports and at least avoid the metered bandwidth costs back to a datacenter.

Have they historically done so though? AWS's market leader position seems to persist even though they're actually quite a bit more expensive than Google Cloud and Azure (in particular due to the billing practices).

Well, check this one out: https://www.runabove.com/titan-x-gpu-servers.xml

That's nice, but the whole point of cloud instances is to be able to spin them up and down in large numbers when you need them, and pay by the hour

Or pay by the minute on Azure

What about comparing against spot market prices? Those AWS GPU instances are frequently $0.10 or $0.20 rather than $0.66. Not clearly a better deal anymore.

Yes, if you are ok with having your neural net suddenly killed mid-training.

you can set your max spot price high, and/or you can periodically save the trained weights (to an EBS volume) and continue from where you left off, same as recovering from a crash. Probably not a huge deal.

Here's how you do it with Tensorflow:

https://www.tensorflow.org/versions/r0.10/how_tos/variables/...

The sweet spot would be updated AWS GPU instances at spot prices. You could spin up XX instances for a relatively short time to train fast. Needs some Kubernetes etc cluster setup for distributed training but should be doable. If the spot price is $0.1-$0.2/hr that's pretty attractive, considering that is close to the price of electricity alone

Does Azure offer spot pricing? You can get those AWS instances for (iirc) 10-15 cents/hr without much interruption.

No, but it doesn't do hourly billing so it can still be cheaper depending on your workload.

Also, you tend to be getting more per instance hour on both Azure and GCE, assuming you do things those environments are good at. In Azure's case if you can get your software running on windows then it's actually quite good performance for I/O intensive things.

Why Windows? You can run Linux on Azure

So... I haven't checked in the last 6 months but when I did do a comparison the Windows vms were noticably faster than the linux vms.

I should check that again, it's been awhile.

But it depends on your platform. Its an unpopular but somewhat verifiable fact that the Windows kernels have much more powerful async I/O support and when software targets that, it is very fast compared to linux counterparts. If your toolkit supports this, then a strong argument can be made that you don't really care so long as SSH works. Powershell is serviceable for maintenance and most deploys are fairly agnostic to other aspects of the platform.

I've been doing skunkworks projects. I tried to get some stuff working in GCE and their docs are a mess and it never worked. Azure has some warts in its git push deploy stuff, but it worked the first time and I managed to diagnose the other issues.

> Windows kernels have much more powerful async I/O

async DISK I/O

Network as well, under certain expectations.

The spot pricing seems to go up pretty quickly if you try to run more than one instance.

I've run 6 of those GPU instances for $0.60ish/hr.

Pricing does go up near computer vision conference deadlines though. Many research labs are too poor to buy our own hardware. :-)

it sure does. I hope AWS and GCE will step up their GPU instance game

GCE should offer Tensorflow instances sometime soon. Closed alpha now. This will be a nice offering from Google which is no doubt part of the reason they open sourced Tensorflow.

Which GPUs they offer and how much, nobody knows.

> Which GPUs they offer and how much, nobody knows.

They might not offer GPUs at all. It may be that Google is planning to build a cloud using their own proprietary "Tensor Processing Unit" chip designed for TensorFlow (https://en.wikipedia.org/wiki/Tensor_processing_unit)

TPUs are only useful for prediction, not for training.

Unless I missed something, whether or not they are for training or inference has not been clarified.

They mention that they use them for inference in one of the TF papers; also the focus on per watt efficiency rather than speed/memory means it's probably for inference.

Nice. Would still prefer GPU instances I can do anything I want with

This is pretty interesting from a video encoding point of view. We're currently using g2.2xlarge AWS instances which are Kepler based. All we really need from those instances is the GPU (more specifically, the NVENC H264 encoding chip), so other specs aren't too important.

It looks like the NV6 Azure instances use a newer Maxwell based chip, which can handle much more (concurrent, or higher bitrate/resolution) videos, compared to the 2 generation older AWS GPUs.

And the NV6 is virtually the same price as the g2.2xlarge... time to do some testing. Hopefully Amazon can catch up soon.

Just for reference:

K1 (AWS): 1 GPUs x 6 H.264 streams (max single stream 720p@30fps)

K2 (Azure K80): 1 GPUs x 6 H.264 streams max single stream 720p@30fps)

M6 (Don't know if it's used by anyone ;)): 1 GPU x 18 H.264 streams (max single stream 1080p@30fps)

M60 (Azure M60): 1 GPU x 18 H.264 streams (max single stream 1080p@30fps)

You can sacrifice encoder bandwidth (max streams) for higher resolutions or frame rates.

Is GPU better than CPU for encoding? I thought performance ended up similar. Also, are you saying that K1 AWS can reencode 6 streams at once?

The performance of the GPU is orders of magnitude better you can do multiple 1080p streams in real time with high bandwidth easily.

The quality on the other hand even for the same bit rate is considerably worse than highly optimized CPU compilers as the GPU does do a lot of hacks along the way.

GPU generally support more codec and faster encoding/decoding times then a CPU. But it really depends on what codec, what CPU, and what GPU.

/r/machinelearning discussion: https://www.reddit.com/r/MachineLearning/comments/4wd5kn/ms_...

AWS: 4 year-old GPUs

Azure: 2 year-old GPUs

Wouldn't it be awesome if a cloud provider could keep up with the GPU roadmap and ship something within a year or so of its release?

The GPUs that hosting providers are buying need to support vGPU features, which are only support by the higher-end workstation and server products.

Which is a software/binning issue (probably 100% software) not inherently HW because M60==GTX 980==GM204 and M40==GTX Titan X==Quadro M6000==GM200.

Interestingly, for the first time ever, GP100 is unique (well, OK, K80 too but K80 was too late). And since Quadro P6000==Titan XP==GP102, it's probably just a software block here as well.

Also, for the first time ever, the high-end Quadro will be the best FP32 GPU of it's generation. That's the most interesting part for me.

It is a software/bios block. Once the driver grabs a desktop card and it's initiated it's locked. The cards also come with a UEFI bios which means if your host initialized the card during boot it's also locked.

To overcome this you need to disable UEFI boot/GPU boot in the BIOS and blacklist the card in the host OS and then create a PCI-stub device that will be used for the passthrough.

This is an utter and complete hack and you can't really use for production grade implementations.

I'm not sure that Quadro actually supports vGPU also AFAIK only Tesla and Grid parts do, Tesla does have some additional in GPU support for virtualization, i don't know how GRID handles it. GRID parts are less for compute and more for game/video streaming so they aren't fully virtualized, AFAIK they do not support P2P gpu communication or shared memory access, they are only more Hypervisor friendly so they can be thin provisioned.

Bare metal grid GPUs do support P2P. Xen breaks that on AWS.

I think supply is the issue... So nvidia and amd are the problem.

I signed up for the beta but haven't heard anything back yet. Anyone else?

I haven't either.

This is interesting. Is there some way for academics to get discounted access for experiments (not for running heavy simulations, but for testing whether our stuff works)?

Microsoft has graned "free" subscriptions to proffessors and researchers before, so it can't hurt to ask :P https://azure.microsoft.com/en-gb/community/education/

Does anyone know of an access protocol (like pcoip) that supports multiple users to have concurrent 3d accelerated sessions on one VM ? Each just 720p

Any idea if it worth to run a bitcoin miner on those like it used to be few years back?

No, profitable bitcoin mining is now only possible using low power ASICs.

It is not profitable to mine Bitcoin but maybe something new like Monero is profitable.

SSL Certificates are a legal tool these days, Microsoft doesn't want to deal with it, the insurance and validation is handled by Verisign/Symantec it's a business decision.

Bluecoat is hardly an "interception" technology it's a mid 90's enterprise grade proxy and internet policy management box that can do SSL interception their early boxes used SSL strip the newer ones probably still do. They don't have a magic backdoor to break SSL all you can do with a Bluecoat box is to install your own enterprise certificate and make it act as an SSL proxy to ensure that your employees do not surf pronhub over SSL at work.

So as far as tinfoil hat rants goes i give this one 3/10 it's not even that creative, you might want to look the "PRISM" partners slides again might give you some inspiration.

Upvoted for the timely Olympic scoring.

Isn't Microsoft's CA used only to sign their software?