Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by tnolet on Monday January 08, 2018 @04:00PM
(#16100503)
Wow, that is pretty substantial difference between AWS and Azure. I’m pretty confident Microsoft is hard at work getting these numbers better, but it shows just how lonely AWS is at the top. I’m pretty deep into AWS-land but I’m rooting for the boys & girls in Redmond; more competition == better service.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by QuinnyPig on Monday January 08, 2018 @05:44PM
(#16101510)
Concur; despite being so deep in AWS I've drowned here, competition makes all of the offerings in this space better. It'd be nice to have a migration strategy in my back pocket if I needed it; today that's a bit of a pipe dream.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by oceanswave on Tuesday January 09, 2018 @07:24AM
(#16105620)
It’s also difficult to run true x64 bit workloads on azure functions as it invokes the x86 version of the dotnet executable, regardless if the application is set to x64
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by lvh on Monday January 08, 2018 @06:53PM
(#16102123)
My experience so far on Google Cloud has been just as bad. I want a simple app that ingests DMARC reports.
- Google Cloud Functions can't ingest e-mail. Also, JS lockin if you care about which platforms are supported.
- Google App Engine can only ingest e-mail in the Standard environment. OK, fine. But the standard environment regularly takes almost a full minute to load and boot my app, after which the request is killed for taking too long. Additionally, the sample code (!) for opening new Stackdriver contexts breaks with a totally inscrutable error message.
You can say "well, maybe your app sucks", but it's been running on AWS (SES + Lambda) for months and every aspect about it has been painless.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by teej on Monday January 08, 2018 @10:50PM
(#16103453)
I don’t know how magic is in Cloud Functions, but I personally set up Mailgun postbacks in AWS and it was a huge pain in the ass to configure correctly. I’d love I know if google does something differently that would have been easier.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by lvh on Tuesday January 09, 2018 @01:13PM
(#16108293)
But the parsing code already works fine, that’s not really the hard bit. Then I get all of the lock-in for questionable benefit and something that Just Works on AWS.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by seangrogg on Monday January 08, 2018 @05:33PM
(#16101425)
GCF still suffers from JS lock-in, as they currently only support Node, thus you can't make a .NET comparison in the first place. That being said, of the various offers I really like the UI and the like.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by spyspy on Monday January 08, 2018 @04:44PM
(#16100920)
If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions. Still "serverless" but more tools and languages to choose from.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by nothrabannosir on Monday January 08, 2018 @07:55PM
(#16102502)
Not the flexible environment iirc. Which is basically the reasonable env to expect, now. Standard env has some aging issues that keep getting worse. most notably jvm 7, if that’s your poison, but just all round crippled env lacking standard libs and with lock in substitutions.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by dragonwriter on Monday January 08, 2018 @05:35PM
(#16101439)
> If you're going to be handling http requests it's almost a given you'll be using AppEngine rather than Cloud Functions
Why? Cloud Functions support HTTP.
I mean, if you can stay within AppEngine’s free tier quotas, there's a clear reason to use it, but otherwise it would seem to be a matter of weighing pros and cons, with HTTP alone not deciding that you should use AE over CF.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by intrasight on Monday January 08, 2018 @05:46PM
(#16101544)
This face-off will be a lot more meaningful in a couple months when we can have both serverless platforms running on .Net Core 2.0. I do look forward to reading the results of that updated face off.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by jamesdrandall on Monday January 08, 2018 @06:31PM
(#16101963)
Hi, author here - completely agree.
I did try the preview version of Core 2.0 on Azure Functions but it's absolutely woeful at the moment and the dev team confirmed they've done no performance work at all. As I recall the single request response time was around 15ms faster for Core than for full fat framework but the response to increased load was dreadful - never falling below a several second response time with the 400 concurrent user test and errors creeping in.
I largely set off down this path as even in isolation the Azure numbers were concerning and not at all what I expected from a serverless architecture and so I wanted a reference point.
As soon as I get an update from MS about Core performance work or it leaves preview I'll be rerunning the tests as I'm very aware that doing this across the two runtimes is less than ideal.
However from what I've observed the issues look to be more about the broader architecture used for Azure Functions than the framework runtimes. I'm hoping to learn more from Microsoft.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by bonesss on Tuesday January 09, 2018 @02:47AM
(#16104325)
In general, from the perspective of a dev in a MS-shop, AWS gives you a bit more 'responsibility' for infrastructure than Azure, but that level of control lets you get further in practice sooner.
Azure has a lot of great things going for it, but I've had a lot of issues getting newer tech out on it without considerable delay. I hope both Azure and GCE give AWS a run for their money in coming years :)
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by appdrag on Monday January 08, 2018 @05:26PM
(#16101358)
The biggest issue with Azure functions is the pricing model where you have to provision and pay for servers to execute your functions... How is that serverless?
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by krisives on Tuesday January 09, 2018 @12:43AM
(#16103896)
Does anyone know if any of these services provide the ability to cycle between IPs? One of the use cases for Lambda (and now Azure Functions) is in web scraping where companies get mad if you access them too many times from a single address. Lambda made it clear while they do use a lot of IPs you're not guaranteed to get distribution among them.
Seems like a long shot but I figured this is a good place to bring it up.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by intrasight on Thursday January 11, 2018 @06:11PM
(#16128955)
I have the same question. Not for scraping but for APIs that are rate-limited by IP address. Like some Google APIs. I certainly wouldn't count on IP distribution from getting around limits for long. Anyway, I'm more interested in understanding how, if at all, we will have to tweak our client code to be well-behaved.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by dronechip on Tuesday January 16, 2018 @10:42AM
(#16158934)
Interesting article, but you don't mention what tier of AppService plan you tested against in Azure (or what memory limit your Lambdas had)? I would also be curious to see the same comparisons, but with Azure Functions in consumption mode which, despite the perceived drawbacks you mention, would really be the closer apples-to-apples test, wouldn't it?
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by mcintyre1994 on Wednesday January 10, 2018 @12:59AM
(#16112856)
Does AWS have first class fully production Python yet? In Azure a single import (pydocumentdb) took a function up to 20-30s on Python, I guess they're serious about it being experimental still sadly. Rewriting in Node it takes 50ms for the entire run, crazy difference.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by carlsborg on Sunday January 21, 2018 @07:43AM
(#16197860)
Have to agree with that part, and am in no way taking away from the commendable engineering on the Azure side. But as a developer, it makes more sense for me to invest time and effort getting good at the platform that innovates faster - and the pace at which AWS has been getting out new functionality in the platform has been breathtaking.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by pwaai on Monday January 08, 2018 @10:24PM
(#16103351)
so does this mean that the cold run startup time thing has really been improved enough to build web applications? To me that is the biggest killer in moving forward, not knowing what the end users experience is going to be like because of cold startups that forces a user to wait a long time.
From the article's data, it's really tempting to use AWS Lambda now given this benchmark.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by wilgertvelinga on Monday January 08, 2018 @04:15PM
(#16100663)
I would say leave the average in but also show standard deviation and 95th percentile average. Of course it is still very unlikely that Azure Functions would come out on top in this case.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by teej on Monday January 08, 2018 @10:54PM
(#16103466)
You are arguing against yourself here. Having the mean and 95th/99th percentile is going to give you a better picture of the distribution than mean alone. Mean alone actually doesn’t give you ANY idea of the distribution. I don’t know why OP is being downvoted, managing to 95th percentile is very common practice and I also thought was a missing addition to the article.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by dmitriid on Tuesday January 09, 2018 @12:32AM
(#16103860)
It’s not the best. There’s a Russian joke that goes “One person is cooling in the morgue, the other is running a high fever. The average temperature in the hospital is 36.6 C though”
Averages are very susceptible to outliers and don’t give a proper insight into your data. What you really want is: how did half of my requests perform, 90% of requests perform, 95% of requests perform, 99% of requests perform
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by e12e on Monday January 08, 2018 @05:21PM
(#16101303)
Also, from webster's:
"av·er·age\ˈa-v(ə-)rij\
noun
: a number that is calculated by adding quantities together and then dividing the total by the number of quantities
: a level that is typical of a group, class, or series : a middle point between extremes
Full Definition
1 a : a single value (as a mean, mode, or median) that summarizes or represents the general significance of a set of unequal values
b : mean 1b
2 a : an estimation of or approximation to an arithmetic mean
(...)"
It's common to see arithmetic mean and average to be treated as synonyms - but the mean is really just a type of average.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by phamilton on Monday January 08, 2018 @05:47PM
(#16101554)
Colloquially, median and mode are used as average more than you'd expect. For example, we might say "the average joe is right handed", which is most likely the mode.
Re:Azure Functions vs. AWS Lambda – Scaling Face Off
by viggity on Monday January 08, 2018 @04:21PM
(#16100709)
this is absolutely not a function of Azure vs AWS, the perf diff is completely between .Net Framework (which was used on Azure) and .Net Core (which was used on AWS). This compares Apples and Watermelons.
.NET Core has way less baggage than .NET 4.6 [1] so you’d be expecting AWS to come out ahead.
It looks like Azure functions supports .NET Core now [2]. (Edit: just noticed the author is aware of this [3].)
Of course it might also be Blob storage having higher latency than S3.
Having used Azure Functions before I agree with the article’s criticisms about scaling.
[1] https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1... Mind this article is talking about comparing ASP.NET with ASP.NET Core, and .NET with .NET Core, not just .NET Core’s performance.
[2] https://github.com/Azure/Azure-Functions/issues/98
[3] https://www.azurefromthetrenches.com/azure-functions-v2-prev...
- Google Cloud Functions can't ingest e-mail. Also, JS lockin if you care about which platforms are supported.
- Google App Engine can only ingest e-mail in the Standard environment. OK, fine. But the standard environment regularly takes almost a full minute to load and boot my app, after which the request is killed for taking too long. Additionally, the sample code (!) for opening new Stackdriver contexts breaks with a totally inscrutable error message.
You can say "well, maybe your app sucks", but it's been running on AWS (SES + Lambda) for months and every aspect about it has been painless.
Disclaimer: I am at Google, but not on GCF.
https://medium.com/@salmaan.rashid/running-net-on-google-clo...
Native support for more languages is coming.
As far as I remember, that is..
Why? Cloud Functions support HTTP.
I mean, if you can stay within AppEngine’s free tier quotas, there's a clear reason to use it, but otherwise it would seem to be a matter of weighing pros and cons, with HTTP alone not deciding that you should use AE over CF.
https://cloud.google.com/free/docs/always-free-usage-limits#...
Disclaimer: I work at GCP.
I did try the preview version of Core 2.0 on Azure Functions but it's absolutely woeful at the moment and the dev team confirmed they've done no performance work at all. As I recall the single request response time was around 15ms faster for Core than for full fat framework but the response to increased load was dreadful - never falling below a several second response time with the 400 concurrent user test and errors creeping in.
I largely set off down this path as even in isolation the Azure numbers were concerning and not at all what I expected from a serverless architecture and so I wanted a reference point.
As soon as I get an update from MS about Core performance work or it leaves preview I'll be rerunning the tests as I'm very aware that doing this across the two runtimes is less than ideal.
However from what I've observed the issues look to be more about the broader architecture used for Azure Functions than the framework runtimes. I'm hoping to learn more from Microsoft.
Azure has a lot of great things going for it, but I've had a lot of issues getting newer tech out on it without considerable delay. I hope both Azure and GCE give AWS a run for their money in coming years :)
Seems like a long shot but I figured this is a good place to bring it up.
From the article's data, it's really tempting to use AWS Lambda now given this benchmark.
Averages are a meaningless stat especially if you benchmark.
For a given especially slow request would you expect a corresponding especially fast request?
Averages are very susceptible to outliers and don’t give a proper insight into your data. What you really want is: how did half of my requests perform, 90% of requests perform, 95% of requests perform, 99% of requests perform
To be fair, it's not clear which form of average is used in the article and your assumption that it is the mean is likely correct.
Is there any foundation to this?
"av·er·age\ˈa-v(ə-)rij\ noun
: a number that is calculated by adding quantities together and then dividing the total by the number of quantities
: a level that is typical of a group, class, or series : a middle point between extremes
Full Definition
1 a : a single value (as a mean, mode, or median) that summarizes or represents the general significance of a set of unequal values b : mean 1b 2 a : an estimation of or approximation to an arithmetic mean (...)"
It's common to see arithmetic mean and average to be treated as synonyms - but the mean is really just a type of average.
It compares mode, median, and mean as each being specific implementations of a more general concept of discrepancy.
https://www.ageofascent.com/2016/02/18/asp-net-core-exeeds-1...
http://codinggorilla.domemtech.com/?p=1499
https://www.azurefromthetrenches.com/azure-functions-v2-prev...
https://www.azurefromthetrenches.com/serverless-blog-christm...