We are finding that unit testing is pretty straightforward with Placebo [0] (also Mitch Garnaat!), for which we use a common "boto client getter" library that the unit test framework can hook into to inject Placebo.

The really hard part is integration testing. You can Chaos-Monkey your Lambda functions, but you can't Chaos-Monkey DynamoDB. We're looking at ways of building tooling to do that.

For deployment, we wanted to use Serverless, but as they started to move away from CloudFormation, that didn't work for our more enterprise needs, so we've been rolling our own, based on ideas from Kappa.

[0] https://github.com/garnaat/placebo

Mitch actually built Placebo alongside kappa, because he needed a way to test kappa. He even submitted a pull request against boto to make it support placebo. Sadly though, placebo is better for mocking the other AWS services, not so much a mesh of Lambda functions.

Have you guys figure out a way to do offline testing of one Lambda calling another? I'd love to see it!

Hey Ben, we're going to build completely on top of Cloudformation in the next version of Serverless. Currently starting to work on it. If you're interested would love to get your feedback: [email protected]

> you can't Chaos-Monkey DynamoDB

I bet you could achieve this effect by either temporarily decreasing provisioned throughput, or inserting clientside middleware that occasionally refuses to do a DynamoDB op.

The middleware would be best option. Put it between your code and the calls and then have it able to throw random errors.

But it means you have to write the middleware and have it be it's own source of bugs, which is still better than nothing, but another risk.

Is Placebo better than moto? It sounds very similar.

https://github.com/spulec/moto

My main issue with lambda so far has been its lack of Python 3 support. I do not understand why there isn't a 3.x runtime available. We have a 3.5 compatible app with parts that depend on 3.4, and we had to cut it in two and cripple it to make it work with lambda.

I've heard that the foot-dragging on Python is due to the senior technologist at AWS not liking the language and/or not taking it seriously.

Anyone else heard the same?

They upgraded their Node runtime from the legacy 0.10 to 4.2LTS... I'm having a hard time imagining a reason to hate Python that doesn't also apply to Node. Lack of a JIT? (It would be awesome to have Lambda + PyPy + 3.5... though that is of course not yet ready on the PyPy side.)

While I can't say I know anything more about python and aws specifically, hate doesn't have to be logical.

Someone might hate python just because.

Lambda can run arbitrary executables, including a Python3.5 interpreter. Admittedly it's a little silly to have a Py2 script that does nothing but start up Py3, but it's better than cutting and crippling, I'd bet.

> Lambda can run arbitrary executables

Linux executables.

During re:Invent 2015 there was talk about supporting a windows runtime environment.

Well if it can run arbitrary executables, it can run Wine!

:)

What is the definition of "Linux executable" that excludes `python3`? Builtins?

???

Arbitrary Executables = Arbitrary Linux Executables.

Since Python3 runs on linux, you can invoke python3 scripts.

Oh, sorry, I agree. I thought you were disagreeing with parent; saying "it doesn't run arbitrary executables, it only runs linux executables".

Although actually, I see what you mean now. D'oh. Windows was so far out of my mind I wasn't even thinking of it as a distinction to be made.

Yeah, that's the biggest problem. It's bugs me too. I'm sure it is coming soon, but the again, most of the big hosted python installs don't support 3.x yet (App engine being the big one).

App engine has 3.4 on their beta flavor right?

But yes in 2016 everyone needs day 1 Python 3 support. Python 2 support is optional.

I don't know why you would say that. I'm sure it would be great to support Python 3, but saying that Python 2 is optional is a little strange.

It's easier to make Python 2 code run on Python 3 than Python 3 code run on 2. Why shouldn't Python 2 be optional at this point? It's the version in maintenance-only mode. Python 2 is the VB6 of Python and it seems clear to me that it should be optional going forward for people spending support time on it.

This is why Python 2 isn't optional yet:

http://py3readiness.org/

Of the top 360 libraries, only 332 of them support Python 3 (but almost all of them still support Python 2).

This analysis was done in March and expected even support for Python 3 and 2 around now:

https://blogs.msdn.microsoft.com/pythonengineering/2016/03/0...

So maybe next year it will be optional, but not yet.

This isn't the thread for that debate, so I'll refrain from a specific response, other than to note that python2 has clearly not been replaced by python3, whatever the language's maintainers wish had happened.

You phrase that as if the language maintainers are the antagonist here. At that point, why continue using a language if you don't agree with the direction the language maintainers believe to be the language's present state? Why would you expect hosting companies (such as Amazon in this example) to necessarily pay support costs to continue hosting a deprecated version of a language side by side with the current version?

So yes, it seems to me that from the standpoint of a hosting company like Amazon for example, Python 2 should be the option for them to host, should not be guaranteed, and maybe even shouldn't cost the same as Python 3 moving forward. That's the present of the language. You are asking them to support the past of the language. Perhaps you should be willing to pay for them to support the past of the language.

(Of course, this particular example is moot because it sounds like Amazon themselves have engineers in the "Python 2 or Die" clade, but it's the example at hand.)

As much as I'd like everybody to be using Python 3, the truth is there's not a lot of immediate benefits, which is why 85% of Python users are still using 2.x.

As a Python user myself, the better, more reliable Unicode-by-default string handling in Python 3 is more than enough of an immediate benefit that I've stopped installing Python 2.x on my devices entirely at this point. Obviously, ymmv, but there seem to me to be a lot of immediate benefits to Python users and what feet dragging remains to stay in 2.x seems more like a political protest of some sort than one focused on actual benefits.

I am fine with supporting both, but if you are only going to do one - pick the way forward.

The more people that say "no python 2" the sooner it will die.

I don't know, python2vs3 thing is weird. Bet someone has a good book on it.

Obviously that is not the case.

Why not?

I can't image rolling out a new python hosting product in 2016 and not even covering python3 ?!?!?

> It's really hard to test if you have more than one function because there is no mocking framework (yet).

I'm not sure how this applies to Lambda, but we've found a workable solution for Service Fabric: the services and actors are almost nothing more than a thin host for the actual implementation (or are DI'd into the implementation) - which is defined in a completely separate package/assembly.

Thanks for kappa, I really like the idea, but it currently seems a bit "broken/unmaintained", is it still actively developed?

that's what I love about Lambda. You CAN test against the online environment, since it costs almost literally zero.

there are very few times where you are offline and would like to run a test suite, but it's much easier to handle those than try to fake the service AWS provides

Why would you use prod lambda in your test suite? Latency and other issues aside, that requires pushing your environment to a lambda (which can take a massive amount of time especially with larger uploads) to run a test suite that's supposed to take a couple of seconds.

we need to create a new build in order to deploy it anyway. The build is hit with some interaction tests. Because config is passed in and not hard coded into the buil it just works. After tests pass the build can be promoted to production alias.

Requiring an internet connection for normal development seems a bit extreme. That rules out using it in many places in the world where the internet is slower or less stable, and even here, we have developers who work on their commute with no internet connection.

That's certainly true. Useful Lambda code (i.e. lots of functions cooperating to do something, likely using a variety of AWS services) can only run in the Lambda context. So it's not appropriate if you need to develop offline.