> first scale vertically (buy better machine)

With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically.

We just won an industry award at work for a multi billion data point spatial analysis project that was all done with Python scripts + Docker on EC2 and PostgreSQL/PostGIS on RDS. A consultant was working in parallel with Hadoop etc and we kept up just fine. Use what works not what is "best".

Edit: A dumbed down version of the Python/Docker piece for anyone interested (https://medium.com/@mbaker/horizontally-scaling-gis-python-a...). It's really easy to scale horizontally with Docker...

> With tools like message queues and Docker making it so easy to scale horizontally you don't even have to go vertically.

That depends entirely on the workload. It's not always a good idea to move from one sql instance, to a cluster of them. Just buy the better machine that gives you time to make a real scalable solution.

What's your firm's name?

It was for my day job and I'd prefer not to have that publicly linked in a searchable format to my HN profile.

My profile has a link to my personal website though and it's the current employer listed on the associated LinkedIn profile.

Why is it searchable if you name them here, but not via linkedin ? When your hn profile explicity links to yr website.

I assume searching for [employer] could direct someone here vs searching for knz/me specifically would take you to my HN profile page.

I'm not ashamed of anything I've said on HN but would rather not have people just searching for my employer ending up here (especially since I work in an office that routinely deals with sensitive political and community issues). It's a minor amount of (perceived) anonymity vs stating my name/job title/employer here!

I love your website.

Ha. Thanks! It's a bit basic but has proven sufficient.

Well, they could start by using something faster than Python. I would tend to use Common Lisp, but Clojure would be the more modern choice.

But yes, scaling up is far easier than scaling out. A box with 72 cores and 1.5TB of DRAM can be had for around $50k these days. I think it would take a startup a while to outgrow that.

Python is plenty fast where it matters. You have heavily optimized numerical and scientific libraries (numpy and scipy) and can easily escape in C if it matters that much to you. But in my experience bad performance is usually a result of wrong architecture and algorithms, sometimes even outright bugs, often introduced by "optimization" hacks which only make code less readable.

This holds for all languages, of course, not only Python. Forget raw speed, it is just the other end of the stick from Hadoop. Believe me, you don't need it. And even when you think you do, you don't. And when you have measured it and you still need it, ok, you can optimize that bottleneck. Everywhere else, choose proper architecture and write maintainable code and your app will leave others in the dust. Because it is never just about the speed anyway.

I agree that it depends on what you're doing, and the speed of the language often doesn't matter -- that has to be the case or Python would never have caught on to begin with.

But you can write code in Common Lisp or Clojure that's just as readable and maintainable (once you learn the language, obviously) as anything you can write in Python, and the development experience is just as good if not better.

Could it play crysis on ultra settings tho ?