One problem I have is that the application is extremely CPU intensive. I can't get past 35-40 requests per second with 500 concurrent users (at 4 CPU cores, 14 GB RAM), which seemed too expensive economically. (I cached as much data as possible, both at the Nginx tier and with Redis) and tuned the number of uWSGI worker processes.
Do I have to try other languages or do you think I have more room for optimization with Python?
Then I'd suggest you use Cython for a speed up or try using PyPy which can be 200% faster without any changes.
When I used Python and worked with financial data, every live algorithm would be recoded as a Cython extension.
Having done web development for over a decade. My experience is that algorithm is far more important than language. After that data structures trumps language. After that doing work you don't have to do trumps language.
First look for really ugly nested loops in your code. I can't tell you the number of times this type of things ends up in your codebase, especially since dev environments often have a small subset of data. Even an O(n!) algorithm is fast for small values of n.
Next look to see what kinds of datastructures you're using. For instance, dicts are a seductive way to store data, they have a theoretical O(1) lookup, but they have a drawback of randomizing memory lookups, which means that for subsequent looks for a large dict, you'll end up with lots of cache misses, can make each lookup 1000 times slower. Meanwhile, a tuple has O(n) lookups, but if you're finding that you need to iterate over the data, you're going to benefit from memory locality. So know your tradeoffs. Also code that had one usage profile at launch, can often have a very different usage profile a year later, so it doesn't hurt to revisit this occasionally.
Last look for code that isn't doing anything. Are you calling the database twice in a row, asking for the same data, and not using it? It sounds obvious, but over time these kinds of things have a way of showing up in a codebase, and they can really add up.
I also should mention caching at this point. It can be incredibly hard to get caching right, but the performance savings are pretty big. It goes under the bucket of not doing things that you don't need to. Do whatever you can to do page caching. Putting an nginx cache in front of the webserver serving python for the majority of your pages could easily get you to hundreds of requests per second.
How did you like Clojure? Does the ecosystem it compare well against Django? I tried it once during an on-site interview but was put off by the multi-minute startup times.
So, comparing the languages, I tried Clojure because a colleague was using it and to me it feels like the best language I have ever used professionally (comparing to C#, C, Obj-C, Python, Java, JavaScript, Ruby, Go) and I'm very happy using it. Great functional programming features, great libraries, I really like the LISP syntax and the stress I'd been having with Python kind of melted away (things like state in objects and the type system (which kind of doesn't really exist most of the time with Clojure)).
In terms of doing web type things, the Liberator and Ring libraries feel a lot like a big chunk of Django's routing and middleware functionality. Selmer does a good job of templating.
For most user interface work I'm using React.js (which I also really like) so my Clojure code is only serving up JSON on an HTTP API. But for simple templated HTML, Selmer works well.
One question is "Django is an entire framework. How does it compare to lots of little libraries?". I have to say that Django is probably the most well constructed pieces of software I have worked with and it hangs together very well. Clojure with a few libraries is very good, but beating Django is a very high bar to reach for any framework or group of libraries.
Startup time is an over-done argument. Once you're running a REPL you don't need to restart it, so development isn't slowed down. Once you're running a server you don't need to restart it, so production isn't slowed down.
Since you work with React, how do you like JS? I'd imagine that, coming from Clojure, you'd be frustrated with its inconsistencies, as I am, coming from Python.
I've tried ClojureScript (without React) and it didn't quite click -- some of JavaScript's weird behaviour still shines through.
And then I thought "why not use it for something web-like". My experience was dreadful. After coming from Python+Django, it felt like writing Java 1.4 (not a good thing). Especially for returning JSON objects with lots of mixed types.
So I switched to Clojure and never looked back (Python would have been an equally valid choice).
Go's a great language, but it's being applied in some very odd (and potentially unsuitable) places.