News for hackers. Stuff that matters.
Re:Write Fast Apps Using Async Python 3.6 and Redis by erikcw on Thursday April 27, 2017 @02:36PM (#14213733)
We've just recently started using Sanic[0] paired with Redis to great effect for a very high throughput web service. It also uses Python 3 asyncio/uvloop at its core. So far very happy with it.

[0] https://github.com/channelcat/sanic

Re:Write Fast Apps Using Async Python 3.6 and Redis by sandGorgon on Thursday April 27, 2017 @02:46PM (#14213786)
There is also ApiStar - https://github.com/tomchristie/apistar

It's built by Tom Christie - the original author of Django Rest Foundation.

http://discuss.apistar.org/t/how-does-api-star-fit-in-with-d...

Re:Write Fast Apps Using Async Python 3.6 and Redis by theptip on Thursday April 27, 2017 @05:05PM (#14214710)
Interesting, hadn't seen that yet; thanks for sharing. Does look like it'll have some nice design concepts -- I've definitely come to view Django/DRF's strong coupling to the ORM as a hinderance to architectural flexibility/sanity as my application has grown.

Interestingly it eschews Swagger/OpenAPI in favour of JSON Schema, wonder how that'll pan out; I like the promise of codegen that swagger offers, but haven't found the generated clients to be particularly usable.

Re:Write Fast Apps Using Async Python 3.6 and Redis by tomchristie on Thursday April 27, 2017 @05:30PM (#14214879)
One thing to clarify here. Swagger/OpenAPI use JSON Schema in order to describe parameters and response structures. The rest of the schema work will start to fall into place pretty quickly now that we've got the groundwork done. Swagger generation based on the annotations will be one of the features, but there'll be plenty more to get excited about too.
Re:Write Fast Apps Using Async Python 3.6 and Redis by theptip on Thursday April 27, 2017 @05:34PM (#14214911)
Isn't Swagger a subset of JSON Schema though? [1]

If APIStar happens to target the same subset, that's not a problem of course.

[1]: http://stackoverflow.com/a/32386131/37481

Re:Write Fast Apps Using Async Python 3.6 and Redis by zebra9978 on Friday April 28, 2017 @12:04AM (#14216797)
any particular reason you are using BSD license ? With all due respect, this does not cover a patent grant like the Apache license and could be a poison pill for companies to adopt.
Re:Write Fast Apps Using Async Python 3.6 and Redis by tomchristie on Friday April 28, 2017 @02:04AM (#14217166)
I just had a quick scan over the licenses of other projects used for server side projects. Projects using BSD/MIT include Node, Go, Rails, Django, and Flask.

I'm happy with the choice.

Re:Write Fast Apps Using Async Python 3.6 and Redis by sandGorgon on Friday April 28, 2017 @04:53AM (#14217897)
Sorry to interject - but that's not completely true. Go comes with a separate patent disclaimer.

https://golang.org/PATENTS

And IMHO nodejs is not a standard BSD license and comes with patent grant. That discussion went on for a year in the TSC . https://github.com/nodejs/node/blob/master/LICENSE

In general, this stuff is not always evident. But the BSD license by itself is not as good as Apache.

While in doubt, use Apache !

P.S. fyi, doing this later is super heavy-duty hard.

Re:Write Fast Apps Using Async Python 3.6 and Redis by Waterluvian on Thursday April 27, 2017 @03:42PM (#14214122)
I have never found a good example of a Python web server that provides some mechanism for statefulness. Is it just fundamentally not possible to have shared state among requests handled by the threads of a process? Sanic's examples seem to be the same as Flask's: self-contained function calls attached to endpoints.

I keep hitting a wall with Python when I want to do something like:

1. subscribe to a websocket connection and keep the last received message in state 2. expose an http endpoint to let a client GET that last message.

Re:Write Fast Apps Using Async Python 3.6 and Redis by bobbyi_settv on Thursday April 27, 2017 @04:49PM (#14214572)
You normally use something like redis to store the state.

If you were going to share state in memory between threads, how would you handle the case where the second request goes to a different server or that the process has restarted? You'd need redis anyway, so you might as well just use it in all cases.

Re:Write Fast Apps Using Async Python 3.6 and Redis by Waterluvian on Thursday April 27, 2017 @06:36PM (#14215389)
I get that everyone's responses are thinking some big public thing. I'm thinking small toy implementation for my home network.

The toy experiment is how to do what's trivial in Node with Python. Mainly because I like working with python. I think the answer might be: Python is the wrong tool for the job.

Re:Write Fast Apps Using Async Python 3.6 and Redis by rcarmo on Thursday April 27, 2017 @06:46PM (#14215457)
Erm, no. You can do shared thread storage, in Python, it's just that it doesn't really scale. I've done it for small daemons without significant hassle, and even wrote my own Go-like CSP helper: https://github.com/rcarmo/python-utils/blob/master/taskkit.p...
Re:Write Fast Apps Using Async Python 3.6 and Redis by zeptomu on Thursday April 27, 2017 @07:03PM (#14215581)
The problem is that accessing shared state concurrently in a multi-process context is a non-trivial problem, so specific software emerged that handles these problems for you.

The simplest solution is to use a small DB system like sqlite. It is built into Python (import sqlite3) performs reasonably well and you do not have to run an additional service.

Now if a small DB like sqlite already feels overblown to you (and it really is simple and small) you might not need concurrent access either, so the simplest solution is to just use a file where you store your state.

Re:Write Fast Apps Using Async Python 3.6 and Redis by icebraining on Thursday April 27, 2017 @07:36PM (#14215762)
I'd say the simplest solution is a global (or just shared) variable using a thread-safe container like queue.Queue. There's also multiprocessing.Queue, which supports sharing the queue across multiple workers.
Re:Write Fast Apps Using Async Python 3.6 and Redis by bpicolo on Thursday April 27, 2017 @07:09PM (#14215612)
Having multiple nodejs processes is the same thing as having multiple python processes w/ regards to sharing state.

What you're referring to works equally well in the single-process case for both.

Re:Write Fast Apps Using Async Python 3.6 and Redis by rbutcher on Thursday April 27, 2017 @07:42PM (#14215787)
This maybe seems complicated because Node has 1 obvious way to run (single threaded with asynchronous functions) but Python has a few ways (single threaded, multithreaded, ioloops kind of like Node, greenlets).

Python is excellent for toy implementations, and real ones too in many cases.

Re:Write Fast Apps Using Async Python 3.6 and Redis by netheril96 on Friday April 28, 2017 @01:58AM (#14217142)
Is multithreading really necessary for a toy?
Re:Write Fast Apps Using Async Python 3.6 and Redis by mkj on Friday April 28, 2017 @02:42AM (#14217318)
It probably is for long-polling or websockets?

https://github.com/mkj/wort-templog/blob/master/web/templog.... is my not-quite-toy example - a single process runs from uwsgi with Bottle (like Flask) and gevent. The long polling waits on a global Event variable that's updated by another request, nice and simple.

Re:Write Fast Apps Using Async Python 3.6 and Redis by raziel2p on Thursday April 27, 2017 @04:38PM (#14214493)
Is that really something you want to do in-memory? Once you have to start multiple worker processes or application servers behind a load balancer, you'll have to re-implement it with some sort of shared persistant store like Redis.
Re:Write Fast Apps Using Async Python 3.6 and Redis by icebraining on Thursday April 27, 2017 @07:43PM (#14215792)
Not really, you can use a multiprocessing.Manager to share a plain old dict or list across multiple worker processes: https://docs.python.org/3/library/multiprocessing.html#shari...
Re:Write Fast Apps Using Async Python 3.6 and Redis by cwilkes on Friday April 28, 2017 @01:56AM (#14217134)
I think the OP was talking about different VMs behind a load balancer where there is no shared memory at all.
Re:Write Fast Apps Using Async Python 3.6 and Redis by icebraining on Friday April 28, 2017 @04:16AM (#14217759)
Wasn't my impression (using worker processes in the same machine is common), but fair enough. On the other hand, message passing across machines is overrated. We run a SaaS service on 25 VMs with no communication between them for regular operation.
Re:Write Fast Apps Using Async Python 3.6 and Redis by weberc2 on Thursday April 27, 2017 @06:55PM (#14215525)
Flask lets you share state between requests; just have the route methods reference some global variable. You could also apply the route decorator to an instance method (although probably not using the decorator syntax).
Re:Write Fast Apps Using Async Python 3.6 and Redis by quinnftw on Thursday April 27, 2017 @04:06PM (#14214262)
You can use caching to mimic this behaviour in Flask, I.e.

http://flask.pocoo.org/docs/0.12/patterns/caching/

I'm not sure how this works with multiple threads though, I imagine you would have to synchronize it yourself.

Re:Write Fast Apps Using Async Python 3.6 and Redis by tyingq on Thursday April 27, 2017 @04:25PM (#14214390)
Redis's lpush, and rpop plus some naming scheme might suffice. Everything is atomic, so the thread bit is covered.
Re:Write Fast Apps Using Async Python 3.6 and Redis by pdmccormick on Thursday May 04, 2017 @03:07AM (#14263092)
Your request has stayed on my mind over the past few days, so I put this together for you: https://github.com/pdmccormick/sample-socketio-chat-app

Enjoy!

Re:Write Fast Apps Using Async Python 3.6 and Redis by chronial on Friday April 28, 2017 @09:41AM (#14219255)
I am a little confused here. What's keeping you from storing your state in a global variable?
Re:Write Fast Apps Using Async Python 3.6 and Redis by vram22 on Friday April 28, 2017 @02:56PM (#14222066)
Any idea how Sanic compares with Falcon? I read somewhere recently that Falcon was quite fast. I tried out Hug, which is built on Falcon, but only for a small demo app, not done any benchmarking.
Re:Write Fast Apps Using Async Python 3.6 and Redis by fafhrd91 on Thursday April 27, 2017 @05:53PM (#14215071)
problem with Sanic that it does not implement streaming properly. so it is very easy to kill any Sanic process, 10-20 seconds. again any.
Re:Write Fast Apps Using Async Python 3.6 and Redis by dalanmiller on Thursday April 27, 2017 @06:04PM (#14215164)
Would you mind elaborating?
Re:Write Fast Apps Using Async Python 3.6 and Redis by fafhrd91 on Thursday April 27, 2017 @07:14PM (#14215639)
check Sanic code, it loads whole incoming payload into memory before processing it. event for 404, so I can write very simple script that would consume all memory. and you can not really protect sanic service with proxy (nginx)
Re:Write Fast Apps Using Async Python 3.6 and Redis by wiremine on Thursday April 27, 2017 @03:33PM (#14214067)
Been doing the same thing: Sanic is great!