News for hackers. Stuff that matters.
Re:Do C10K testing with gevent by eugenejen on Friday March 04, 2011 @11:01AM (#2288459)
I have a stupid question here. When you have a large memory machine, why can't you serve 1M clients on 1 ip without IP aliases?

I asked this question because if one uses 17 instances to handle 1M connections, for each instance is around 50k. I am more interested in knowing the possibility and performance for one machine to handle 1M connections.

Re:Do C10K testing with gevent by nivertech on Friday March 04, 2011 @11:54AM (#2288695)
the real problem is not to generate 1M outgoing connections, but to handle 1M incoming connections on a single node (C1M), while in addition also doing some meaningful processing.

It's a solved problem for me. That is why I don't understand why many high profile realtime web startups (like Convore), still using inferior solutions, like Python eventlib or gevent, which at most let them handle hundreds-to-thousands of clients.

EDIT: metabrew has very good tutorial - which should be a starting point for everybody new to this space!

Re:Do C10K testing with gevent by eugenejen on Friday March 04, 2011 @06:42PM (#2290292)
Are you referring to http://www.metabrew.com/article/a-million-user-comet-applica...

http://www.metabrew.com/article/a-million-user-comet-applica...

http://www.metabrew.com/article/a-million-user-comet-applica...

By Richard Jones of the Last.fm?

Thank you for your replying. The original C10k from Dan Kegel was a guideline for me when I worked on project during 2003 - 2006. I am wondering what is the latest progress in this area. I am glad this problem is solved to you. I will check out niver tech in future when the problem comes up.

Re:Do C10K testing with gevent by wmf on Friday March 04, 2011 @11:51AM (#2288680)
In the real world your 1M connections will be coming from ~1M clients, so there's no problem. When doing load testing with one client and one server you run into port number limits; this is most realistically solved by using multiple IP addresses on the client (or multiple clients if your cloud is crippled).
Re:Do C10K testing with gevent by stonemetal on Friday March 04, 2011 @11:13AM (#2288506)
Port numbers are 2byte values. To track a connection as an IP:port number combination(the way your typical OS does it) you need multiple IPs to give you enough port numbers.
Re:Do C10K testing with gevent by nivertech on Friday March 04, 2011 @12:00PM (#2288727)
each socket can be uniquely identified by 4-tuple: {LocalIP,LocalPort,ServerIP,ServerPort}

LocalIP:LocalPort -> ServerIP,ServerPort

So even if number of ports is limited by 64K, total number of incoming sockets per IP address isn't.

Re:Do C10K testing with gevent by nichol4s on Friday March 04, 2011 @12:06PM (#2288771)
Yes, but when you are load testing a specific server, as in this case, you are bound to a specific serverIp and serverPort.

And thus, the total amount of sockets you can have open to that server is limited by the amount of IP's your machine has.

Re:Do C10K testing with gevent by stonemetal on Monday March 07, 2011 @02:38PM (#2298359)
If we are testing a web server on port 80, such that ServerIP,and port are fixed, then the only way you are getting more than 64K connections is through multiple client IP addresses.