News for hackers. Stuff that matters.
Re:Scaling HipChat Using ElasticSearch And Redis by taspeotis on Tuesday January 07, 2014 @01:41AM (#7016010)

    Stats
     * 60 messages per second
    ...
      * 26 front end proxy serves. Double that in backend app servers.
78 servers for 60 messages per second?

We have one MSSQL box at work here that is regularly doing 2,500 queries/sec and that's an OLTP workload so easily more than 60 writes/sec.

serves makes me think that someone didn't proof-read the article...

Re:Scaling HipChat Using ElasticSearch And Redis by georgebarnett on Tuesday January 07, 2014 @02:24AM (#7016145)
How did it even make sense in your head to compare an OLTP workload to a chat service?

The proxy servers are there to terminate the large number of persistent connections. Of course it's possible to do it using less servers, but given the Hipchat guys are smart (disclosure - I'm an Atlassian and know the internals) I would give them the benefit of the doubt rather than engaging in armchair architecture.

Re:Scaling HipChat Using ElasticSearch And Redis by taspeotis on Tuesday January 07, 2014 @02:52AM (#7016195)
The point was to ask whether the 60 in 60 messages per second is wrong by comparison.
Re:Scaling HipChat Using ElasticSearch And Redis by mtrimpe on Tuesday January 07, 2014 @03:20AM (#7016258)
The frontend needs to keep open thousands of persistent connections; 99.9% of which will time out with receiving a single message.

Practically speaking the number of frontend servers is related to the number of active listening clients and not messages.

Only 60 messages per second does seem very small from a business perspective however.

Re:Scaling HipChat Using ElasticSearch And Redis by taspeotis on Tuesday January 07, 2014 @03:34AM (#7016287)
If you take the frontend servers out of the equation it's still 52 servers for 60 messages per second.
Re:Scaling HipChat Using ElasticSearch And Redis by msmithstubbs on Tuesday January 07, 2014 @04:30AM (#7016404)
My guess: the 60 messages per second is an average. Monday morning in the US timezones might see several multiples of that. Weekends could be a lot lower.
Re:Scaling HipChat Using ElasticSearch And Redis by powdahound on Tuesday January 07, 2014 @08:51AM (#7017253)
It's definitely very bursty. On weekends and holidays things are much more quiet. During peak load we'll be in the hundreds/sec. Also keep in mind that chat messages don't actually make up the majority of the traffic we serve; it's presence information (away, idle, available), people connecting/disconnecting, etc. (I'm one of the HipChat co-founders.)
Re:Scaling HipChat Using ElasticSearch And Redis by hox on Tuesday January 07, 2014 @07:51PM (#7021042)
And? The article makes no mention of what each server is responsible for, how much redundancy is built in (this is AWS where nodes die on a whim), whether all of these servers are used for production or if there is a spread across dev and staging environments, etc.

When all of these factors are considered, 52 servers isn't a huge number.

Re:Scaling HipChat Using ElasticSearch And Redis by tveita on Tuesday January 07, 2014 @05:24AM (#7016533)
Looks about right from their graph:

https://s3.amazonaws.com/uploads.hipchat.com/10804/368466/10...

That traffic is going to be highly bursty, though.

Re:Scaling HipChat Using ElasticSearch And Redis by Kiro on Tuesday January 07, 2014 @05:41AM (#7016562)
May I ask what kind of company you work at where you have that kind of queries/s?
Re:Scaling HipChat Using ElasticSearch And Redis by taspeotis on Tuesday January 07, 2014 @06:37AM (#7016729)
Australian childcare software provider.

I imagine an outsider wouldn't equate "childcare software" with "database intensive" but the online reporting childcare services have to do is quite involved [1].

Their online reporting to the government is a separate concern to day-to-day tasks in their childcare service. So childcare software has to do more than just interface with the online reporting interface.

[1] http://docs.education.gov.au/system/files/doc/other/service_... (The attendance information of children is the most complicated part, see page 160 for what an attendance record looks like.)

Re:Scaling HipChat Using ElasticSearch And Redis by Jamsii on Tuesday January 07, 2014 @06:43AM (#7016755)
Is it hubworks?
Re:Scaling HipChat Using ElasticSearch And Redis by taspeotis on Tuesday January 07, 2014 @06:47AM (#7016776)
No. I don't know what RDBMS HubWorks use but to my knowledge they run their software with AWS and I think there's better fits to AWS than MSSQL.
Re:Scaling HipChat Using ElasticSearch And Redis by GamblersFallacy on Tuesday January 07, 2014 @05:56AM (#7016604)
On my shit dev box, localhost Elasticsearch is indexing (AKA writing) @ approx 2,500 msg/sec.
Re:Scaling HipChat Using ElasticSearch And Redis by est on Tuesday January 07, 2014 @07:02AM (#7016824)
Can you still process that many while holding like 2M connections open?