News for hackers. Stuff that matters.
Re:The Technology Behind Convore by ericflo on Wednesday February 16, 2011 @04:11PM (#2228324)
Yeah, I've been following that work for quite a while, it's really impressive! I'm a big fan of Cassandra in general (in fact, I wrote an example application to help teach beginners about it: https://github.com/ericflo/twissandra).

The decision to go with PostgreSQL over Cassandra (or another distributed system like Riak or HBase) was simply because it gave us the most flexibility to change our product quickly while we operate at low scale. And if I'm honest with myself, we're operating at very low scale right now.

In the future as we scale up, Cassandra's distributed counters will be one of the first places we look.

Re:The Technology Behind Convore by saurik on Wednesday February 16, 2011 @08:13PM (#2229203)
Are you willing to say how many simultaneous users you are handling at this point? (I obviously totally understand if you aren't.)
Re:The Technology Behind Convore by ericflo on Wednesday February 16, 2011 @08:30PM (#2229249)
Unfortunately I can't give out any exact numbers :(
Re:The Technology Behind Convore by saurik on Thursday February 17, 2011 @04:01AM (#2230339)
I realize you may also not want to answer this, and that's also alright, but I figure I may as well ask, as I really am curious (and maybe someone else will answer from the perspective of their company): why not? Whenever anyone has asked me for a stat on Cydia, whether it be active devices over some time range, daily revenue, costs related to <insert-subproject-here>, or what have you, I tend to break out a quick SQL query and provide an exact answer (assuming the question is of something I an measure: not all are); is this stupid of me? I've noticed a ton of companies refuse to disclose numbers, and I've always assumed that the result will be that anyone listening will just assume "ok, so almost none then" unless you give them a good answer and back up how you calculated it, but does this actually "hurt the cause"? (I do not have the benefits of the years if startup experience that you have access to by being a part of Y Combinator, so I try to get advice whenever I can ;P.)
Re:The Technology Behind Convore by axod on Thursday February 17, 2011 @05:07AM (#2230448)
The top 20 public groups topped out at about 250 users last night, add on a bit for the other public groups and a little for private groups. Maybe 400 concurrent users max so far?

Just my guess, but I'd be surprised if it's orders of magnitude different.

Re:The Technology Behind Convore by foobarbazetc on Wednesday February 16, 2011 @08:28PM (#2229243)
Just FYI: we run a very large web service with purely PostgreSQL, and have no need for sharding or anything of the sort.

Scale up first. Use redis or memcached to store 'overview' stats. If you lose those stats, recalc from Pg.

Re:The Technology Behind Convore by ericflo on Wednesday February 16, 2011 @08:32PM (#2229256)
That's exactly the approach we're taking. We're not going to spend any time prematurely worrying about various scaling things until we can forecast that it's legitimately going to be a problem.
Re:The Technology Behind Convore by foobarbazetc on Wednesday February 16, 2011 @08:45PM (#2229295)
Great to hear. :)

While we're here, here are some PG tips:

1. Make sure your PG install is fully optimized. I've seen countless postgresql.conf's at major companies with no opts what so ever. Take a look at this book: http://www.2ndquadrant.com/books/postgresql-9-0-high-perform... .

2. Make sure you're using PG 9+.

3. Use pgbouncer for connection pooling and try to write your app so you can use 'pool_mode = transaction'.

4. Please don't run PG virtualized.

Re:The Technology Behind Convore by ericflo on Wednesday February 16, 2011 @09:07PM (#2229355)
We're good on all fronts except 4, and the transaction pool mode :)