News for hackers. Stuff that matters.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jessaustin on Friday September 09, 2016 @01:53PM (#12464528)
MongoDB -- as our core application data store. It's popular to make fun of Mongo these days, but we've found it to be a great database technology as long as you don't store too many things in it. Anything you can count on your fingers and toes should be fine.

I suspect this is a joke, but I'm still confused? If 4B things happen every day, how is that finger-toe countable? Does this mean that relatively slowly-changing things like "customer name" are stored in mongo while events etc. are stored elsewhere?

Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @01:54PM (#12464545)
I was being cheeky :)

Events are stored in DynamoDB. We use mongo for "core" data like accounts, feature flags, etc, but none of the high throughput stuff touches MongoDB.

Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by koolba on Friday September 09, 2016 @02:44PM (#12464980)
> Events are stored in DynamoDB. We use mongo for "core" data like accounts, feature flags, etc, but none of the high throughput stuff touches MongoDB.

Curious, why the choice of MongoDB for anything? In my mind "core" data tends to be very structured and require true transactional updates, both of which are cornerstones of a relational database (cough Postgres). Sure it's possible to use MongoDB, but what was the driver for it?

Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @03:13PM (#12465242)
We had a lot of operational experience with Mongo from our previous jobs at Atlassian, and started out storing almost everything in Mongo. As we scaled out, we migrated all of our high volume data into other stores-- analytics into DynamoDB, searchable data into ElasticSearch, etc.

The last remaining piece is our core data, and there's no strong push to move it out of Mongo.

Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by kornish on Friday September 09, 2016 @06:14PM (#12466426)
So, just to clarify, you use DynamoDB and ElasticSearch as primary datastores for those types of data and don't replicate data from Mongo into them?
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @06:24PM (#12466479)
That's mostly right. We don't use ES as a primary datastore, though.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jonaf on Friday September 09, 2016 @10:32PM (#12467358)
Are there any good resources for managing feature flags like this? I am currently using Netflix Archaius and it's working great, but I find it's easy to forget what values are set or what flags exist. Over time, it's easy to make a mistake and deploy something broken. I'm currently working on instituting a kanban policy to replace feature flags with static configurations ~1-2 months after their initial deployment where possible. But I'm thinking there's a better way to avoid this trap.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by siliconc0w on Friday September 09, 2016 @04:01PM (#12465615)
It's a pretty question architecture that outsources feature flags to a third party. You're creating a critical path hard dependency and for most apps this is going to be hit multiple times on every request and is something you're going to want to run locally.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @04:43PM (#12465903)
We've thought quite a bit about how to make this work as a service. The key to our architecture is that evaluating a feature flag for a user does not involve a remote call. We make that work by embedding a rule evaluation engine in our SDKs. When you request a flag, the user is compared against these rules (in memory) and served the appropriate variation.

We then use a streaming API to serve rule changes, so when you make a change to your dashboard, the new rules are streamed to all your backend servers within a few hundred milliseconds.

If you need even more resiliency, you can deploy a small service in your own infrastructure (https://github.com/launchdarkly/ld-relay) that allows you to persist these flag configurations in Redis.

Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by rch on Friday September 09, 2016 @05:46PM (#12466279)
Your rules engine sounds interesting - definitely worth a closer look.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by edoceo on Friday September 09, 2016 @02:32PM (#12464868)
My logs show frequent timeouts when looking up setting through their API. Their SDK default timeout is 3s. Wouldn't want to wait longer. Maybe not serving all 4B
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by chatmasta on Friday September 09, 2016 @01:54PM (#12464540)
Is averaging 4.6k requests per second really that much?
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @01:56PM (#12464565)
I think it depends on the workload. Serving 4.6k static pages per second, cached on a CDN, is not too difficult. However, handling an analytics workload of 4.6K RPS is a little harder.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by wsh91 on Friday September 09, 2016 @05:06PM (#12466048)
Albeit doable with Lambda and Cassandra. ;)
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by jkodumal on Friday September 09, 2016 @05:09PM (#12466073)
I mentioned this briefly in the article, but we thought about doing something with Lambda + API Gateway. But doing the math, 5k RPS pushed through API Gateway is about $1500 daily just to authenticate.
Re:How LaunchDarkly Serves Over 4B Feature Flags Daily by blibble on Friday September 09, 2016 @03:59PM (#12465594)
that's almost 512mb of 0/1 bits a day!

(not serious, I suspect they're wrapped in a 5kb HTTP request)