Claude can 100% build you your own basecamp quickly, but there is a lot of good thinking that has gone into a product. Edge cases. Integrations. Clarity of thinking and ways to extend it. Agents love standing on the shoulders of good crystalized cognition (grep, curl etc), but I think that extends even beyond base CLI tools.
Which is not to say that "cheapest alternative" won't be much more important than it is now, but the tools that will succeed are the ones that solve clear problems with accessible CLI/tool-call like interfaces and take significant load off the agent running, while doing it efficiently and cost effectively so they are clearly better than building it yourself.
To be fair, there will be a ton of margin collapse and that will be unachievable by a lot of current orgs / structures.
entitlements is a huge one. Sooooo many orgs are in an absolute rats nest of confusion because half their entitlements are in something like an auth system, half are in something like billing and the third half is hacked into feature flags.
One particular pain point is that finance or something "do users with access to feature X retain better and it's super hard to figure out.
My overall feel is that developers are trying to solve a problem and that it turns out the configuration is as fundamentally as important as the software itself. I define "dynamic configuration" as "a key value store which takes context and has rules based system to give you a value". This primitive turns out to be extraordinarily useful and powerful. Rather than try to split it into N different systems, each custom / don't have telemetry / aren't available to all services SDKs. What if you have ONE BIG CONFIG system that really does a bang up job.
(Note: copying the big boys is not always the right move, but https://research.facebook.com/publications/holistic-configur... does give some credence that this is a decent approach)
By focussing on one terrific system, you can put all your eggs into the basket of making that reliable / comprehensible / flexible / strict.
The one on your list that I think DOESN'T belong would be permission checks at user level. The N of that is not a good fit for config (though when you look at the facebook paper, it's pretty wild how they've scaled it (but it doesn't do permissions afaik)).
First one got acquired. But also I wish I'd built it differently from the start (git based, other lessons learned), so started over.
It's true that it's a pretty simple thing, but you do want them to be of the utmost level of reliability. Its ideal if you can use feature flags & dynamic config (which I think of as the same problem) right at boot time. But then of course you have an external dependency to booting you app. So the reliability angle becomes imperative. So I do think the SaaS can play a good (or bad) role in overall reliability vs something you implement in house.
The other major advantage is telemetry. Once you have things toggling on and off, immediately engineers will get confused about whether something is on or off and why. Basic internal apps usually don't have any good telemetry / debugging. So a good SaaS should give you all kinds of ways to ensure that you can see what context is being passed around to the SDK to do their eval. Sanity checking etc.
My og experience was at HubSpot where we used the hell out of dynamic config. Once you get used to things like instantly targeting debug log levels for a single class to a single org its hard to go back.
For Quonfig I landed on:
- Data model wise they are identical. Flags and configs can both be targeted. They can both use segments. They can both do partial rollouts. They can both have the same range of values (bool/number/duration/json/json-w-schema/etc)
- Its fine to use Flags for your Experiments/AB Tests, but that's just the "allocation engine". The rest of experimentation is the exposure tracking and goal tracking. Those ought live in your product analytics stack, because they are really just events. But its helpful to have a single place (flags) for the experiment allocation because then you can re-use segments for things like hold out groups / and just general targeting.
- The only real difference is that Flags are intended to be ephemeral and configs are intended to be permanent. A good UI should show you how long a flag has been alive and help you clean it up (or convert it to a config) if it's been true for everyone for too long.
- The use cases are different enough that it's worth keeping them in two separate UI.
Every project should have flags, but many projects need just the basics and a service is overkill.
Rolling your own JSON still feels like something we ought avoid though. Yes to start out it’s 95% booleans. But then you want a rollout. And then you want some targeting rules. And then you want non booleans… maybe some json. Oo wouldn’t it be nice if the json could conform to a schema… and then eventually you are like damn I really want to change these without deploying. Or you want to read the same flag from multiple services.
I’ve tried to incorporate this lowest common denominator into https://quonfig.com Use it totally free & open source as SDK, and it’s just loading JSON that you can track in git. Agents love it, hot reloads, SDK in lots of languages. But vs rolling your own you’ve got a lot of headroom on the design. A bunch of targeting operators. Segments etc. And then if you do want to get a nice UI / delivery network for real time updates, then you can use the paid side of things.
Local use description: https://docs.quonfig.com/docs/how-tos/open-source-local
And taking a second crack at a feature flags with https://quonfig.com because I seem to be unable to put this problem down and finally feel like I have the right git native and agent first approach.
I had a $5/month thing. Made a $20/month option that barely had more features. Sold lotsa seats.
Rewrote it. $10 and $50 plans. $50 is most popular.
But a day after I launched I got a new payment for $360. Damn, I thought, Claude must've screwed something up.
Nope, just somebody going straight into the Yearly option of the $50 (that I hadn't realized Claude added (with a discount)).
I would imagine that for a non-engineer trying to code it would be quite useful / deliver a better result / less liable to end up in total mess. But for experienced engineers it quickly felt like overkill / claude itself just gets better and better. Particularly once we got agent swarms I left GSD and don't think I'll be back. But I would recommend it to non coders trying to code.
I've done 100s of interviews over the years. But haven't been doing any for the past 3.But now we're scaling, so need to figure out how to do a technical interview again. But... man, all my technical interview methods feel obsolete / inadequate. Seems pretty silly to not let a candidate use Cursor/Claude when I would 100% be expecting them to use it every day. But also if we just vibe something...
Context: I run a FF company (https://prefab.cloud/)
There are multiple distinct benefits to be had from feature flagging. Because it's the "normal" path, most FF products bundle them all together, but it's useful to split them out.
- The code / libraries for evaluating rules. - The UI for creating rules, targeting & roll outs. - The infrastructure for hosting the flags and providing real-time updates. - Evaluation tracking / debugging to help you verify what's happening.
If you don't need #1 and #2 there, you might decide to DIY and build it yourself, but I think you shouldn't have to. Most feature flag tools today are usable in an offline mode. For Prefab it is: https://docs.prefab.cloud/docs/how-tos/offline-mode You can just do a CLI command to download the flags. Then boot the client off a downloaded file. With our pricing model that's totally free because we're really hardly doing anything for you. Most people use this functionality for CI environments, but I think it's a reasonable way to go for some orgs. It has 100% reliability and that's tough to beat.
You can do that if you DIY too, but there's so many nice to haves in actually having a tool / UI that has put some effort into it that I would encourage people not to go down that route.
Saying gRPC hides the internals is a joke. You’ll get internals all right, when you’re blasting debug logging trying to figure out what the f is going on causing 1/10 requests to fail and fine tuning 10-20 different poorly named and timeout / retry settings.
Hours lost fighting with maven plugins. Hours lost debugging weird deadline exceeded. Hours lost with LBs that don’t like the esoteric http2. Firewall pain meaning we had to use Standard api anyway. Crappy docs. Hours lost trying to get error messages that don’t suck into observability.
I wish I’d never heard of it.
It's particularly worth taking a look if you have a postgres table that has the potential to become enormous.
I remember spending an extra $30 to at 2am to overnight "Gripe Water" in the hope that it would get my kid to sleep. #takeMyMoney
I hope 2024 is the year where we realize that if we make the log levels dynamically update-able we can have our cake and eat it too. We feel stuck in a world where all logging is either useless bc it's off or on and expensive. All you need is a way to easily modify log level off without restarting and this gets a lot better.
Feels like 100 years ago.
Just got meta glasses. Curious whether they'll steal another 15% of my traffic. Primarily the "looking up a random fact for my kids" volume.
If you’re lookin for a book that has absolutely kept this sense of wonder, “Immune” by Philipp Dettmer has this in spades. Highly recommended.
we screwed around with it on a live stream: https://www.youtube.com/live/3YhBoox4JvQ?si=dkni5LY3EALnWVuE...
If you're writing something that will run on someone's local machine I think we're at the point where you can start building with the assumption that they'll have a local, fast, decent LLM.
I won't spoil the surprise, but I did end up with one takeaway that I'll remember going forward when diagnosing.
1. Some of these are primarily updated in a UI, some are driven from some other system. Flags are customer UI, but Auth is often customer's customer (admins setting RBAC) or derived from SAML Role / etc.
2. There are very different personas / different blast-radiuses for different changes. This means that there should be different interfaces for the creation and administration of the different type of things.
3. Different scales. Dynamic configuration / feature flags are great, but you don't want the rule payload to be GB, which you would have if you tried to store fine grained permissions inside it. Zanzibar is awesome and should be available to more people.
4. It would be best if we could all agree on the names & types of these objects. I think schema files and code-generation are underused.
5. My inclination is that entitlements are just feature flags under the covers, but that there's enough different that a custom UX is warranted.
6. It's easy to add a ton of latency to applications with this stuff. First we spend 50ms getting the user... then we go get the billing for 50ms... then we get the entitlements for 50ms... There are big gains to be had by having this solved holisticaly.
hmu anytime jdwyer at prefab.cloud would love to hear your thoughts.
I'm really interested in getting stronger opinions here about how to set people up for success. Being the founder of a feature flag company I feel like I'm a rope dealer.
I'm really interesting in providing good authorization primitives as well. This is kinda the whole reason/vision for why Prefab exists. If you have an authorization tool and a separate feature flag tool and a separate billing tool. You're going to be tempted to do some weird un-holy things. If you can use one provider for it, then we can really help when it comes to putting things in the right place. My dream is that you can type user.can_do?(:thing) and get back a response that has checked the authorization, ff & entitlements system and gives you a clear and comprehensible answer.
Sooo many usage based pricing things out there (ironically with totally non-transparent pricing), but I agree that it doesn't feel like the right solution has been built yet.
I'm excited/experimenting with attaching https://ollama.com/ though. It is so much faster it's starting to feel ok. A lot changes when we just assume that everyone's computer can run LLM locally I think.