> Make feature flags short-lived. Do not confuse flags with application configuration.
This is my current battle.
I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them.
Then another dev decided that these Feature Flags look like a great way to implement permanent application configs for different subsets of entities in our system. In fact, he evangelized it in his design for a major new project (I was not invited to the review).
Now I have to stand back and watch as the feature flags are being used for long-term configurations. I objected when I saw the misuse- in a code review I said "hey that's not what these are for"- and was overruled by management. This is the design, there's no time to update it, I'm sure we can fix it later, someday.
Lesson learned: make it very hard to misuse meta-features like feature flags, or someone will use them to get their stuff done faster.
Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse.
- Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI)
- Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its own mess), people are going to find it vastly easier to create feature flags with per-use override lists (almost all of them let you override on primary token). They will use your feature flags for:
- Preview features: "is this user in the preview group?"
- rollouts that might not ever go 100%: "should this organization use the old login flow?"
- business-critical attributes that it would be a major incident to revert to defaults: "does this user operate under the alternate tax regime?"
You can try to fight this (indeed, especially for that last one, you most definitely should!), but you will not ever completely win the feature flag ideological purity war!
We have an interesting hybrid between the two that I'd like your take on. When we release new versions of our web client static assets we have a version number that we bump that moves folks over to the new version.
1. We could stick it in a standard conf system and serve it up randomly based on what host a client hits. (Or come up with more sophisticated rollouts)
2. Or we can put it as "perm" conf in the feature flag system and roll it out based on different cohorts/segments.
I'm leaning towards #2 but I'd love to understand why you want to prohibit long lived keys so I can make a more informed choice. The original blog posts main reasons were that FF systems favor availability over consistency so make a pour tool if you need fast converging global config, which somewhat becomes challenging here during rollbacks but is likely not the end of the world.
Or... see them for what they are: runtime configuration. The name implies a use case scenario, but in reality it's just a configuration knob. With a good UI, it's a pretty damn convenient way to do runtime configuration.
So of course they'll be used for long-term configuration purposes, especially under pressure and for gradual rollouts of whole systems, not just A/B testing features.
There is a need for runtime configurations, yes, but it's important to put them behind an interface intended for that, and not one intended for something else.
Our FF system uses our config system as its system of record. There's some potential for misuse, and it's difficult to apply deadlines. On the plus side all our settings are captured in version control. Before they were spread out over several systems, one of which had an audit system that was pure tribal knowledge for years.
The main challenge is when things go wrong. Feature flags are designed for high-rate evaluation with low latency responses. Configuration usually doesn't care that much about latency as it's usually read once at startup. This context leads to some very specific tradeoffs such as erring to availability over consistency, which in the case of configuration management could be a bad choice
Yeah, and assuming they are done well, they probably have better analytics and insights attached to them than anything else except perhaps your experiments!
Long lived features flags is a development process bug, I'm not sure we can solve it with the feature toggle system.
I'm at the point of deciding that Scrum is fundamentally incompatible with feature flags. We demo the code long before the flag has been removed, which leads to perverse incentives. If you want flags to go away in a timely manner you need WIP limits, and columns for those elements of the lifecycle. In short: Kanban doesn't (have to) have this problem.
And even the fixes I can imagine like the above, I'm not entirely sure you can stop your bad actor, because it's going to be months before anyone notices that the flags have long overstayed their welcome.
I'm partial to flags being under version control, where we have an audit trail. However time and again what we really need is a summary of how long each flag has existed, so they can be gotten rid of. The Kanban solution I mention above is only a 90% solution - it's easy to forget you added a flag (or added 3 but deleted 2)
I faced something similar, and I think it's unavoidable. Give people a screwdriver and they'll find a way of using it as a hammer.
The best you can do is expect the feature flagging solution to give some kind of warning for tech debt. Then equip them with alternative tools for configuration management. Rather than forbidding, give them options, but if it's not your scope, I'd let them be (I know as engineers this is hard to do :P).
> Give people a screwdriver and they'll find a way of using it as a hammer.
I feel like feature flags aren't that far off though. They're fantastic for many uses of runtime configuration as mentioned in another comment.
There's multiple people in this thread complaining about "abuse" of feature flags but no one has been able to voice why it's abuse instead of just use beyond esoteric dogma.
I feel like this is a solvable problem:
1) make feature flags be configured to have an expiration date. If over the expiration date, auto-generate a task to clean up your FF
2) If you want to be extra fancy, set up a codemod to automatically clean up the FF once it's expired
I don't see the problem with developers using flags for configuration as a stopgap until there's a better solution available.
Sounds like "other dev" found some business case they could unblock with existing system, and you thought the business was better off not solving that, or finding a more expensive solution.
Curious how you plan to justify cost to "fix it" to management. If it ain't broke...
I think it's better to admit they actually are config, just a different kind of config that comes with an expiration date.
Accepting reality in this way means you'll design a config management system that lets you add feature flags with a required expiration date, and then notifies you when they're still in the system after the deadline.
I agreed. My perspective is that there are two kinds of feature flags: temporary and permanent.
Temporary ones can be used to power experiments or just help you get to GA and then can be removed.
Permanent ones can be configs that serve multiple variations (e.g. values for rate limits), but they can also be simple booleans that manage long term entitlements for customers (like pricing tiers, regional product settings, etc.)
We did the same. We were early adopters of unleash and wrangled it to also host long term application configuration and even rule based application config.
The architecture of unleash made it so simple to do in unleash vs having to evaluate, configure, and deploy a separate app config solution.
It’s one of the main reasons to start with something like unleash because they have stale flag warnings built in. Plus, since you already have a UI it’s harder for it to be hijacked.
For those that dont know about the project, check out Open Feature https://openfeature.dev/ which is sort of like Open Telemetry but for feature flags. Helps avoid vendor lock in. We're a young project and looking for help and to build the community!
This feels a bit like the dicta on 12 Factor: rules handed down from a presumed authority without any discussion of the tradeoffs. Engineering is tradeoff evaluation. Give me some discussion about the alternatives, when and why they're inferior and don't pretend like the proposed solution doesn't have shortcomings or pitfalls.
I agree with you that tradeoff evaluation is crucial in engineering, but I don't see the 12 Factor methodology as a set of strict rules. They're more like guidelines that are generally a good idea to follow for building modern applications or services. Some of the suggestions apply for any type of software, like having a single version controlled codebase, separate build/release/run stages, and using stateless processes.
So it's good to be aware of _why_ those guidelines are considered a good thing, but as with any methodology, an engineer should be pragmatic in deciding when to follow it strictly, and when to adapt or ignore some of it.
That said, I wouldn't want to work on software that completely ignores 12 Factor.
It's true that there are more long-lived use cases, but if you have the ability to choose, runtime controlled ones cover both cases, while compile time only cover some use cases. But fair point
I dedicated a day to evaluating feature flag software based on specific criteria:
- Must support multiple SDKs, including Java and Ruby.
- Should be self-hosted with PostgreSQL database support.
- Needs to enable remote configuration for arbitrary values (not just feature flags). I don't run two separate services for this.
- Should offer some UI functionality.
- it should cache flag values locally and, ideally, provide live data updates (though pooling is acceptable).
Here are the four options that met these basic criteria and underwent detailed evaluation:
- Unleash: Impressive and powerful, but its UI is more complex than needed, and it lacks remote configuration.
- Flagsmith: Offers remote configuration but appears less polished with some features not working smoothly; Java SDK error reporting needs improvement.
- Flipt: Simple and elegant, but lacks remote configuration and local caching for Java SDK.
- FeatureHub: Offers fewer features than Unleash and Flagsmith; its Java API seems somewhat enterprisly but supports remote configuration and live data updates.
Currently, I'm leaning towards FeatureHub. If remote configuration isn't necessary, Unleash offers more features, and if simplicity is key and local caching isn't needed, Flipt is an attractive option.
Hey thanks for giving Flipt a look! I'm the creator of Flipt so would love to chat more about your needs to see how we could make it work for your use case! We're actively looking into providing local caching for all our SDKs btw and would love to learn more about what your requirements are for remote configuration as it's also on our radar!
Feel free to send me an email at: mark (at) flipt.io.
As an engineer, I am generally against feature flags.
They fracture your code base, are sometimes never removed, and add complexity and logic that at best is a boolean check and at worse is something more involved.
I'd love a world where engineers are given time to complete their feature in its entirety, and the feature is released when it is ready.
Sadly, we do not live in that world and hence: feature flags.
This misses the point. A big point of feature flags is that you don't yet know how features will be perceived until you get them in front of real users.
I get what you'd like "as an engineer", but it ignores the needs of the business.
Isn't that the job a product manager? There are other means and methodologies for gathering user sentiment before you go and build something.
You should get as close as you can, release the product, and iterate.
Todays world is release the product in some ramshackle form or fashion, collect feedback, iterate. To do that introduces a new construct of Feature Flags that would otherwise not be necessary.
That is not what feature flags are typically used for.
They're typically used as a way of enabling a change for a subset of your services to allow for monitoring of the update and easier "rollback" if it becomes necessary.
They can be used for A/B testing, but this is not what they're typically used for.
I just item 1 (“Enable run-time control. Control flags dynamically, not using config files”) and it’s almost exclusively focused on what to do but not on why to do it.
It seems to be skipping past the use-cases and assumptions, in particular, describing what a system with feature flags looks and acts like, what the benefits and drawbacks are.
Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly.
I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause production problems. One thing you can do to help with this is run proxies closer to your services. For example, LaunchDarkly has an open source "Relay".
Local evaluation seems to be pretty standard at this point, although I'd argue that delivering flag definitions is (relatively) easy. One of the real value-add of a product like LaunchDarkly is all the things they can do when your applications send evaluation data upstream: unused flags, only-ever-evaluated-to-the-default flags, only-ever-evaluated-to-one-outcome flags, etc.
One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). I'd argue the same practice should be taken with metrics names.
One of the most useful things to know (but seldom communicated clearly near landing pages) is a basic sketch of the architecture. It's necessary to know how things will behave if there is trouble. For instance: our internal system uses ZK to store (protobuf) flag definitions, and applications set watches to be notified of changes. LaunchDarkly clients download all flags[1] in the project on connection, then stream changes.
If I were going to build a feature flag system, I would ensure that there is a global, incrementing counter that is updated every time any change is made, and make it a fundamental aspect of the design. That way, clients can cache what they've seen, and easily fetch only necessary updates. You could also imagine annotating that generation ID into W3C Baggage, and passing it through the microservices call graph to ensure evaluation at a consistent point in time (clients would need to cache history for a minute or two, of course).
One other dimension in which feature flag services vary is by the complexity of the rules they allow you to evaluate. Our internal system has a mini expression language (probably overkill). LaunchDarkly's arguably better system gives you an ordered set of rules within which conditions are ANDed together. Both allow you to pass in arbitrary contexts of key/value pairs. Many open source solutions (Unleash, last I checked, some time ago) are more limited: some of them don't let you vary on inputs, some only a small set of prescribed attributes.
I think the time is ripe for an open standard client API for feature flags. I think standardizing the communication mechanisms would be constricting, but there's no reason we couldn't create something analogous to (or even part of) the Open Telemetry client SDK for feature flags. If you are seriously interested in collaborating on that, please get in touch. (I'm "zellyn" just about everywhere)
[1] Yes, this causes problems if you have too many flags in one project. They have a pretty nice filtering solution that's almost fully ready.
One more update. I spent a little time the other day trying to find all the feature flag products I could. I'm sure I missed a ton. Let me know in the comments!
Here's my first draft of the questions you'd want to ask about any given solution:
Questionnaire
- Does it seem to be primarily proprietary, primarily open-source, or “open core” (parts open source, enterprise features proprietary)?
- If it’s open core or open source with a service offering, can you run it completely on your own for free?
- Does it look “serious/mature”?
- Lots of language SDKs
- High-profile, high-scale users
- Can you do rules with arbitrary attributes or is it just on/off or on/off with overrides?
- Can it do complex rules?
- How many language SDKs (one, a few, lots)
- Do feature flags appear to be the primary purpose of this company/project?
- If not, does it look like feature flags are a first-class offering, or an afterthought / checkbox-filler? (eg. split.io started out in experimentation, and then later introduced free feature flag functionality. I think it’s a first-class feature now.)
- Does it allow approval workflows?
- What is the basic architecture?
- Are flags evaluated in-memory, locally? (Hopefully!)
- Is there a relay/proxy you can run in your own environment?
- How are changes propagated?
- Polling?
- Streaming?
- Does each app retrieve/stream all the flags in a project, or just the ones they use?
- What happens if their website goes down?
- Do they do experiments too?
- As a first-class offering?
- Are there ACLs and groups/roles?
- Can they be synced from your own source of truth?
- Do they have a solution for mobile and web apps?
- If so, what is the pricing model?
- Do they have a mobile relay type product you can run yourself?
- What is the pricing model?
- Per developer?
- Per end-user? MAU?
> One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant).
Can you elaborate on this? As a programmer, I would think that using something like a constant would help us find references and ensure all usage of the flag is removed when the constant is removed.
One of the most common things you want to do for a feature flag or metric name is ask, "Where is this used in code?". (LaunchDarkly even has a product feature that does this, called "Code References".) I suppose one layer of indirection (into a constant) doesn't hurt too much, although it certainly makes things a little trickier.
The bigger problem is when the code constructs metric and flag names programmatically:
That kind of thing makes it very hard to find references to metrics or flags. Sometimes it's impossible, or close to impossible to remove, but it's worth trying hard.
Not OP but multiple code bases may refer to the same flag by a different constant. Having a single string that can be searched accross all repos in an organization is quite handy to find all places where it's referenced.
If you create your own service to evaluate a bunch of feature flags for a given user/client/device/location/whatever and return the results, for use in mobile clients (everyone does this), PLEASE *make sure the client enumerates the list of flags it wants*. It's very tempting to just keep that list server-side, and send all the flags (much simpler requests, right?), but you will have to keep serving all those flags for all eternity because you'll never know which deployed versions of your app require which flags, and which can be removed.
> I'd argue that delivering flag definitions is (relatively) easy.
I'd argue that coming up with good UI that nudges developers towards safe behavior, as well as useful and appropriate guard rails -- in other words, using the feature flag UI to reduce likelihood of breakage -- is difficult, and one of the major value propositions of feature flag services.
Readit News
This is my current battle.
I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them.
Then another dev decided that these Feature Flags look like a great way to implement permanent application configs for different subsets of entities in our system. In fact, he evangelized it in his design for a major new project (I was not invited to the review).
Now I have to stand back and watch as the feature flags are being used for long-term configurations. I objected when I saw the misuse- in a code review I said "hey that's not what these are for"- and was overruled by management. This is the design, there's no time to update it, I'm sure we can fix it later, someday.
Lesson learned: make it very hard to misuse meta-features like feature flags, or someone will use them to get their stuff done faster.
1. We could stick it in a standard conf system and serve it up randomly based on what host a client hits. (Or come up with more sophisticated rollouts)
2. Or we can put it as "perm" conf in the feature flag system and roll it out based on different cohorts/segments.
I'm leaning towards #2 but I'd love to understand why you want to prohibit long lived keys so I can make a more informed choice. The original blog posts main reasons were that FF systems favor availability over consistency so make a pour tool if you need fast converging global config, which somewhat becomes challenging here during rollbacks but is likely not the end of the world.
So of course they'll be used for long-term configuration purposes, especially under pressure and for gradual rollouts of whole systems, not just A/B testing features.
The term "feature flag" has come to inherently have a time component because features are supposed to eventually be fulled GA'd.
What I've seen in practice is feature flags are never removed so a better way to think about them is as a runtime configuration.
I'm at the point of deciding that Scrum is fundamentally incompatible with feature flags. We demo the code long before the flag has been removed, which leads to perverse incentives. If you want flags to go away in a timely manner you need WIP limits, and columns for those elements of the lifecycle. In short: Kanban doesn't (have to) have this problem.
And even the fixes I can imagine like the above, I'm not entirely sure you can stop your bad actor, because it's going to be months before anyone notices that the flags have long overstayed their welcome.
I'm partial to flags being under version control, where we have an audit trail. However time and again what we really need is a summary of how long each flag has existed, so they can be gotten rid of. The Kanban solution I mention above is only a 90% solution - it's easy to forget you added a flag (or added 3 but deleted 2)
The best you can do is expect the feature flagging solution to give some kind of warning for tech debt. Then equip them with alternative tools for configuration management. Rather than forbidding, give them options, but if it's not your scope, I'd let them be (I know as engineers this is hard to do :P).
I feel like feature flags aren't that far off though. They're fantastic for many uses of runtime configuration as mentioned in another comment.
There's multiple people in this thread complaining about "abuse" of feature flags but no one has been able to voice why it's abuse instead of just use beyond esoteric dogma.
I don't see the problem with developers using flags for configuration as a stopgap until there's a better solution available.
Um what? How could that ever work. It's like you are trying to find new exciting ways to break prod.
Curious how you plan to justify cost to "fix it" to management. If it ain't broke...
Accepting reality in this way means you'll design a config management system that lets you add feature flags with a required expiration date, and then notifies you when they're still in the system after the deadline.
Temporary ones can be used to power experiments or just help you get to GA and then can be removed.
Permanent ones can be configs that serve multiple variations (e.g. values for rate limits), but they can also be simple booleans that manage long term entitlements for customers (like pricing tiers, regional product settings, etc.)
The architecture of unleash made it so simple to do in unleash vs having to evaluate, configure, and deploy a separate app config solution.
So it's good to be aware of _why_ those guidelines are considered a good thing, but as with any methodology, an engineer should be pragmatic in deciding when to follow it strictly, and when to adapt or ignore some of it.
That said, I wouldn't want to work on software that completely ignores 12 Factor.
For a more nuanced and careful discussion of the topic I like to reference: https://martinfowler.com/articles/feature-toggles.html
They fracture your code base, are sometimes never removed, and add complexity and logic that at best is a boolean check and at worse is something more involved.
I'd love a world where engineers are given time to complete their feature in its entirety, and the feature is released when it is ready.
Sadly, we do not live in that world and hence: feature flags.
I get what you'd like "as an engineer", but it ignores the needs of the business.
You should get as close as you can, release the product, and iterate.
Todays world is release the product in some ramshackle form or fashion, collect feedback, iterate. To do that introduces a new construct of Feature Flags that would otherwise not be necessary.
They're typically used as a way of enabling a change for a subset of your services to allow for monitoring of the update and easier "rollback" if it becomes necessary.
They can be used for A/B testing, but this is not what they're typically used for.
It seems to be skipping past the use-cases and assumptions, in particular, describing what a system with feature flags looks and acts like, what the benefits and drawbacks are.
This is a great feedback. Our intention was to describe how such a system work at scale, but I see we could do better in this section, thanks!
Do you have some use-cases in mind?
I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause production problems. One thing you can do to help with this is run proxies closer to your services. For example, LaunchDarkly has an open source "Relay".
Local evaluation seems to be pretty standard at this point, although I'd argue that delivering flag definitions is (relatively) easy. One of the real value-add of a product like LaunchDarkly is all the things they can do when your applications send evaluation data upstream: unused flags, only-ever-evaluated-to-the-default flags, only-ever-evaluated-to-one-outcome flags, etc.
One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). I'd argue the same practice should be taken with metrics names.
One of the most useful things to know (but seldom communicated clearly near landing pages) is a basic sketch of the architecture. It's necessary to know how things will behave if there is trouble. For instance: our internal system uses ZK to store (protobuf) flag definitions, and applications set watches to be notified of changes. LaunchDarkly clients download all flags[1] in the project on connection, then stream changes.
If I were going to build a feature flag system, I would ensure that there is a global, incrementing counter that is updated every time any change is made, and make it a fundamental aspect of the design. That way, clients can cache what they've seen, and easily fetch only necessary updates. You could also imagine annotating that generation ID into W3C Baggage, and passing it through the microservices call graph to ensure evaluation at a consistent point in time (clients would need to cache history for a minute or two, of course).
One other dimension in which feature flag services vary is by the complexity of the rules they allow you to evaluate. Our internal system has a mini expression language (probably overkill). LaunchDarkly's arguably better system gives you an ordered set of rules within which conditions are ANDed together. Both allow you to pass in arbitrary contexts of key/value pairs. Many open source solutions (Unleash, last I checked, some time ago) are more limited: some of them don't let you vary on inputs, some only a small set of prescribed attributes.
I think the time is ripe for an open standard client API for feature flags. I think standardizing the communication mechanisms would be constricting, but there's no reason we couldn't create something analogous to (or even part of) the Open Telemetry client SDK for feature flags. If you are seriously interested in collaborating on that, please get in touch. (I'm "zellyn" just about everywhere)
[1] Yes, this causes problems if you have too many flags in one project. They have a pretty nice filtering solution that's almost fully ready.
[Update: edited to make 70% of it not italics ]
LaunchDarkly Split Apptimize CloudBees ConfigCat DevCycle FeatBit FeatureHub Flagsmith Flipper Flipt GrowthBook Harness Molasses OpenFeature Posthog Rollout Unleash
Here's my first draft of the questions you'd want to ask about any given solution:
I will toss our hat in the ring but we are early in this space! https://lekko.com
This seems like a MUST rather than a SHOULD, right?
Can you elaborate on this? As a programmer, I would think that using something like a constant would help us find references and ensure all usage of the flag is removed when the constant is removed.
The bigger problem is when the code constructs metric and flag names programmatically:
That kind of thing makes it very hard to find references to metrics or flags. Sometimes it's impossible, or close to impossible to remove, but it's worth trying hard.Of course, this is just, like, my opinion, man!
If you create your own service to evaluate a bunch of feature flags for a given user/client/device/location/whatever and return the results, for use in mobile clients (everyone does this), PLEASE *make sure the client enumerates the list of flags it wants*. It's very tempting to just keep that list server-side, and send all the flags (much simpler requests, right?), but you will have to keep serving all those flags for all eternity because you'll never know which deployed versions of your app require which flags, and which can be removed.
[Edit: speling]
I'd argue that coming up with good UI that nudges developers towards safe behavior, as well as useful and appropriate guard rails -- in other words, using the feature flag UI to reduce likelihood of breakage -- is difficult, and one of the major value propositions of feature flag services.