Go FlagZ
Dynamic, thread-safe flag variables that can be modified at runtime through etcd
or Kubernetes.
For a similar project for JVM languages (Java, scala) see java-flagz
This sounds crazy. Why?
File-based or command-line configuration can only be changed when a service restarts. Dynamic flags provide
flexibility in normal operations and emergencies. Two examples:
- A new feature launches that you want to A/B test. You want to gradually enable it for a certain fraction of user
requests (1%, 5%, 20%, 50%, 100%) without the need to restart servers. - Your service is getting overloaded and you want to disable certain costly features. You can't afford
restarting because you'd lose important capacity.
All of this can be done simultaneously across a whole shard of your services.
Features
- compatible with popular
flagreplacementspf13/pflag(e.g. ones usingspf13/cobra) - dynamic
flagthat are thread-safe and efficient:DynInt64DynFloat64DynStringDynDurationDynStringSliceDynJSON- aflagthat takes an arbitrary JSON structDynProto3- aflagthat takes aproto3struct in JSONpb or binary form
validatorfunctions for eachflag, allows the user to provide checks for newly set valuesnotifierfunctions allow user code to be subscribed toflagchanges- Kubernetes
ConfigMapwatcher, see configmap/README.md. etcdbased watcher that syncs values from a distributed Key-Value store into the program's memory- Prometheus metric for checksums of the current flag configuration
- a
/debug/flagzHandlerFunc endpoint that allows for easy inspection of the service's runtime configuration
Here's a teaser of the debug endpoint:
Examples
Declare a single pflag.FlagSet in some public package (e.g. common.SharedFlagSet) that you'll use throughout your server.
Dynamic JSON flag with a validator and notifier
var (
limitsConfigFlag = flagz.DynJSON(
common.SharedFlagSet,
"rate_limiting_config",
&rateLimitConfig{ DefaultRate: 10, Policy: "allow"},
"Config for service's rate limit",
).WithValidator(rateLimitConfigValidator).WithNotifier(onRateLimitChange)
)
This declares a JSON flag of type rateLimitConfig with a default value. Whenever the config changes (statically or dynamically) the rateLimitConfigValidator will be called. If it returns no errors, the flag will be updated and onRateLimitChange will be called with both old and new, allowing the rate-limit mechanism to re-tune.
Dynamic feature flags
var (
featuresFlag = flagz.DynStringSlice(common.SharedFlagSet, "enabled_features", []string{"fast_index"}, "list of enabled feature markers")
)
...
func MyHandler(resp http.ResponseWriter, req *http.Request) {
...
if existsInStringSlice("fast_index", featuresFlag.Get()) {
doFastIndex(req)
}
...
}
All access to featuresFlag, which is a []string flag, is synchronised across go-routines using atomic pointer swaps.
Watching for changes from etcd
// First parse the flags from the command line, as normal.
common.SharedFlagSet.Parse(os.Args[1:])
w, err := watcher.New(common.SharedFlagSet, etcdClient, "/my_service/flagz", logger)
if err != nil {
logger.Fatalf("failed setting up %v", err)
}
// Read flagz from etcd and update their values in common.SharedFlagSet
if err := w.Initialize(); err != nil {
log.Fatalf("failed setting up %v", err)
}
// Start listening of dynamic flags from etcd.
w.Start()
The watcher's go-routine will watch for etcd value changes and synchronise them with values in memory. In case a value fails parsing or the user-specified validator, the key in etcd will be atomically rolled back.
More examples:
Status
This code is production quality. It's been running happily in production at Improbable for a few months.
Features planned:
- - #11 monitoring of
FlagSetchecksus using a Prometheus handler - - #12 support for standard
flag(requires changes inspf13/pflaginterfaces)
License
go-flagz is released under the Apache 2.0 license. See the LICENSE file for details.