Feature flags play an important role in our ability to release quickly and minimise risk. However, maintaining several feature flags in parallel means we increase testing complexity – there are many permutations of features being enabled/disabled. This presents us with a challenge for our automation suite. We want to make sure we’re covering the functionality under the influence of the flags, but we can only test this if it’s enabled. Therefore, we need to make our tests understand and adapt to the current feature flag configuration.
Back to basics
When we’re increasing the complexity of our suite, it’s sensible to remember some fundamentals to our automation approach to ensure we keep control of our suite.
Planning
When a feature has been specified, and we know the design includes the use of one or more feature flags, we should identify and communicate that there is a need to extend our tests. We should aim to build in support as soon as we can. Initially, we are likely to expect them to be disabled, but this doesn’t mean we can’t read the flag settings and begin to make changes to conditions.
Documentation
When we start stringing together multiple conditions and paths through our scripts, it can be quite difficult to follow. To help others understand the test coverage, it’s sensible to keep some basic documentation that describes variable usage and decisions. Remember to update it with new feature coverage and variables.
Maintenance
With most feature flags, we should plan to retire them once the functionality is trusted and established. If we just leave our tests when this happens, we’re going to carry unnecessary complexity. Therefore, we need to make sure we continue to monitor the persistence of the flags and react according. Removing conditions should be a quick task. If we’ve been keeping our documentation current, this should help us to quickly identify what need to be removed and where it is used.
Now we’ve reiterated the fundamentals, let’s work through an example to explain how we might cover the testing of feature flagged functionality.
Worked Example
Let’s use ebuyer.com as the basis for our worked example. We’re going to pretend their two deferred payment methods, PayPal Credit and Klarna, are each controlled by feature flags. This is our feature flags configuration page:
We’re going to focus on the product detail page (PDP) and cart page for our script examples.
For the PDP, we’re going to assume the following functional logic applies:
- The general “0% Interest Offers Available” message appears if either main flag is enabled
- The PayPal Credit section appears if both the main PayPal Credit flag and the child PDP flag are enabled
- Both options appear available if the item price is £99 or above, otherwise they are unavailable
- The Klarna section appears if both the main Klarna flag and the child PDP flag are enabled
- If the item value is between £49 and £750, Pay In Three is available
- If the item value is £300 or above, 0% Finance is available

For the cart, this is our understanding of the related functionality:
- The PayPal Credit section appears if both the main PayPal Credit flag and the cart flag are enabled
- Both options appear available if the cart total is £99 or above, otherwise they are unavailable
- The Klarna section appears if both the main Klarna flag and the child PDP flag are enabled
- If the cart total is between £49 and £750, Pay In Three is available
- If the cart total is £300 or above, 0% Finance is available
Extracting the feature flag configuration
The first thing we’re going to do is define our variables and make sure they’re all set to 0 (false) to begin with. This isn’t a required step, but it’s useful to have a list of all our variables in one place and we can be certain they all have the starting value we expect.
Then we’re going to create a script module to check which feature flags are enabled. To do this, we make use of conditional steps (see the dynamic tests article for an intro to these). As our flags use check boxes, we’re just looking for the ‘checked’ attribute, and setting the related variable to 1 if it’s present. We want to inherit the steps from this script in any journey that reacts to the feature flag configuration (which may well be all of them).
The basic JavaScript for the conditional step has the same structure for all steps (see below). If the condition is met then we set our variable to 1.
return document.querySelector("#ppCreditMain[checked]");
Code language: JavaScript (javascript)
Testing with the features disabled
On the assumption that we’re starting out with the feature flags disabled, we should first write our tests to make sure the new functionality remains hidden with the flags disabled. We need to make sure the “0% Interest Offers Available” is not shown and that neither deferred payment method is shown. For our conditions, we’re simply checking if our main PayPal Credit and Klarna flags are set to false.
if ( ({{paypalcredit}} === 0) && ({{klarna}} === 0) ) {
return true;
}
Code language: JavaScript (javascript)
We then check each each flag in isolation to make sure the related section on the PDP and cart is hidden. Of course, as well as the specific PDP and cart flags, the section would also be hidden if the top level payment method flag was disabled:
if ( ({{paypalcredit}} === 0) || ({{paypalcreditpdp}} === 0) ) {
return true;
}
Code language: JavaScript (javascript)
Testing the new features
The basic flag-on tests are really just the opposite of the above, but need additional consideration for the more granular messaging. For PayPal Credit, the availability of both payment options is determined by the same logic. We check the combination of our feature flags and the item price:
if ( ({{paypalcredit}} === 1) && ({{paypalcreditpdp}} === 1) && ({{price}} >= 99) ) {
return true;
}
Code language: JavaScript (javascript)
The class on the <li> lets us differentiate between available and unavailable, so we use this to check whether the element is visible on the page.
Of course, we also want to check the negative cases, where the payment options are unavailable because they fall outside of the supported price ranges. This is a simple case of reversing some of the conditions – the example below checks outside of the accepted price range for the 3 payments option in the cart:
if ( ({{klarna}} === 1) && ({{klarnacart}} === 1) && (({{totalprice}} < 49) || ({{totalprice}} > 750)) ) {
return true;
}
Code language: JavaScript (javascript)
As you can see, the conditions and script structure for the cart tests will be very similar. The primary difference is that we use the total price rather than item price. Import the full script to see the full set of tests. We now have a set of verification steps in place that will adapt to the different feature flag configurations.
Configuring the tests in Ghost Inspector
Suite setup
Script steps example
Extracting the feature flag configuration
PayPal Credit on the PDP
Klarna in the cart
Script export
The full script contains many more steps than are shown here. Feel free to set the start URL to any ebuyer.com product page to see how the test adapts. Note that the Feature Flags script will not run successfully, as the flag configuration page doesn’t actually exist. Simply import the .json files into Ghost Inspector.