Test Design Feature Flags Title

Test Design: Feature Flags

Feature flags play an important role in our ability to release quickly and minimise risk. However, maintaining several feature flags in parallel means we increase testing complexity – there are many permutations of features being enabled/disabled. This presents us with a challenge for our automation suite. We want to make sure we’re covering the functionality under the influence of the flags, but we can only test this if it’s enabled. Therefore, we need to make our tests understand and adapt to the current feature flag configuration.

Back to basics

When we’re increasing the complexity of our suite, it’s sensible to remember some fundamentals to our automation approach to ensure we keep control of our suite.

Planning

When a feature has been specified, and we know the design includes the use of one or more feature flags, we should identify and communicate that there is a need to extend our tests. We should aim to build in support as soon as we can. Initially, we are likely to expect them to be disabled, but this doesn’t mean we can’t read the flag settings and begin to make changes to conditions.

Documentation

When we start stringing together multiple conditions and paths through our scripts, it can be quite difficult to follow. To help others understand the test coverage, it’s sensible to keep some basic documentation that describes variable usage and decisions. Remember to update it with new feature coverage and variables.

Maintenance

With most feature flags, we should plan to retire them once the functionality is trusted and established. If we just leave our tests when this happens, we’re going to carry unnecessary complexity. Therefore, we need to make sure we continue to monitor the persistence of the flags and react according. Removing conditions should be a quick task. If we’ve been keeping our documentation current, this should help us to quickly identify what need to be removed and where it is used.

Now we’ve reiterated the fundamentals, let’s work through an example to explain how we might cover the testing of feature flagged functionality.

Worked Example

Let’s use ebuyer.com as the basis for our worked example. We’re going to pretend their two deferred payment methods, PayPal Credit and Klarna, are each controlled by feature flags. This is our feature flags configuration page:

eBuyer Feature Flag Config Screenshot

We’re going to focus on the product detail page (PDP) and cart page for our script examples.

For the PDP, we’re going to assume the following functional logic applies:

  • The general “0% Interest Offers Available” message appears if either main flag is enabled
  • The PayPal Credit section appears if both the main PayPal Credit flag and the child PDP flag are enabled
    • Both options appear available if the item price is £99 or above, otherwise they are unavailable
  • The Klarna section appears if both the main Klarna flag and the child PDP flag are enabled
    • If the item value is between £49 and £750, Pay In Three is available
    • If the item value is £300 or above, 0% Finance is available
eBuyer.com PDP Deferred Payments Screenshot
on the PDP, there’s a message under the price indicating finance is available, and a section for each method lower down

For the cart, this is our understanding of the related functionality:

  • The PayPal Credit section appears if both the main PayPal Credit flag and the cart flag are enabled
    • Both options appear available if the cart total is £99 or above, otherwise they are unavailable
  • The Klarna section appears if both the main Klarna flag and the child PDP flag are enabled
    • If the cart total is between £49 and £750, Pay In Three is available
    • If the cart total is £300 or above, 0% Finance is available
eBuyer.com Cart Deferred Payments Screenshot
in the cart, we have a similar section for both deferred payment methods

Extracting the feature flag configuration

The first thing we’re going to do is define our variables and make sure they’re all set to 0 (false) to begin with. This isn’t a required step, but it’s useful to have a list of all our variables in one place and we can be certain they all have the starting value we expect.

Feature Flag Variables Screenshot

Then we’re going to create a script module to check which feature flags are enabled. To do this, we make use of conditional steps (see the dynamic tests article for an intro to these). As our flags use check boxes, we’re just looking for the ‘checked’ attribute, and setting the related variable to 1 if it’s present. We want to inherit the steps from this script in any journey that reacts to the feature flag configuration (which may well be all of them).

eBuyer.com Feature Flag Config Source Screenshot
this is the basic markup for the feature flags configuration page

The basic JavaScript for the conditional step has the same structure for all steps (see below). If the condition is met then we set our variable to 1.

return document.querySelector("#ppCreditMain[checked]");
Code language: JavaScript (javascript)

Testing with the features disabled

On the assumption that we’re starting out with the feature flags disabled, we should first write our tests to make sure the new functionality remains hidden with the flags disabled. We need to make sure the “0% Interest Offers Available” is not shown and that neither deferred payment method is shown. For our conditions, we’re simply checking if our main PayPal Credit and Klarna flags are set to false.

if ( ({{paypalcredit}} === 0) && ({{klarna}} === 0) ) { return true; }
Code language: JavaScript (javascript)

We then check each each flag in isolation to make sure the related section on the PDP and cart is hidden. Of course, as well as the specific PDP and cart flags, the section would also be hidden if the top level payment method flag was disabled:

if ( ({{paypalcredit}} === 0) || ({{paypalcreditpdp}} === 0) ) { return true; }
Code language: JavaScript (javascript)

Testing the new features

The basic flag-on tests are really just the opposite of the above, but need additional consideration for the more granular messaging. For PayPal Credit, the availability of both payment options is determined by the same logic. We check the combination of our feature flags and the item price:

if ( ({{paypalcredit}} === 1) && ({{paypalcreditpdp}} === 1) && ({{price}} >= 99) ) { return true; }
Code language: JavaScript (javascript)

The class on the <li> lets us differentiate between available and unavailable, so we use this to check whether the element is visible on the page.

ebuyer.com Enabled vs Disabled Markup

Of course, we also want to check the negative cases, where the payment options are unavailable because they fall outside of the supported price ranges. This is a simple case of reversing some of the conditions – the example below checks outside of the accepted price range for the 3 payments option in the cart:

if ( ({{klarna}} === 1) && ({{klarnacart}} === 1) && (({{totalprice}} < 49) || ({{totalprice}} > 750)) ) { return true; }
Code language: JavaScript (javascript)

As you can see, the conditions and script structure for the cart tests will be very similar. The primary difference is that we use the total price rather than item price. Import the full script to see the full set of tests. We now have a set of verification steps in place that will adapt to the different feature flag configurations.

Configuring the tests in Ghost Inspector

Suite setup

GI Feature Flags Script Setup Screenshot

Script steps example

Extracting the feature flag configuration

GI Feature Flag Extraction Steps Screenshot
set our variables to 1 if the feature checkbox is checked

PayPal Credit on the PDP

GI PayPal PDP Conditions Screenshot
we use the same price-based condition for both checks

Klarna in the cart

GI Klarna Cart Conditions Screenshot
the Klarna conditions are slightly more complex

Script export

The full script contains many more steps than are shown here. Feel free to set the start URL to any ebuyer.com product page to see how the test adapts. Note that the Feature Flags script will not run successfully, as the flag configuration page doesn’t actually exist. Simply import the .json files into Ghost Inspector.

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.