Checkmango

Our signup flow has three steps:

1. Email address and password
2. Personalisation (segmentation) questions
3. Email verification

We're seeing a steep (40% abandonment) drop-off at step 2: many users are entering their email address and creating a password, and then just can't be bothered with the personalisation questions.

Our hypothesis (an absolute no-brainer, a door knob would struggle to think up worse) is that by removing step 2, we'll get a whole heap more people completing the signup flow.

I came across checkmango.com which promises an easy way to do A/B testing – perfect for what we want.

This post documents our setup and use of Checkmango inside our Laravel app.

Experiment Setup

After registering for a Checkmango account I created a new experiment: test_signup_hide_personalisation.

The experiment has two variants:

1. var_signup_show_personalisation the control variant (what we're doing already)
2. var_signup_hide_personalisation the new variant we're going to test

I chose a 50-50 split ("equal distribution") for participants, and the goal ("primary metric") I created was signup_complete.

Click "Start" in the Checkmango dashboard to begin your experiment.

Install the PHP SDK

Installing the SDK is as simple as: composer require checkmango/phpsdk. The documentation for the SDK lives here: github.com/checkmango/php-sdk. I pinned my version of the SDK to dev-master.

Creating Participants

Each Checkmango Participant need to be identified using a unique string (not a numerical ID). We're using the format cm_{userId} as it's a string, easily reproducible, and doesn't need another database column. This works for us, you might want to do something different.

To create a participant:

$client = new Checkmango\Client();
$client->authenticate($apiKey);

$participantId = '...'; // You figure this out

$client->teams($teamId)->participants()->create([
	'key' => $participantId,
]);

This API call takes ~1 second to complete. I think you're best to complete this in a queued job, but you could do the request inline if you're desperate (or running the experiment early in a user's journey).

Enrol the Participant in an Experiment

To pre-empt any confusion: I code in Queen's English, so I'm using enrol. If you speak North American your native spelling is enroll.

Each time a user is created in our application (in step 1 of our signup flow) we need to enrol the participant in our experiment:

$client->teams($teamId)
	->experiments()
	->enrol('test_signup_hide_personalisation', $participantId);

This API call returns the enrollment information, which includes the variant that we should show to the user. Checkmango decides the variant for us, based on the algorithm that we chose when we setup the experiment.

[
  "id" => "cm_...",
  "type" => "participants",
  "attributes" => [
    ...
    "enrollments" => [
      "test_signup_hide_personalisation" => "var_signup_show_personalisation",
    ],
    ...
  ],
  ...
]

In our case, Checkmango tells us the user should be shown the var_signup_show_personalisation variant (or, shown step 2 in our sign up flow).

Remember which Variant the Participant sees

We set up a simple database table to keep track of which users are in which experiments, and which variant for each experiment the user should see. You might want to do something similar.

Schema::create('checkmango_variants', function (Blueprint $table) {
    $table->id();
    $table->foreignId('user_id');
    $table->string('experiment')->index();
    $table->string('variant')->index();
    $table->timestamps();
});

When we've enrolled the participant in an experiment, we then record the all the user's experiments and variants like so:

$data = $client->teams($teamId)
	->experiments()
	->enrol($experiment, $participantId);

$enrolments = data_get($data, 'attributes.enrollments');

foreach ($enrolments as $experiment => $variant) {
    CheckmangoVariant::updateOrCreate(
        [
            'user_id' => $user->id,
            'experiment' => $experiment,
        ],
        [
            'variant' => $variant,
        ]
    );
}

You can recall the variant the user should see for a given experiment quickly:

public static function variant(User $user, string $experiment)
{
    $variant = CheckmangoVariant::where([
        'user_id' => $user->id,
        'experiment' => $experiment,
    ])->first();
    return $variant ? $variant->variant : false;
}

Using the Variant

Our Blade templates we check which variant the user should be seeing, and make adjustments as required. In our controller:

return view('template', [
    'variant' => Checkmango::variant($user, $experiment),
    ...
]);

And in our Blade template, remembering that var_signup_show_personalisation is our control:

@if (! $variant || $variant === 'var_signup_show_personalisation')

	{{-- show personalisation, control --}}

@else

	{{-- hide personalisation --}}

@endif

Tracking Impressions

WIP