CLEVER

IDM for Microsoft Active Directory

Earned Clever $1M ARR by building empathy with Active Directory admins

TIMELINE

Dec 2022 - Apr 2023

TEAM

1 product manager
1 product marketing manager
4 engineers

ROLE

User researcher
Designer

FOCUS

Survey design
Usability testing

Context

CONTEXT

Adding Microsoft to the Identity Management (IDM) product

  • In 2021, Clever releases its first identity management product, Clever IDM, which automates Google account provisioning for school districts. 

  • The product is successful, but customers need an account provisioning solution for Microsoft as well as Google.

  • I join the team at the end of 2022 as a researcher and designer.

CONTEXT

Opportunity

  • Although the team had already identified the need to support Microsoft, they did not yet know which Microsoft identity provider to focus on first: Active Directory or Azure.

  • My task as a researcher was to deeply understand customer context and needs for the Microsoft identity provider that Clever IDM would support in its next release.

Assessing the market landscape

PLANNING

Research goals

  • The team knew there was an opportunity for us to expand our IDM product to support customers who use Microsoft as an identity provider, but they needed a deeper understanding of customer needs in this segment. 

  • The purpose of setting goals and criteria was to ensure we had the information we needed to make decisions about how to approach the market.

Research goal

Determine the market share of Microsoft Active Directory vs. Microsoft Azure as a school district’s primary identity provider

Describe our competitive positioning for identity provisioning in school districts.

Determine the biggest gaps school districts have with identity provisioning today.

METHODOLOGY

Survey

We have enough data from districts to understand the size of the addressable market and who our main competitors are.

Since our research goals focused on developing a broad understanding of the market landscape and where our identity provisioning solution would be competitive, I chose to send a survey to gather quantitative information at scale.

RECRUITMENT

Building confidence with current customers

  • We chose to reach out to our current customers because we were only serving a small fraction of them with our identity provisioning solution.

  • To feel confident in our survey results, we wanted to recruit at least 100 district administrator participants. We ultimately exceeded our recruitment goal and had 199 survey participants.

  • To go as broad as possible and understand the market landscape we reached out to all segments of our customer base.

Big

Districts with 60+ schools

Mid

Districts with 20-59 schools

I found that AD was the the leading primary identity provider across district segments

We have enough data from school districts to prioritize features that will differentiate our solution

Scaled

Districts with 1-19 schools

SURVEY INSIGHT

Data

Success criteria

We have enough data from districts to make a decision on which Microsoft identity provider to support first

For Microsoft districts, Active Directory (AD) is the source of truth identity provider

  • As their primary identity provider, 52% of participants reported AD and <1% reported Azure. Of those who reported AD as their primary identity provider, 41% reported Azure as an additional identity provider they use.

  • Further, of the customers who reported Google as their primary identity provider, which our product already supported, those who reported using an additional identity provider unanimously reported AD.

Decision

The team prioritized supporting AD as the next identity provider for our product to support

When asked how they create user accounts, mid districts had the highest rate of choosing “Automated process - internal scripts” or “Manual account creation”

SURVEY INSIGHT

Data

Clever has the greatest addressable market with the mid district segment

  • 74% of mid districts did not have an existing third party automated solution for identity provisioning.

  • The same was true for 65% of scaled districts and 0% of big districts.

Decision

The team focused on the mid district segment as our primary market

I used a 4-point likert scale without a neutral option to ensure a response about a specific opinion. Hands off, secure, and low cost revealed the biggest gaps admins had with identity provisioning.

SURVEY INSIGHT

Data

The most compelling value propositions are hands off, secure, and low cost

These value propositions received the highest amount of “very compelling” or “somewhat compelling” ratings.

Decision

The team led with these value propositions on our in-product sign up page for IDM

Understanding how admins use AD

PLANNING

Research goals

  • Now that we had a broad view of the market landscape and knew we would focus on AD as the next identity provider to support, we wanted to get a deeper understanding of how customers used AD.

  • The purpose of setting goals and criteria was to ensure we had the information we needed to prioritize specific product and design decisions.

Research goal

Describe the top needs of districts that use AD when provisioning accounts

Illustrate how OU organization is different in districts that use AD than districts that use Google

Determine if districts use multiple identity systems and how the differences are managed

Describe what districts think about the differences between AD and Azure as well as how they rank provisioning systems.

We have enough data to decide whether the OU structuring will be the same or different than our existing product that supports Google.

We have enough data to decide if we should prioritize supporting multiple identity systems and, if so, how the design should support multiple systems.

We have enough data to understand how AD and Azure should interact with one another in the design.

Success criteria

We have enough data to prioritize the features of our design.

METHODOLOGY

User interviews

Because we had identified AD as the next identity provider to support from our survey data, and wanted to dig into how district administrators use AD, I chose to do user interviews to get deeper qualitative insights. 

RECRUITMENT

Follow-up with survey participants

  • In our survey, we included a question that asked participants if they would be open to a follow-up interview.

  • We emailed all 33 participants who indicated that they would be open to an interview and reported using AD, 24 scheduled a call with us, and 10 showed up.

An admin explains how they manually provision accounts today: “Accounts are built in Active Directory then passed over to Google. The same OU exists in both places.”

INTERVIEW INSIGHT

Data

The top needs for districts when provisioning AD accounts are:

  • Automated syncs between their school information system and Active Directory.

  • One product that can do account provisioning for Google, Active Directory, and Azure.

  • Prevention of duplicate instances of accounts once provisioned.

Decision

Leverage what we built for Google to automate syncs and prevent duplicate instances of accounts. Prioritize multi-destination to ensure IDM can do provisioning for more than one identity provider per district.

An admin shows us their complex, legacy AD instance where they manage accounts.

INTERVIEW INSIGHT

Data

Admins prefer cloud-based systems, but AD is so deeply embedded into their processes, they will continue using it

  • Cloud-based systems like Google and Azure are easy to connect to other systems and manage.

  • Admins don’t get these same benefits with AD because it is a legacy, on-premise platform, but it is deeply embedded into their existing processes.

Decision

The team doubled-down on AD as the next identity provider IDM would support and focused on making the connection to Clever step simple.

Evaluating our initial solution

PLANNING

Research goals

  • Once we had our initial IDM for AD flow built, we wanted to evaluate how well our initial solution was meeting customer needs. 

  • The purpose of setting goals and criteria was to ensure we had the information we needed to make adjustments to the IDM for AD flow based on user feedback.

Research goal

Describe how admins move through the flow

Illustrate how self-serve support tools are used in the flow

Determine admins understanding of what they are doing in the flow

We have enough data to make decisions on how to modify the self-serve support tools included in the flow

We have enough data to make decisions on the language and framing of steps in the flow

Success criteria

We have enough data to make a decision on which steps need usability improvements

METHODOLOGY

Moderated usability testing

Since we had built an initial solution and wanted to evaluate how well it would meet customer needs, I chose moderated usability testing. This method would allow us to get honest behavioral feedback from users paired with an opportunity to ask follow-up questions and go deeper on particular usability issues. 

RECRUITMENT

In-product pathway to capture interest 

  • To recruit for usability testing we include a “Email me when AD is available” button to capture districts that were interested in adding AD to IDM. 

  • We reached out to all 23 districts that expressed interest through this pathway to see if they would be interested in testing our flow, 11 districts responded, and 9 and completed testing.

USABILITY INSIGHT

The two biggest hurdles in the flow are the connect and preview steps

  • For the connect step, all participants either completed with difficulty or did not complete.

  • For the participants who made it to the preview step, all completed with difficulty.

USABILITY INSIGHT

Admins can’t easily connect Clever to AD because they don’t know where to find what we’re asking for

Finding specific required information such as their LDAPS URL, Bind account, and Base DN were challenging for admins to find because they rarely connected their AD instances to other systems.

An admin struggles to complete part of the connect step:
“Bind account? Oh boy, I’m gonna have to look that up…”

USABILITY INSIGHT

The language in the preview step confuses admins, which makes them hesitant to provision accounts

  • After admins completed the IDM for AD flow, they could download a csv file that showed a preview of how Clever would manage their AD identities.

  • Showing a user email rather than user principal name conflated Google account provisioning with AD, which confused admins.

  • Admins weren’t clear on what the headings “forward data” and “reverse data” referred to.

When admins didn’t understand the headings of their preview file, they were hesitant to provision accounts.

USABILITY INSIGHT

For many of the participants, what we’re asking them to do in our IDM for AD flow is disruptive and scary.

  • Many of the participants had these legacy AD systems passed to them and their role is to keep them running without full knowledge of how it works.

  • We’re putting their core job at risk by asking them to do specific things without supporting them with advice and documentation.

An admin has difficulty because this legacy AD system was passed on to them:
“I’ve added the Clever IPs to our firewall, but I’m not sure about the LDAPS URL…”

SHARING AND ACTIVATION

Our team needed to learn more about the AD information we were asking for

  • By the end of usability testing our team knew that the connect step was causing the most problems for admins: they didn’t know where to find what we were asking for.

  • To provide accurate advice and documentation we had to learn where admins could find the AD information we were asking for.

I had hit a wall with my research methods up to this point. We needed to learn more about the AD information we were asking admins to provide Clever.

Becoming an AD admin to build empathy

PLANNING

Research goals

  • This phase of our research focused on building our understanding of where to find the specific pieces of AD information we were asking admins to provide Clever.

  • The purpose of setting goals and criteria was to ensure we had the information we needed to make decisions on how we guided admins to provide the AD information we needed.

Research goal

Describe the steps admins must take to find the AD information Clever needs to connect

Success criteria

We have enough data to create walkthrough materials that guide admins to provide Clever the required AD information

METHODOLOGY

Secondary research on Active Directory

  • Since both admins and our team were unclear on where to find specific information in an AD instance, I chose secondary research to empathize with admins and put myself in their shoes as they navigated AD to find the information Clever was requesting.

  • As a result, I shared the usability testing results with my engineering team and asked them to set up our own AD instance. This way, I could find every piece of information we were asking customers to provide Clever in the connect step.

I asked my eng team to set up our own AD instance so that I could become the customer and experience how an admin would find the AD information Clever was requesting.

SECONDARY RESEARCH INSIGHT

All of the information Clever asks for can be found, but it takes some digging

  • After a few days of digging around the AD instance my team had set up, I found where all of the information we were requesting lived.

  • I recorded step-by-step videos of where each piece of information lived to document what I had learned.

In this step-by-step video, I showed where admins could find one of the pieces of information required for the connect step: a bind account username and password.

Impact

IMPACT

Research informed product improvements

Using my findings from the usability testing and secondary research, I collaborated with my product manager to prioritize product improvements and my engineers to build the improvements I designed.

The flow conflated asking for an LDAPS URL with allow listing Clever’s IP addresses on a firewall. This confused admins and didn’t proactively address a common error in the LDAPS step.

Before

After

The flow separated allow listing Clever’s IP addresses on a firewall as step one and providing an LDAPS URL as step two. This reduced cognitive load and proactively addressed the most common error in the LDAPS step.

The flow directed admins to contact support if they were having trouble with any of the information we were asking for in the connect step, but our support team often could not help.

Before

After

The flow directed admins to a help center page, specific to the information Clever was requesting, including the step-by-step videos I had recorded. We made this update for LDAPS URL, bind account, and base DN.

After (help center page for bind account)

The help center page that showed admins how to find their bind account username and password in AD, then provide those credentials to Clever.

The final step of the flow asked admins to run a final test of their AD configuration. The errors associated with this test were vague and provided minimal paths to recovery.

Before

After

The final test of the AD configuration included errors that highlighted the specific step that had an issue, provided a more detailed explanation of the error, and provided a link to the help center article specific to that error.

Admins could download a csv file that showed a preview of how Clever would manage their AD identities. The email, forward data, and reverse data headings confused admins.

Before

After

The csv file had headings that matched what admins reported would make more sense: UPN, existing data, and planned data. We also switched the heading order to match admins’ expectations: existing before planned data.

IMPACT

Iteration to IDM helped team achieve business objective

Business objective: Grow the IDM business by supporting customers who use a Microsoft identity provider.

$1.03M

Annual recurring revenue (ARR)
after adding AD functionality

103% to key result

70%

Conversion rate for AD
(completed onboarding and purchased)

116% to key result

IMPACT

Start with desk research first

Reflections

Going into this project, we assumed that admins would know where specific information lived in their AD instances. That assumption was wrong but, even if it was true, I should have built my own understanding of where that information lived prior to beginning other research to have a clearer sense of the admin’s context surrounding the Clever customer experience. If I was to do this project again, I would have started with desk research first to build my understanding of the admin's context in AD.

Get a sense of participants’ familiarity with AD before usability testing

In our recruitment for usability testing we relied solely on customer interest without using any screener or questionnaire to understand participants’ familiarity with AD. This led to a group of participants that tended to be more experienced with AD. If I were to do this project again, I would have gathered this information on familiarity in a screener or questionnaire prior to usability testing. That way, we could have created a more balanced pool of participants that included both admins that were experienced and inexperienced with AD.