CLEVER
IDM for Microsoft Active Directory
Earned Clever $1M ARR by building empathy with Active Directory admins
TIMELINE
Dec 2022 - Apr 2023
TEAM
1 product manager
1 product marketing manager
4 engineers
ROLE
User researcher
Designer
FOCUS
Survey design
Usability testing
Context
CONTEXT
Adding Microsoft to the Identity Management (IDM) product
In 2021, Clever releases its first identity management product, Clever IDM, which automates Google account provisioning for school districts.
The product is successful, but customers need an account provisioning solution for Microsoft as well as Google.
I join the team at the end of 2022 as a researcher and designer.
CONTEXT
Opportunity
Although the team had already identified the need to support Microsoft, they did not yet know which Microsoft identity provider to focus on first: Active Directory or Azure.
My task as a researcher was to deeply understand customer context and needs for the Microsoft identity provider that Clever IDM would support in its next release.
Assessing the market landscape
PLANNING
Research goals
The team knew there was an opportunity for us to expand our IDM product to support customers who use Microsoft as an identity provider, but they needed a deeper understanding of customer needs in this segment.
The purpose of setting goals and criteria was to ensure we had the information we needed to make decisions about how to approach the market.
Research goal
Determine the market share of Microsoft Active Directory vs. Microsoft Azure as a school district’s primary identity provider
Describe our competitive positioning for identity provisioning in school districts.
Determine the biggest gaps school districts have with identity provisioning today.
METHODOLOGY
Survey
We have enough data from districts to understand the size of the addressable market and who our main competitors are.
Since our research goals focused on developing a broad understanding of the market landscape and where our identity provisioning solution would be competitive, I chose to send a survey to gather quantitative information at scale.
RECRUITMENT
Building confidence with current customers
We chose to reach out to our current customers because we were only serving a small fraction of them with our identity provisioning solution.
To feel confident in our survey results, we wanted to recruit at least 100 district administrator participants. We ultimately exceeded our recruitment goal and had 199 survey participants.
To go as broad as possible and understand the market landscape we reached out to all segments of our customer base.
Big
Districts with 60+ schools
Mid
Districts with 20-59 schools
I found that AD was the the leading primary identity provider across district segments
We have enough data from school districts to prioritize features that will differentiate our solution
Scaled
Districts with 1-19 schools
SURVEY INSIGHT
Data
Success criteria
We have enough data from districts to make a decision on which Microsoft identity provider to support first
For Microsoft districts, Active Directory (AD) is the source of truth identity provider
As their primary identity provider, 52% of participants reported AD and <1% reported Azure. Of those who reported AD as their primary identity provider, 41% reported Azure as an additional identity provider they use.
Further, of the customers who reported Google as their primary identity provider, which our product already supported, those who reported using an additional identity provider unanimously reported AD.
Decision
The team prioritized supporting AD as the next identity provider for our product to support
When asked how they create user accounts, mid districts had the highest rate of choosing “Automated process - internal scripts” or “Manual account creation”
SURVEY INSIGHT
Data
Clever has the greatest addressable market with the mid district segment
74% of mid districts did not have an existing third party automated solution for identity provisioning.
The same was true for 65% of scaled districts and 0% of big districts.
Decision
The team focused on the mid district segment as our primary market
I used a 4-point likert scale without a neutral option to ensure a response about a specific opinion. Hands off, secure, and low cost revealed the biggest gaps admins had with identity provisioning.
SURVEY INSIGHT
Data
The most compelling value propositions are hands off, secure, and low cost
These value propositions received the highest amount of “very compelling” or “somewhat compelling” ratings.
Decision
The team led with these value propositions on our in-product sign up page for IDM
Understanding how admins use AD
PLANNING
Research goals
Now that we had a broad view of the market landscape and knew we would focus on AD as the next identity provider to support, we wanted to get a deeper understanding of how customers used AD.
The purpose of setting goals and criteria was to ensure we had the information we needed to prioritize specific product and design decisions.
Research goal
Describe the top needs of districts that use AD when provisioning accounts
Illustrate how OU organization is different in districts that use AD than districts that use Google
Determine if districts use multiple identity systems and how the differences are managed
Describe what districts think about the differences between AD and Azure as well as how they rank provisioning systems.
We have enough data to decide whether the OU structuring will be the same or different than our existing product that supports Google.
We have enough data to decide if we should prioritize supporting multiple identity systems and, if so, how the design should support multiple systems.
We have enough data to understand how AD and Azure should interact with one another in the design.
Success criteria
We have enough data to prioritize the features of our design.
METHODOLOGY
User interviews
Because we had identified AD as the next identity provider to support from our survey data, and wanted to dig into how district administrators use AD, I chose to do user interviews to get deeper qualitative insights.
RECRUITMENT
Follow-up with survey participants
In our survey, we included a question that asked participants if they would be open to a follow-up interview.
We emailed all 33 participants who indicated that they would be open to an interview and reported using AD, 24 scheduled a call with us, and 10 showed up.
An admin explains how they manually provision accounts today: “Accounts are built in Active Directory then passed over to Google. The same OU exists in both places.”
INTERVIEW INSIGHT
Data
The top needs for districts when provisioning AD accounts are:
Automated syncs between their school information system and Active Directory.
One product that can do account provisioning for Google, Active Directory, and Azure.
Prevention of duplicate instances of accounts once provisioned.
Decision
Leverage what we built for Google to automate syncs and prevent duplicate instances of accounts. Prioritize multi-destination to ensure IDM can do provisioning for more than one identity provider per district.
An admin shows us their complex, legacy AD instance where they manage accounts.
INTERVIEW INSIGHT
Data
Admins prefer cloud-based systems, but AD is so deeply embedded into their processes, they will continue using it
Cloud-based systems like Google and Azure are easy to connect to other systems and manage.
Admins don’t get these same benefits with AD because it is a legacy, on-premise platform, but it is deeply embedded into their existing processes.
Decision
The team doubled-down on AD as the next identity provider IDM would support and focused on making the connection to Clever step simple.
Evaluating our initial solution
PLANNING
Research goals
Once we had our initial IDM for AD flow built, we wanted to evaluate how well our initial solution was meeting customer needs.
The purpose of setting goals and criteria was to ensure we had the information we needed to make adjustments to the IDM for AD flow based on user feedback.
Research goal
Describe how admins move through the flow
Illustrate how self-serve support tools are used in the flow
Determine admins understanding of what they are doing in the flow
We have enough data to make decisions on how to modify the self-serve support tools included in the flow
We have enough data to make decisions on the language and framing of steps in the flow
Success criteria
We have enough data to make a decision on which steps need usability improvements
METHODOLOGY
Moderated usability testing
Since we had built an initial solution and wanted to evaluate how well it would meet customer needs, I chose moderated usability testing. This method would allow us to get honest behavioral feedback from users paired with an opportunity to ask follow-up questions and go deeper on particular usability issues.
RECRUITMENT
In-product pathway to capture interest
To recruit for usability testing we include a “Email me when AD is available” button to capture districts that were interested in adding AD to IDM.
We reached out to all 23 districts that expressed interest through this pathway to see if they would be interested in testing our flow, 11 districts responded, and 9 and completed testing.
USABILITY INSIGHT
The two biggest hurdles in the flow are the connect and preview steps
For the connect step, all participants either completed with difficulty or did not complete.
For the participants who made it to the preview step, all completed with difficulty.
USABILITY INSIGHT
Admins can’t easily connect Clever to AD because they don’t know where to find what we’re asking for
Finding specific required information such as their LDAPS URL, Bind account, and Base DN were challenging for admins to find because they rarely connected their AD instances to other systems.
An admin struggles to complete part of the connect step:
“Bind account? Oh boy, I’m gonna have to look that up…”
USABILITY INSIGHT
The language in the preview step confuses admins, which makes them hesitant to provision accounts
After admins completed the IDM for AD flow, they could download a csv file that showed a preview of how Clever would manage their AD identities.
Showing a user email rather than user principal name conflated Google account provisioning with AD, which confused admins.
Admins weren’t clear on what the headings “forward data” and “reverse data” referred to.
When admins didn’t understand the headings of their preview file, they were hesitant to provision accounts.
USABILITY INSIGHT
For many of the participants, what we’re asking them to do in our IDM for AD flow is disruptive and scary.
Many of the participants had these legacy AD systems passed to them and their role is to keep them running without full knowledge of how it works.
We’re putting their core job at risk by asking them to do specific things without supporting them with advice and documentation.
An admin has difficulty because this legacy AD system was passed on to them:
“I’ve added the Clever IPs to our firewall, but I’m not sure about the LDAPS URL…”
SHARING AND ACTIVATION
Our team needed to learn more about the AD information we were asking for
By the end of usability testing our team knew that the connect step was causing the most problems for admins: they didn’t know where to find what we were asking for.
To provide accurate advice and documentation we had to learn where admins could find the AD information we were asking for.
I had hit a wall with my research methods up to this point. We needed to learn more about the AD information we were asking admins to provide Clever.
Becoming an AD admin to build empathy
PLANNING
Research goals
This phase of our research focused on building our understanding of where to find the specific pieces of AD information we were asking admins to provide Clever.
The purpose of setting goals and criteria was to ensure we had the information we needed to make decisions on how we guided admins to provide the AD information we needed.
Research goal
Describe the steps admins must take to find the AD information Clever needs to connect
Success criteria
We have enough data to create walkthrough materials that guide admins to provide Clever the required AD information
METHODOLOGY
Secondary research on Active Directory
Since both admins and our team were unclear on where to find specific information in an AD instance, I chose secondary research to empathize with admins and put myself in their shoes as they navigated AD to find the information Clever was requesting.
As a result, I shared the usability testing results with my engineering team and asked them to set up our own AD instance. This way, I could find every piece of information we were asking customers to provide Clever in the connect step.
I asked my eng team to set up our own AD instance so that I could become the customer and experience how an admin would find the AD information Clever was requesting.
SECONDARY RESEARCH INSIGHT
All of the information Clever asks for can be found, but it takes some digging
After a few days of digging around the AD instance my team had set up, I found where all of the information we were requesting lived.
I recorded step-by-step videos of where each piece of information lived to document what I had learned.
In this step-by-step video, I showed where admins could find one of the pieces of information required for the connect step: a bind account username and password.
Impact
IMPACT
Research informed product improvements
Using my findings from the usability testing and secondary research, I collaborated with my product manager to prioritize product improvements and my engineers to build the improvements I designed.
The flow conflated asking for an LDAPS URL with allow listing Clever’s IP addresses on a firewall. This confused admins and didn’t proactively address a common error in the LDAPS step.
Before
After
The flow separated allow listing Clever’s IP addresses on a firewall as step one and providing an LDAPS URL as step two. This reduced cognitive load and proactively addressed the most common error in the LDAPS step.
The flow directed admins to contact support if they were having trouble with any of the information we were asking for in the connect step, but our support team often could not help.
Before
After
The flow directed admins to a help center page, specific to the information Clever was requesting, including the step-by-step videos I had recorded. We made this update for LDAPS URL, bind account, and base DN.
After (help center page for bind account)
The help center page that showed admins how to find their bind account username and password in AD, then provide those credentials to Clever.
The final step of the flow asked admins to run a final test of their AD configuration. The errors associated with this test were vague and provided minimal paths to recovery.
Before
After
The final test of the AD configuration included errors that highlighted the specific step that had an issue, provided a more detailed explanation of the error, and provided a link to the help center article specific to that error.
Admins could download a csv file that showed a preview of how Clever would manage their AD identities. The email, forward data, and reverse data headings confused admins.
Before
After
The csv file had headings that matched what admins reported would make more sense: UPN, existing data, and planned data. We also switched the heading order to match admins’ expectations: existing before planned data.
IMPACT
Iteration to IDM helped team achieve business objective
Business objective: Grow the IDM business by supporting customers who use a Microsoft identity provider.
$1.03M
Annual recurring revenue (ARR)
after adding AD functionality
103% to key result
70%
Conversion rate for AD
(completed onboarding and purchased)
116% to key result
IMPACT
Start with desk research first
Reflections
Going into this project, we assumed that admins would know where specific information lived in their AD instances. That assumption was wrong but, even if it was true, I should have built my own understanding of where that information lived prior to beginning other research to have a clearer sense of the admin’s context surrounding the Clever customer experience. If I was to do this project again, I would have started with desk research first to build my understanding of the admin's context in AD.
Get a sense of participants’ familiarity with AD before usability testing
In our recruitment for usability testing we relied solely on customer interest without using any screener or questionnaire to understand participants’ familiarity with AD. This led to a group of participants that tended to be more experienced with AD. If I were to do this project again, I would have gathered this information on familiarity in a screener or questionnaire prior to usability testing. That way, we could have created a more balanced pool of participants that included both admins that were experienced and inexperienced with AD.