Before you even start thinking about performing an experiment, its value must be determined. Why are you conducting it? What do you hope to get out of it?
There are three possible things to experiment:
- Technology
- Processes
- Product
We’ve tried all three!
Yea or nay?
The goal was to see if the development of features would speed up if our design would become more consistent, and the quality of our code better. In time, it turned out it was quite useful, therefore the experiment was successful, and we kept on using it!
As far as processes are concerned, we felt our teams were quite disconnected among themselves. Everyone was doing their part, but they lacked the clarity on what other teams were working on. So we decided to bring everything together in one place, visually.
Our Scrum Masters got together and created a Timeline, with duct tape and sticky notes. It showed what every single team was working on, and deadlines by which some tasks needed to be completed, or certain decisions taken.
Last, but not least, product. Since it’s something we already talked about in our previous posts, we’d like to get very particular here.
Prepping for an experiment
We assume that our visitors whose native language isn’t English won’t create a trial because they don’t wish to use software whose primary and only language is, in fact, English. Perhaps, if they were informed right from the start that other languages were available as well, the percentage of created trials vs. the number of visits would grow.
Then we do some research, and come up with a hypothesis: following the launch of the experiment, we’ll have an increase of a minimum of 10% in the number of trials created. The difference between the initial question and the hypothesis is: the latter is actionable.In our case, this means that if the hypothesis proves to be correct, and the number of trials increases following the start of our experiment, we should invest more resources into translating the entire website. We should also consider communicating these options to our customers better, and enhance the localization of the app.
Which door to choose?
Option n°1. The ActiveCollab website informs its users: “Did you know that you can use AC in more than 20 languages?”, and a button leads to a Help page that explains how to switch to a certain language. The experiment goes on for 7 days.
Option n°2. As part of a remarketing campaign, we display to certain people on certain markets the possibility to translate ActiveCollab. There’s a blog post that explains how, and the banner can lead to it.
Option n°3. We make an A/B test, where we communicate to one group of people ActiveCollab is available in 25 languages. We’ll need a budget, and we’ll decide each week whether to continue with the experiment or not, depending on the results it gives.
It’s important to choose the simplest possible way to test the hypothesis and discover relevant results. Find out in our next post which path we took to reach this goal!
