A problem with test and learn
Prefer to watch this as a video? Click here.
If your career in marketing to date has been from a traditional stand point, you probably like the idea of certainty. You want things to be knowable, and to be known as far in advance of them happening as possible. The problem is that moving to an agile approach takes away a lot of this certainty, so this can make the change feel pretty scary and unpleasant. You’ve been fed the myth of certainty your entire life, and like an alcoholic who suddenly goes cold turkey, you start looking for all sorts of ways to get your little fix of certainty again.
This is where test and learn gets really dangerous. It’s a big part of agile marketing, and should be a part of any agile approach really. Testing to see how things go down, using the learnings to refine what you do, or even abandoning what you thought would work and doing something completely different. Testing is a good thing, and you should always do lots of it.
The danger comes more in the idea of learning. Learning, especially to people who’ve not done much of it before, is associated with facts, and with facts come certainty. Now of course, facts are just one part of learning, and many theories of knowledge dispute the idea that there are knowable facts anyway, but this is generally the point at which I get accused of being too academic, so i’ll stop.
However, I think it’s safe to say that if you’re working in the kind of environment that is complex, fast paced and uncertain enough to require an agile approach, then even if you learn something now, you can’t be sure that that learning will hold true in the future. But it’s this fact that people so often forget. They do a test, they learn something from it, and they then conclude that that learning is a fact and cling onto it for far longer than they should. Even worse, they do a test in one context or environment, then assume that the results of the test will apply just as well in other contexts and environments too. In a complex, fast-paced and uncertain environment, if you want to say something is true, you can only say it’s true here and true now.
There’s another problem with the ‘learn’ part of test and learn as well. Testing is often got wrong, but running valid and useful tests is actually pretty easy once you know how. It’s also reasonably easy to show your methodology to others and have them agree (or not) that your tests were valid. However, learning is a much more subjective and personal thing. Due to the whole load of personal biases we all hold and the different ways we learn, giving the same test results to different people could see them learn different elements of the results, or learn them to different degrees. Just the same way a class of students don’t all get the same results on an exam despite all having been exposed to the same learning process.
So in a way, the phrase ’test and learn’ isn’t that great, and I love the idea of instead calling it ‘test and identify’. The test part stays the same, but instead of saying you’re definitely going to learn from the process, you say the tests only identify learnings, which you will have to work long and hard at fully understanding and learning. The same holds true for the idea of listing out some ‘lessons learned’ at things like scrum retrospectives, really we should be calling them ‘lessons identified’.
So there we are then. I don’t want to be too down on the idea of ’test and learn’. It’s certainly far better than approaches such as ‘guess and ignore’ that have been all too common in fields like marketing and communications to date. However, it’s really important to remember that your learnings are not certainties or eternal truths, they may become invalid as time passes and contexts change. It’s also important to remember that the process can only identify a learning for you, the actual learning is something only you can do yourself, and you may be doing it differently and with different outcomes to others.
As scary as it can initially feels, always remember that if you’re feeling too certain, you’re probably doing it wrong.