Friday, 22 December 2006

Taguchi Testing A Far Cry From A/B

A/B testing is a staple of the direct marketing industry. Two versions of a direct mail piece are mailed to representative groups of the audience to see which version generates the most responses. The most successful is then mailed to the full mailing list. The assumption being that either version A or B is the best possible answer - it rarely is.
Genichi Taguchi developed a process to test several variables at the same time by carefully selecting different combinations so that the total test reveals which version of each variable is best. This approach is called multivariate testing. For example, if you want to test five versions of the offer and five versions of the copy on your mailing piece, you can use different combinations of each so that you can test all 25 combinations at the same time, getting your answers for both variables at once.
Multivariate testing software can select which version of each variable should be tested on each page and can boil down those billions of possibilities to a few dozen. After testing those few dozen, you can run a second multivariate test with a dozen versions, then get it down to three or four for the last wave before you decide the right variants for every variable.