How to Test Anything, Part 1: Maximize Learning, Minimize Unexpected Consequences
Step 1: Write down what you want to test, and list all the expected outcomes and their relative probabilities.
Step 2: List all possible variations. They may be obvious, or they may be nearly invisible. If they are hidden, check all the places that might be relevant.
Step 3: Choose a variation, either at random or based on judgment. First identify preconceptions that may bias the test. If you know the result, you have once raised a barrier to learning (your own inability to see reality).
Step 4: Measure the metric(s) of interest, and document your reasoning. Then measure everything else. Document your reasoning for everything else.
Step 5: Document again. You need to be reminded that you remembered to measure what you thought you were measuring.
Step 6: Analyze the results. Learn to tell when you are learning as fast as possible, and when you need a better hypothesis or additional variations of the test.
Step 7: Revise your model. Your models, not reality, are the unbiased sources of your learning. One of us (DHH) has suggested that the world does indeed need more anthropomorphized quadratic equations.
Step 8: If there are connections between what you are testing and other areas of interest, consider parallel tests in those areas.
Step 9: Share what you have learned as widely as possible. You can take steps to reduce the risk of negative outcomes, if you learn enough.
Step 10: Set up a process to follow the most important results even after you have forgotten about them. If no one is interested, announce the results and explain why they are important to people who care.
