File:Banner25AoverBHISTredline.png

English: This graph is from a statistics exercise of the Wikimedia Foundation fundraising department. We went back to one of our largest test samples, where we ran two banners -- A vs B -- for a very long time. The response rates across a range of low-donation-rate countries were 56 and 50 per 100,000 respectively. We chopped that long test up into 25 smaller samples that are closer to the size of our typical tests — in this case with about 3.5 million banner views per banner per test. Then we checked how often those short tests accurately represented the “true”(er) result of the full test. The graph shows by how much A beat B in each test (each subset of the larger test actually). The red vertical line represents the true(er) value of how much A beats B based on the entire large sample. Each dot represents A’s winning margin in a different test — 1.1 means A beat B by 10 percent. This kind of graph is called a histogram. The bars show how many results fit into different ranges. You can see that most of the tests fall around a central value. This is good to see! Our stats methods assume the data conforms to a certain pattern, which is called a “normal distribution.” And this is one indication that our data is normal.