Thursday, November 21, 2013

Benford's Law: Can it Negate Cheating?

Imagine that you begin your day at an auction, purchasing a myriad of equipment for your laboratory. You then spend the afternoon recording data on the radioactive half-life of uranium, and you end the evening by looking at your tax return. The prices you paid for the auction items, the values comprising your research data and the depressing numbers on your tax return all follow a pattern called Benford’s law.

Benford’s law describes the frequency of leading digits. Lower numbers such as one or two appear as the first digit more often than higher numbers, and the fall-off frequency follows a logarithmic scale. Meaning that more than 50 percent of the prices you paid for the auction items will likely begin with the number one, two or three. Research indicates that the same rule applies to large samples of numbers describing radioactive half-lives, tax returns, statistical physics distributions, geological stream-flow rates and more.

Credit: Alberto G.

Numerous numerical datasets follow Benford’s law. A team of scientists at Trent University and Brock University in Ontario, Canada specialize in multiple-choice test assessment. They began to wonder if Benford’s Law could be used to gain an unfair advantage during undergraduate test taking, specifically in physics. Could Benford’s law offer students the next best strategy to “When in doubt, choose C”?

The team tested the idea using an introductory multiple-choice physics testbank as well as three undergraduate physics and chemistry textbooks. They recorded the leading digits of solutions in the testbank and end-of-chapter textbook solutions and found that both sets closely followed Benford’s law. Their data suggested that over 50 percent of the solutions would begin with the number one, two or three.

More disturbing, this implied that by blindly following Benford’s Law, a test-taker could earn a score of 51 percent on a 3-option multiple choice exam, 41 percent on a 4-option exam, and 33 percent on a 5-option exam. Random guessing for the same exams yields scores of 33 percent, 25 percent and 20 percent respectively.

Any student armed with this knowledge might use it to achieve higher scores on multiple-choice exams. But they would be disappointed with the results. The team did exactly what any student might and took a four-option multiple-choice physics test selecting the option with the lowest leading digit. If a question had more than one option with identical lowest digits, then the team would guess amongst those options.

Using Benford’s Law as a test-taking guide, the team earned a score of 24.6 percent, proving that Benford’s Law did not provide any significant advantage over random guessing. At first, the result confused them since it contradicted their initial predictions, said lead author of the paper and Assistant Professor at Trent University Aaron Slepkov.

“After some thought we realized that the answer was obvious,” Slepkov said. “The distractors [i.e. incorrect options] also follow Benford’s Law. So the answers following Benford’s Law are hiding among a forest of distractors. It’s kind of ironic, in some way our paper is a negative result.”

As long as both the right and wrong options in a physics multiple-choice exam follow Benford’s Law, no advantage exists in using the law as a strategic test-taking guide.

1 comment: