“Damage caused by cybercrime is estimated at $100 billion annually, said Kilian Strauss, of the Organization for Security and Cooperation in Europe (OSCE),” New Scientist reported back in 2008.

Where are those supposed billions going? According to a Cisco report: “In Russia, for instance, social networks were used to create an online marketplace for stolen credit cards. This has allowed the ‘sellers’ to specialize in areas such as acquisition, while the ‘buyers’ focus their efforts in exploitation.”

Journalists like me read these sorts of reports from vendors, analyst firms and law enforcement and pass along the information to our readers and pass along a simple message: cybercriminals must be getting really rich off all this stolen data. But according to a paper written by Dinei Florêncio and Cormac Herley and published by Microsoft Research, the surveys used in these reports are seriously flawed.

As the researchers put it in their paper, cybercrime “defies large-scale direct observation and the estimates we have of it are derived almost exclusively from surveys.” But cybercrime surveys pose some statistical challenges. In many surveys, errors can cancel each other out. That doesn’t work in a cybercrime survey, where respondents are only giving information about how much money they’ve lost. I can claim to have lost $2 million more to cybercrime than I actually did, but no one can balance out my exaggerated claim because they can’t say they lost “negative $2 million.” This necessarily makes the estimates skew upwards.

Things get really out of hand when you have large erroneous outliers in the sample group being extrapolated to the larger population. In their summary of their findings for The New York Times, Florêncio and Herley wrote:

Suppose we asked 5,000 people to report their cybercrime losses, which we will then extrapolate over a population of 200 million. Every dollar claimed gets multiplied by 40,000. A single individual who falsely claims $25,000 in losses adds a spurious $1 billion to the estimate. And since no one can claim negative losses, the error can’t be canceled.

THE [sic] cybercrime surveys we have examined exhibit exactly this pattern of enormous, unverified outliers dominating the data. In some, 90 percent of the estimate appears to come from the answers of one or two individuals. In a 2006 survey of identity theft by the Federal Trade Commission, two respondents gave answers that would have added $37 billion to the estimate, dwarfing that of all other respondents combined.

The two conclude in the paper that cybercrime surveys “are so compromised and biased that no faith whatever can be placed in their findings,” and cite other researchers who have come to similar conclusions.

Florêncio and Herley warn that this doesn’t mean that security isn’t a problem. In fact, the costs of cleaning up malware and changing passwords can be steep, which leads to the researchers’ conclusion: “Surveys that perpetuate the myth that cybercrime makes for easy money are harmful because they encourage hopeful, if misinformed, new entrants, who generate more harm for users than profit for themselves.”

So don’t throw away your firewalls or stop installing antivirus software on your users’ desktops quite yet, but don’t exaggerate the danger either.

Photo by Circo de Invierno