TPRC has ended
Back To Schedule
Sunday, September 29 • 12:20pm - 1:00pm
Spam and Botnet Reputation Randomized Control Trials and Policy

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Download Paper

Designing randomized control trials (RCT) of reputational effects of spam and botnet rankings as proxies for Internet security has interesting challenges. These challenges are related to the policy issues such reputation is intended to address. Building on preliminary results and the public SpamRankings.net top 10 rankings per country by spam volume from two anti-spam blocklists (see TPRC 2012 and 2011 papers), formal RCT experiments provide another level of evidence. However, using RCT with thousands of organizations in treatment and control groups raises numerous difficulties in non-homogeneous legal and organizational regimes and potential active opposition. Fortunately most of these difficulties can be turned to advantages, and all have policy implications.

These complications compared to RCTs of more traditional econometric one-shot surveys with single publication arise because the subject of these field experiments is the live Internet in real time with ongoing updated treatments. The experimental treatments themselves act as information security (infosec), since their purpose is to cause improvements in infosec in treated companies through reputation. The treatments thus must adapt to changes in conditions in the Internet as they happen. Like other infosec, to be effective the treatments must also be portable across departments within treated organizations plus customers and investors, and the experimental team itself crosses economics, MSIS, and computer science.

If the experiments demonstrate statistical evidence that this reputational approach works, such results will provide a new policy approach of reputational rankings, plus the beginnings of tools to apply that approach, ranging from the public treatments themselves to drilldowns into underlying details of the symptoms causing good or bad reputation.

Difficulties encountered include:

1) Differing sensitivities of different blocklists to spam from certain sources; sensitivities that change over time as the blocklists adapt to new miscreant behavior.

Approach: A weighted composite ranking based on both spam volume and spamming address count from at least two different blocklists.

2) Heterogeneity of legal regimes and other characteristics across countries.

Approach: Initial experiments within a single country (the U.S.), perhaps followed by clustered RCT using countries as clusters.

3) Availability of organizational characterization information for stratification by industry (finance, medical, etc.) and within industry (ISPs or hosting, telephone company or cable company, etc.).

Approach: Start with the U.S., for which this information is relatively readily available in homogeneous form.

4) Public visibility is necessary for reputation so that customers and investors of treated rganizations can see the treatments, yet limits flexibility of experimental treatments, since an ongoing, regularly updated treatment once deployed is hard to retract.

Approach: Start with a subset of the universe of spamming organizations and deploy more treatments for other organizations later, plus potential additional treatments for already-treated organizations, while tuning existing treatments like product releases.

5) Spammers or bot herders could choose to migrate away from treated organizations to untreated (control) organizations, interfering with independence of treated and control groups.

Approach: Use botnet volume and address data to observe whether this actually happens.

6) Miscreants may actively retaliate with DDoS or other attacks.

Approach: Harden the treatment websites by hosting them in a cloud provided by a very large organization.

Preliminary RCT results are expected by the paper completion deadline, and will be presented, while the series of experiments supported by NSF grant 0831338 will continue, and the usual disclaimers apply.

Sunday September 29, 2013 12:20pm - 1:00pm PDT
GMUSL Room 332

Attendees (0)