Guidelines for Performing Safe Measurement on the Internet

1.1. Scope of this document

These are guidelines for how to measure the internet safely. When performing research on a platform shared with live traffic from other users, that research is considered safe if and only if other users are protected from or unlikely to experience danger, risk, or injury arising due to the research, now or in the future.¶

Following the guidelines contained within this document is not a substitute for institutional ethics review processes, although these guidelines could help to inform that process. It is particularly important for the growing area of research that includes internet measurement to better equip review boards to evaluate internet measurement methods [SIGCOMM], and we hope that this document is part of that larger effort.¶

Similarly, these guidelines are not legal advice. Local laws must be considered before starting any experiment that could have adverse impacts on user safety.¶

The scope of this document is restricted to guidelines that mitigate exposure to risks to user safety when measuring properties of the internet: the network, its constituent hosts and links, or user traffic.¶

1.2. Terminology

Threat model: A threat is a potential for a security violation, which exists when there is a circumstance, capability, action, or event that could breach security and cause harm [RFC4949].¶

User: For the purpose of this document, an internet user is an individual or organisation whose data is used in communications over the internet, and most broadly, those who use the internet to communicate.¶

Active measurement: Active measurements generate or modify traffic.¶

Passive measurement: Passive measurements involve the observation of existing traffic without active intervention.¶

On/off-path: A measurement that is on-path happens on the network. Off-path indicates activity in a side-channel, end-point or at other points where the user, their connection, or their data can be accessed.¶

One-/two-ended: A one-ended measurement is like a probe or a trace. A measurement with two-ended control requires the cooperation of both endpoints, which might include the network itself if that is the measurement target.¶

1.3. User impact from measurement studies

Any conceivable internet measurement study might have an impact on an internet user's safety. The measurement of generated traffic may also lead to insights into other users' traffic indirectly as well. It is always necessary to consider the best approach to mitigate the impact of measurements, and to balance the risks of measurements against the benefits to impacted users.¶

Some possible ways in which users can be affected as a result of an internet measurement study:¶

Breach of privacy: User privacy can be violated in the context of data collection. This impact also covers the case of an internet user's data being shared beyond that for which a user had given consent. First-order data that distinguishes a person, such as name, as well as second-order data that can be used to track behaviour, such as IP address, should be considered. [Kenneally]¶

Inadequate data protection: A scenario where data, either in transit or at rest, lacks sufficient protection from disclosure. Failure to meet user expectations for data protection is a concern, even if it does not result in unauthorized access to the data. This includes cases of improper access control (i.e. people having access to user data who do not need it).¶

Traffic generation: A scenario where undue traffic is generated to traverse the internet.¶

Traffic modification: A scenario where users' on-path internet traffic is nonconsensually modified.¶

Impersonation: A scenario where a user is impersonated during a measurement.¶

Legal: Users and service providers are bound by a wide range of policies from Terms of Service to rule of law, each according to context and jurisdiction. A measurement study may violate these policies, and the consequences of such a violation may be severe.¶

Unavailability: Users or other entities may rely on the information or systems that are involved in the research and they may be harmed by unexpected or planned unavailability of that information or systems [Menlo].¶

System or data corruption: A scenario where generated or modified traffic causes the corruption of a system. This covers cases where a user's data may be lost or corrupted, and cases where a user's access to a system may be affected as a result.¶

Emotional trauma: A scenario where a measurement of or exposure to content or behaviour in an internet measurement study causes a user emotional or psychological harm.¶

2.1. Attribute

Proactively identify your measurement to others on the network. "This allows any party or organization to understand what an unsolicited probe packet is, what its purpose is, and, most importantly, who to contact."[RFC9511]¶

Example: For a layer 3 IP packet probe you could mark measurements with a probe description URI as defined in RFC9511.¶

2.2. Obtain consent

Accountability and transparency are fundamentally related to consent. As per the Menlo Report, "Accountability demands that research methodology, ethical evaluations, data collected, and results generated should be documented and made available responsibly in accordance with balancing risks and benefits." [Menlo] A user is best placed to balance the risks and benefits for themselves therefore consent must be obtained. From most transparent to least, there are a few options for obtaining consent.¶

2.3. Share responsibly

Further to use of measurement data, data is often shared with other researchers. Measurement data sharing comes with its own set of expectations and responsibilities of the provider. Likewise there are responsibilities that come with the use of others’ measurement data. One obvious expectation is around end-user consent (see "Implied consent" above). Allman and Paxson [Allman] provide "a set of guidelines that aim to aid the process of sharing measurement data... [in] a framework under which providers and users can better attain a mutual understanding about how to treat particular datasets."¶

Their guidance for data providers is to:¶

• explicitly indicate the terms of a dataset’s acceptable use¶
• convey what interactions they desire or will accommodate.¶

Their guidance for researchers is to:¶

• be thoughtful in the reporting of potentially sensitive information gleaned from providers’ data.¶
• comply with the indications and interactions of the data providers.¶

Example: Researchers have obtained network measurement data from more than one provider for purposes of conducting analysis of protocol use on both. Where privacy paritioning techniques are used, the researchers' findings may inadvertently collude to uncover private information about users. Once realised, researchers should mitigate this privacy risk to end users as well as disclosing this result to the data providers themselves.¶

2.4. Isolate risk with a dedicated testbed

Wherever possible, use a testbed. An isolated network means that there are no other users sharing the infrastructure you are using for your experiments.¶

When measuring performance, competing traffic can have negative effects on the performance of your test traffic and so the testbed approach can also produce more accurate and repeatable results than experiments using the public internet.¶

Example: WAN link conditions can be emulated through artificial delays and/or packet loss using a tool like [netem]. Competing traffic can also be emulated using traffic generators.¶

2.5. Be respectful of others' infrastructure

If your experiment is designed to trigger a response from infrastructure that is not your own, consider what the negative consequences of that may be. At the very least your experiment will consume bandwidth that may have to be paid for.¶

In more extreme circumstances, you could cause traffic to be generated that causes legal trouble for the owner of that infrastructure. The internet is a global network that crosses many legal jurisdictions and so what may be legal for one is not necessarily legal for another.¶

If you are sending a lot of traffic quickly, or otherwise generally deviating from typical client behaviour, a network may identify this as an attack which means that you will not be collecting results that are representative of what a typical client would see.¶

One possible way to mitigate this risk is transparency, i.e. mark measurement-related data or activity as such. For example, the popular internet measurement tool ZMap hardcodes its packets to have IP ID 54321 in order to allow identification [ZMap].¶

2.6. Maintain a "Do Not Scan" list

When performing active measurements on a shared network, maintain a list of hosts that you will never scan regardless of whether they appear in your target lists. When developing tools for performing active measurement, or traffic generation for use in a larger measurement system, ensure that the tool will support the use of a "Do Not Scan" list.¶

If complaints are made that request you do not generate traffic towards a host or network, you must add that host or network to your "Do Not Scan" list, even if no explanation is given or the request is automated.¶

You may ask the requester for their reasoning if it would be useful to your experiment. This can also be an opportunity to explain your research and offer to share any results that may be of interest. If you plan to share the reasoning when publishing your measurement results, e.g. in an academic paper, you must seek consent for this from the requester.¶

Be aware that in publishing your measurement results, it may be possible to infer your "Do Not Scan" list from those results. For example, if you measured a well-known list of popular websites then it would be possible to correlate the results with that list to determine which are missing. This inference might leak the fact that those websites specifically requested to not be scanned.¶

2.7. Minimize data

When collecting, using, disclosing, and storing data from a measurement, use only the minimal data necessary to perform a task. Reducing the amount of data reduces the amount of data that can be misused or leaked.¶

When deciding on the data to collect, assume that any data collected might be disclosed. There are many ways that this could happen, through operational security mistakes or compulsion by a state authority.¶

When directly instrumenting a protocol to provide metrics to a passive observer, see section 6.1 of RFC6973 [RFC6973] for the data minimization considerations enumerated below that are specific to the use case.¶

2.8. Reduce accuracy

There are various techniques that can be used to reduce the accuracy of the collected data and make it less identifying.¶

The use of binning to group numbers of more-or-less continuous values, coarse categorization in modeling, reduction in concentrations of IP address by geography (geoip) or other first- or second-order identifiers, the introduction of noise and all privacy-preserving measurement techniques that allow researchers to safely conduct internet measurement experiments without risking harm to real users [Janson].¶

2.9. Analyze risk

The benefits of internet measurement should outweigh the risks. [Menlo] [Kenneally]¶

Consider auxiliary data (e.g. third-party data sets) when assessing the risks. Consider that while a privacy risk may not be immediately apparent or realisable, in the future increased computing power may then make something possible.¶

Example: A research project releases encrypted payloads as a method for minimising exposure of sensitive user data. However the encryption could be trivially broken in the future with typical increases in computing power.¶

2.10. Transparency and accountability

Have “a general policy of openness about developments, practices and policies with respect to personal data.” [OECD]¶

Despite best intentions, things fall apart. In the case of any adverse events, researchers should “responsibly inform affected stakeholders.” [Menlo] Where contacting users is not feasible, information about discovery of security vulnerabilities, improvements and fixes, data compromise, instances of unauthorized access to information should be published publicly.¶

If possible, institute a regular accountability and reporting mechanism, such as an annual review by an advisory board. Such regular accountability mechanisms can include an assessment of the experiment's impact on user safety and privacy, analysis of security events, and an evaluation of whether the experiment is meeting its expected goals.¶