Welcome to Measurement Lab

All data collected through M-Lab is intended to be made publicly available and placed in the public domain. Currently, data from NDT and NPAD are available, and other tools' datasets will become available in the future. You can access these datasets through Amazon's Public Data Sets, and we hope to make the data available in other ways as well.

• Access NDT data here
• Access NPAD data here

Below, you can find more information about how the datasets are structured and their contents.

1. Web100 Variables used by NDT and NPAD
2. NDT data
3. NPAD data

Web100 Variables used by NDT and NPAD

Both NDT and NPAD make use of Web100, and it is necessary to understand Web100 variables in order to interpret the data.

The web100 variables stored in a "snaplog" are documented in the file tcp-kis.txt contained in the Web100 kernel tarball. For reference, these can be obtained from: http://www.web100.org/download/kernel/2.5.25/tcp-kis.txt (kis = kernel instrument set).

People might find it easier to read RFC 4898, the TCP extended statistics MIB, which was the final product of the Web100 project. The RFC was also generated in part from tcp-kis.txt. Several of the variable names differ and just a few were completely revised between Web100 and the published RFC.

The snaplog is a binary file, however it starts out with a plain-text ascii description of the following Web100 snapshots. Each line has the variable name followed its byte offset in the structure, a data type code, and its size in bytes. The Web100 library provides tools to fully decode the snaplog.

NDT data structure

The NDT client/server program streams data for 10 seconds from the client to the server and a 2nd 10 sec test streams data in the opposite direction. The measured throughput (bytes transfered / time) for each test is reported to the user.

NDT records the following for each test run:

A tcpdump trace for every upload/download tests. This is a standard trace file for all packets (except the initial syn & syn/ack exchange). The tcpdump and tcptrace programs can read these files.

A web100 snaplog trace for every upload/download test. This file contains all the the web100 kernel MIB variables in 5 msec increments. The NDT package comes with a utility (called genplot) that can convert these trace files into text or xplot graph files. Alternatively the user can write their own analysis program using the web100 library functions.

The server logs the final value of 50 web100 variables. It also reports some analysis details. This data is written out in a comma (,) separated line, so parsing the data is a matter of 'knowing' what variable is in what position in the line. The details for this are in the code.

To make it easier to use this data, the NDT server writes out a meta data file (*.meta). This metadata file contains the names of the above files, the clients & servers IP addr and FQDN, and the encoded line of web100/analysis data. A sample of that data is

[iupui_ndt@mlab1 ~]$ cat /usr/local/ndt/serverdata/
2009/10/09/20091009T20:46:28.
141927000Z_149.20.53.166:63251.meta
Date/Time: 20091009T20:46:28.141927000Z
c2s_snaplog file: 20091009T20:46:28.141927000Z_149.20.53.166:60102.c2s_snaplog.gz
c2s_ndttrace file: 20091009T20:46:28.141927000Z_149.20.53.166:60102.c2s_ndttrace.gz
s2c_snaplog file:
s2c_ndttrace file: 20091009T20:46:28.141927000Z_149.20.53.166:61730.s2c_ndttrace.gz
cputime file:
server IP address: 4.71.254.147
server hostname: mlab1.atl01.measurement-lab.org
server kernel version: 2.6.22.19-vs2.3.0.34.32.mlab.pla
client IP address: 149.20.53.166
client hostname: nb.tech.org
client OS name:
client_browser name:
Summary data:313,2246,1658,0,69963,1020,19,8,1961,0,1448,392,1428,525600,112560,
43440,0,10001122,25307,2899352,0,15,15,21720,270,525600,100,0,0,0,1,6,2,8,10,8,1961,
66,10,22,964,0,166,0,-1208754176,3,0,31,0,360,1448,43440,8

The NDT server also contains a Java application (which can be downloaded here) that can look through the NDT log files and link log entries to trace files, and it also simplifies the viewing of this data. The utility will need to be rewritten slightly to take advantage of the meta files instead of (in addition to) the log files.

The detailed tcpdump and snaplog files can be used to re-examine the test to see what happened. I.E.:

a) was there packet loss during the s2c test, and if so when/how often did it occur?
b) what was the maximum throughput during TCP's slow-start growth phase?
c) how many times did TCP oscillate in the Congestion Avoidance phase?
d) did the test compete with other traffic?
e) was there non-congestive loss on the path?
f) was the test limited by the user's PC (default tunable settings)?
g) how often did TCP retransmit packets, and were any of these unnecessary?
h) were packets being reordered (e.g., sent 1, 2, 3, 4, 5 but received 1, 2, 4, 3, 5)
i) what was the capacity of the bottleneck link in this path?
j) did the test produce the expected results?
k) was the client connected to a wired or wireless (WiFi) network?
l) is there a firewall and/or NAT box in the path?

More information about NDT:
http://www.internet2.edu/performance/ndt/

NPAD data structure

All of the NPAD data is placed in the archive:
YYYY/MM/DD/<server_name>/<iso8601time>_<client_address>_<client_hostname>[<suffix>].<ext>
Where:
YYYY/MM/DD/ -- year/month/date (UTC).
server_name -- Name of the MLab server where the data was collected.
iso8601time -- UTC timestamp in ISO8601 format (e.g. output of 'date -u +%Y%m%dT%T').
client_address -- IP address of the system being tested.
client_hostname -- Reverse DNS lookup of the client IP address (or IP address if none).
[suffix].ext -- File name suffix (optional) and extension.

For primary data collection files <suffix> and <ext> are as follows :
TIME_IPaddr_HOSTNAME.ctrl : metadata and fine grained Web100 controls (python/ascii)
TIME_IPaddr_HOSTNAME.log : log of the web100 instruments (binary)
TIME_IPaddr_HOSTNAME.out : stdout and stderror, for debugging only

Files generated from the above, during the report generation phase:
TIME_IPaddr_HOSTNAME.html : The report
TIME_IPaddr_HOSTNAME.live : a duplicate of the .html for regression testing
TIME_IPaddr_HOSTNAME.plt : tabular data, suitable for importing into a spread sheet
TIME_IPaddr_HOSTNAMEsmry.html : one row to be inserted into /summary.html
TIME_IPaddr_HOSTNAMEloss.png : (optional) plot of the loss rate
TIME_IPaddr_HOSTNAMEpow.png : (optional) plot of the power = Rate/RTT
TIME_IPaddr_HOSTNAMErate.png : (optional) plot of the data rate
TIME_IPaddr_HOSTNAMErtt.png : (optional) plot of the Round Trip Time

Note: Plotting is currently tuned off.

For NPAD the primary data for each run is stored in a Web100 snaplog (*.log). These are generally 1 second samples. The *.ctrl files start with control parameters that correspond to each saved web100 snapshot in the snaplog. These include some of the most important measurement parameters (duration of this sample, bytes transferred, etc) as well as the web100 controls used to manipulate the flow (most important is LimCwnd, which clamps cwnd to some value less than what would be chosen by TCP's congestion control). The final section of the .ctrl file dumps various parameters which were computed from the run, (e.g. peak data rate) and a whole bunch of parameters that are very specific to the code.

Other files are various fragments of the .html report and index page, or used by the tools that I use for regression testing.

More information about NPAD:
http://www.psc.edu/networking/projects/pathdiag/

For a description of the tool and algorithm people should see: M. Mathis, J. Heffner, P. O'Neil, P. Siemsen, "Pathdiag: Automated TCP Diagnosis", PAM 2008. Obtain from http://staff.psc.edu/mathis/papers/mathis08pathdiag.pdf