The Packet Header Service is an M-Lab core service that runs alongside every hosted measurement test. For every TCP connection that reaches an M-Lab server, it captures the packet headers (not the payload) and saves them as a .pcap file. Each file is named after the UUID of the TCP flow, making it straightforward to correlate packet captures with their corresponding measurement results.
This service is a passive sidecar: it does not generate any traffic itself and does not affect the measurements. It simply records what arrives.
What Is Captured
The packet-headers service saves headers only, not payload content. This means:
- IP headers (source/destination addresses, TTL, flags)
- TCP headers (sequence numbers, acknowledgment numbers, flags, window size, options)
- Timestamps with sub-millisecond precision
Payload bytes are deliberately excluded to protect user privacy and reduce storage volume. The headers alone are sufficient for TCP behavior analysis — retransmits, congestion signals, window scaling, and timing can all be reconstructed from headers.
For WeHe tests only - IP address anonymization is supported via a command-line flag and can be enabled on a per-deployment basis.
How It Works
The service runs as a separate binary (packet-headers) on each M-Lab server. It uses a packet capture library (libpcap) to monitor all incoming TCP flows. When a new flow is established, it gets the flow’s UUID from the tcp-info service and writes the packet headers to a compressed .pcap.gz file named after the UUID. By default, only the first 30 seconds of each flow are saved.
The headers are not published in BigQuery, but the index (needed to find the right .tgz archive, see below) is: measurement-lab.ndt_raw.pcap and contains id (the UUID), and parser.ArchiveURL of the relevant pcap test data.
Because file naming is UUID-based, you can take any measurement result from BigQuery (e.g., an NDT test), extract its UUID, and use the measurement-lab.ndt_raw.pcap index to find the .tgz archive in GCS that contains the corresponding .pcap.gz file.
Accessing Packet Header Data
Google Cloud Storage
Packet captures are stored in GCS, organized by the measurement service that generated the connection:
Files are organized by date.
Open them with Wireshark, tcpdump, or any tool that reads standard .pcap format.
BigQuery
Packet header data is not published to BigQuery. Analysis requires working with the raw .pcap files in GCS.
How People Use Packet Header Data
Deep TCP analysis — packet captures enable analysis at the sub-RTT level: tracing individual retransmissions, observing congestion window evolution, detecting spurious retransmits, and characterizing buffer sizing.
Reconstructing congestion events — when NDT or MSAK reports unexpectedly high loss or low throughput, the corresponding PCAP can show exactly which packets were dropped and when.
Validating TCP metrics — tcp-info provides periodic kernel-level statistics; PCAPs provide a complementary view that can validate or extend those statistics with packet-level timing.
Research requiring raw packet timing — studies of queuing latency, jitter, or packet reordering require sub-millisecond precision that only packet captures can provide.
Protocol behavior studies — researchers studying TCP options (SACK, timestamps, window scaling) or congestion control algorithm behavior use PCAPs to observe these mechanisms in the wild.
Source Code
Citing Packet Header Data
The M-Lab Packet Header Data Set, <date range used>. https://measurementlab.net/tests/pcap