Characterization of the Traffic between SLAC and the Internet
Connie Logg
July 17, 2003, August 8, 2001, March 1, 2001

INDEX
[Analysis] [Challenges] [Introduction] [Methodology]
[Data mining]
Index to the SLAC Netflow Analysis

Introduction

The SLAC <-> Internet connection is instrumented with a Cisco Catalyst 6509 and an MSFC router. Cisco's Netflow is enabled on the switch and Netflow records are sent to a Linux server running. The Netflow records are collected, buffered, and written to disk every 10 minutes by Cisco's Netflow Collector Software. The Netflow data is then analyzed by perl scripts written at SLAC.

There are several aspects to the analysis including (* indicates that the information is only viewable withon SLAC):

Protocols: Characterizing the protocols used
Applications: Characterizing the applications as defined by the Micsmon Committee
Programs: Identifying the program the traffic is related to (SSRL or HEP)
Flow sizes: Characterizing the size of the flows
Flow times: Characterizing the length of the flows
Flow throuputs: Characterizing the throughputs of the flows
Packet sizes: Characterizing the packet sizes of the flows
Destination:* Characterizing the destination of the traffic coming into SLAC
Sources: Characterizing the external sources/destinations of the traffic to/from SLAC
Security:* Analysis of the traffic for compliance with security and appropriate use requirements
Resources:* Analysis of the resources required to process the Netflow data

Analysis

Analysis performed includes:

Protocols: A breakdown of the traffic by protocol
Applications: breakdown as defined by the Micsmon committee
Programs: breakdown by research programs at SLAC.
Flow sizes:
- Analysis of the distribution of flow byte sizes and packet sizes for a day. This analysis also includes power law parameter (a,b) fits (f(x)=100000*a*10^x*b)
- Time series analysis for a day of in/out tcp/udp packets and bytes per flow
Destination: SLAC network is partitioned into virtual lans (VLANs) which serve specific functions. The destination analysis creates a graph which shows the volume of traffic associated with each of these VLANs.
Sources: The "source" and destination of the traffic to/from SLAC is analyzed in two ways.
- An analysis of the top 2 domain levels is done for collaborators in SLAC efforts for the IEPM and PINGER projects. The graph includes a breakdown of the type of traffic.
- An analysis of the top level domains for all the traffic is graphed. Note that not all of the "top" domains are actually likely to be valid. "DNS" is a pseudo domain defined by me to account for IP name resolution traffic.
Resources: - Between 6 and 19 million Netflow records are generated in a weekday. As the analysis occurs, a record of the cpu time, elapsed time, and disk space required is created. Each day, the complete Netflow data collected for the previous day is analyzed. It takes about 12-18 hours of elapsed time for the analysis to be done on a Linux server with 2 cpus. The various analyses scripts are invoked serially (not concurrently). The disk space required for 8 million records is approximately:
- Binary data : 313 megabytes
- CSV ASCII data : 590 megabytes
- Data analysis results: 350 kilobytes
The CSV form of the data is kept around for about 6 months for convenient mining in the validation of the analysis.

Methodology

Terminology:

$nfcsrc = "/afs/slac/package/netmon/netflow/src" - The SLAC developed code which performs the analysis resides here
$nfcdata = "/nfs/slac/get/net/lanmon/netflow" - The output of the analysis resides here

Analysis methodology:

The data is collected by the Cisco Netflow data collector and it calls a script "$nfcsrc/nfc-receiver" passing the binary file name as an argument.
Each day, the complete Netflow data collected for the previous day is analyzed by a call to '$nfcsrc/run-analysis -d yesterday' from a crontab shortly after midnight.

Challenges

Netflow running on the SLAC DMZ can generate over 8 million records a day. A record can represent more than one flow, so the challenge of analyzing the data correlates with the number of records cut per day, not the number of flows.

DMZ1 is only used for the route going to Stanford (i.e. not ESnet routes).

Application Port Identification - Please note that this analysis is evolving as research continues into how the various TCP and UDP ports are used. The various pieces of documentation on port usage often indicates that a given port is used for a specific application in both TCP and UDP. Recent research by looking at the actual packets indicates in some cases that this is not really the case. For example, the graphs above for TCP show some TCP AFS traffic. An examination of these packets indicates that they are not AFS. In addition, ports which have been "defined" may also by used for other purposes.

Identifying the "Application" - Which port (source or destination) should be used to classify the application traffic? To make this decision, the following algorithm is used:

Is either port 0? If so, call it port0 traffic.
Is the lowest number port "defined"? If so, use it.
Is the highest number port "defined"? If so, use it.
If neither port is 0, or defined, use the lowest numbered port to classify the traffic.

Naming the TCP and UDP application ports - What names should be used to name the TCP and UDP application ports? Different documents use different names.

IP Address to IP Name Translation - By far the greatest challenge in analyzing the Netflow data lies in resolving the IP addresses in a record to IP names. It can vary from less than 1 second to translate a name up to several minutes. Given 2 ip addresses to a record, it has been necessary to develop techniques for optimizing this process. Some trickes used are:

A table (referred to as the "hosttable") of all known SLAC IP addresses is loaded by the code when it starts the conversation analysis. This eliminates almost 50% of the necessary nameserver lookups.
When a name is looked up, an entry is created in the "hosttable". This entry is then used while processing the rest of the days data.
Analysis of the IP addresses which required lookup (non-SLAC IP addresses) showed that those requiring the longest lookup time were the ones for which no translation was available. As one of these is encountered, an entry is made in "hosttable" so that an attempt is not made to look it up again.
Analysis of the IP addresses for which lookups failed, showed two things:
- Many were ip addresses of nameservers - To take advantage of this characteristic, a check is made to see if the source port or destination port is the DNS port (53). If it is, an entry is made into "hosttable" so that the address is not looked up.
- Many were ISP addresses that had no translation - An analysis of 2 months ipaddress to name translations showed that in most cases there were few exceptions seen (exceptions were thrown out) to the hypothesis that many translations could be avoided by building up a list of top 2 or 3 level domain fields. For example: for .GOV, .EDU, .MIL, .NET, .ORG, & .COM, 2 level domain name fields (e.g. ATT.NET, HARVARD.EDU, AF.MIL) could be indexed by the first two octets of the IP address; and for many other sites (primarily international) 3 level domain name fields correlations could be made (e.g. ROMA1.INFN.IT, PD.INFN.IT, RL.AC.UK). A table (called "shorthost") is created for these. If the first two octets of the IP address are in this table, the name is wildcarded with the top 2 (or 3) levels. For example: "?.HARVARD.EDU".

Understanding How Netflow Works - The documentation on the actual operation of Netflow is minimal and conflicting. Some examples are:

How or when are flow records cut? The references indicate the following are the parameters that determine when a flow expires and a record is cut:

Transport is completed (TCP FIN or RST)
The cache is becoming full
After 15 sec of traffic inactivity (the only way for UDP).
After 30 min of traffic activity.

However analysis of the actual data shows that there are actually many flows where the active time is reported as being longer than 1800 seconds, and there are some where the active time is "infinite". I suspect the latter is actually a bug in the record generation.

How are IP fragments counted? The analysis of the data shows that a major portion of the UDP traffic and occasionally some TCP traffic is labeled in its Netflow record as being to and from port 0. Capture of these packets in stream shows them to actually be IP fragments. SLAC has a lot of AFS traffic to and from the outside world, and the UDP AFS traffic generally consists of a header packet followed by IP fragment packets. Netflow loads the source and destination ports of the IP fragment as 0 and 0. This has recently been verified. Here is an example of an AFS transfer.

Data Mining and Validation Script

FLOWMINE - /afs/slac/package/netmon/netflow/src/flowmine is a script which facilitates independent mining of the Netflow data. It can either take arguments on the command line or from a file. Arguments passed on the command line superceed any arguments in the file. The switch name must be provided. The only required argument is the switch name. Note that the netflow data is read protected, and only authorized users can access it.