%include "default.mgp"
%tab bullet icon arc blue 40
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%nodefault
%bgrad 0 0 32 0 1 cyan magenta
%xfont "utopia-bold-r", size 12

%center, fore "black"
cflowd and arts++
%size 5
Cisco flow-export collection

%xfont courier-bold-r





dwm@caida.org

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Background

%fore black, size 5

&bullet host software for collecting Cisco version 5 flow-export data
&bullet aggregates data into tables for continuous collection of\
 summary data in time-series
&bullet stores raw flow data in rotating log files
&bullet provides client/server collection of tabular data in time series

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Changes from 1.3b2 to 2.0

%fore black, size 5

&bullet central collector included
&bullet uses arts++ package
&bullet arts++ adds signficant functionality

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black
%center

cflowd

%image "ciscoToCflowd.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

flow export version 5

%fore black, size 5

&bullet sends UDP packets to a specified host address and port number
&bullet each UDP packet contains a number of flow entries

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black
%center

flow-export packets

%image "flowPacket.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black
%center

flow header

%image "flowHeader.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black
%center

flow entry

%image "flowEntry.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

cflowd aggregation

%fore black, size 5

&bullet cflowd primarily designed to aggregate flow data into\
 tabular data to be used for capacity planning
&bullet AS matrix, net matrix, port matrix and protocol table aggregation\
 across all flows

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Why so much aggregation?

%fore black, size 5
&bullet data reduction
&bullet scale of intended use (backbone-wide)
&bullet Add a cflowd host, you may add it to central collection of\
 tabular data.
&bullet unreliable transport from Cisco to cflowd encourages\
 deploying multiple cflowd hosts; use them.
&bullet disk and bandwidth requirements for raw flow data\
 in a backbone network.  Only transfer tabular data back to the\
 central collector.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

cfdcollect

%fore black, size 5

&bullet collects cflowd data at regular intervals
&bullet produces time-series data for each of the tabular data\
 types (AS matrix, net matrix, port matrix and protocol table)
&bullet stores data in ARTS files

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black
%center

centralized collection

%image "centralCollection.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

arts++

%fore black, size 5

&bullet C++ class library for subset of ARTS data
&bullet supports reading/writing of ARTS data via iostreams and\
 UNIX file descriptors
&bullet supports simple time-domain aggregation for several data types
&bullet simple command-line utilities included for viewing ARTS data\
 files and time domain aggregation

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS data files

%fore black, size 5

&bullet efficient data archival (binary, simple size-reducing techniques)
&bullet data files are portable; always written in network byte order,\
 the arts++ class library is the interface
&bullet extensible for additional data types
&bullet versioning of data objects for different storage formats (typically\
 used for space/CPU tradeoff)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS data handled by arts++

%fore black, size 5

&bullet AS matrix (version 0)
&bullet net matrix (version 2)
&bullet port matrix (version 0)
&bullet selected port table (version 0)
&bullet port table (version 2)
&bullet protocol table (version 2)
&bullet forward IP path (version 0)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS AS matrix (version 0)

%fore black, size 5

&bullet counters for traffic (packets and bytes) from\
 source ASes to destination ASes
&bullet sparse matrix, having only\
 entries for which traffic information is stored

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS AS matrix example data

%center
%fore black
%image "asMatrixData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS net matrix (version 2)

%fore black, size 5

&bullet counters for traffic (packets and bytes) from\
 source networks to destination networks
&bullet networks are identified by network number and netmask length
&bullet sparse matrix, having only entries for which traffic\
 information is stored

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS net matrix example data

%center
%fore black

%image "netMatrixData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS port matrix (version 2)

%fore black, size 5

&bullet counters for (packets and bytes) from source transport layer\
 port number to destination port number
&bullet matrix is sparse; there are no entries for ports on which no traffic\
 was seen

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS port matrix example data

%center
%fore black

%image "portMatrix.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS port table (version 2)

%fore black, size 5

&bullet counters for input and output traffic (packets\
 and bytes) versus transport layer port number
&bullet input counters represent traffic destined for the port\
 while output counters represent traffic sourced from the port
&bullet table is sparse; there are no entries for ports on which no traffic\
 was seen

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS port table example data

%center
%fore black

%image "portTableData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS protocol table (version 2)

%fore black, size 5

&bullet counters (packets and bytes) versus IP protocol\
 (TCP, UDP, ICMP, IGMP, et. al.)
&bullet sparse table, there are no entries for protocols\
 that were not seen in the measured traffic

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS protocol example data

%center
%fore black

%image "protocolTableData.gif"


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS IP forward path (version 0)

%fore black, size 5

&bullet contains IP addresses of hops in forward path from a\
 source to a destination
&bullet contains an RTT value for the source to destination
&bullet may be extended in the future to hold more information

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

ARTS forward IP path example data

%center
%fore black
%image "ipPathData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Time Domain Aggregation

%fore black, size 5
Time domain aggregation reduces time granularity in time-series data.
Examples:

&bullet convert 5-minute data to 1-hour data for a summary bar chart
&bullet convert 1-hour data to one large aggregate for a pie chart

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Aggregation Utilities

%fore black, size 5
&bullet artsasagg - aggregates AS matrix data
&bullet artsnetagg - aggregates net matrix data
&bullet artsportmagg - aggregates port matrix data
&bullet artsprotoagg - aggregates protocol table data

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Simple display utilities

%fore black, size 5

&bullet artsdump
&bullet artsases
&bullet artsnets
&bullet artsportms
&bullet artsports
&bullet artsprotos

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Future Tools

%fore black, size 5

&bullet plotting utilities using XRT/PDS
&bullet Web reporting tools (probably java-based)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont utopia-bold-r
%bgrad 0 0 32 0 0 white cyan
%fore black

Open Questions

%fore black, size 5

&bullet what types of aggregation are useful to network service providers?
&bullet what additional tabular data types are needed?
&bullet are there desired applications for flow-export outside of capacity\
 planning and usage/billing?
&bullet what real time features would providers find useful?