direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Sie sind hier

TU Berlin

Inhalt des Dokuments

Es gibt keine deutsche Übersetzung dieser Webseite.

Time Machine (Old information)

The time machine is a joint project of the Technische Universität Berlin, the Technische Universität München, and the ICSI (University of California Berkeley). It is open-source and published under the BSD license.

Outline

There are times when it would be extraordinarily convenient to record the entire contents of a high-volume network traffic stream, in order to later "travel back in time" and inspect activity that has only become interesting in retrospect. Two examples are security forensics—determining just how an attacker compromised a given machine—and network trouble-shooting, such as inspecting the precursors to a fault after the fault.

To perform this task efficiently, the packets are first stored in a ring buffer in the memory (RAM), later the packets are copied to (hard) disk. This allows the time machine to smoothen capture bandwidth peaks in memory and store huge amounts of traffic on disk, covering several days of network traffic. The time machine is designed to work in Gbps environments.

Since it is not feasible to capture the complete load of a fully utilized Gbps link to disk, the time machine utilizes a mechanism called "connection cutoff" to reduce the the amount of data to process. This "connection cutoff" only records the first X bytes of every monitored connection (identified via the 5-tupel of source and destination IP and Port and the transport protocol). Indeed this approach it does not impair the analysis capabilities (unless the cutoff is set to low) because most of the "interessting" data is located in the first few packets of a connection. The effiency of this approach comes from leveraging the heavy-tailed nature of network traffic: because the bulk of the traffic in high-volume streams comes from just a few connections.

To take full advantage of this recording it is import to be able to quickly locate certain packets. For example one might be interested in all packets of a specific connection or all packets from one IP address. This is achieved by indexing stored packets. The indexes to create can be specified, for example one could create indexes for the connection 5-tupel, for IP address pairs, for IP addresses, etc. One can than issue a queries for a specific index to the time machine and the time machine will lookup the query in its index and will return all stored packets matching the query.

To further streamline the analysis capabilities we have coupled the TimeMachine with the Bro network intrusion detection system (IDS) (www.bro-ids.org). Thus the IDS can directly interact with the TimeMachine and request historic traffic to represent it to a security analyst or to do retrospective analysis.

Documentation

Download

Please note, that the current release of the time machine is in an early development stage. Bug reports and comments on the functionality and handling of the time machine and its documentation are appreciated. Please do not hesitate to send an email with your question or comment to tmlists.net.t-labs.tu-berlin.de. Developer release: Download tm-20090206-0.tar.gz Most notable changes since 20080206:

  • Bugfixes in broccoli communication code
  • GCC 4.3 compatability

Most notable changes since 20061220:

  • Many bugfixes and performance improvements.

    • New connection table code (with less locking)
    • New index hash tables
    • Disk write performance

  • Coupling with IDS through broccoli.
  • Support for dynamic classes: The TM can be instructed to (temporarily) assign a particular host to a different storage class (e.g., if an IDS detected suspicious behavior from that host)
  • Better logging facilities.
  • Subscription support for all indexes.

Most notable changes since 20061111:

  • Huge increase in performance due to

    • Changes in internal data structures
    • Index generation and aggregation
    • using ptmalloc on FreeBSD
    • Thread scheduling

  • Documentation Updates
  • Support for running tm in the background as a daemon

(Be sure to subscribe to tm-announce.)

Previous releases:

If you are experiencing packet losses, you might perhaps want to take a look at our recommendations for best packet capturing systems.

Users Mailinglist

For up-to-date Informations on the Time Machine project, new versions, and improvments please be sure to subscribe to tm-announce mailinglist subscription page

Contributors

Authors:

  • Gregor Maier (TU Berlin / DT Laboratories)
  • Stefan Kornexl (TU München)

Contributors:

All of us can be reached via the time machine list: .

Publications

Operational Network Intrusion Detection: Resource-Analysis Tradeoffs
Zitatschlüssel D-ONIDRAT-07
Autor Dreger, Holger
Jahr 2007
Schule Technische Universität München, Munich, Germany
Zusammenfassung Network Intrusion Detection Systems (NIDS) span an area of massive research and commercial interest. Modern systems offer a wide range of capabilities and parameters to adapt the analysis to the needs of the operator. If one deploys a NIDS in a high volume network environment (1Gbps or more) however, one notices that some capabilities are not usable as the available resources (CPU cycles and memory) of the NIDS are not sufficient for such detailed analysis. In this thesis, we target a thorough understanding of the dependencies and tradeoffs between NIDS resource usage and detection capabilities. We base this work on our operational experience with NIDS in three large research network environments, among them the M¨unchener Wissenschaftsnetz (MWN), Germany. We demonstrate, that operational network intrusion detection in high-volume network environments raises a host of resource management issues. We explore tradeoffs between resource usage and analysis, that range from predictive to adaptive to retrospective. Predicting the resource usage of a NIDS is difficult. We set out to develop a performance model of a NIDS that allows to determine the appropriate analysis depth and parametrization of the system. The model can be used in two ways: First, to help determining a configuration of the NIDS based on the measured traffic characteristics of the network environment. Second, it can be used to predict the NIDS resource usage based on a known or guessed trend of the network traffic development. Connection oriented NIDS do not analyze every connection the same. However the decision process of how to analyze a connection is rather hard-configured into the NIDS. This means, at run-time the NIDS cannot adapt the analysis per connection. We develop a framework for connection oriented NIDS to decide at run-time per connection what analysis to perform. We use this framework for dynamically performing the appropriate application layer protocol decoding. Using this enhancement, a NIDS is for example able to reliably detect applications not using their standard ports, do payload inspection of FTP data connections and to reliably detect IRC based botnet clients and servers. If a NIDS alerts in high volume environments, it does not have the possibility to provide a lot of context to the operator. For trading off disk space against forensic capability, we develop a NIDS-supplementary tool called Time Machine. The tool records a prioritized yet comprehensive packet trace in high level environments. Our approach leverages the heavy tailed connection size distribution: it prioritizes small connections over large ones which greatly reduces the volume of traffic to record while retaining the largest fraction of the connections transferred. For our operational evaluation of the tradeoffs and the developed mechanisms, throughout this thesis, we use the open source NIDS Bro. Its design is targeted to maximum flexibility, which makes it an ideal platform for powerful extensions and for use in a wide range of experiments.
Typ der Publikation Doktorarbeit
Link zur Publikation Download Bibtex Eintrag

Zusatzinformationen / Extras

Direktzugang:

Schnellnavigation zur Seite über Nummerneingabe