New Script-Level Data Input API

Overview

This is the companion to the logging framework and is looking as though it may be very similar in many respects. The idea behind the API is that it helps users import and use data without having to write Zeek scripts containing the actual data. It is also intended to provide an easier way to manage the flow of data as it’s imported and updated at runtime. It should use existing data structures to hold the data once it’s pulled into Zeek to reuse existing table/set membership (and other) operators.

Desired Features

  • Create mappings between external data sources and Zeek
    1. Solely event based. i.e. just get the data into Zeek as an event!
    2. Easy and automatic mapping into an internal data structure.
  • Plugin support for pulling data from a variety of sources (files, databases)
    • Support the same files,databases,etc that the logging framework supports.
  • Automatic data refresh
    • Polling, "tailing", and on-demand pull should all be supported.
  • Live queries using the ‘when’ statement
    • Not sure how this would look or work in the cluster context since the data would all be streaming from the manager most likely.
  • Regarding automatic mapping to internal data structures
    • Filter records from the external data source before inserting into internal data structures.
    • Automatic data expiration based on Source’s refresh interval.

Proposal

The input interface is built around three main abstractions:

Data sources
A source identifies a set of data that is grouped into a cohesive data set. Mostly referring to a single data file or single table in an SQL database.
Filters
A filter takes a specific data input source as its input and decides what of the received information to put into the memory store.
Readers

A reader interfaces with the external repository of data (file/database/etc) and turns the key-value data contained within into records which can then be passed to the associated data source.

Note that readers will be implemented in C++, not on the script level.

API

module Input;

export {
## The default interval for refreshing the data from data sources.
const default_refresh = 1hr &redef;
}

# The set of writers Zeek provides.
type Reader: enum {
READER_DEFAULT, # See default_writer below.
READER_ASCII,
READER_DATA_SERIES,
READER_SYSLOG
};

## The default reader to use if a filter does not specify
## anything else.
const default_reader = READER_ASCII &redef;

## Each source gets a unique ID. This type will be extended by
## other scripts.
type ID: enum {
Unknown
};

## Type defining a data source.
type Source = record {
## The ID of the source.
ID id;

## The reader to use.
reader: Reader &default=READER_DEFAULT;

## The path name for the particular dataset. The exact definition and
## implementation of this value is left up to the discretion of the
## reader. A file reader might make this a file name while an SQL database
## reader might make this a complete SQL query.
path: string;

## A record type defining the source’s input columns.
columns: any;

## Refresh interval to refresh the data from the original data source.
## What this means exactly is left to the discretion of the reader. If
## the reader implies some data type that is fed in live (like syslog),
## this option may be irrelevant, but if the reader is pulling from a file,
## there my be some polling interval that you’d like to stick to. But, it’s
## also possible that you may want to tail a file as well.
refresh: interval &default=default_refresh;

# An event that is raised whenever the filter is applied
# to an entry. The event receives the same parameter
# as the predicate. It will always be generated,
# independent of what the predicate returns.
ev: event &optional;
};

## A filter defining which records to import into a local data structure.
type Filter = record {
## A name to reference this filter.
name: string;

## A predicate returning True if the filter wants to accept this
## data record. If not given, an implicit True is assumed
## for all entries. The predicate receives one parameter:
## an instance of the input datas record type.
pred: function &optional;

## A table/set/vector where the data that is accepted by the predicate
## will be saved. A write_expire will also be applied to the data based
## on the refresh interval of the Source to which this filter is applied.
##
## I think we may want to support all data types here. This could turn
## into a way to store configuration-type data externally from Zeek scripts.
store: any;

## If the $store value is a type with an index (table/set), then this value
## will need to be supplied to provide the input framework with the
## information about which field(s) from the Source should be used as the
## index value(s). If the types in the index don’t match the types of the
## fields given here, Zeek will complain or not start.
##
## If the store is a table, the entire Source’s record will be used as
## the yield value. Otherwise, if the store is a set or vector, only
## the values given here will be used.
index: vector of string &optional;
};

Page Contents

Twitter @zeekurity

Blog Icon

    Search