base/frameworks/broker/store.bro

Broker

The Broker-based data store API and its various options.

Namespace:Broker
Imports:base/bif/data.bif.bro, base/bif/store.bif.bro, base/frameworks/broker/main.bro
Source File:/scripts/base/frameworks/broker/store.bro

Summary

Redefinable Options

Broker::default_clone_mutation_buffer_interval: interval &redef The maximum amount of time that a disconnected clone will buffer data store mutation commands.
Broker::default_clone_resync_interval: interval &redef The default frequency at which clones will attempt to reconnect/resynchronize with their master in the event that they become disconnected.
Broker::default_clone_stale_interval: interval &redef The duration after which a clone that is disconnected from its master will begin to treat its local cache as stale.

Types

Broker::BackendOptions: record Options to tune the particular storage backends.
Broker::BackendType: enum Enumerates the possible storage backends.
Broker::QueryResult: record The result of a data store query.
Broker::QueryStatus: enum Whether a data store query could be completed or not.
Broker::RocksDBOptions: record Options to tune the RocksDB storage backend.
Broker::SQLiteOptions: record Options to tune the SQLite storage backend.

Functions

Broker::append: function Extends an existing string with another.
Broker::clear: function Deletes all of a store’s content, it will be empty afterwards.
Broker::close: function Close a data store.
Broker::create_clone: function Create a clone of a master data store which may live with a remote peer.
Broker::create_master: function Create a master data store which contains key-value pairs.
Broker::data: function Convert any Bro value to communication data.
Broker::data_type: function Retrieve the type of data associated with communication data.
Broker::decrement: function Decrements an existing value by a given amount.
Broker::erase: function Remove a key-value pair from the store.
Broker::exists: function Check if a key exists in a data store.
Broker::get: function Lookup the value associated with a key in a data store.
Broker::get_index_from_value: function Retrieve a specific index from an existing container value.
Broker::increment: function Increments an existing value by a given amount.
Broker::insert_into_set: function Inserts an element into an existing set.
Broker::insert_into_table: function Inserts an element into an existing table.
Broker::is_closed: function Check if a store is closed or not.
Broker::keys: function Returns a set with all of a store’s keys.
Broker::pop: function Removes the last element of an existing vector.
Broker::push: function Appends an element to an existing vector.
Broker::put: function Insert a key-value pair in to the store.
Broker::put_unique: function Insert a key-value pair in to the store, but only if the key does not already exist.
Broker::record_assign: function Replace a field in a record at a particular position.
Broker::record_create: function Create communication data of type “record”.
Broker::record_iterator: function Create an iterator for a record.
Broker::record_iterator_last: function Check if there are no more elements to iterate over.
Broker::record_iterator_next: function Advance an iterator.
Broker::record_iterator_value: function Retrieve the data at an iterator’s current position.
Broker::record_lookup: function Lookup a field in a record at a particular position.
Broker::record_size: function Get the number of fields within a record.
Broker::remove_from: function Removes an element from an existing set or table.
Broker::set_clear: function Remove all elements within a set.
Broker::set_contains: function Check if a set contains a particular element.
Broker::set_create: function Create communication data of type “set”.
Broker::set_insert: function Insert an element into a set.
Broker::set_iterator: function Create an iterator for a set.
Broker::set_iterator_last: function Check if there are no more elements to iterate over.
Broker::set_iterator_next: function Advance an iterator.
Broker::set_iterator_value: function Retrieve the data at an iterator’s current position.
Broker::set_remove: function Remove an element from a set.
Broker::set_size: function Get the number of elements within a set.
Broker::store_name: function Get the name of a store.
Broker::table_clear: function Remove all elements within a table.
Broker::table_contains: function Check if a table contains a particular key.
Broker::table_create: function Create communication data of type “table”.
Broker::table_insert: function Insert a key-value pair into a table.
Broker::table_iterator: function Create an iterator for a table.
Broker::table_iterator_last: function Check if there are no more elements to iterate over.
Broker::table_iterator_next: function Advance an iterator.
Broker::table_iterator_value: function Retrieve the data at an iterator’s current position.
Broker::table_lookup: function Retrieve a value from a table.
Broker::table_remove: function Remove a key-value pair from a table.
Broker::table_size: function Get the number of elements within a table.
Broker::vector_clear: function Remove all elements within a vector.
Broker::vector_create: function Create communication data of type “vector”.
Broker::vector_insert: function Insert an element into a vector at a particular position, possibly displacing existing elements (insertion always grows the size of the vector by one).
Broker::vector_iterator: function Create an iterator for a vector.
Broker::vector_iterator_last: function Check if there are no more elements to iterate over.
Broker::vector_iterator_next: function Advance an iterator.
Broker::vector_iterator_value: function Retrieve the data at an iterator’s current position.
Broker::vector_lookup: function Lookup an element in a vector at a particular position.
Broker::vector_remove: function Remove an element from a vector at a particular position.
Broker::vector_replace: function Replace an element in a vector at a particular position.
Broker::vector_size: function Get the number of elements within a vector.

Detailed Interface

Redefinable Options

Broker::default_clone_mutation_buffer_interval
Type:interval
Attributes:&redef
Default:2.0 mins

The maximum amount of time that a disconnected clone will buffer data store mutation commands. If the clone reconnects before this time, it will replay all stored commands. Note that this doesn’t completely prevent the loss of store updates: all mutation messages are fire-and-forget and not explicitly acknowledged by the master. A negative/zero value indicates to never buffer commands.

Broker::default_clone_resync_interval
Type:interval
Attributes:&redef
Default:10.0 secs

The default frequency at which clones will attempt to reconnect/resynchronize with their master in the event that they become disconnected.

Broker::default_clone_stale_interval
Type:interval
Attributes:&redef
Default:5.0 mins

The duration after which a clone that is disconnected from its master will begin to treat its local cache as stale. In the stale state, queries to the cache will timeout. A negative value indicates that the local cache is never treated as stale.

Types

Broker::BackendOptions
Type:

record

sqlite: Broker::SQLiteOptions &default = [path=] &optional

rocksdb: Broker::RocksDBOptions &default = [path=] &optional

Options to tune the particular storage backends.

Broker::BackendType
Type:

enum

Broker::MEMORY
Broker::SQLITE
Broker::ROCKSDB

Enumerates the possible storage backends.

Broker::QueryResult
Type:

record

status: Broker::QueryStatus

Whether the query completed or not.

result: Broker::Data

The result of the query. Certain queries may use a particular data type (e.g. querying store size always returns a count, but a lookup may return various data types).

The result of a data store query.

Broker::QueryStatus
Type:

enum

Broker::SUCCESS
Broker::FAILURE

Whether a data store query could be completed or not.

Broker::RocksDBOptions
Type:

record

path: string &default = "" &optional

File system path of the database. If left empty, will be derived from the name of the store, and use the ‘.rocksdb’ file suffix.

Options to tune the RocksDB storage backend.

Broker::SQLiteOptions
Type:

record

path: string &default = "" &optional

File system path of the database. If left empty, will be derived from the name of the store, and use the ‘.sqlite’ file suffix.

Options to tune the SQLite storage backend.

Functions

Broker::append
Type:function (h: opaque of Broker::Store, k: any, s: string, e: interval &default = 0 secs &optional) : bool

Extends an existing string with another.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
S:the string to append.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::clear
Type:function (h: opaque of Broker::Store) : bool

Deletes all of a store’s content, it will be empty afterwards.

Returns:false if the store handle was not valid.
Broker::close
Type:function (h: opaque of Broker::Store) : bool

Close a data store.

H:a data store handle.
Returns:true if store was valid and is now closed. The handle can no longer be used for data store operations.
Broker::create_clone
Type:function (name: string, resync_interval: interval &default = Broker::default_clone_resync_interval &optional, stale_interval: interval &default = Broker::default_clone_stale_interval &optional, mutation_buffer_interval: interval &default = Broker::default_clone_mutation_buffer_interval &optional) : opaque of Broker::Store

Create a clone of a master data store which may live with a remote peer. A clone automatically synchronizes to the master by receiving modifications and applying them locally. Direct modifications are not possible, they must be sent through the master store, which then automatically broadcasts the changes out to clones. But queries may be made directly against the local cloned copy, which may be resolved quicker than reaching out to a remote master store.

Name:the unique name which identifies the master data store.
Resync_interval:
 the frequency at which a clone that is disconnected from its master attempts to reconnect with it.
Stale_interval:the duration after which a clone that is disconnected from its master will begin to treat its local cache as stale. In this state, queries to the clone will timeout. A negative value indicates that the local cache is never treated as stale.
Mutation_buffer_interval:
 the amount of time to buffer data store update messages once a clone detects its master is unavailable. If the clone reconnects before this time, it will replay all buffered commands. Note that this doesn’t completely prevent the loss of store updates: all mutation messages are fire-and-forget and not explicitly acknowledged by the master. A negative/zero value indicates that commands never buffer.
Returns:a handle to the data store.
Broker::create_master
Type:function (name: string, b: Broker::BackendType &default = Broker::MEMORY &optional, options: Broker::BackendOptions &default = [sqlite=[path=], rocksdb=[path=]] &optional) : opaque of Broker::Store

Create a master data store which contains key-value pairs.

Name:a unique name for the data store.
B:the storage backend to use.
Options:tunes how some storage backends operate.
Returns:a handle to the data store.
Broker::data
Type:function (d: any) : Broker::Data

Convert any Bro value to communication data.

Note

Normally you won’t need to use this function as data conversion happens implicitly when passing Bro values into Broker functions.

D:any Bro value to attempt to convert (not all types are supported).
Returns:the converted communication data. If the supplied Bro data type does not support conversion to communication data, the returned record’s optional field will not be set.
Broker::data_type
Type:function (d: Broker::Data) : Broker::DataType

Retrieve the type of data associated with communication data.

D:the communication data.
Returns:The data type associated with the communication data. Note that broker represents records in the same way as vectors, so there is no “record” type.
Broker::decrement
Type:function (h: opaque of Broker::Store, k: any, a: any &default = 1 &optional, e: interval &default = 0 secs &optional) : bool

Decrements an existing value by a given amount. This is supported for all numerical types, as well as for timestamps.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
Amount:the amount to decrement the value by.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::erase
Type:function (h: opaque of Broker::Store, k: any) : bool

Remove a key-value pair from the store.

H:the handle of the store to modify.
K:the key to remove.
Returns:false if the store handle was not valid.
Broker::exists
Type:function (h: opaque of Broker::Store, k: any) : Broker::QueryResult

Check if a key exists in a data store.

H:the handle of the store to query.
K:the key to lookup.
Returns:True if the key exists in the data store.
Broker::get
Type:function (h: opaque of Broker::Store, k: any) : Broker::QueryResult

Lookup the value associated with a key in a data store.

H:the handle of the store to query.
K:the key to lookup.
Returns:the result of the query.
Broker::get_index_from_value
Type:function (h: opaque of Broker::Store, k: any, i: any) : Broker::QueryResult

Retrieve a specific index from an existing container value. This is supported for values of types set, table, and vector.

H:the handle of the store to query.
K:the key of the container value to lookup.
I:the index to retrieve from the container value.
Returns:For tables and vectors, the value at the given index, or failure if the index doesn’t exist. For sets, a boolean indicating whether the index exists. Returns failure if the key does not exist at all.
Broker::increment
Type:function (h: opaque of Broker::Store, k: any, a: any &default = 1 &optional, e: interval &default = 0 secs &optional) : bool

Increments an existing value by a given amount. This is supported for all numerical types, as well as for timestamps.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
A:the amount to increment the value by.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::insert_into_set
Type:function (h: opaque of Broker::Store, k: any, i: any, e: interval &default = 0 secs &optional) : bool

Inserts an element into an existing set.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
I:the index to insert into the set.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::insert_into_table
Type:function (h: opaque of Broker::Store, k: any, i: any, v: any, e: interval &default = 0 secs &optional) : bool

Inserts an element into an existing table.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
I:the index to insert into the table
V:the value to associate with the index.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::is_closed
Type:function (h: opaque of Broker::Store) : bool

Check if a store is closed or not.

Returns:true if the store is closed.
Broker::keys
Type:function (h: opaque of Broker::Store) : Broker::QueryResult

Returns a set with all of a store’s keys. The results reflect a snapshot in time that may diverge from reality soon afterwards. When acessing any of the element, it may no longer actually be there. The function is also expensive for large stores, as it copies the complete set.

Returns:a set with the keys. If you expect the keys to be of non-uniform type, consider using Broker::set_iterator to iterate over the result.
Broker::pop
Type:function (h: opaque of Broker::Store, k: any, e: interval &default = 0 secs &optional) : bool

Removes the last element of an existing vector.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::push
Type:function (h: opaque of Broker::Store, k: any, v: any, e: interval &default = 0 secs &optional) : bool

Appends an element to an existing vector.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
B:the value to append to the vector.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::put
Type:function (h: opaque of Broker::Store, k: any, v: any, e: interval &default = 0 secs &optional) : bool

Insert a key-value pair in to the store.

H:the handle of the store to modify.
K:the key to insert.
V:the value to insert.
E:the expiration interval of the key-value pair.
Returns:false if the store handle was not valid.
Broker::put_unique
Type:function (h: opaque of Broker::Store, k: any, v: any, e: interval &default = 0 secs &optional) : Broker::QueryResult

Insert a key-value pair in to the store, but only if the key does not already exist.

H:the handle of the store to modify.
K:the key to insert.
V:the value to insert.
E:the expiration interval of the key-value pair.
Returns:the result of the query which is a boolean data value that is true if the insertion happened, or false if it was rejected due to the key already existing.
Broker::record_assign
Type:function (r: Broker::Data, idx: count, d: any) : bool

Replace a field in a record at a particular position.

R:the record to modify.
D:the new field value to assign.
Idx:the index to replace.
Returns:false if the index was larger than any valid index, else true.
Broker::record_create
Type:function (sz: count) : Broker::Data

Create communication data of type “record”.

Sz:the number of fields in the record.
Returns:record data, with all fields uninitialized.
Broker::record_iterator
Type:function (r: Broker::Data) : opaque of Broker::RecordIterator

Create an iterator for a record. Note that this makes a copy of the record internally to ensure the iterator is always valid.

R:the record to iterate over.
Returns:an iterator.
Broker::record_iterator_last
Type:function (it: opaque of Broker::RecordIterator) : bool

Check if there are no more elements to iterate over.

It:an iterator.
Returns:true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
Broker::record_iterator_next
Type:function (it: opaque of Broker::RecordIterator) : bool

Advance an iterator.

It:an iterator.
Returns:true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
Broker::record_iterator_value
Type:function (it: opaque of Broker::RecordIterator) : Broker::Data

Retrieve the data at an iterator’s current position.

It:an iterator.
Returns:element in the collection that the iterator currently references.
Broker::record_lookup
Type:function (r: Broker::Data, idx: count) : Broker::Data

Lookup a field in a record at a particular position.

R:the record to query.
Idx:the index to lookup.
Returns:the value at the index. The optional field of the returned record may not be set if the field of the record has no value or if the index was not valid.
Broker::record_size
Type:function (r: Broker::Data) : count

Get the number of fields within a record.

R:the record to query.
Returns:the number of fields in the record.
Broker::remove_from
Type:function (h: opaque of Broker::Store, k: any, i: any, e: interval &default = 0 secs &optional) : bool

Removes an element from an existing set or table.

H:the handle of the store to modify.
K:the key whose associated value is to be modified. The key must already exist.
I:the index to remove from the set or table.
E:the new expiration interval of the modified key. If null, the current expiration time isn’t changed.
Returns:false if the store handle was not valid.
Broker::set_clear
Type:function (s: Broker::Data) : bool

Remove all elements within a set.

S:the set to clear.
Returns:always true.
Broker::set_contains
Type:function (s: Broker::Data, key: any) : bool

Check if a set contains a particular element.

S:the set to query.
Key:the element to check for existence.
Returns:true if the key exists in the set.
Broker::set_create
Type:function () : Broker::Data

Create communication data of type “set”.

Broker::set_insert
Type:function (s: Broker::Data, key: any) : bool

Insert an element into a set.

S:the set to modify.
Key:the element to insert.
Returns:true if the key was inserted, or false if it already existed.
Broker::set_iterator
Type:function (s: Broker::Data) : opaque of Broker::SetIterator

Create an iterator for a set. Note that this makes a copy of the set internally to ensure the iterator is always valid.

S:the set to iterate over.
Returns:an iterator.
Broker::set_iterator_last
Type:function (it: opaque of Broker::SetIterator) : bool

Check if there are no more elements to iterate over.

It:an iterator.
Returns:true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
Broker::set_iterator_next
Type:function (it: opaque of Broker::SetIterator) : bool

Advance an iterator.

It:an iterator.
Returns:true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
Broker::set_iterator_value
Type:function (it: opaque of Broker::SetIterator) : Broker::Data

Retrieve the data at an iterator’s current position.

It:an iterator.
Returns:element in the collection that the iterator currently references.
Broker::set_remove
Type:function (s: Broker::Data, key: any) : bool

Remove an element from a set.

S:the set to modify.
Key:the element to remove.
Returns:true if the element existed in the set and is now removed.
Broker::set_size
Type:function (s: Broker::Data) : count

Get the number of elements within a set.

S:the set to query.
Returns:the number of elements in the set.
Broker::store_name
Type:function (h: opaque of Broker::Store) : string

Get the name of a store.

Returns:the name of the store.
Broker::table_clear
Type:function (t: Broker::Data) : bool

Remove all elements within a table.

T:the table to clear.
Returns:always true.
Broker::table_contains
Type:function (t: Broker::Data, key: any) : bool

Check if a table contains a particular key.

T:the table to query.
Key:the key to check for existence.
Returns:true if the key exists in the table.
Broker::table_create
Type:function () : Broker::Data

Create communication data of type “table”.

Broker::table_insert
Type:function (t: Broker::Data, key: any, val: any) : Broker::Data

Insert a key-value pair into a table.

T:the table to modify.
Key:the key at which to insert the value.
Val:the value to insert.
Returns:true if the key-value pair was inserted, or false if the key already existed in the table.
Broker::table_iterator
Type:function (t: Broker::Data) : opaque of Broker::TableIterator

Create an iterator for a table. Note that this makes a copy of the table internally to ensure the iterator is always valid.

T:the table to iterate over.
Returns:an iterator.
Broker::table_iterator_last
Type:function (it: opaque of Broker::TableIterator) : bool

Check if there are no more elements to iterate over.

It:an iterator.
Returns:true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
Broker::table_iterator_next
Type:function (it: opaque of Broker::TableIterator) : bool

Advance an iterator.

It:an iterator.
Returns:true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
Broker::table_iterator_value
Type:function (it: opaque of Broker::TableIterator) : Broker::TableItem

Retrieve the data at an iterator’s current position.

It:an iterator.
Returns:element in the collection that the iterator currently references.
Broker::table_lookup
Type:function (t: Broker::Data, key: any) : Broker::Data

Retrieve a value from a table.

T:the table to query.
Key:the key to lookup.
Returns:the value associated with the key. If the key did not exist, then the optional field of the returned record is not set.
Broker::table_remove
Type:function (t: Broker::Data, key: any) : Broker::Data

Remove a key-value pair from a table.

T:the table to modify.
Key:the key to remove from the table.
Returns:the value associated with the key. If the key did not exist, then the optional field of the returned record is not set.
Broker::table_size
Type:function (t: Broker::Data) : count

Get the number of elements within a table.

T:the table to query.
Returns:the number of elements in the table.
Broker::vector_clear
Type:function (v: Broker::Data) : bool

Remove all elements within a vector.

V:the vector to clear.
Returns:always true.
Broker::vector_create
Type:function () : Broker::Data

Create communication data of type “vector”.

Broker::vector_insert
Type:function (v: Broker::Data, idx: count, d: any) : bool

Insert an element into a vector at a particular position, possibly displacing existing elements (insertion always grows the size of the vector by one).

V:the vector to modify.
D:the element to insert.
Idx:the index at which to insert the data. If it is greater than the current size of the vector, the element is inserted at the end.
Returns:always true.
Broker::vector_iterator
Type:function (v: Broker::Data) : opaque of Broker::VectorIterator

Create an iterator for a vector. Note that this makes a copy of the vector internally to ensure the iterator is always valid.

V:the vector to iterate over.
Returns:an iterator.
Broker::vector_iterator_last
Type:function (it: opaque of Broker::VectorIterator) : bool

Check if there are no more elements to iterate over.

It:an iterator.
Returns:true if there are no more elements to iterator over, i.e. the iterator is one-past-the-final-element.
Broker::vector_iterator_next
Type:function (it: opaque of Broker::VectorIterator) : bool

Advance an iterator.

It:an iterator.
Returns:true if the iterator, after advancing, still references an element in the collection. False if the iterator, after advancing, is one-past-the-final-element.
Broker::vector_iterator_value
Type:function (it: opaque of Broker::VectorIterator) : Broker::Data

Retrieve the data at an iterator’s current position.

It:an iterator.
Returns:element in the collection that the iterator currently references.
Broker::vector_lookup
Type:function (v: Broker::Data, idx: count) : Broker::Data

Lookup an element in a vector at a particular position.

V:the vector to query.
Idx:the index to lookup.
Returns:the value at the index. If the index was larger than any valid index, the optional field of the returned record is not set.
Broker::vector_remove
Type:function (v: Broker::Data, idx: count) : Broker::Data

Remove an element from a vector at a particular position.

V:the vector to modify.
Idx:the index to remove.
Returns:the value that was just evicted. If the index was larger than any valid index, the optional field of the returned record is not set.
Broker::vector_replace
Type:function (v: Broker::Data, idx: count, d: any) : Broker::Data

Replace an element in a vector at a particular position.

V:the vector to modify.
D:the element to insert.
Idx:the index to replace.
Returns:the value that was just evicted. If the index was larger than any valid index, the optional field of the returned record is not set.
Broker::vector_size
Type:function (v: Broker::Data) : count

Get the number of elements within a vector.

V:the vector to query.
Returns:the number of elements in the vector.
Copyright 2016, The Bro Project. Last updated on December 19, 2018. Created using Sphinx 1.8.2.