Ingest

Table of Contents

Client

class Ingest.Client

A client object representing ‘Ingest’ Service:

import almdrlib

client = almdrlib.client('ingest')

Available methods:

get_type_attributes(**kwargs)

Return the attributes of a data type.

All type attributes are static with the exception of packet and packet_versions. As such, defining <<”latest”>> as the version to fetch for packet will return the most recent version. Leaving version undefined while fetching packet will also return the same as <<”latest”>>

Request Syntax

response = client.get_type_attributes(
    attributes=[
        'None'
    ]
    type='string'
    version='string'
)
Parameters

  • attributes (list) –

    A list of type attributes to return. This option is used to return a specified set of type attributes instead of all possible type attributes.

    Valid values: name, encodings, packet, packet_versions, export

    Default: [name, encodings, packet, packet_versions, export]

    • (list) –

      A list of type attributes to return. This option is used to return a specified set of type attributes instead of all possible type attributes.

      Valid values: name, encodings, packet, packet_versions, export

      Default: [name, encodings, packet, packet_versions, export]

  • type (string) –

    [REQUIRED]

    A data type managed by the service. The currently known types are available by calling get_types API

  • version (string) –

    This is the version of the packet schema which the user would like to return. Values can be any of the form int.int.int

    Default: 1.0.0

Return type

dict

Returns

Response Syntax

{
    'doc': {
        'description': 'string',
        'fields': {},
        'name': 'string'
    },
    'encodings': [
        'string'
    ],
    'export': {
        'description': 'string',
        'properties': 'string',
        'schema': 'string',
        'title': 'string',
        'type': 'string'
    },
    'interval_size': 'integer',
    'name': 'string',
    'packet': {
        'associated': [
            {
                'fields': [
                    {}
                ],
                'key': 'string',
                'name': 'string'
            }
        ],
        'index': [
            {}
        ],
        'interval_size': 'integer',
        'name': 'string',
        'primary': {
            'fields': [
                {}
            ],
            'key': [
                'string'
            ],
            'name': 'string'
        },
        'version': 'string'
    },
    'packet_versions': [
        'string'
    ],
    'searchable': 'False|True'
}

Response Definitions

  • doc (dict) –

    • description (string) –

    • fields (dict) –

    • name (string) –

  • encodings (list) –

    • (string) –

  • export (dict) –

    • description (string) –

    • properties (string) –

    • schema (string) –

    • title (string) –

    • type (string) –

  • interval_size (integer) –

  • name (string) –

  • packet (dict) –

    • associated (list) –

      • (dict) –

        • fields (list) –

          • (dict) –

        • key (string) –

        • name (string) –

    • index (list) –

      • (dict) –

    • interval_size (integer) –

    • name (string) –

    • primary (dict) –

      • fields (list) –

        • (dict) –

      • key (list) –

        • (string) –

      • name (string) –

    • version (string) –

  • packet_versions (list) –

    • (string) –

  • searchable (boolean) –

get_types(**kwargs)

Return a list of all supported data stream types.

Request Syntax

response = client.get_types(
)
Return type

list

Returns

Response Syntax

[
    {
        'name': 'string'
    }
]

Response Definitions

  • (dict) –

    Data Type Object

    • name (string) –

      Data Type Name

send_data(**kwargs)

Send data to the Ingest Service

Request Syntax

response = client.send_data(
    account_id='string'
    chkpt='string'
    content_encoding='deflate'|'chunk'
    content_type='alertlogic.com/json'|'alertlogic.com/syslog'|'snort.org/unified2'
    data='binary|binary|list'
    environ='string'
    return_ingest_ids='False|True'
    sid='string'
    type='string'
)
Parameters

  • account_id (string) –

    [REQUIRED]

    AIMS Account ID

  • chkpt (string) – The caller may use this for tracability of the records they send.

  • content_encoding (string) –

    Use the content_encoding to leverage both chunk encoding to stream larger bodies, as well as the standard compression types when sending data type encodings that do not natuarally compress the data. This will result in a much more efficent interaction for both the client as well as the service.

    Valid values: deflate, chunk

  • content_type (string) –

    [REQUIRED]

    The content-type header should be set to the encoding format of the data being sent.

    Each data type, supports a specific set of data encodings that can be specified, and they can be retrieved via the Get Type Attributes API

    Below is a list of supported content-types:

    • alertlogic.com/json

      • Use to send auditmsgs, observation data types

    • alertlogic.com/syslog

      • Use to send syslog data

    • snort.org/unified2

      • Use to send snort unified2 data

    Valid values: alertlogic.com/json, alertlogic.com/syslog, snort.org/unified2

    Valid values: alertlogic.com/json, alertlogic.com/syslog, snort.org/unified2

    Note

    Following parameters depend on the``content_type`` value: send_data.data

  • data (list | binary | binary) –

    Below is specification of data parameter as it relates to the content_type parameter’s value

    content_type == alertlogic.com/json

    • (list) –

      Data to send to the Ingest service

      • (dict) –

    content_type == alertlogic.com/syslog

    • (binary) –

      Syslog messages separated by \n

    content_type == snort.org/unified2

    • (binary) –

      SnortUnified2 records

  • environ (string) – The deployment environment to target with the API request. This parameter only applies when the service is running in the integration accounts, and is not supported in production.

  • return_ingest_ids (boolean) –

    The caller may use this to receive the ingest_ids which is the unique identifier generated when the message is ingested. This can later be used to find the ingested message in daccess.

    This can then be decoded by first base64 decoding the message, decompress using the zlib library, and getting the unique ids through the use of the uuid library on every 16 bytes of the returned ingest_ids.

    The message can later be retrieved from daccess via the Read Messages API

    Valid values: true, false

    Default: false

  • sid (string) –

    The appliance identifier

    This query parameter is required if the source of the data type is an appliance, and not a agent running on an customer system. This appliance to data types, such as, snmsgs which are IDS events generated on the appliance itself.

  • type (string) –

    [REQUIRED]

    Ingestion Data Type Below is the mapping between type and data models:

    • ’auditmsgs’ -> AuditMsgs-0.0.2

    • ’logmsgs’ -> SyslogMsg

    • ’snmsgs’ -> SnortUnified2

    • ’observation’ -> Observation-1.0.1

Returns

None