UP | HOME

ZIO messages

Table of Contents

ZIO messages are ultimately ZeroMQ messages but overlay a schema convention for a portion of these messages. On this convention, a taxonomy of ZIO behavior is built.

1 Schema

A ZIO message is composed of several "segments" (avoiding "frame" and "part" here, see 2 below). The segments fall into one of two categories which are called:

header
leading data, follows a fixed schema determined by ZIO.
payload
trailing data in application-defined schema.

A general message is illustrated in the following table where each subtable represents one message segment.

name type bytes
magic "ZIO" 3
level "%d" 1
form (eg "JSON", "TEXT") 4
label free string N0
origin uint64 8
granule uint64 8
seqno uint64 8
payload per form N1
payload per form N2
…    

The first two segments comprise the ZIO message header. The first header segment is called the prefix header. The prefix header contains these fields:

magic
three ASCII characters: ZIO marks this as a ZIO message
level
a single digit, ASCII encoded gives the message level
form
a four byte form ID specifying the payload format, ZIO reserves a few.
label
optional, application dependent string

Up to the label (but often including it as well) the prefix header can be interpreted as an ASCII string and is suitable for use as a PUB/SUB subscription prefix (thus the segment name). Except for the restrictions above, the prefix header contents may be freely provided.

The second header segment is called the coordinate header and has three 64 bit unsigned integer fields with the following interpretations:

origin
identify the entity that produced the message.
granule
provide an ordering value for the message
seqno
index this message in a sequence of related messages.

Additional details about these header quantities as well as information on the payload segments follow.

1.1 Levels

ZIO message have a level indicated by a single ASCII digit. The general interpretation is that of an importance level where a higher value indicates greater importance and also less frequency. ASCII digits "0" through "8", inclusive, are allowed in ZIO messages. No other value for level is allowed in this version of ZIO.

The value "0" is explicitly reserved to me interpreted as undefined and used in cases where no meaningful level exists. The following interpretations are recommended for the remaining levels and they are provided in an enumeration defined in libzio.

  1. trace, messages are emitted from nested loops or occurring with frequency far higher than any "events".
  2. verbose, messages will be emitted from contexts occurring more than once per event.
  3. debug, potentially multiple messages of different types but each type occurring once per event.
  4. info, a rare message, occurring no more than once event.
  5. summary, a message holding information about a collection of events, emitted less than once per event.
  6. warning, a problem was encountered and handled.
  7. error, a problem was encountered, not handled (eg ignored) but the emitting component continues to run.
  8. fatal, an error occurred and the component gives up (eg, component throws/raises an exception or aborts shortly after).

Here an "event" is some driving impetus to the component emitting the messages and which sets some time scale. For example, a message processing component might consider an "event" to be each input message. Or, it may be interpreted as each message which satisfies some criteria.

1.2 Form

The form field provides a hint for the parsing of the payload, and possibly its further interpretation. That is, given the form, application code consuming the payload should be able to read the payload into memory in some suitable type of data structure and possibly react to its content.

Except for the forms reserved by ZIO and described in format, the application developer is free to set the form field to any four byte value. It is suggested to try to make use of the reserved formats and to not proliferate them due to conflating need for a new format hint with need for schema hints. The latter is discussed below in section 1.6. If custom forms are used, it is further recommended to use four ASCII characters to facilitate prefix matching.

1.3 Origin

In general, the interpretation of the value of the origin field is application specific and a ZIO node and its ports have support for managing it. The recommended interpretation of origin is to identify "where" a message came from.

1.4 Granule

In general, the interpretation of the value of the granule field is application specific and ZIO provides support for interpreting granule as the time at which the message was created, measured in microseconds from the Unix Expoch. The intent is for granule to provide a means to synchronize related messages from different origins.

1.5 Seqno

In general, the interpretation of the value of the seqno field is application specific. The seqno value is intended to provide an index into some logical sequence which spans messages in some set. The set may be associated with all messages from a given port, a given origin, or across some larger set of sources. In a simple case, it can count the messages from one port in order to detect message loss (eg, if PUB/SUB is used).

1.6 Payload

All segments that follow the header are called payload. Ultimately, the payload is fully in the domain of the application developer. ZIO does not dictate their use or format beyond the ZIO-reserved forms and providing the form field in the ZIO header.

The header field form is intended to provide the hint needed to at least parse the payload if not also to interpret it. The field label may be used to provide further hinting.

For example, it is expected that payload of a message with form set to JSON can be parsed as a string in JSON format. Interpreting the resulting data structure may then rely on a variety of application-dependent hints. These include

  • Existence of some predefined attributes in the JSON object,
  • Auxiliary information from the message label or
  • Header information from the peer headers from the node that sent the message.

2 Single-part vs Multi-part

ZeroMQ exposes a concept of single-part vs multi-part messages. This section describes how ZIO deals with this dichotomy.

2.1 ZeroMQ part-cardinality

In ZeroMQ, a "part" represents one atomic transfer of data between the application and the ZeroMQ socket (or vice versa) in the process of constructing a single message which will be transferred atomically between ZeroMQ sockets.

The different types of ZeroMQ sockets place different requirements on this "part-cardinality" of the messages they process. So called "thread-safe" sockets1 (CLIENT/SERVER, RADIO/DISH) require single-part messages while others (REP/REQ/ROUTER/DEALER) require multi-part and yet others (PAIR, PUB/SUB, PUSH/PULL) require neither and can process either.

Footnotes:

1

Here, "thread-safe" means the ability to use a socket from different threads over its lifetime. All ZeroMQ sockets can be used from threads but "thread-unsafe" sockets must reside in the same thread for their lifetime. Some corner-cases may work but ZIO adheres strictly to the recommendation by ZeroMQ documentation.

When ZIO message are transferred via ZIO ports the socket type is taken into account in order to assure proper part-cardinality. A ZIO message will be encoded into a single part prior to sending by sockets that require single-part messages and will be decoded subsequent to being received from a cohort. All other sockets will see a ZIO message in multiple parts.