DUNE DAQ Command Object Creation

The simplest schema class is boolean. The appfwk schema currently includes no examples. We can however, create one from whole cloth by dropping down to moo.

import moo
moo.otypes.make_type(schema="boolean", name="MyBool", doc="Example Boolean", path="test.junk")
from test.junk import MyBool
MyBool(True)

<boolean MyBool: True>

And, let the REPL show some docs:

In [17]: MyBool?
Init signature: MyBool(val)
Docstring:     
A MyBool boolean
Example Boolean

Every type of appfwk queue has a "capacity" which we will provide from instances of the type QueueCapacity which is of the schema class number.

In [12]: cmd.QueueCapacity?
Init signature: cmd.QueueCapacity(val)
Docstring:      A number type QueueCapacity

We saw an example of a string schema class type CmdId above in the section Interrogating types so we will not repeat that here.

An enum is a string-like type that may take on a value from a predetermined set. The defining schema may specify a default value from this set or if not given then the first value is taken as default.

The appfwk schema uses an enum type QueueKind to identity a "kind" of queue. It's documentation:

In [21]: cmd.QueueKind?
Init signature: cmd.QueueKind(val=None)
Docstring:     
Enum type QueueKind [['Unknown', 'StdDeQueue', 'FollySPSCQueue', 'FollyMPMCQueue']]
The kinds (types/classes) of queues

We can see the default value in action with a default construction:

cmd.QueueKind()

<enum QueueKind: 'Unknown' of ['Unknown', 'StdDeQueue', 'FollySPSCQueue', 'FollyMPMCQueue']>

Let's try to break it:

In [25]: cmd.QueueKind("MagicQueue")
...
ValueError: illegal enum QueueKind value MagicQueue

The Python type which corresponds to a moo oschema class record has some extra functionality to allow instances to mimic Python data objects while assuring type constraints.

One simple record in the appfwk command schema is QueueSpec. Let's let the REPL tell us about it:

In [19]: cmd.QueueSpec?
Init signature:
cmd.QueueSpec(
    *args,
    kind: dunedaq.appfwk.cmd.QueueKind = 'Unknown',
    inst: dunedaq.appfwk.cmd.InstName = None,
    capacity: dunedaq.appfwk.cmd.QueueCapacity = None,
)
Docstring:     
Record type QueueSpec with fields: "kind", "inst", "capacity"

Queue specification

We see that kind is a QueueKind, introduced previously, and that the default value for this enum has been forwarded to be a default value for the field. This is a special handling for enum fields. The schema could have set a field default different than the enum default.

The appfwk schema happens to not provide defaults for the other two fields: inst of a string type InstName and capacity a number type and also introduced previously. We must provide at least these latter two at some point to have a fully constructed instance of the type QueueSpec. Let's start by making an instance with just the capacity record field given.

qs = cmd.QueueSpec(capacity=10)
(qs, qs.kind, qs.capacity)

(<record QueueSpec, fields: {kind, inst, capacity}>, 'Unknown', 10)

If we try to get the qs.inst attribute or call qs.pod() (which will try to access all required attributes) at this time we will get an error as the record is not yet complete.

print(qs.inst)
...
AttributeError: no such attribute inst

Let's set that last attribute now:

qs.inst = "hose"
qs.pod()

{'kind': 'Unknown', 'inst': 'hose', 'capacity': 10}

Records follow Postel's law by being "generous in what they accept and strict in what they produce". This allows us to delay filling in all the required fields and relying on defaults where desired. As we saw, strictness comes in when accessing the values of the record type instance.

However, when explicitly setting a value, the record type instance will be strict so that no "illegal" values will be accepted. The type InstName can not be just any string but must be short identifier such as might be a valid variable name in a C-like syntax. Let's try to break it:

In [30]: qs.inst = "very wrong"
...
ValidationError: 'very wrong' does not match '^[a-zA-Z][a-zA-Z0-9_]*$'
...
ValueError: format mismatch for string InstName

Like with the string example above, setting the field of a record which is itself a string needs to be done properly.

A sequence schema class models an ordered array or list of elements which are themselves each of the same type. The appfwk schema has a type called QueueSpecs which, as the pluralized name implies, is a sequence of record type QueueSpec which we learned above. Our REPL tells us:

In [31]: cmd.QueueSpecs?
Init signature: cmd.QueueSpecs(val)
Docstring:     
A QueueSpecs sequence holding type dunedaq.appfwk.cmd.QueueSpec.
A sequence of QueueSpec

Let's make one with as single item of type QueueSpec.

qs = cmd.QueueSpec(capacity=10, inst="hose")
qss = cmd.QueueSpecs([qs])
(qss, qss.pod())

(<sequence QueueSpecs 1:[dunedaq.appfwk.cmd.QueueSpec]>,
[{'kind': 'Unknown', 'inst': 'hose', 'capacity': 10}])

A sequence type instance may be empty:

qss = cmd.QueueSpecs([])
(qss, qss.pod())

(<sequence QueueSpecs 0:[dunedaq.appfwk.cmd.QueueSpec]>, [])

And, later updated:

qss.update([qs])
(qss, qss.pod())

(<sequence QueueSpecs 1:[dunedaq.appfwk.cmd.QueueSpec]>,
[{'kind': 'Unknown', 'inst': 'hose', 'capacity': 10}])

An any type is a simple form of a dynamic type. It is kind of like a void* in C or a pointer to a polymorphic base class in C++ but there is a twist. Two different any types are not compatible. That is, an instance of any type Data1 can not be used to construct or update an instance of any type Data2.

Another twist is that an instance of an any type may only be constructed or updated with any other non-any "otype" (or an instance of the same any type) but not any POD. Despite that, an instance of an any type can still be turned into POD with the usual .pod() method.

In the appfwk schema, we currently have one catch-all any type called Data.

In [7]: cmd.Data?
Init signature: cmd.Data(val)
Docstring:     
The any type Data.

Can hold any oschema type except Any types that are not Data.

An opaque object holding lower layer substructure

As above, we must use some schema type instance to set an any type instance. Here we use CmdId for simplicity. In a "real" command object, the any type Data tends to hold some larger record type.

d1 = cmd.Data(cmd.CmdId("conf"))
d1.update(cmd.QueueKind("StdDeQueue"))
d1.pod()

'StdDeQueue'

Let's try to break it:

In [23]: d2 = cmd.Data("this should not work")
...
ValueError: any type Data requires oschema type

In [25]: moo.otypes.make_type(schema="any", name="Other", doc="Another any", path="test.junk")
In [26]: from test.junk import Other
In [28]: d3 = Other(d1)
...
ValueError: cross any updates not allowed

The init command provides all the information to take a fully generic appfwk application into one that has all of its queues and modules constructed. The queues are also fully configured by this init command. The modules receive notice of the init and may even require custom init information but such is best left for conf.

The substructure for Command.data for an init command is a:

In [30]: cmd.Init?
Init signature:
cmd.Init(
    *args,
    queues: dunedaq.appfwk.cmd.QueueSpecs = None,
    modules: dunedaq.appfwk.cmd.ModSpecs = None,
)
Docstring:     
Record type Init with fields: "queues", "modules"

The app-level init command data object struction

The ModSpecs are new:

In [32]: cmd.ModSpec?
Init signature:
cmd.ModSpec(
    *args,
    plugin: dunedaq.appfwk.cmd.PluginName = None,
    inst: dunedaq.appfwk.cmd.InstName = None,
    data: dunedaq.appfwk.cmd.Data = None,
)
Docstring:     
Record type ModSpec with fields: "plugin", "inst", "data"

Module specification

The simplest, and most boring, init command:

myinit = cmd.Command(id=cmd.CmdId("init"), data=cmd.Init(queues=[], modules=[]))
myinit.pod()

{'id': 'init', 'data': {'queues': [], 'modules': []}}

That code has some redundancies so we can replace it with a little helper:

import ddcmd
myinit = ddcmd.init()
myinit.pod()

{'id': 'init', 'data': {'queues': [], 'modules': []}}

As such, this init command would initialize an "empty" and rather useless app instance. The next two sections describe how to make the queue and module sequences with a working example.

Let's use the little ddcmd.init() helper again to build a full init command.

import ddcmd
myinit = ddcmd.init(queues=queues, modules=modules)
myinit.pod()

{'id': 'init',
'data': {'queues': [{'kind': 'StdDeQueue', 'inst': 'hose', 'capacity': 10}],
'modules': [{'plugin': 'FakeDataProducerDAQModule',
'inst': 'source',
'data': {'inst': 'hose', 'name': 'output', 'dir': 'output'}},
{'plugin': 'FakeDataConsumerDAQModule',
'inst': 'sink',
'data': {'inst': 'host', 'name': 'input', 'dir': 'input'}}]}}

The conf command's any typed .data holds module-level configuration (sub) objects in a .modules attribute. This attribute is of types AddressedCmds which is a sequence of AddressedCmd.

In [14]: cmd.AddressedCmd?
Init signature:
cmd.AddressedCmd(
    *args,
    match: dunedaq.appfwk.cmd.Match = None,
    data: dunedaq.appfwk.cmd.Data = None,
)
Docstring:     
Record type AddressedCmd with fields: "match", "data"

General, non-init module-level command data structure

The Match type is a string to provide a regular expression. All that match will receive the object held in the any type .data. For this conf command we will explicitly match the module names.

A skeleton for making the conf command object might look like:

cmd.AddressedCmds([
  cmd.AddressedCmd(match="source", data=...),
  cmd.AddressedCmd(match="sink", data=...)])

We now turn to making the module-specific configuration objects.