DUNE DAQ REPL

Introduction
REPL with the File Command Facility
- Shell REPL
- Python REPL
REPL with the REST Command Facility

Introduction

This document describes some approaches to interactively feeding commands to appfwk applications via various types of read-eval-print loop (REPL) and with different types of command transport. See also how to create commands.

REPL with the File Command Facility

The appfwk has a "command facility called the File Command Facility (FCF) (actually provided by cmdlib). It is what is used when the application is called like:

$ daq_application --commandFacility file:///path/to/commands.json
## or
$ daq_application --commandFacility file:commands.json

You may actually omit the file: scheme for relative files or the file:// for absolute paths.

You may also give the file format explicitly instead of relying on the file extension, eg:

$ daq_application --commandFacility file:commands.json?fmt=jstream

When given a file of commands, the FCF will normally "play" those commands as fast as possible (though has a batch time delay feature). To provide some interactive pacing of commands we may use a REPL.

Shell REPL

The simplest REPL is no REPL. Or, rather using simply your Unix shell and some basic Unix tools like cat. To use this type of REPL we first create a "named pipe" aka a FIFO file. The dap_application will read from this pipe and we will write to this pipe from our shell. Making a FIFO file is simple:

$ mkfifo foo.fifo

Next, start the DAQ app and tell it to read from the FIFO file. Like before you can tell it what format to assume with a URL query parameter fmt. For REPL we should use a JSON Stream:

$ daq_application --commandFacility 'file:foo.fifo?fmt=jstream'

One could just as well do mkfifo foo.jstream and use that. The example here shows how format may be set independently from file name extension.

Now, prepare some commands. Let's assume we have one command object per JSON file. Then, in another shell play one or more of them:

$ cat cmd1.json > foo.fifo
$ cat cmd2.json > foo.fifo

If you have a JSON Array file you can also let jq select individual command elements of a command sequence:

$ cat fdpc-job.json |jq '.[0]' > foo.fifo
$ cat fdpc-job.json |jq '.[1]' > foo.fifo

When done, you may wish to manually clean up the FIFO file:

$ rm foo.fifo

Python REPL

A second approach to using a REPL with the File Command Facility is implemented in Python by the ddrepl Python module. This approach integrates well with making your command objects in Python as described in DUNE DAQ Command Object Creation document.

Under the hood, this method works much the same way as the shell REPL above. Python tooling is provided to:

make the FIFO file
start the daq_application as a Python subprocess
provide an interface to send individual commands through the FIFO
cleanup FIFO file

Here is an example as of Python you may put in a script or interactively type in a Python REPL such as ipython or ptipython. The value 0.1 given to the da.output() is in seconds and represents how long to wait (block) while receiving any stdout back from daq_application.

import json
import ddrepl
cmds = json.load(open("fdpc-job.json"))
da = ddrepl.DaqApp()
for cmd in cmds:
    da(cmd)
    print(da.output(0.1))
del(da)

using fifo: /tmp/tmpk82viwoc/commands.jstream
ready to send
2021-Jan-08 16:46:33,260 INFO [fileCommandFacility::fileCommandFacility(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:236] uri: file:///tmp/tmpk82viwoc/commands.jstream
2021-Jan-08 16:46:33,261 INFO [fileCommandFacility::fileCommandFacility(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:242] url: scheme:file path:/tmp/tmpk82viwoc/commands.jstream
2021-Jan-08 16:46:33,261 INFO [fileCommandFacility::fileCommandFacility(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:269] open: file:///tmp/tmpk82viwoc/commands.jstream as jstream
2021-Jan-08 16:46:33,262 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:301] {"data":{"modules":[{"data":{"qinfos":[{"dir":"output","inst":"hose","name":"output"}]},"inst":"fdp","plugin":"FakeDataProducerDAQModule"},{"data":{"qinfos":[{"dir":"input","inst":"hose","name":"input"}]},"inst":"fdc","plugin":"FakeDataConsumerDAQModule"}],"queues":[{"capacity":10,"inst":"hose","kind":"StdDeQueue"}]},"id":"init"}
2021-Jan-08 16:46:33,262 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:303] DAQModuleManager execution complete
2021-Jan-08 16:46:33,271 INFO [dunedaq::appfwk::DAQModuleManager::execute(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:113] Command id:init
2021-Jan-08 16:46:33,272 INFO [dunedaq::appfwk::DAQModuleManager::init_queues(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:73] Adding queue: hose
2021-Jan-08 16:46:33,272 INFO [dunedaq::appfwk::DAQModuleManager::init_modules(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:36] construct: FakeDataProducerDAQModule : fdp

2021-Jan-08 16:46:33,372 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:301] {"data":{"modules":[{"data":{"ending_int":14,"nIntsPerVector":10,"queue_timeout_ms":100,"starting_int":-4,"wait_between_sends_ms":1000},"match":"fdp"},{"data":{"ending_int":14,"nIntsPerVector":10,"queue_timeout_ms":100,"starting_int":-4},"match":"fdc"}]},"id":"conf"}
2021-Jan-08 16:46:33,372 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:303] DAQModuleManager execution complete
2021-Jan-08 16:46:33,376 INFO [dunedaq::appfwk::DAQModuleManager::execute(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:113] Command id:conf

2021-Jan-08 16:46:33,476 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:301] {"data":{"modules":[{"data":{"run":42},"match":""}]},"id":"start"}
2021-Jan-08 16:46:33,476 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:303] DAQModuleManager execution complete
2021-Jan-08 16:46:33,477 INFO [dunedaq::appfwk::DAQModuleManager::execute(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:113] Command id:start

2021-Jan-08 16:46:33,577 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:301] {"data":{"modules":[{"data":{},"match":"fdp"},{"data":{},"match":"fdc"}]},"id":"stop"}
2021-Jan-08 16:46:33,577 INFO [fileCommandFacility::run(...) at /home/bv/dev/dune-daq/sv/filecmd/plugins/fileCommandFacility.cpp:303] DAQModuleManager execution complete
2021-Jan-08 16:46:33,578 INFO [dunedaq::appfwk::DAQModuleManager::execute(...) at /home/bv/dev/dune-daq/sv/appfwk/src/DAQModuleManager.cpp:113] Command id:stop

REPL with the REST Command Facility

The above approach requires the DAQ application and the source of commands to reside on the same host. Ie, both ends must access the FIFO file.

Another approach exploits the REST command facility (RCF) which provides an HTTP server running in the DAQ application. The ddrepl provides a simple client to send the server commands. It requires the daq_application to have already been started like:

$ daq_application --commandFacility rest://localhost:12345

The ddrepl Python interface is then almost identical to the local file-based one. After starting as above, one may run:

import json
import ddrepl
cmds = json.load(open("fdpc-job.json"))
da = ddrepl.DaqRestClient()
for cmd in cmds:
    res = da(cmd)
    print(res.content.decode())

Command received

Command received

Command received

Command received

You may see the usual logging show up in the terminal running daq_application.

The ddrepl.DaqRestClient() takes two options:

In [2]: ddrepl.DaqRestClient?
Init signature:
ddrepl.DaqRestClient(
    url='http://localhost:12345/command',
    answer_port=12333,
)
Docstring:      Access app REST API