moo 無 oschema
Describing schema as objects

Table of Contents

Concepts

A schema defines information about the type of some value. It may describe an atomic type ("integer", "boolean") or types which are aggregates of other types ("record", "sequence"). The language forms we choose to use to provide this information defines a schema system (essentially itself a "meta schema"). moo supports a multiple schema systems but the predominant one is called oschema.

The "o" stands for "object" as oschema describes a type using an object representation. A lesser used moo schema system is fschema which as you may guess uses a functional representation.

Many schema systems will specify a persistent representation (file format) for expressing a schema. Eg, XML has XSD and JSON has JSON Schema (itself expressed as JSON). moo oschema system is defined in terms of a transient representation (structured data in memory). Thus, a variety of persistent representations may be used to provide oschema schema. Users may select file formats they know and love if they wish. Below we will describe two (Jsonnet and Python) for which moo provides direct support.

Schema classes

moo oschema is defined conceptually starting with a fixed set of schema classes that are listed below. A moo oschema type (called in some parts of moo an otype) is considered an instance of exactly one schema class. Dropping down one rung of the semantic ladder, a model (aka "value") is an instance of a type.

The moo oschema schema classes are summarized:

boolean
a type which may take value "true" or "false"
number
a numeric type of given format and size and optional numeric constraints
string
a character string type possibly matching some pattern or format
sequence
an array or vector with elements of one type
tuple
(not yet supported, but you can guess what it will be)
record
a collection of fields (named type references) directly held or indirectly held by zero or more records named as bases.
enum
a type that may take one value from a predefined, limited set of values
any
a type that may take a value of any type (eg such as void*, boost::any, nlohmann::json)
anyOf
a type that may take a value of any type in a predefined, limited set of types.
namespace
a collection of named types, (distinct from record to match eg C++/Python semantics)

Every oschema type then provides a set of attributes as determined by its schema class. Some attributes are common across all schema classes and may be required or optional:

name
(required) type name unique to the type context (see path)
schema
(required) string identifying the schema class taken from above list
doc
(optional, default empty) document string briefly describing the type
path
(required, potentially empty), ordered array of names representing the absolute context of the type (eg as a C++ namespace or Python module path).

The following sections go through this list of schema classes in more detail.

Atomic types

A type in moo schema is considered atomic if it holds no references to other types. Some details of the atomic types are provided in this section.

Boolean

An instance of the boolean schema class is a type that may hold a true or false value.

Like all schema classes, the user is free to make multiple instances of boolean. However, unlike the other schema classes, these types will not actually provide any variety of meaning. One boolean type is like any other even if they are differentiated by their type name.

Number

Instances of the moo number schema class describe numeric types in terms of representation size, format and optional constraints.

The size and format is specified as code which should be a valid numpy.dtype. For example, i8 is a 8 byte (not bit!) signed integer type and f4 is a single-precision floating point type. Numpy supports a wide variety of "spelling" of the format and size codes however moo schema is restricted to 2-character dtype codes.

The dtype is intended for codegen. For validation, a number schema instance may also supply numeric constraints with names matching those from JSON Schema numeric types. Specifically:

multipleOf
a valid value must be a multiple of the given number
exclusiveMaximum
a valid value must be strictly less than the given number
exclusiveMinimum
a valid value must be strictly more than the given number
maximum
a valid value must be less than or equal to the given number
minimum
a valid value must be more than or equal to the given number

To produce a schema that validates a number has having an integral value, specify the constraint multipleOf=1.

The JSON Schema "draft 4" interpretation of the exclusive variants is rejected. That is, all numeric constraints themselves take a numeric value and not Boolean.

Numeric constraints are given as literal numeric values and thus are subject to limitations of the language in which they are specified. In particular, Jsonnet treats all numeric values as IEEE754 64-bit floating point numbers.

String

Enum

Aggregate types

Some types will hold references to one or more other types. These are aggregate types. A type reference is represented as a fully-qualified type name (FQTN) which is formed as the dot-separated concatenation of the elements of path (if any) and the name. A FQTN shall not begin nor end with a period. As the path is absolute there is no concept of a "relative" type reference. See the section Namespace below for related concepts.

Thus, every type exists at an absolutely determined location in a name hierarchy, and it carries this location with it as path + name. Types that must reference another type do so by its path and name. Of course, to resolve a reference the referred type must be available in order to match the type reference against possible path and name. For this reason a schema is considered complete if every type referenced by any of the schema's types are also included.

Sequence

Record

The moo record schema class is analogous to Python class or C++ struct but with it we may only define data members, which in moo are called fields, and not methods.

Each field is itself a small data structure but moo does not considered it a first class "type". A field merely associates the following information in the context of the record:

name
unique identifier (native type string type)
item
reference the type of the value that the field represents.
default
optionally provide a value directly in the schema (see note)
doc
optionally provide a brief description of the field.
optional
optionally indicate if this field is optional (true value) or required (false).

When optional values are not provided, the attribute will be omitted from the resulting record structure.

A field may be given a default value expressed literally in the language used to define the schema (ie, Jsonnet or Python). This value may be used by whatever consumes the schema (eg, a template) to provide a value for the field.

When providing a default value, care must be given to assure it is in a valid form for the field type held by item. In particular, when a field is itself of a type record type, a full and matching object must be provided.

Special care is typically needed when handling a default value by the consumer of a schema due to its representation as a literal value. For example, a default value which is an empty string does not itself carry any quotes.

A record may also be constructed with an attribute called bases which provides an array of zero or more references to other record types. The fields of all record types referenced in bases should be considered held by the record itself. That is, bases provides a simple form of object inheritance.

Like all aspects of schema, it is up to a consumer of the schema to determine how to reflect this inheritance information. As examples, the ostructs.hpp.j2 template, described more below, directly reflects bases into a list of base struct's in simple C++ inheritance. Somewhat differently the onljs.hpp.j2 template which provides serialization an ostruct.hpp.j2 generated struct and nlohman::json representations interpret bases simply as providing additional fields. Thus it enacts a "duck-typed" interface between the type-free JSON object and the strongly-typed C++ struct.

Any

The any schema class provides the "type erasure" pattern. That is an any type may hold any type. It is like a void* in C or a std::any in C++. As it may represent any type it holds no type information other than name and doc.

With the ostructs.hpp.j2 and onljs.hpp.j2 templates, the any type is mapped to a nlohmann::json type. This can be used to delay serialization of specific parts of a record type until the instance can be passed into some more specifically typed C++ context.

The any type must hold a value which is itself of a type under the schema. As there is currently no support in moo to realize moo schema types in Jsonnet there is no way to provide a default value to a record with a field of type any. See the otypes document for details about realizing types and their values in Python.

xxxOf

Three similar aggregate schema classes are: anyOf, allOf and oneOf. Each type is a collection of references to other types for which

anyOf
any may apply
allOf
all apply
oneOf
exactly one may apply

As always, how they reflect depends on the consumers and not all may have useful meanings in all contexts. Generally, all three have meaningful reflections in a validation context. Indeed they are in moo in order to support JSON Schema validation. The oneOf could be considered to map to union types such as std::variant. The allOf could be considered to represent a component with all of the required interfaces. Given those two definitions, anyOf lacks an obvious use for code generation.

Type containers

Namespace

Every type is defined in a context called a path. Conceptually, a namespace is a path shared by all types "in" or "under" the namespace. moo provides Python and Jsonnet helper code which provides a namespace as a type factory constructing types in a given namespace.

Schema array

The most simple way to express a set of moo types is as a schema array. This is simply a Jsonnet or JSON array or Python list holding moo type structures. Generally, schema arrays must be topologically sorted so that no type structure references any other type later than itself in the array. moo provides Python and Jsonnet code to perform this topological sort and examples are given below.

Schema Examples

This section describes how to define a schema using either the Jsonnet or the Python programming language as a persistent representation. It also describes how to convert moo schema (from any format) into other schema systems in particular JSON Schema.

To illustrate some of the patterns seen in real, large-scale projects the example will factor the schema into two parts:

sys
a set of types relevant to some system (eg a framework)
app
a set of types relevant to an application based on the system

Jsonnet

moo oschema may be described easily in the Jsonnet language as Jsonnet was designed for defining data structures.

We start by simply presenting the sys schema: 

// examples/oschema/sys.jsonnet
local moo = import "moo.jsonnet";
local sys = moo.oschema.schema("sys");
[sys.number("Count", "u4")]

Let's walk through this short example line by line: 

local moo = import "moo.jsonnet";

This shows Jsonnet's "module" system in action. A file is loaded and its contents available via the moo variable. The moo.jsonnet file holds various Jsonnet functions and data that will help build our oschema. 

local sys = moo.oschema.schema("sys");

This call of the schema() function of the moo.oschema object returns another object held by the local variable sys which will provide sort of a "schema factory" that operates "in" the type path of "sys". We see it in action: 

[sys.number("Count", "u4")]

This last line defines the data structure which is the "return" value of the entire sys.jsonnet file. That is, this file "compiles" to an array holding a single oschema type.

All moo "schema files" are expected to provide an array-of-type-objects. We will later see how the order of this array matters and how to assure it is correct. We will also later see how this array becomes just one element of a more complex structure called a model which we will form prior to applying to a template.

We can see how the sys schema expands or compiles to a JSON representation using moo: 

moo compile examples/oschema/sys.jsonnet

[
    {
        "deps": [],
        "dtype": "u4",
        "name": "Count",
        "path": [
            "sys"
        ],
        "schema": "number"
    }
]

This is then a schema with a single type called Count which is of schema class number that has numeric Numpy-style type code dtype of an unsigned integer in four bytes "u4" and is in a context path of simply ["sys"].

The final structural form of oschema (ie, the JSON above) is not something that a developer of a schema strictly needs to know. moo support for Jsonnet and Python provide native language code to assist in building a schema without exposing all the details of the type object structure. But these details shall need to be understood if any new native language support is developed.

Next, we imagine an application schema with a more rich set of types: 

// examples/oschema/app.jsonnet
local moo = import "moo.jsonnet";
local sa = import "sys.jsonnet";
local sh = moo.oschema.hier(sa);
local as = moo.oschema.schema("app");
local hier = {
    counts: as.sequence("Counts",sh.sys.Count,
                        doc="All the counts"),

    email: as.string("Email", format="email",
                    doc="Electronic mail address"),
    affil: as.any("Affiliation",
                  doc="An associated object of any type"),
    mbti: as.enum("MBTI",["introversion","extroversion",
                          "sensing","intuition",
                          "thinking","feeling",
                          "judging","perceiving"]),
    make: as.string("Make"),
    model: as.string("Model"),
    autotype: as.enum("VehicleClass", ["boring", "fun"]),
    vehicle : as.record("Vehicle", [
        as.field("make", self.make, default="Subaru"),
        as.field("model", self.model, default="WRX"),
        as.field("type", self.autotype, default="fun")]),

    person: as.record("Person", [
        as.field("email",self.email,
                 doc="E-mail address"),
        as.field("email2",self.email,
                 doc="E-mail address", default="me@example.com"),
        as.field("counts",self.counts,
                 doc="Count of some things"),
        as.field("counts2",self.counts,
                 doc="Count of some things", default=[0,1,2]),
        as.field("affil", self.affil,
                 doc="Some affiliation"),
        as.field("mbti", self.mbti,
                 doc="Personality"),
        as.field("vehicle", self.vehicle, doc="Example of nested record"),
        as.field("vehicle2", self.vehicle, default={model:"CrossTrek", type:"boring"},
                 doc="Example of nested record with default"),
        as.field("vehicle3", self.vehicle, default={model:"BRZ"},
                 doc="Example of nested record with default"),
    ], doc="Describe everything there is to know about an individual human"),
};
sa + moo.oschema.sort_select(hier, "app")

We will go through some of these lines of Jsonnet in order to explain some of the forms. Starting with the first few lines: 

local moo = import "moo.jsonnet";
local sa = import "sys.jsonnet";
local sh = moo.oschema.hier(sa);
local as = moo.oschema.schema("app");

As with sys we import the support module moo.jsonnet. We also import the sys schema that we made above. Thus, sa holds a single-element array.

Next we call the moo.oschema.hier() method on this array. This transforms the ordered array of types into an (unordered) object where each type is available by its name. We'll see sh in use next.

Finally for this preamble, we make another "schema factory" which is this time "in" the path: app. We'll use as many times to build up elements of our schema.

Next we build our sys schema and do so inside a "working object". This lets us use Jsonnet language feature to refer to one of our types in another. Let's look at the start: 

local hier = {
    counts: as.sequence("Counts",sh.sys.Count,
                        doc="All the counts"),

Here we start our object and save it in a local variable hier and give it an initial entry. The key name counts is a temporary convenience and the value is what will ultimately matter. The type we make is a sequence that references the sys.Count type made in the sys schema. This shows how "cross schema" references can be made.

The next set of types are nothing special but illustrate how instances of some of the different schema classes in Jsonnet are made:

Next, let's jump to the definition of the Person type which begins with: 

email: as.string("Email", format="email",
                doc="Electronic mail address"),
affil: as.any("Affiliation",
              doc="An associated object of any type"),
mbti: as.enum("MBTI",["introversion","extroversion",
                      "sensing","intuition",
                      "thinking","feeling",
                      "judging","perceiving"]),

Now we define an instance of the record schema class: 

make: as.string("Make"),
model: as.string("Model"),
autotype: as.enum("VehicleClass", ["boring", "fun"]),
vehicle : as.record("Vehicle", [
    as.field("make", self.make, default="Subaru"),
    as.field("model", self.model, default="WRX"),
    as.field("type", self.autotype, default="fun")]),

In addition to showing how an instance of a record schema class this example shows how to reference types within our "working object" through a self object. We will skip the res of our "working object" other than to say that this Vehicle type is referenced in the Person type described in the remaining and that this shows an example of "nested" records.

We end the app schema file by producing a "sorted" array of types: 

sa + moo.oschema.sort_select(hier, "app")

Like any oschema, this array must include all of the types needed to satisfy any type references and in an order such that any referenced types come first. That is, a topological sort must be applied to the graph built between types and their references. This is not so trivial of an operation, but the moo Jsonnet support provides the required algorithm. As the sys schema is simple and independent from app by construction we merely prepend it.

Sparing the long output here, the full schema compiled to JSON can be produced with this command: 

moo compile examples/oschema/app.jsonnet

While developing a schema, it is very useful to run moo compile frequently in order to check for Jsonnet syntax or logic errors.

Python

moo provides support for defining oschema in Python which has some similarities to its Jsonnet support. However, Python is a far more expressive language than Jsonnet and thus moo Python support provides more options to the developer.

Like the Jsonnet support there are layers of representation of schema information, transformations between the layers. These are summarized:

Object representation
the moo.oschema module provides a set of Python classes, each associated with one oschema class and object instances of which represent types.
POD representation
the plain-old-data representation corresponds closely to JSON. In fact, one may copy-paste an oschema type in JSON representation into a Python file and it works as POD. POD and Object representations can be inter-transformed.

Like with the Jsonnet "schema factory" object, a moo.oschema.Namespace may be constructed with a path and then be used to construct type objects which are "in" that path.

Python is far more expressive than Jsonnet and that leaves the developer many choices how to work with oschema representations in Python. The example presented here make some reasonable choices that systems may adopt but other approaches are certainly possible.

In general we will make files which are analogous to the Jsonnet files but which may imported as Python modules. We take the convention that a .schema module variable will hold the array of types. These types will be in Object form.

Starting with the simple sys schema we can make something like: 

#!/usr/bin/env python3
import moo.oschema as mo
ns = mo.Namespace("sys")
schema = [ns.Number("Count", "u4")]

We name the module systypes to not conflict with Python's sys module.

The app schema is structured similarly, if a fair bit longer. 

import moo.oschema as mo
import systypes

ss = {typ.name.lower(): typ for typ in systypes.schema}

ns = mo.Namespace("app")

counts = ns.sequence("Counts", ss['count'])
email = ns.string("Email", format="email",
                  doc="Electronic mail address")
affil = ns.any("Affiliation",
               doc="An associated object of any type")
mbti = ns.enum("MBTI", ["introversion", "extroversion",
                        "sensing", "intuition",
                        "thinking", "feeling",
                        "judging", "perceiving"])

make = ns.string("Make")
model = ns.string("Model")
autotype = ns.enum("VehicleClass", ["boring", "fun"])
vehicle = ns.record("Vehicle", [
    ns.field("make", make, default="Subaru"),
    ns.field("model", model, default="WRX"),
    ns.field("type", autotype, default="fun")])


person = ns.record("Person", [
    ns.field("email", email, doc="E-mail address"),
    ns.field("email2", email, doc="E-mail address", default="me@example.com"),
    ns.field("counts", counts, doc="Count of some things"),
    ns.field("counts2", counts, doc="Count of some things", default=[0, 1, 2]),
    ns.field("affil", affil, doc="Some affiliation"),
    ns.field("mbti", mbti, doc="Personality"),
    ns.field("vehicle", vehicle, doc="Example of nested record"),
    ns.field("vehicle2", vehicle, default=dict(model="CrossTrek", type="boring"),
             doc="Example of nested record with default"),
    ns.field("vehicle2", vehicle, default=dict(model="BRZ"),
             doc="Example of nested record with default"),
], doc="Describe everything there is to know about an individual human")


schema = systypes.schema + mo.depsort({k: v for k, v in globals().items() if isinstance(v, mo.BaseType)})

Comparing this to the app.jsonnet above, one can see it is a near transliteration of syntax and so we will not dwell on the details. But, one thing to call out is that like with app.jsonnet we must prepend the sys schema array to our result and perform a topological sort on the types we make here. The sort is provided by moo.oschema.depsort and we play a bit of a Python trick to collect all the oschema type objects made in the module using a filter on globals().

TODO call from moo.

  • add an import based Python loader to moo

JSON Schema

moo provides support to convert a moo oschema into JSON Schema schema form. To do this, we must specify the fully-qualified type reference, a moo schema array containing the referenced type and any others it references. An instance of a JSON Schema may also specify an unique identifier usually in the form of a URL.

On the command line this may look like: 

moo -A msa=examples/oschema/app.jsonnet \
    -A typeref=app.Person \
    compile moo2jschema.jsonnet | jq '."$ref"'                    

"#/definitions/app/Person"

Here we pipe to jq just to filter down the rather verbose result and show the command works.

Models

The main use of moo is to apply a data structure ("model") to a template in order to generate a file (eg, a C++ header file). The template must have an understanding ("contract") of the structure of the model. The "oschema" structure described here is likely not enough information, or not in a convenient form, for templates to be easily defined.

We must then have means to transform and possibly augment the initial data structure into a model expected by the template and moo supports several strategies to supply that.

DIY

In some cases, transformation and augmentation can may be done at the input data structure level (ie, in Jsonnet). moo "supports" this in general by not restricting the structure of the input data. Users are free to come up with their own solutions. Typically this requires accepting a fluid contract between models and templates as one iterates both.

Jsonnet

In the case of using Jsonnet to describe the input data structure, the moo CLI supports the passing "top level arguments" (TLA) to the Jsonnet code. This requires the Jsonnet to evaluates to a top level function.

This simple example shows how TLAs work: 

function(arg="", def="default") {
    arg: arg, def:def,
}

moo -A arg="hi" compile examples/oschema/tla.jsonnet

{
    "arg": "hi",
    "def": "default"
}

As shown, multiple TLAs may be used and default TLA values may be given in the Jsonnet and omitted on the CLI. A TLA may also be specified as a Jsonnet file in which case the contents of that file will be evaluated and the resulting structure passed to the top level function. Reusing the above example and the sys schema file: 

moo -A arg=examples/oschema/sys.jsonnet compile examples/oschema/tla.jsonnet

{
    "arg": [
        {
            "deps": [],
            "dtype": "u4",
            "name": "Count",
            "path": [
                "sys"
            ],
            "schema": "number"
        }
    ],
    "def": "default"
}

Thus, with TLAs one may construct a somewhat general Jsonnet file that transforms and augments some initial data structure to something more specific.

TLAs have one more useful trick. The function to which TLAs are given is just a normal Jsonnet function. This gives us the option to "bake in" some TLAs in a Jsonnet file that calls the original function. Consider: 

local sys = import "sys.jsonnet";
local tla = import "tla.jsonnet";
tla(sys)

Thus we may get the same output with a simpler command: 

moo compile examples/oschema/tla-sys.jsonnet

{
    "arg": [
        {
            "deps": [],
            "dtype": "u4",
            "name": "Count",
            "path": [
                "sys"
            ],
            "schema": "number"
        }
    ],
    "def": "default"
}

This pattern of "baking in" of TLAs can be useful, for example, if one has a codegen system where a package must supply the specific information but otherwise relies on a common model. By "hiding" the TLAs to that model in a Jsonnet file, the build system layer can be made simpler. See build sys document for some pointers on integrating moo into popular build systems.

Python

Jsonnet is a "small" language (one of its charms) and some model transformations may be complex enough that its simplicity poses a limitation. moo thus allows transformations to be defined in Python and this opens up the ability to form the model in a more strong and object-oriented manner.

To do this, the user may tell the moo CLI to apply a Python function to transform the input data just prior to the application of the template. The function is specified as a "dot" path naming the function in its module. We may use the moo dump command to illustrate a transformation applied to the app schema: 

moo -M examples/oschema -t moo.oschema.typify dump -f pretty app.jsonnet

[<Number "sys.Count">,
 <Any "app.Affiliation">,
 <Sequence "app.Counts" items:sys.Count>,
 <String "app.Email">,
 <Enum "app.MBTI">,
 <String "app.Make">,
 <String "app.Model">,
 <Enum "app.VehicleClass">,
 <Record "app.Vehicle" fields:{make, model, type}>,
 <Record "app.Person" fields:{email, email2, counts, counts2, affil, mbti, vehicle, vehicle2, vehicle3}>]

The built-in moo.oschema.typify() function illustrated here converts a schema type as a "raw" data structure into a corresponding Python object. In a template, the Python object should be usable everywhere the "raw" data structure. The typify() transform is thus only useful if the extensions that the Python object provides are needed in a template or if subsequent transforms require objects instead of raw data structures (an example of which we'll see next).

We may also pipeline transformations. Here is an example that will use the output of typify(), form a graph from the type reference dependencies, and perform a topological sort to produce an array which are ordered from least dependent to most. 

moo -T examples/oschema -M examples/oschema \
    -t 'moo.oschema.typify|moo.oschema.graph|moo.oschema.depsort' \
      render app.jsonnet ool.txt.j2

Iterate over list of types:
<Number "sys.Count">
<Any "app.Affiliation">
<Sequence "app.Counts" items:sys.Count>
<String "app.Email">
<Enum "app.MBTI">
<String "app.Make">
<String "app.Model">
<Enum "app.VehicleClass">
<Record "app.Vehicle" fields:{make, model, type}>
<Record "app.Person" fields:{email, email2, counts, counts2, affil, mbti, vehicle, vehicle2, vehicle3}>

Note the Person type comes after the types that it refers to in its fields due to the topological sort. Also, note that this particular transform may also be performed in the Jsonnet layer and so is used here as an illustration of the functionality.

This example also shows that the pipeline of transformations may becomes rather complex. At some point, developing a composite transformation function in Python and referring to it on the moo CLI may be useful to keep the command argument list small.

But, let us now move on to codegen.

Codegen

Some trivial templates were introduced above in order to dump out some of the information in their models. Here we develop two "real" templates and apply them to the app schema to generate code.

ostructs.hpp.j2
generate a C++ header a defining a C++ namespace scope and holding definition of a C++ struct type for each type instance of the schema class record with a path in that scope.
onljs.hpp.j2
for each C++ struct defined above, produce functions that will allow the struct to participate in nlohmann::json (nljs) based serialization.

But, before developing the templates we first define a contract or model on which the template development may depend.

Model

The omodel contract is embodied in this Jsonnet file: 

local oschema = import "oschema.jsonnet";

function(os, path=[], ctxpath=[]) {

    // The "path" determines which schema types in the "os" array will
    // be considered to be directly "in" the model.  The "path" may be
    // give either as a literal list of string or encoded as a
    // dot-separated string.  It is used, eg, to form a surrounding
    // C++ namespace containing the codegen'ed types.
    path: oschema.listify(path),

    // The "context path" is a prefix of the "path" to be removed when
    // refering to this model from external resources and in a
    // relative way.  Eg, the ctxpath is removed from the path when
    // forming relative #include statements between "sibling" headers
    // generated from this model.  It may either be a litteral list of
    // strings or a list of string encoded as a dot-separated string.
    ctxpath: oschema.listify(ctxpath),

    // Select out the types which are "in" the path for consideration.
    types: [t for t in os if oschema.isin(self.path, t.path)],

    // Also provide the super set of all types so that referenced
    // types may be resolved.  This super set should be complete to
    // any type referenced by a type in the "types" array above.
    all_types: os,

    // Reference any type by its FQN.
    byref: {[oschema.fqn(t)]:t for t in $.all_types},

    // Collect the types of interest by their schema class name
    byscn: {[tn]:[oschema.fqn(t) for t in $.types if t.schema == tn]
                   for tn in oschema.class_names},

    // Find external type references
    local extypref = [
        d for d in std.flattenArrays([t.deps for t in $.types])
          if !oschema.isin($.path,oschema.listify(d))],
    extrefs: std.uniq(std.sort([oschema.relpath(oschema.basepath(t), self.ctxpath) for t in extypref]))
}

The top-level arguments are described below. What is produced is an object (ie, an instance of the model) with these attributes:

path
the namespace path scope to focus on as a list/array
nspre
namespace prefix with trailing dot
types
array of type data structures which are in scope
byref
full type information retrieved via a type reference
byscn
references to types collected by schema class
extref
list of references to types outside the scope

Top-level arguments

os
bring in the oschema array
path
the namespace path with which select a branch on the full schema tree

We can test out some TLAs and test that the model compiles using the moo CLI: 

moo -M examples/oschema \
  -A os='app.jsonnet' -A path='app' \
     compile omodel.jsonnet

We want to apply a transform to the types attribute and can test that with. This can be done by with this command. We will hold off on showing the output until the next example CLI. 

moo -M examples/oschema \
  -A os='app.jsonnet' -A path='app' \
  -t '/types:moo.oschema.typify|moo.oschema.graph|moo.oschema.depsort' \
     dump -f pretty omodel.jsonnet

We are almost ready to turn to the template but one last detail is needed. As we will find there are some utilities that will simplify developing the template and which are specific to the target-language (eg C++) and which the rest of the model does not depend. We will bring these in as a model "graft". 

moo -g '/lang:ocpp.jsonnet' \
    -M examples/oschema \
    -A os='app.jsonnet' -A path='app' \
    -t '/types:moo.oschema.typify|moo.oschema.graph|moo.oschema.depsort' \
    dump -f types omodel.jsonnet

all_types <class 'list'>
byref <class 'dict'>
byscn <class 'dict'>
ctxpath <class 'list'>
extrefs <class 'list'>
path <class 'list'>
types <class 'list'>
lang <class 'dict'>

If you squint you'll see the lang attribute added with others from the model. Let's now move to the template.

Template

The ostructs.hpp.j2 template file gets applied to the omodel to produce a C++ header file defining a struct for each record instance in the model and any supporting types via a using type alias. It also uses the extref info to #include any required external headers that themselves are also generated from other parts of the overall schema.

Take particular note that this #include pattern bakes in a specific mapping from a type's path array to file locations. For the resulting C++ code to compile, this pattern must of course actually be honored in some way. This may be done manually by properly placing the generated files according to this mapping or, better, be automatically assured via a build system. Future work may generate this file-system level assurance itself from schema. For now, we must simply be careful.

Finally, note that the grafting of ocpp.jsonnet selects a particular mapping from schema class names to their C++ equivalents. Eg, nlohman::json for any. If we wished to generate code using a different mapping, such as boost::any for any we would need to modify or fork this grafted data structure while the rest of the structure may be left as-is.

We can finally generate code by changing the above CLI call from dump to render and adding the template file name. 

moo -g '/lang:ocpp.jsonnet' \
    -M examples/oschema \
    -A os='app.jsonnet' -A path='app' \
    render omodel.jsonnet ostructs.hpp.j2

/*
 * This file is 100% generated.  Any manual edits will likely be lost.
 *
 * This contains struct and other type definitions for shema in 
 * namespace app.
 */
#ifndef APP_STRUCTS_HPP
#define APP_STRUCTS_HPP

#include <cstdint>
#include "sys/Structs.hpp"

#include <nlohmann/json.hpp>
#include <string>
#include <vector>
#include <string>

namespace app {

    // @brief An associated object of any type
    using Affiliation = nlohmann::json;

    // @brief All the counts
    using Counts = std::vector<sys::Count>;

    // @brief Electronic mail address
    using Email = std::string;

    // @brief 
    enum class MBTI: unsigned {
        introversion,
        extroversion,
        sensing,
        intuition,
        thinking,
        feeling,
        judging,
        perceiving,
    };
    // return a string representation of a MBTI.
    inline
    const char* str(MBTI val) {
        if (val == MBTI::introversion) { return "introversion" ;}
        if (val == MBTI::extroversion) { return "extroversion" ;}
        if (val == MBTI::sensing) { return "sensing" ;}
        if (val == MBTI::intuition) { return "intuition" ;}
        if (val == MBTI::thinking) { return "thinking" ;}
        if (val == MBTI::feeling) { return "feeling" ;}
        if (val == MBTI::judging) { return "judging" ;}
        if (val == MBTI::perceiving) { return "perceiving" ;}
        return "";                  // should not reach
    }
    inline
    MBTI parse_MBTI(std::string val, MBTI def = MBTI::introversion) {
        if (val == "introversion") { return MBTI::introversion; }
        if (val == "extroversion") { return MBTI::extroversion; }
        if (val == "sensing") { return MBTI::sensing; }
        if (val == "intuition") { return MBTI::intuition; }
        if (val == "thinking") { return MBTI::thinking; }
        if (val == "feeling") { return MBTI::feeling; }
        if (val == "judging") { return MBTI::judging; }
        if (val == "perceiving") { return MBTI::perceiving; }
        return def;
    }

    // @brief 
    using Make = std::string;

    // @brief 
    using Model = std::string;

    // @brief 
    enum class VehicleClass: unsigned {
        boring,
        fun,
    };
    // return a string representation of a VehicleClass.
    inline
    const char* str(VehicleClass val) {
        if (val == VehicleClass::boring) { return "boring" ;}
        if (val == VehicleClass::fun) { return "fun" ;}
        return "";                  // should not reach
    }
    inline
    VehicleClass parse_VehicleClass(std::string val, VehicleClass def = VehicleClass::boring) {
        if (val == "boring") { return VehicleClass::boring; }
        if (val == "fun") { return VehicleClass::fun; }
        return def;
    }

    // @brief 
    struct Vehicle {

        // @brief 
        Make make = "Subaru";

        // @brief 
        Model model = "WRX";

        // @brief 
        VehicleClass type = app::VehicleClass::fun;
    };

    // @brief Describe everything there is to know about an individual human
    struct Person {

        // @brief E-mail address
        Email email = "";

        // @brief E-mail address
        Email email2 = "me@example.com";

        // @brief Count of some things
        Counts counts = {};

        // @brief Count of some things
        Counts counts2 = {0, 1, 2};

        // @brief Some affiliation
        Affiliation affil = {};

        // @brief Personality
        MBTI mbti = app::MBTI::introversion;

        // @brief Example of nested record
        Vehicle vehicle = {"Subaru", "WRX", app::VehicleClass::fun};

        // @brief Example of nested record with default
        Vehicle vehicle2 = {"Subaru", "CrossTrek", app::VehicleClass::boring};

        // @brief Example of nested record with default
        Vehicle vehicle3 = {"Subaru", "BRZ", app::VehicleClass::fun};
    };

} // namespace app

#endif // APP_STRUCTS_HPP

And, here are the corresponding nlohmann::json serialization functions, produced by applying the onljs.hpp.j2 template to the same model. 

moo -g '/lang:ocpp.jsonnet' \
    -M examples/oschema \
    -A os='app.jsonnet' -A path='app' \
    render omodel.jsonnet onljs.hpp.j2

/*
 * This file is 100% generated.  Any manual edits will likely be lost.
 *
 * This contains functions struct and other type definitions for shema in 
 * namespace app to be serialized via nlohmann::json.
 */
#ifndef APP_NLJS_HPP
#define APP_NLJS_HPP

// My structs
#include "app/Structs.hpp"

// Nljs for externally referenced schema
#include "sys/Nljs.hpp"

#include <nlohmann/json.hpp>

namespace app {

    using data_t = nlohmann::json;    NLOHMANN_JSON_SERIALIZE_ENUM( MBTI, {
            { app::MBTI::introversion, "introversion" },
            { app::MBTI::extroversion, "extroversion" },
            { app::MBTI::sensing, "sensing" },
            { app::MBTI::intuition, "intuition" },
            { app::MBTI::thinking, "thinking" },
            { app::MBTI::feeling, "feeling" },
            { app::MBTI::judging, "judging" },
            { app::MBTI::perceiving, "perceiving" },
        })
    NLOHMANN_JSON_SERIALIZE_ENUM( VehicleClass, {
            { app::VehicleClass::boring, "boring" },
            { app::VehicleClass::fun, "fun" },
        })


    inline void to_json(data_t& j, const Vehicle& obj) {
        j["make"] = obj.make;
        j["model"] = obj.model;
        j["type"] = obj.type;
    }

    inline void from_json(const data_t& j, Vehicle& obj) {
        if (j.contains("make"))
            j.at("make").get_to(obj.make);    
        if (j.contains("model"))
            j.at("model").get_to(obj.model);    
        if (j.contains("type"))
            j.at("type").get_to(obj.type);    
    }

    inline void to_json(data_t& j, const Person& obj) {
        j["email"] = obj.email;
        j["email2"] = obj.email2;
        j["counts"] = obj.counts;
        j["counts2"] = obj.counts2;
        j["affil"] = obj.affil;
        j["mbti"] = obj.mbti;
        j["vehicle"] = obj.vehicle;
        j["vehicle2"] = obj.vehicle2;
        j["vehicle3"] = obj.vehicle3;
    }

    inline void from_json(const data_t& j, Person& obj) {
        if (j.contains("email"))
            j.at("email").get_to(obj.email);    
        if (j.contains("email2"))
            j.at("email2").get_to(obj.email2);    
        if (j.contains("counts"))
            j.at("counts").get_to(obj.counts);    
        if (j.contains("counts2"))
            j.at("counts2").get_to(obj.counts2);    
        obj.affil = j.at("affil");
        if (j.contains("mbti"))
            j.at("mbti").get_to(obj.mbti);    
        if (j.contains("vehicle"))
            j.at("vehicle").get_to(obj.vehicle);    
        if (j.contains("vehicle2"))
            j.at("vehicle2").get_to(obj.vehicle2);    
        if (j.contains("vehicle3"))
            j.at("vehicle3").get_to(obj.vehicle3);    
    }

} // namespace app

#endif // APP_NLJS_HPP

Objects

Besides generating code from a schema, data objects may be constructed with the help of and validated against schema. For information about this usage pattern see the otypes document which describes how to use the moo.otypes module.

Author: Brett Viren

Created: 2021-02-24 Wed 15:25

Validate