moo 無 ovalid
Validating objects with moo oschema types

The most simple case is to validate a single "atomic" unit of data with no structure such as a number or a string. To keep the number of files small we bundle both the model and schema in a single file.

local moo = import "moo.jsonnet";
local as = moo.oschema.schema("ovalid.atomic");
{
    model: 42,
    target: as.number("Count", dtype="u4")
}

We tell moo where in that single file to find the model and the target schema via "data path prefixed file names":

moo validate --passfail \
    -t target:examples/ovalid/atomic.jsonnet \
    model:examples/ovalid/atomic.jsonnet

[
    true
]

If you read the --help output above you may note taht there is no "context schema" provided via the -s/--schema option. This can be avoided in this case because the target schema is provided as an object directly and because that schema depends on no other schema (it is atomic). In examples below we will show how the "context" schema becomes required.

Before increasing the complexity of the schema we can use the "atomic" example to show how moo validate can also utilize schema in JSON Schema form in addition to moo oschema form. It is convenient here to provide the JSON Schema via the moo jsonschema command but the JSON Schema could just as well be provided in some other manner. Just to show what JSON Schema gets used in validation we emit the intermediate results:

moo jsonschema target:examples/ovalid/atomic.jsonnet

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$defs": {},
    "type": "integer",
    "minimum": 0
}

More complex JSON Schema can be generated from moo oschema via this command. If the schema is compound then a target schema likely must be provided via -t/--target option. We will see more about target schema in the later moo validate examples below. Or, as always, see the help:

moo jsonschema -h

Usage: moo jsonschema [OPTIONS] OSCHEMA

  Convert from moo oschema to JSON Schema

Options:
  -o, --output FILE  Output file, default is stdout
  -t, --target TEXT  Specify target schema
  -h, --help         Show this message and exit.

In any case, here shows that the JSON Schema form also validates our simple atomic model:

moo jsonschema  target:examples/ovalid/atomic.jsonnet > atomic.json
moo validate --passfail \
    -t atomic.json \
    model:examples/ovalid/atomic.jsonnet

[
    true
]

A schema describing an atomic model is not very expressive. Within the rules of moo oschema, arbitrarily complex structure can be described. In this example we minimally extend the atomic case to include a few more atoms and a record type named Object with fields composed of these other types.

As described more in the oschema doc we typically build any moderately complex schema in the context of a "working object" (often named a "hier" as in "hierarchy of types"). This allows constructing one type with references to others via referencing features provided by the Jsonnet language. Here is a simple such schema:

local moo = import "moo.jsonnet";
local as = moo.oschema.schema("ovalid.simple");
{
    name: as.string("Name"),
    count: as.number("Count", dtype="u4"),
    real: as.number("Real", dtype="f4"),
    any: as.any("Data"),

    obj: as.record("Object", [
        as.field("rname", self.name, doc="required string"),
        as.field("rany", self.any, doc="required any"),
        as.field("oname", self.name, optional=true, doc="optional string"),
        as.field("oany", self.any, optional=true, doc="optional any"),                
        as.field("dname", self.name, default="", doc="default string"),
        ///NOTE: can not currently provide a default to an any!
        //as.field("dany", self.any, default=???, doc="default any"),
    ]),

    counts: as.sequence("Counts", self.count),
    cobj: as.record("CountsObject", [
        as.field("counts", self.counts),
    ]),

}

And here is an example of a model that matches the Object schema:

{
    rname: "required_name",
    rany: ["anything",4,"you"],
}

Let's check if it indeed matches:

moo validate \
    --passfail \
    --target Object \
    --schema examples/ovalid/simple-schema-hier.jsonnet \
    examples/ovalid/simple-model.jsonnet

[
    true
]

Here we have provided moo validate schema information in two ways that are different from the atomic example.

1. We have provided a "context schema" with the -s/--schema option.
2. We have identified the target inside this context via its type name Object

An example below shows some other ways to provide target schema.

So far, the examples are all valid and true is returned. Let's make a failure.

moo validate \
    --passfail \
    --target Count \
    --schema examples/ovalid/simple-schema-hier.jsonnet \
    examples/ovalid/simple-model.jsonnet

[
    false
]

Now a false is printed. The model is really meant to be of type Object but we validate it against type Count. We can see what the underlying JSON Schema validation engine thinks of this situation by omitting the --passfail option. Here is an example:

moo validate \
    --target Count \
    --schema examples/ovalid/simple-schema-hier.jsonnet \
    examples/ovalid/simple-model.jsonnet 2>&1 | awk /Failed/,EOF

Failed validating 'type' in schema:
    {'$defs': {},
     '$schema': 'http://json-schema.org/draft-07/schema#',
     'minimum': 0,
     'type': 'integer'}

On instance:
    {'rany': ['anything', 4, 'you'], 'rname': 'required_name'}

We use the awk bit to avoid cluttering this display with the Python traceback that precedes the more useful bits.

By default moo validate uses the jsonschema Python module to perform ovalid type validation. Optionallhy it may apply fastjsonschema like so:

moo validate \
    --target Count \
    --validator fastjsonschema \
    --schema examples/ovalid/simple-schema-hier.jsonnet \
    examples/ovalid/simple-model.jsonnet 2>&1 | grep '^fastjsonschema'

fastjsonschema.exceptions.JsonSchemaValueException: data must be integer

As can be seen, fastjsonschema provides a rather more terse explanation of validation failures.

Getting back to our "simple" schema, we identified a target schema in the above examples by providing a schema name of Object. Because as that schema was provided in a "hier" schema object form we can also give an object key:

moo validate \
    --passfail \
    --target obj \
    --schema examples/ovalid/simple-schema-hier.jsonnet \
    examples/ovalid/simple-model.jsonnet

[
    true
]

You can learn the key name by reading the Jsonnet source but it can sometimes be easier to compile the Jsonnet to JSON and examine that. We won't do that here but the command would be:

moo dump -f json examples/ovalid/simple-schema-hier.jsonnet

In may cases, the context schema is provided not as a "hier" object but as a sequence which has been topologically sorted according to type dependency information. The "simple" hier object is transformed into such a sequence with this example:

local moo = import "moo.jsonnet";
local hier = import "simple-schema-hier.jsonnet";
moo.oschema.sort_select(hier)

In order to identify a target schema in a sequence context schema one can still provide the schema name (ie Object, Count, etc) or we may identify the target in the sequence by providing an index as an integer counting in the usual "Python" way:

moo validate \
    --passfail \
    --target -2 \
    --schema examples/ovalid/simple-schema-seq.jsonnet \
    examples/ovalid/simple-model.jsonnet

[
    true
]

And, again, moo dump may provide an easy way to learn which index to supply.

So far the examples validated in the default "scalar mode" of moo validate. This mode assumes both the target schema and the model are singular, be they atomic or an aggregate. moo validate has a second mode called "sequence mode" where a pair of matched sequences of individual models and schema is assumed.

The "sequence mode" is really not much more than a glorified loop calling moo validate in "scalar mode" on each singular, matched model/schema from their individual sequences. User may implement this loop themselves by calling moo validate many times in "scalar mode". The benefit in moving this loop inside of moo validate is that the user may provide more concise and fewer files holding schema and model information. In particular, sequence mode is useful to develop unit tests which exercise various portions of a larger schema.

We now look at an example of applying sequence mode using the "simple" example. We provide a single, concise file that brings both context and a sequence of target schema together with their sequence of models:

local h = import "simple-schema-hier.jsonnet";
{
    schema: h,
    targets: ["real", h.count, "Name", "Counts", "Counts", "Counts", "Count", "Count"],
    models: [6.9, 42, "Arthur", [1,2,3], [1.1, 2.2, 3.3], ["one", "two", "three"], 1.1, "one"]
}

Note the targets attribute of the top-level object which this file produces is an array of same length as that of the models array. The validation walk down both arrays in step. The targets array holds a schema name or the key by which the schema can be found in the context schema, provided by the schema attribute.

We validate the entire sequence with this command:

moo validate \
    --passfail \
    -t targets:examples/ovalid/simple-sandm.jsonnet \
    -s schema:examples/ovalid/simple-sandm.jsonnet \
    models:examples/ovalid/simple-sandm.jsonnet

[
    true,
    true,
    true,
    true,
    false,
    false,
    false,
    false
]

Compared to examples above, we identify target and context schema and the models all as attributes of the same file. Because the data provide by -t/--target resolves to more than one target schema, moo validate enters sequence mode automatically. Sequence mode is also detected if more than one -t/--target option is given or the user can explicitly request it with the --sequence flag. Strictly speaking, this flag is only required if one processes a sequence of exactly one target.

As can be seen, half the validations failed. This is contrived and you can examine simple-sandm.jsonnet to understand why, In looking at that file, not that several ways to "spell" a target are used. As described in moo validate --help you can provide a target in at least these ways:

• the key name into a context schema object as in "real"
• directly a moo oschema or JSON Schema object as in h.count
• the schema name as in "Name"

If the context schema is provided as an array of schema objects one can specify a target as an integer index. It is even possible to specify a target as another data path prefixed file name.

A function is needed to load data files. Without going into details here is a one example. If search paths, top-level-arguments and datapath prefixes are not important, they can be omitted.

def load_file(fn, path=(), **tlas):
    dpath, filename = moo.util.unprefix(filename)
    sp = moo.util.search_path(filename, path)
    return moo.io.load(filename, sp, dpath, **tlas)

The context schema, if used, can then be loaded:

context = load_file(context_filename)

The target schema can be given as described above in many forms. They can be resolved with code like:

targets = [...] # list of targets in various forms
targets = moo.util.resolve_schema(targets, context, load_file)

Models are loaded like schema

models = ctx.obj.load(model)
# or with multiple models:
models = [ctx.obj.load(model) for model in models]

Finally, validation, assuming a matched sequence of target schema and models:

res = moo.ovalid.validate(models, targets, context, throw=False)