moo 無 build
Integrating moo with build systems

A reasonably simple way to "integrate" moo into a build system is to not. One may run moo manually, or partly automated via a shell script, and arrange for the generated files to be placed in the source areas and committed. Absent any apparent knowledge of moo, the software project will simply appear to have an impeccably consistent code developer.

Pros:

• requires moo only at develop time, not build (nor run) time.
• easy to implement
• (manually) assures correct build dependencies
• generated code can be placed alongside hand-made code which assists in browsing/understanding of the code base
• building the project is decoupled from development occurring in moo itself or in templates provided by moo that are used by the project

Cons:

• a developer can change a local template or model and forget to regenerate (though presumably, this is caught as a matter-of-course).

Another reasonable solution is to run moo commands as part of the native build of the package. From the point of view of developing a build system, moo is seen as a compiler or possibly more accurately a pre processor.

While g++ may convert C++ files to .o object files, moo converts .j2 and .jsonnet files into .hpp, .cpp, etc files (which then themselves must go on to be further compiled. Thus, moo commands must be inserted into the build graph that the build system (Waf, CMake, etc) creates in a more novel manner than merely adding more source files.

The "pros" and "cons" are largely reversed from the manual option with the following additional "pros":

• correct build dependencies may be assured, given a quality build system
• generated code may be placed in or outside the source area

The rest of this document gives information to help build system developers understand how to fit moo into their system. It focuses on what input and output files with which moo may interact as most build systems use a file-based dependency graph. It ends with some Examples of integrating moo into a few of the more popular build systems (which are also understood by the author).

When integrating moo to a build system it can be useful to query what search paths moo will use to locate a given file and moo path will report these.

It is important to know that moo builds a search path depending on the type of file to be found. For now, there are two classes of file search paths: those to locate data files (includes both model and schema files) and those to locate template files. The latter are identified by a file name ending in j2 and any other ending will be considered a data file.

In addition, for either class of files, the complete search path is composed of a builtin path and augmented with any additional path given by the user.

Some examples makes this clear. First, display the builtin search paths:

moo path j2

/home/bv/dev/moo/moo/templates

moo path jsonnet

/home/bv/dev/moo/moo/jsonnet-code

Next we show how the user may augment the search paths.

moo -M /tmp path jsonnet

/home/bv/dev/moo/moo/jsonnet-code
/tmp

moo -T $HOME/dev/moo/test path j2

/home/bv/dev/moo/moo/templates
/home/bv/dev/moo/test

Like the path command, a build system integrator may wish to test if a file is properly found without necessarily processing anything. The moo resolve command goes the extra step:

moo resolve ocpp.hpp.j2
moo resolve moo.jsonnet
moo -M $HOME/dev/moo/test resolve test-any.jsonnet

/home/bv/dev/moo/moo/templates/ocpp.hpp.j2
/home/bv/dev/moo/moo/jsonnet-code/moo.jsonnet
/home/bv/dev/moo/test/test-any.jsonnet

The moo command imports is provided to help discover intermediate dependency files which have been imported through Jsonnet's or Jinja's file inclusion mechanisms. This is analogous to CPP's -M option to produce .d files for make.

Here is an example of how a build system may call moo to "scan" a Jsonnet file so that the build dependency graph may properly insert the implicit dependencies:

echo 'local a = import "b.jsonnet"; a' > buildsys/top.jsonnet
echo '{b:69}' > buildsys/b.jsonnet

moo imports buildsys/top.jsonnet

/home/bv/dev/moo/buildsys/b.jsonnet

When a model is a function, all TLAs that lack default values must be satisfied in order for evaluation to succeed. Thus all TLAs must be satisfied when the compile, render or imports commands are used.

moo -A more=less imports buildsys/topf.jsonnet

/home/bv/dev/moo/buildsys/b.jsonnet
/home/bv/dev/moo/buildsys/top.jsonnet

A Jinja template file can be scanned similarly and will be detected as such if its file name ends in .j2.

moo imports buildsys/top.txt.j2

By default, moo will print the list of imported files to standard output. The -o option to imports can be provided to output to a specific file.

moo compile will take a model in Jsonnet or other format and output JSON. This takes the same main options as render (next) and so is a way test the intermediate result of grafting and/or applying TLAs or simply providing a common file type from a variety of supported input types.

A build system may choose to explicitly expose this intermediate representation but that must come at a cost of more file I/O and file production. It should only be employed if needed.

The main goal of moo is to generate code and this is the main command do perform that. Like compile (and imports) it prints the result to standard output but can also write to an explicit file given with the -o command option. A minimal example to show the command line is:

moo render -o output.hpp model.jsonnet template.hpp.j2

Both Waf and moo are implemented in Python and that gives the developers options in how to integrate moo into their Waf-based build system. One may either call moo as a CLI program (see section cmd) or use it as a Python module (see section mod).

Ah, venerable make. This example currently does not capture intermediate dependencies. Contributions/fixes welcome.

The Makefile:

model-dump.txt : model.jsonnet dump.txt.j2 
        moo render -o $@ $^
clean:
        rm -f model-dump.txt

And it getting exercised:

cd buildsys/make
make clean
sed -i 's/fortran/FORTRAN/' sub/sub.jsonnet
make
echo -e '\nNext we edit the intermediate files and note no rebuild\n'
sed -i 's/FORTRAN/fortran/' sub/sub.jsonnet
make
echo -e '\nIn any case, first build gave us:\n'
ls -l model-dump.txt
cat model-dump.txt

rm -f model-dump.txt
moo render -o model-dump.txt model.jsonnet dump.txt.j2

Next we edit the intermediate files and note no rebuild

make: 'model-dump.txt' is up to date.

In any case, first build gave us:

-rw-rw-r-- 1 bv bv 93 Feb 12 10:56 model-dump.txt
sub = {'a': 1, 'b': 2, 'c': 'hello'}
subsub = {'a': 1, 'b': 2, 'c': 'hello', 'd': 'FORTRAN'}

This section describes one way to integrate moo with a CMake-based build. It shows what one must do to insert a code generator into the build graph in a way that respects intermediate dependencies using moo imports like the Waf examples.

The current example CMakeLists.txt file using the moo CMake support is:

project("moo-buildsys-cmake")
cmake_minimum_required(VERSION 3.4...3.17) # older may also work

# fixme: needs to turn into a find_package(moo)!
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})
include(moo)

moo_codegen(MODEL model.jsonnet TEMPL dump.txt.j2 CODEGEN model-dump.txt)

## We need some target to trigger the codegen.  Normally, this would
## be done by some "real" code target depending on the CODEGEN output.
##   cmake --build build/ -t DOIT
add_custom_target(DOIT ALL DEPENDS model-dump.txt)

And, here it is getting exercised:

cd buildsys/cmake
rm -rf build
echo '{a:1,b:2,c:"hello"}' > sub.jsonnet
cmake -B build -S .

echo -e '\nDo initial build\n'  
sed -i 's/fortran/FORTRAN/' sub/sub.jsonnet
cmake --build build
ls -l --full-time build/model-dump.txt

echo -e '\nNext we edit the intermediate files and notice a rebuild\n'
sed -i 's/FORTRAN/fortran/' sub/sub.jsonnet
cmake --build build
ls -l --full-time build/model-dump.txt

echo -e '\nDo do a second build, output is not rebuilt\n'  
cmake --build build
ls -l --full-time build/model-dump.txt

echo -e '\nAdd new implicit dependency, which should cause a rebuild\n'  
echo -e 'local ss = import "sub/sub2.jsonnet";\nss' > sub.jsonnet
echo '{a:1,b:2,c:"hello"}' > sub/sub2.jsonnet
cmake --build build
ls -l --full-time build/model-dump.txt

echo -e '\nUpdate new implicit dependency, which should cause a rebuild\n'  
echo '{a:1,b:2,c:"see me"}' > sub/sub2.jsonnet
cmake --build build
ls -l --full-time build/model-dump.txt

-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bv/dev/moo/buildsys/cmake/build

Do initial build

Scanning dependencies of target DOIT
[100%] generate code model-dump.txt
[100%] Built target DOIT
-rw-rw-r-- 1 bv bv 93 2021-02-12 10:56:29.982843529 -0500 build/model-dump.txt

Next we edit the intermediate files and notice a rebuild

[100%] generate code model-dump.txt
[100%] Built target DOIT
-rw-rw-r-- 1 bv bv 93 2021-02-12 10:56:30.694823972 -0500 build/model-dump.txt

Do do a second build, output is not rebuilt

[100%] Built target DOIT
-rw-rw-r-- 1 bv bv 93 2021-02-12 10:56:30.694823972 -0500 build/model-dump.txt

Add new implicit dependency, which should cause a rebuild

[100%] generate code model-dump.txt
[100%] Built target DOIT
-rw-rw-r-- 1 bv bv 93 2021-02-12 10:56:31.442803424 -0500 build/model-dump.txt

Update new implicit dependency, which should cause a rebuild

[100%] Built target DOIT
-rw-rw-r-- 1 bv bv 93 2021-02-12 10:56:31.442803424 -0500 build/model-dump.txt