Makex Files

Naming

The name of the Makex File should be one of the following:

• Makexfile

• makexfile

The default names to search can be changed with the makex.makex_files configurable.

If several of these files exist in a directory, they shall be searched in the order specified. The first match shall be used as the folder’s primary (or default) Makex file.

The extension of Makex files is .mx.

Syntax

Makex Files are a restricted subset of The Python Programming Language. See the differences for more details.

Tip

Keep your Makex Files simple. Don’t be too clever.

The Makex file format/language is designed to be simple (simpler than Python) and easy/fast to process (almost statically, if necessary).

Makex files are designed to evaluate quickly without running subprocesses.

Commenting

Comments are identical to python.

Example of the syntax below:

# This is a single line comment.

"""
This is a 
multiline comment.
"""

Strings

Strings are the same syntax as in Python, surrounded by quotation markers (' or "). Multiline strings may be surrounded by triple quotation markers (""" or ''').

String objects defined in Makex files have the following structure:

class String

The makex string object. You cannot instantiate this object directly. Use quotes and/or f-strings to define strings.

replace(value, substitute)

Replaces all value with substitute in a String.

Return type:

String

Note

A wide range of built-in methods for Python strings and other primitive types (such as lists and dictionaries) are not defined or enabled.

Formatted Strings

Quotation marks prefixed with the letter f denote a formatted string (For example, f"Example string"). Formatted strings may contain placeholders for variables that shall be rendered when necessary. For example, f"Hello {name}", defines a string with a placeholder called name.

Depending on how formatting strings are defined and used, their rendering may be deferred. This allows embedding references or paths in formatted strings which can be expanded correctly. A formatted string with any type of UnresolvedPath objects can not be rendered outside the scope of a Task definition.

Lists

As in Python, lists in Makex may be defined using the open and closing brackets to define items (For example, ["item1", "item2"]).

The only supported method is append. Lists may be concatenated using the + or += operators.

It is not recommended to modify (or mutate) lists away from their definition site.

Mappings

As in Python, Mappings (or Dictionaries) in Makex may be defined using the open and closing braces to define keys and values (For example, {"key1": "value1", "key2": "value2", ...}).

Mappings must use Strings for the keys, and the values may be any type as required/used.

It is not recommended to modify (or mutate) Mappings away from their definition site.

None

The name None is used as a null value.

A None value in Makex is typically ignored or skipped. None values are not serialized to strings.

Built-in Functions

task()

This function defines a Task which will become part of the execution graph.

The task function is as follows:

task(name, requires=None, inputs=None, steps=None, outputs=None, environment=None)

Parameters:

• name (String) – Name of the task.

• requires (list[PathLike]) – A list of requirements (Task names or Locators). A string with a : will be parsed as a Task Locator. Any values which evaluate to None will be skipped.

• steps (list[Action | list[Action]]) – A list of Actions. These are actions, executables, or scripts run in sequence as part of the task.

• inputs (PathLike | list[PathLike] | dict[String, PathLike]) – A file, list of files or mapping of existing files required by this task.

• outputs (Union[PathLike, list[PathLike], dict[String, PathLike]]) – A file or list of the files this task produces. If a task produces any files that are to be consumed by any dependents of the task, they should be defined here. Defining outputs makes the Task a candidate for caching.

• environment (Mapping[str, Union[String, PathLike,Number]]) – Environment variables to set for the Task and any executables it runs. Values must be simple or immediate, and serializable to strings.

Example of creating a task named world which depends on hello:

task(
    name="world",
    requires=[
        "hello"
    ],
    steps=[
        print("World!")
    ],
)

task(
    name="hello",
    steps=[
        print("Hello")
    ],
)

Running the world task will run the hello task first (using makex run world). The printed output will be:

Hello
World!

Paths

Makex standardizes on paths with / as a separator.

Paths in Makex files may not contain the .. (double dot) marker.

Note

Any specifics or problems in the Python pathlib module will show up in Makex.

path()

To refer to arbitrary paths use the path() function to produce a Path object.

Paths may be relative to the Makex file, absolute to the Workspace (starting with two slashes //), or absolute (starting with a single slash /)

Paths in Makex files may be joined with a separator using the / operator, as a form of shorthand.

An example of using this function an operator is below:

# functional:
BASH_PATH = path("/usr/bin").join("bash")

# or, using operator:
BASH_PATH = path("/usr/bin") / "bash"

path(*path: PathLike) → Path

Return a Path object.

Parameters:

path – One or more path components.

class Path

A path object. Similar to pathlib.Path. Paths can be concatenated with separators by using the / operator.

name: String

The name of the path; typically a file name. This is the last component/part of any given path. The name of the root (/) is an empty string.

join(*parts: String)

Joins the path with one or more String components specified with the parts argument.

Return type:

Path

task_path()

The task_path() function will return the output Path for the specified task. See the Tasks and the Cache documentation for more about Task output paths.

task_path(name: String, path: PathLike = None) → Path

Returns an output path for a task with name. If the path argument is specified, returns the output path corresponding to the task with the matching name inside of a Makex file at path.

The output path of a task is typically MakexFile.folder / “_output_” / Task.identity.

Parameters:

• name – Name of a task.

• path – Optional. A folder containing a Makex file. Workspace relative paths are accepted here.

Note

If DIRECT_REFERENCES_TO_MAKEX_FILES is enabled, the path argument may be a path to a Makex file.

home()

home(*path:PathLike=None) → Path

Return a Path that evaluates to the current user’s home folder.

Parameters:

path – Optional. A subpath in the home folder.

source()

source(*path:PathLike=None) → Path

Return a Path that evaluates to the current source folder (the folder containing the makex file).

This may be used to write back files to the source folder (for example, via the copy or write actions).

Parameters:

path – Optional. A subpath in the in the source folder.

find()

find(path: PathLike, include: Pattern | Glob = None) → list[Path]

Finds files (recursively). If path is relative, it will be resolved relative to the folder of the makex file (the source folder).

This function is typically intended to arbitrarily find source files to operate on.

Note: this function is intended to be used as part of Task or its actions; find will not return/resolve otherwise.

Parameters:

• path – A path to search for files in.

• include (Union[Pattern,Glob]) – Files to include. If specified, only files matching the include will be returned.

glob()

glob(pattern: String = None) → Glob

Prepares/compiles a glob pattern.

Note: this function is intended to be used as part of Task or its actions; glob will not return/resolve otherwise.

When used alone within a task (e.g. as part of a Tasks inputs/dependencies), globs will match files within the source folder of the task (or, the same folder in which the makex file exists).

Parameters:

pattern – Patterns of files to include.

Globs support the following syntax:

• / to separate path segments.

• * to match zero or more characters in a path segment.

• ? to match on one character in a path segment.

• ** to match zero or more folders. (e.g. **.py will match python files in any directory)

• [] to declare a range of characters to match.

• {} to declare a set of patterns to match.

• [!...] to negate a range of characters to match.

Tip

Match a file with extension: *.c

Match a file with extension recursively: **.c

Match files with multiple extensions: *.{py,md,txt}

Actions

Each Task can accept a list of “Actions” that are run when the Task is executed.

For example:

task(
    name="example",
    steps=[
        copy(...),
        execute(...),
        print(...),
    ]
)

execute()

execute(*executable: str | PathLike | list[str | PathLike])

Execute an executable.

Executables are run from the task’s source folder (the input root).

The first value in executable is the executable to run. This may be a relative path (to a source folder), an absolute path, the name of an executable, or the reference/output of another task. If it’s a name (without any slashes), it will be searched for in the folders defined inside the PATH environment variable.

Followed by the executable is a variable number of arguments passed to the executable.

List values in any arguments are “flattened”.

Any arguments which evaluate to None will not be included when running the executable.

To execute the output of another Task, you must specify the task name, followed by a colon, followed by an optional path of the Task (for example, execute("task_name://path", ...) or execute("task_name:", ...)). See Tasks as executables for more information.

Note

Arguments should be quoted and separated as required by the executable. Typically, this means passing each argument and value as a separate string.

Note

Arguments are passed to the executable as is, with no shell expansion. If you use ~ to denote a home directory, you will need to expand it, or use the home() or shell() functions.

Note

When the executable is a reference to the output of another task, currently, the first declared unnamed output of the referred Task is used as the executable. The referenced Task should be included in executing task’s requirements, though it may be automatically added (implicitly, by configuration). For more on this, see Executing the Output of a Task from another Task.

Tip

You may group arguments/values together using tuples. For example:

execute("example", ("--argument1", "argument1_value"), ...).

copy()

copy(paths, destination=None, /)

Copy paths (files or folders) to the Task’s output folder, or the specified folder destination. The paths argument may be a list of paths, or a single path.

If destination is a relative path, it will be resolved relative to the Task’s output path; this may be used to prefix items in the output. Any directories specified in destination (by using a directory separator) will be created before copying.

If the destination is empty, the specified paths will be copied directly into the Task’s output path.

If the destination doesn’t exist, it will be created.

An Execution error will be raised if the destination exists, and it is not a folder.

The contents of folders will be copied as-is. Symbolic links shall remain unchanged.

Parameters:

• paths (Union[PathLike,list[PathLike]]) – Paths to the file(s) or folder(s) to copy. Relative paths are resolved relative to the makex file (or source folder).

• destination (PathLike) – The destination. May be a path relative to the task output path, or an absolute path.

Note

Makex uses file cloning/copy-on-write/reflinks for lightweight copies of files on supporting filesystems (bcachefs, btrfs, XFS, OCFS2, ZFS (unstable), APFS and ReFSv2).

The copy function has 7 valid forms:

# copy a file to the Task output
copy(file)

# copy a folder to the Task output
copy(folder)

# copy folder to the specified Folder (relative to the task output)
copy(folder, folder)

# list forms:

# copy a list of files to the Task output
copy(files)

# copies a set of files to the specified folder (relative to the task output).
copy(files, folder)

# copy a list of folders to the Task output
copy(folders) 

# copies a set of folders to the specified folder (relative to the task output).
copy(folders, folder) 

print()

print(message)

Print a message to standard output.

Parameters:

message (String) – The message to print.

shell()

shell(*script)

Run script in a system shell.

Warning

You should seriously avoid use of this function. Shells may introduce unexpected behavior in Makex.

For example, the line shell(f”rm {SOME_VARIABLE}/bin”)) will attempt to remove your /bin directory if SOME_VARIABLE is defined as an empty string.

shell() is there if you really need it, and additional mechanisms will be employed in the future to increase safety. Keep your scripts simple.

Makex may a adopt a “strict” mode where all shell scripting is disabled.

By default, Makex will use the detected/system shell (usually, sh, or the Bourne Shell bash). A shell can be specified in configuration, but it’s recommened to leave it to autodetect based on platform.

The passed script is prefixed with a preamble by default:

set -Eeuo pipefail

Parameters:

script (Union[String,list[String]]) – The script/command line to run. If a list of strings, the strings will concatenated with new line separators and run in one process/shell.

Note

The syntax of the script depends on the system’s shell. Variables are expanded according the specified shell’s rules.

write()

write(file: PathLike, data: str | list[str], executable: bool = False)

Writes data to file.

Any intermediate folders specified in the file path will be created as necessary.

Parameters:

• file (PathLike) – The destination file. May be a Workspace path, an absolute path (if allowed), or relative path within the Task’s output path.

• data (Union[str,list[str]]) – The data to write, may be a list of strings which will be concatenated.

• executable (bool) – Ensure the file is executable.

Self Documentation

A multi-line comment string may be included at the top of the Makex file to document it.

This string may be written in markdown with restructured text to provide help or description in other formats/renderings (see MyST).

Differences from Python Syntax

• Import statements (import ... or from ... import ...) to any standard libraries are not allowed in Makex Files. Future versions of Makex shall include modularization features.

• String syntax and functions are restricted.

• Makex has modified syntax for defining functions (called macros). Defining/using functions without a macro decorator is not allowed.

• Several methods on objects such as strings and lists are not [yet] provided.

Formatting

When calling functions or constructing lists or dictionaries you should leave a trailing comma at the end of the list or dictionary. This helps when adding or changing values later.

It is preferred to break functions/callables with keyword arguments into separate lines with a keyword argument per line. Actions like execute() or copy() may be left on a single line if they fit, and may omit the trailing comma.

We plan to introduce automatic formatting. Until then, tools like yapf or black are recommended to keep your Makex files formatted.

Keep your Makex files simple.

# bad
alist = [1,2,3]

# good
alist = [
    1,
    2,
    3,
]

# bad 
task(
    name="example1", requires=[
        "bad", "Bad"
    ],
    ...
)

# good
task(
    name="example1",
    requires=[
        "bad", 
        "Bad",
    ],
    ...
)

Magic marker/Hashbang/Shebang

A marker #!makex at the top of the file both serves to mark the file as a script and to differentiate makex files from other types of files.

At the moment, this marker is entirely optional.

#!makex

The word makex can be anywhere on the line after the #!.