Makex Files¶

Naming¶

The name of the Makex File should be one of the following:

Makexfile
makexfile

The default names to search can be changed with the makex.makex_files configurable.

If several of these files exist in a directory, they shall be searched in the order specified. The first match shall be used as the folder’s primary (or default) Makex file.

The extension of Makex files is .mx.

Syntax¶

Makex Files are a restricted subset of The Python Programming Language. See the differences for more details.

Tip

Keep your Makex Files simple. Don’t be too clever.

The Makex file format/language is designed to be simple (simpler than Python) and easy/fast to process (almost statically, if necessary).

Makex files are designed to evaluate quickly without running subprocesses.

Magic marker/Hashbang/Shebang¶

A marker #!makex at the top of the file both serves to mark the file as a script and to differentiate makex files from other types of files.

At the moment, this marker is entirely optional.

#!makex

The word makex can be anywhere on the line after the #!.

Commenting¶

# This is a single line comment.

"""
This is a 
multiline comment.
"""

Strings¶

Strings are defined as in Python, surrounded by quotation markers (' or "). Multiline strings may be surrounded by triple quotation markers (""" or ''').

Quotation marks prefixed with the letter f denote a formatted string (For example, f"Example string"). Formatted strings may contain placeholders for variables that shall be rendered when necessary. For example, f"Hello {name}", defines a string with a placeholder called name.

Depending on how formatting strings are defined and used, their rendering may be deferred. This allows embedding references or paths in strings which can be expanded correctly. Typically, a formatted string with any type of UnresolvedPath objects can not be rendered outside the scope of a task definition.

String objects defined in Makex files have the following structure:

class String¶

The makex string object. You cannot instantiate this object directly. Use quotes and/or f-strings to define strings.

replace(value, substitute)¶

Replaces all value with substitute in a String.

Return type:: String

Note

A wide range of built-in methods for Python strings and other primitive types (such as lists and dictionaries) are not defined or enabled.

Lists¶

As in Python, lists in Makex may be defined using the open and closing brackets to define items (For example, ["item1", "item2"]).

The only supported method is append. Lists may be concatenated using the + or += operators.

It is not recommended to modify (or mutate) lists away from their definition site.

Mappings¶

As in Python, Mappings (or Dictionaries) in Makex may be defined using the open and closing braces to define keys and values (For example, {"key1": "value1", "key2": "value2", ...}).

Mappings must use Strings for the keys, and the values may be any type as required/used.

It is not recommended to modify (or mutate) Mappings away from their definition site.

None¶

The name None is used as a null value.

A None value in Makex is typically ignored or skipped. None values are not serialized to strings.

Functions¶

Makex files have functions available which do different things:

`task()`¶

This function defines a Task which will become part of the execution graph.

The task function is as follows:

task(name, requires=None, steps=None, outputs=None)¶

Parameters:

name (String) – Name of the task.
requires (list[PathLike]) – A list of requirements. Can be files or other tasks using a task locator or reference. A string with a : will be parsed as a task reference. Any values which evaluated to None will be skipped.
steps (list[Union[Action,list[Action]]]) – A list of Actions. These are actions/task/executables/scripts run in sequence as part of the task.
outputs (Union[PathLike, list[PathLike], dict[String, PathLike]]) – A file or list of the files this task outputs. If a task produces any files that are to be consumed by any dependents of the task, they should be defined here. Defining outputs makes the task a candidate for caching.

Note

In previous versions of Makex, this function was named target(); this is not supported anymore. Please rename to task().

Example of creating a task named world which depends on hello:

task(
    name="world",
    requires=[
        ":hello"
    ],
    steps=[
        print("World!")
    ],
)

task(
    name="hello",
    requires=[
    ],
    steps=[
        print("Hello")
    ],
)

Running the world task will run the hello task first (using makex run :world). The printed output will be:

Hello
World!

Paths¶

Makex standardizes on paths with / as a separator.

Paths in Makex files may not contain the .. (double dot) marker.

Note

Any specifics or problems in the Python pathlib module will show up in Makex.

path()¶

To refer to arbitrary paths use the path() function to produce a path object.

Any paths in Makex files can be joined with a separator using the / operator.

path(*path: PathLike) → Path¶

Return a Path object.

Parameters:: path – One or more path components.

class Path¶

A path object. Similar to pathlib.Path. Paths can be concatenated with separators by using the / operator.

name: String¶: The name of the path; typically a file name. This is the last component/part of any given path. The name of the root (/) is an empty string.

task_path()¶

The task_path() function will return the output Path for the specified task. See the Tasks and the Cache documentation for more about Task output paths.

task_path(name: String, path: PathLike = None) → Path¶

Returns an output path for a task with name. If the path argument is specified, returns the output path corresponding to the task with the matching name inside of a Makex file at path.

The output path of a task is typically MakexFile.folder / “_output_” / Task.identity.

Parameters:

name – Name of a task.
path – Optional. A folder containing a Makex file. Workspace relative paths are accepted here.

Note

If DIRECT_REFERENCES_TO_MAKEX_FILES is enabled, the path argument may be a path to a Makex file.

home()¶

home(*path:PathLike=None) → Path¶

Return a Path that evaluates to the current user’s home directory.

Parameters:: path – Optional. A subpath in the home directory.

find()¶

find(path: PathLike, include: Pattern | Glob = None) → list[Path]¶

Finds files (recursively). If path is relative, it will be resolved relative to the folder of the makex file (the source folder).

This function is typically intended to arbitrarily find source files to operate on.

Note: this function is intended to be used as part of Task or its actions; find will not return/resolve otherwise.

Parameters:

path – A path to search for files in.
include (Union[Pattern,Glob]) – Files to include. If specified, only files matching the include will be returned.

glob()¶

glob(pattern: String = None) → Glob¶

Prepares/compiles a glob pattern.

Note: this function is intended to be used as part of Task or its actions; glob will not return/resolve otherwise.

When used alone within a task (e.g. as part of a tasks inputs/dependencies), globs will match files within the source folder of the task (or, the same folder in which the makex file exists).

Parameters:: pattern – Patterns of files to include.

Globs support the following syntax:

/ to separate path segments.
* to match zero or more characters in a path segment.
? to match on one character in a path segment.
** to match zero or more folders. (e.g. **.py will match python files in any directory)
[] to declare a range of characters to match.
{} to declare a set of patterns to match.
[!...] to negate a range of characters to match.

Tip

Match a file with extension: *.c

Match a file with extension recursively: **.c

Match files with multiple extensions: *.{py,md,txt}

Actions¶

Each task can accept a list of “Actions” that are run when the task is executed.

For example:

task(
    name="example",
    steps=[
        copy(...),
        execute(...),
        print(...),
    ]
)

execute()¶

execute(*executable: str | PathLike | list[str | PathLike])¶

Execute an executable.

Executables are run from the task’s source folder (the input root).

The first value in executable is the executable to run. This may be a relative path (to a source folder), an absolute path, the name of an executable, or the reference/output of another task. If it’s a name (without any slashes), it will be searched for in the folders defined inside the PATH environment variable.

Followed by the executable is a variable number of arguments passed to the executable.

List values in any arguments are “flattened”.

Any arguments which evaluate to None will not be included when running the executable.

To execute the output of another Task, you must specify the task name and path of the Task (for example, execute(“//path:task_name”, …)). See {ref}`Tasks as executables<tasks-as-executables>` for more information.

Note

Arguments should be quoted and separated as required by the executable. Typically, this means passing each argument and value as a separate string.

Note

Arguments are passed to the executable as is, with no shell expansion. If you use ~ to denote a home directory, you will need to expand it, or use the home() or shell() functions.

Note

When the executable is a reference to the output of another task, currently, the first declared unnamed output of the referred Task is used as the executable. The referenced Task should be included in executing task’s requirements, though it may be automatically added (implicitly, by configuration). For more on this, see Executing the Output of a Task from another Task.

copy()¶

copy(paths, destination=None, /)¶

Copy paths (files or folders) to the Task’s output folder, or the specified folder destination. paths may be a list of paths or a single path.

If destination is a relative path, it will be resolved relative to the Task’s output path; this may be used to prefix items in the output. Any directories specified in destination (by using a directory separator) will be created before copying.

If the destination is empty, the specified paths will be copied directly into the Task’s output path.

If the destination doesn’t exist, it will be created.

An Execution error will be raised if the destination exists, and it is not a folder.

The contents of folders will be copied as-is. Symbolic links shall remain unchanged.

Parameters:

paths (Union[PathLike,list[PathLike]]) – Paths to the file(s) or folder(s) to copy. Relative paths are resolved relative to the makex file (or source folder).
destination (PathLike) – The destination. May be a path relative to the task output path, or an absolute path.

Note

Makex uses file cloning/copy-on-write/reflinks for lightweight copies of files on supporting filesystems (bcachefs, btrfs, XFS, OCFS2, ZFS (unstable), APFS and ReFSv2).

The copy function has 7 valid forms:

# copy a file to the Task output
copy(file)

# copy a folder to the Task output
copy(folder)

# copy folder to the specified Folder (relative to the task output)
copy(folder, folder)

# list forms:

# copy a list of files to the Task output
copy(files)

# copies a set of files to the specified folder (relative to the task output).
copy(files, folder)

# copy a list of folders to the Task output
copy(folders) 

# copies a set of folders to the specified folder (relative to the task output).
copy(folders, folder) 

print()¶

print(message)¶

Print a message to standard output.

Parameters:: message (String) – The message to print.

shell()¶

shell(*script)¶

Run script in a system shell.

Warning

You should seriously avoid use of this function. Shells may introduce unexpected behavior in Makex.

For example, the line shell(f”rm {SOME_VARIABLE}/bin”)) will attempt to remove your /bin directory if SOME_VARIABLE is defined as an empty string.

shell() is there if you really need it, and additional mechanisms will be employed in the future to increase safety. Keep your scripts simple.

Makex may a adopt a “strict” mode where all shell scripting is disabled.

By default, Makex will use the detected/system shell (usually, sh, or the Bourne Shell bash). A shell can be specified in configuration, but it’s recommened to leave it to autodetect based on platform.

The passed script is prefixed with a preamble by default:

set -Eeuo pipefail

Parameters:: script (Union[String,list[String]]) – The script/command line to run. If a list of strings, the strings will concatenated with new line separators and run in one process/shell.

Note

The syntax of the script depends on the system’s shell. Variables are expanded according the specified shell’s rules.

write()¶

write(file: PathLike, data: str | list[str], executable: bool = False)¶

Writes data to file.

Any intermediate folders specified in the file path will be created as necessary.

Parameters:

file (PathLike) – The destination file. May be a Workspace path, an absolute path (if allowed), or relative path within the Task’s output path.
data (Union[str,list[str]]) – The data to write, may be a list of strings which will be concatenated.
executable (bool) – Ensure the file is executable.

Self Documentation¶

A multi-line comment string may be included at the top of the Makex file to document it.

This string may be written in markdown with restructured text to provide help or description in other formats/renderings (see MyST).

Differences from Python Syntax¶

Import statements (import ... or from ... import ...) to any standard libraries are not allowed in Makex Files. Future versions of Makex shall include modularization features.
String syntax and functions are restricted.
Makex has modified syntax for defining functions (called macros). Defining/using functions without a macro decorator is not allowed.
Several methods on objects such as strings and lists are not [yet] provided.

Formatting¶

When calling functions or constructing lists or dictionaries you should leave a trailing comma at the end of the list or dictionary. This helps when adding or changing values later.

It is preferred to break functions/callables with keyword arguments into separate lines with a keyword argument per line. Actions like execute() or copy() may be left on a single line if they fit, and may omit the trailing comma.

We plan to introduce automatic formatting. Until then, tools like yapf or black are recommended to keep your Makex files formatted.

Keep your Makex files simple.

# bad
alist = [1,2,3]

# good
alist = [
    1,
    2,
    3,
]

# bad 
task(
    name="bad", requires=[
        ":bad", ":Bad"
    ],
    ...
)

# good
task(
    name="bad",
    requires=[
        ":bad", 
        ":Bad",
    ],
    ...
)

Makex Documentation

Table of Contents

Makex Files¶

Naming¶

Syntax¶

Magic marker/Hashbang/Shebang¶

Commenting¶

Strings¶

Lists¶

Mappings¶

None¶

Functions¶

`task()`¶

Paths¶

path()¶

task_path()¶

home()¶

find()¶

glob()¶

Actions¶

execute()¶

copy()¶

print()¶

shell()¶

write()¶

Self Documentation¶

Differences from Python Syntax¶

Formatting¶

Links

Table of Contents

Search

Need support?

This Page

Table of Contents

Makex Files¶

Naming¶

Syntax¶

Magic marker/Hashbang/Shebang¶

Commenting¶

Strings¶

Lists¶

Mappings¶

None¶

Functions¶

task()¶

Paths¶

path()¶

task_path()¶

home()¶

find()¶

glob()¶

Actions¶

execute()¶

copy()¶

print()¶

shell()¶

write()¶

Self Documentation¶

Differences from Python Syntax¶

Formatting¶

`task()`¶