Specification builder¶

Creating a specification, especially in external applications written in Python, can be significantly simplified with the SpecificationBuilder class. This tool:

Provides methods to update the contents of the specification
Provides an API that allows modifying the specification without worrying about the changes in the format
Provides sanity checks of URLs, prevents duplicates and validates the specification with the frontend.

Note that this chapter only presents the initial steps of constructing a specification in Pipeline Manager. Check the specification format for some more comprehensive details.

Example usage of the `SpecificationBuilder`¶

Creating the builder¶

Import the package, specify the specification version to use and create an instance:

from pathlib import Path
from pipeline_manager.specification_builder import SpecificationBuilder

SPECIFICATION_VERSION = '20230830.11'
ASSETS_DIRECTORY = Path("./assets")

specification_builder = SpecificationBuilder(
    spec_version=SPECIFICATION_VERSION,
    assets_dir=ASSETS_DIRECTORY,
    check_urls=True
)

assets_dir is a path to additional assets (icons, visualizations) - it is used during validation to inform the user that e.g. paths provided in the specification do not have corresponding files. check_urls determines whether the SpecificationBuilder should check URLs provided in the specification for availability and raise errors when pages are not found.

Creating node types¶

Node types can be created with:

specification_builder.add_node_type(
    name='Digital camera',
    category='Video provider'
)

This method requires a unique name of the node type. We can also specify category, parent classes, layer and abstract fields. Follow the documentation of the class for more details.

Add node type “as category”¶

To create a node type that can act as a category grouping the child node types, we can use add_node_type_as_category:

specification_builder.add_node_type_as_category(
    categoryname='Filter',
)

To add a node that extends this node type as category, just provide its name in the extends field in the child node:

specification_builder.add_node_type(
    name='GaussianFilter',
    extends='Filter'
)

Adding interfaces to node types¶

The following snippets show how to add interfaces to node types. Note that one or more values can be passed to the interfacetype argument, signifying which types of other interfaces can be connected to it.

specification_builder.add_node_type_interface(
    name='Digital camera',
    interfacename='Video output',
    interfacetype='video',
    direction='output'
)

specification_builder.add_node_type_interface(
    name='Filter',
    interfacename='Video',
    interfacetype=['video', 'filtered_output'],
    direction='input'
)

specification_builder.add_node_type_interface(
    name='Filter',
    interfacename='Output',
    interfacetype='filtered_output',
    direction='output',
)

Adding properties to node types¶

To create a property in Pipeline Manager, we need to pass three required arguments:

name - the name of the property
type - the type of the property, which has nine possible values:
- text, constant, number, integer, select, bool, slider, list, hex,
default - specifies a default value, its type depends on the type of the property

specification_builder.add_node_type_property(
    name='Digital camera',
    propname='Focal length',
    proptype='slider',
    min=14,
    max=135,
    default=50,
    description='Value signifying a camera\'s angle of view'
)

specification_builder.add_node_type_property(
    name='Filter',
    propname='Kernel size',
    proptype='select',
    values=['3x3', '5x5', '7x7'],
    default='3x3',
    description='Small matrix used for convolution'
)

There are some additional arguments, available in the specification format.

Adding node type descriptions¶

Adding descriptions is just a matter of providing a node name and a description string.

specification_builder.add_node_description(
    name='Digital camera',
    description='Camera providing digital video output'
)

Adding metadata¶

Metadata specifies additional editor options, like connectionStyle or movementStep. All parameters are defined in the specification format.

specification_builder.metadata_add_param(
    paramname='connectionStyle',
    paramvalue='orthogonal'
)

Adding URLs to node types¶

To add a URL, first specify the URL group, which is part of the metadata. The icon argument can be a path to an asset file or a link.

specification_builder.metadata_add_url(
    groupname='wikipedia',
    displayname='Wikipedia',
    icon='https://en.wikipedia.org/static/favicon/wikipedia.ico',
    url='https://en.wikipedia.org/'
)

Then, add the node URL by defining the suffix that will be appended to the group URL.

specification_builder.add_node_type_url(
    name='Digital camera',
    urlgroup='wikipedia',
    suffix='wiki/Digital_camera'
)

Constructing and validating the specification¶

This function will generate and check the validity of the specification:

specification = specification_builder.create_and_validate_spec()

The possible arguments are:

workspacedir - The path to the workspace directory for Pipeline Manager, the same that has been provided for the build script
fail_on_warnings - Determines whether the validation should fail on warnings
dump_spec - A path to where the specification should be dumped as a file before validation. Useful for debugging purposes.
sort_spec - Return specification that has introduced ordering to dictionaries and lists. In lists containing dictionaries the unique field of each dictionary is used to sort entries.

The created specification upon successful run should contain a full specification based on SpecificationBuilder. It is a regular Python dictionary that can be saved to a JSON file using the json.dump method.

Note

To make sure the entries in the specification are sorted (mostly dictionaries) after using sort_spec, in case of Python’s json.dump and json.dumps methods use sort_keys=True, e.g.:

json.dumps(spec, sort_keys=True, indent=4)

`SpecificationBuilder` documentation¶

class pipeline_manager.specification_builder.SpecificationBuilder(spec_version: str, assets_dir: Path | None = None, check_urls: bool = False)¶

Creates a specification file and checks validity of files.

This class allows to:

Create and modify specification entries using simple API
Merge existing specifications with some consistency checks
Check URLs to remote resources in terms of availability
Checks correctness of paths to assets from the given directory

This class also performs very strict checking, e.g. it does not allow creating duplicates. In case of any errors spotted it raises a SpecificationBuilderException.

add_node_description(name: str, description: str)¶

Sets description for the node if it is not defined.

Parameters:¶

name : str¶: Name of the node type
description : str¶: Description

Raises:¶

SpecificationBuilderException – Raised if description is already defined

Adds a node type to the specification.

Parameters:¶

name : str¶: Name of the node type
category : Optional[str]¶: Category of the node
layer : Optional[str]¶: Name of the layer metatype
extends : Optional[Union[str, List[str]]]¶: Base classes for the node type
abstract : Optional[bool]¶: Tells if the type is abstract or not. Abstract types do not need to be complete, they are also not added to the final specification. They are templates for other classes.

Raises:¶

SpecificationBuilderException – Raised if the node type is redefined.

add_node_type_additional_data(name: str, additionaldata: Any)¶

Adds additional data to the node, in JSON-like format.

Parameters:¶

name : str¶: Name of the node type
additionaldata : Any¶: Any JSON-like construct

Raises:¶

SpecificationBuilderException – Raised if additionalData already exists.

add_node_type_as_category(categoryname: str, categoryparent: str = '', layer: str | None = None, extends: str | List[str] | None = None)¶

Adds a node type “as category” to the specification.

Parameters:¶

categoryname : str¶: Name of the newly added category
categoryparent : str¶: Path to the new category
layer : Optional[str]¶: Name of the layer metatype
extends : Optional[Union[str, List[str]]]¶: Base classes for the node type

Raises:¶

SpecificationBuilderException – Raised when node type is redefined.

add_node_type_category(name: str, category: str)¶

Adds a category to the given node. Raises an error if different category is already added.

Parameters:¶

name : str¶: name of the node type
category : str¶: category for the node type

Raises:¶

SpecificationBuilderException – Raised if there is already a node type category.

add_node_type_from_spec(node)¶: Adds single node type defined in JSON-like format.

add_node_type_icon(name: str, iconpath: dict | str) → None¶

Adds icon for the node type.

Parameters:¶

name : str¶: Name of the node type
iconpath : Union[dict, str]¶: Icon path

Raises:¶

SpecificationBuilderException – Raised when given icon is not available in metadata or is invalid.

Adds interface to the node type.

Parameters:¶

name : str¶: Name of the node type
interfacename : str¶: Name of the interface
interfacetype : Optional[Union[str, List[str]]]¶: List of matching types for interfaces
direction : str¶: Direction of the connection
side : Optional[str]¶: On which side the interface should be placed by default
maxcount : Optional[int]¶: The maximum connections to the given interface
override : Optional[bool]¶: Determines whether interface should be overridden
array : Optional[List[int]]¶: Creates an array of interfaces with given name. Accepts two integers - minimal and maximal value

Raises:¶

SpecificationBuilderException – Raised when interface already exists.

add_node_type_parent(name: str, parent_names: str | List[str])¶

Adds a parent class to the node type. Raises an exception if the base name does not exist.

Parameters:¶

name : str¶: name of the node type
parent_names : Union[str, List[str]]¶: name of the base class, or list of base classes’ names.

Raises:¶

SpecificationBuilderException – Raised when base/parent class is not found.

add_node_type_property(name: str, propname: str, proptype: str, default: Any, description: str | None = None, min: Any = None, max: Any = None, values: List[Any] | None = None, dtype: str | None = None, override: bool | None = None)¶

Adds property to the node.

Parameters:¶

name : str¶: Name of the node type
propname : str¶: Name of the property
proptype : str¶: Type of the property
default : Any¶: Default value of the property
description : Optional[str]¶: Optional description for the property
min : Any¶: Minimal value
max : Any¶: Maximal value
values : Optional[List[Any]]¶: List of allowed values
dtype : Optional[str]¶: Type of elements in property type is list
override : Optional[bool]¶: Determines whether property should be overridden

Raises:¶

SpecificationBuilderException – Raised when the property exists already.

add_node_type_property_group(name: str, propgroupname: str, propname: str, proptype: str, default: Any, description: str | None = None, min: Any = None, max: Any = None, values: List[Any] | None = None, dtype: str | None = None, override: bool | None = None)¶

Adds a property to a property group.

Parameters:¶

name : str¶: Name of the node type
propgroupname : str¶: Name of the group of property
propname : str¶: Name of the property
proptype : str¶: Type of the property
default : Any¶: Default value of the property
description : Optional[str]¶: Optional description for the property
min : Any¶: Minimal value
max : Any¶: Maximal value
values : Optional[List[Any]]¶: List of allowed values
dtype : Optional[str]¶: Type of elements in property type is list
override : Optional[bool]¶: Determines whether property should be overridden

Raises:¶

SpecificationBuilderException – Raised when no properties are found that could be bound to the group.

add_node_type_url(name: str, urlgroup: str, suffix: str)¶

Adds URL for the node type.

Parameters:¶

name : str¶: Name of the node type
urlgroup : str¶: Name of the URL group the entry is defined for. URL group defines prefix, and icon for the URL.
suffix : str¶: Appended suffix to the URL group

Raises:¶

SpecificationBuilderException – Raised if provided urlgroup does not exist.

add_subgraph_from_spec(subgraph)¶: Adds subgraph defined in JSON-like format.

create_and_validate_spec(workspacedir: Path | None = None, resolved_specification: Path | None = None, fail_on_warnings: bool = True, sort_spec: bool = False, dump_spec: Path | None = None) → Dict¶

Creates a specification and validates it using schema.

Parameters:¶

workspacedir : Optional[Path]¶: Path to the workspace directory for Pipeline Manager
resolved_specification : Optional[Path]¶: Path to specification that has have resolved inheritance, i.e. resolved ‘extends’ attributes. If none then the specification is not resolved.
fail_on_warnings : bool¶: Tells if the specification creation should fail on warnings
sort_spec : bool¶: True if the entries in the specification should be sorted.
dump_spec : Optional[Path]¶: Tells where the specification should be dumped to file before validation and resolving for debugging purposes.

Returns:¶

Built specification, if successful

Return type:¶

Dict

Raises:¶

SpecificationBuilderException – Raised when specification is not valid or when warnings appeared.

create_property(propname: str, proptype: str, default: Any, description: str | None = None, min: Any = None, max: Any = None, values: List[Any] | None = None, dtype: str | None = None, override: bool | None = None) → dict¶

Creates and returns a property.

Parameters:¶

propname : str¶: Name of the property
proptype : str¶: Type of the property
default : Any¶: Default value of the property
description : Optional[str]¶: Optional description for the property
min : Any¶: Minimal value
max : Any¶: Maximal value
values : Optional[List[Any]]¶: List of allowed values
dtype : Optional[str]¶: Type of elements in property type is list
override : Optional[bool]¶: Determines whether property should be overridden

Returns:¶

Creates a single property for the node type

Return type:¶

dict

get_node_description(name: str) → str¶

Gets description for the node type.

Parameters:¶

name : str¶: Name of the node type

Returns:¶

Description of the node type

Return type:¶

str

metadata_add_interface_styling(interfacename: str, interfacecolor: str | None = None, interfaceconnpattern: str | None = None, interfaceconncolor: str | None = None)¶

Adds interface styling to metadata.

interfacename: str: Name of the interface type
interfacecolor: Optional[str]: Color of the interface
interfaceconnpattern: Optional[str]: Interface connection line pattern
interfaceconncolor: Optional[str]: Color of the interface connection line

metadata_add_layer(name: str, nodelayers: str | List[str] | None = None, nodeinterfaces: str | List[str] | None = None)¶

Adds nodes’ layer to metadata.

Parameters:¶

name : str¶: Name of the layer
nodelayers : Optional[Union[str, List[str]]]¶: List of node layers in the layer
nodeinterfaces : Optional[Union[str, List[str]]]¶: List of interface types in the layer

Raises:¶

SpecificationBuilderException – Raised when provided layer already exists.

metadata_add_param(paramname: str, paramvalue: Any, metadata: Dict | None = None)¶

Sets parameter in metadata. Modifies the metadata dictionary in place.

The following rules apply: * If the parameter is a list and the value is a list, the parameter is extended by the value. * If the parameter is a dictionary and the value is a dictionary, the parameter is recursively updated by the value. * Otherwise, the parameter is set to the value.

Parameters:¶

paramname : str¶: Name of the metadata parameter
paramvalue : Any¶: Value of the parameter
metadata : Optional[Dict]¶: Metadata dictionary to modify. if None, the class metadata is used

metadata_add_url(groupname: str, displayname: str, icon: str, url: str)¶

Adds URL group to metadata.

Parameters:¶

groupname : str¶: Name of the URL group
displayname : str¶: Displayable name of the URL group
icon : str¶: Path in assets directory or URL to the icon
url : str¶: URL prefix for the URL group

Raises:¶

SpecificationBuilderException – Raised when URL group already exists and differs in format with the existing entry.

register_category(categorypath: str)¶

Adds a category to the set of available categories.

Parameters:¶

categorypath : str¶: Category. Subcategories are added via path-like string.

reset()¶: Resets all fields for the specification.

set_node_description(name: str, description: str)¶

Sets description for the node type.

Parameters:¶

name : str¶: Name of the node type
description : str¶: Description for the node

update_spec_from_other(otherspec: Any)¶

Updates the specification from other specification. It can be also used to merge partial specifications.

Parameters:¶

otherspec : Any¶: JSON-like structure with other specification

Last update: 2024-05-07

Specification builder¶

Example usage of the SpecificationBuilder¶

Creating the builder¶

Creating node types¶

Add node type “as category”¶

Adding interfaces to node types¶

Adding properties to node types¶

Adding node type descriptions¶

Adding metadata¶

Adding URLs to node types¶

Constructing and validating the specification¶

SpecificationBuilder documentation¶

Example usage of the `SpecificationBuilder`¶

`SpecificationBuilder` documentation¶