Specification builder¶
Creating a specification, especially in external applications written in Python, can be significantly simplified with the SpecificationBuilder class.
This tool:
Provides methods to update the contents of the specification
Provides an API that allows modifying the specification without worrying about the changes in the format
Provides sanity checks of URLs, prevents duplicates and validates the specification with the frontend.
Note that this chapter only presents the initial steps of constructing a specification in Pipeline Manager. Check the specification format for some more comprehensive details.
Example usage of the SpecificationBuilder¶
Creating the builder¶
Import the package, specify the specification version to use and create an instance:
from pathlib import Path
from pipeline_manager.specification_builder import SpecificationBuilder
SPECIFICATION_VERSION = '20240723.13'
ASSETS_DIRECTORY = Path("./assets")
specification_builder = SpecificationBuilder(
spec_version=SPECIFICATION_VERSION,
assets_dir=ASSETS_DIRECTORY,
check_urls=True
)
assets_dir is a path to additional assets (icons, visualizations) - it is used during validation to inform the user that e.g. paths provided in the specification do not have corresponding files.
check_urls determines whether the SpecificationBuilder should check URLs provided in the specification for availability and raise errors when pages are not found.
Creating node types¶
Node types can be created with:
specification_builder.add_node_type(
name='Digital camera',
category='Video provider'
)
This method requires a unique name of the node type. We can also specify category, parent classes, layer and abstract fields. Follow the documentation of the class for more details.
Add node type “as category”¶
To create a node type that can act as a category grouping the child node types, we can use add_node_type_as_category:
specification_builder.add_node_type_as_category(
categoryname='Filter',
)
To add a node that extends this node type as category, just provide its name in the extends field in the child node:
specification_builder.add_node_type(
name='GaussianFilter',
extends='Filter'
)
Adding interfaces to node types¶
The following snippets show how to add interfaces to node types.
Note that one or more values can be passed to the interfacetype argument, signifying which types of other interfaces can be connected to it.
specification_builder.add_node_type_interface(
name='Digital camera',
interfacename='Video output',
interfacetype='video',
direction='output'
)
specification_builder.add_node_type_interface(
name='Filter',
interfacename='Video',
interfacetype=['video', 'filtered_output'],
direction='input'
)
specification_builder.add_node_type_interface(
name='Filter',
interfacename='Output',
interfacetype='filtered_output',
direction='output',
)
Adding properties to node types¶
To create a property in Pipeline Manager, we need to pass three required arguments:
name- the name of the propertytype- the type of the property, which has nine possible values:text,constant,number,integer,select,bool,slider,list,hex,
default- specifies a default value, its type depends on thetypeof the property
specification_builder.add_node_type_property(
name='Digital camera',
propname='Focal length',
proptype='slider',
min=14,
max=135,
default=50,
description='Value signifying a camera\'s angle of view'
)
specification_builder.add_node_type_property(
name='Filter',
propname='Kernel size',
proptype='select',
values=['3x3', '5x5', '7x7'],
default='3x3',
description='Small matrix used for convolution'
)
There are some additional arguments, available in the specification format.
Adding node type descriptions¶
Adding descriptions is just a matter of providing a node name and a description string.
specification_builder.add_node_description(
name='Digital camera',
description='Camera providing digital video output'
)
Adding metadata¶
Metadata specifies additional editor options, like connectionStyle or movementStep.
All parameters are defined in the specification format.
specification_builder.metadata_add_param(
paramname='connectionStyle',
paramvalue='orthogonal'
)
Adding URLs to node types¶
To add a URL, first specify the URL group, which is part of the metadata. The icon argument can be a path to an asset file or a link.
specification_builder.metadata_add_url(
groupname='wikipedia',
displayname='Wikipedia',
icon='https://en.wikipedia.org/static/favicon/wikipedia.ico',
url='https://en.wikipedia.org/'
)
Then, add the node URL by defining the suffix that will be appended to the group URL.
specification_builder.add_node_type_url(
name='Digital camera',
urlgroup='wikipedia',
suffix='wiki/Digital_camera'
)
Constructing and validating the specification¶
This function will generate and check the validity of the specification:
specification = specification_builder.create_and_validate_spec()
The possible arguments are:
workspacedir- The path to the workspace directory for Pipeline Manager, the same that has been provided for the build scriptfail_on_warnings- Determines whether the validation should fail on warningsdump_spec- A path to where the specification should be dumped as a file before validation. Useful for debugging purposes.sort_spec- Return specification that has introduced ordering to dictionaries and lists. In lists containing dictionaries the unique field of each dictionary is used to sort entries.
The created specification upon successful run should contain a full specification based on SpecificationBuilder.
It is a regular Python dictionary that can be saved to a JSON file using the json.dump method.
Note
To make sure the entries in the specification are sorted (mostly dictionaries) after using sort_spec, in case of Python’s json.dump and json.dumps methods use sort_keys=True, e.g.:
json.dumps(spec, sort_keys=True, indent=4)
SpecificationBuilder documentation¶
-
class pipeline_manager.specification_builder.SpecificationBuilder(spec_version: str, assets_dir: Path | None =
None, check_urls: bool =False)¶ Creates a specification file and checks validity of files.
This class allows to:
Create and modify specification entries using simple API
Merge existing specifications with some consistency checks
Check URLs to remote resources in terms of availability
Checks correctness of paths to assets from the given directory
This class also performs very strict checking, e.g. it does not allow creating duplicates. In case of any errors spotted it raises a SpecificationBuilderException.
- add_include_subgraph(dataflow: dict[tuple[str, str], str])¶
Adds dataflow to the include subgraph defined by url.
- add_node_description(name: str, description: str)¶
Sets description for the node if it is not defined.
-
add_node_type(name: str, category: str | None =
None, layer: str | None =None, extends: str | list[str] | None =None, abstract: bool | None =False)¶ Adds a node type to the specification.
- Parameters:¶
- name : str¶
Name of the node type
- category : Optional[str]¶
Category of the node
- layer : Optional[str]¶
Name of the layer metatype
- extends : Optional[Union[str, List[str]]]¶
Base classes for the node type
- abstract : Optional[bool]¶
Tells if the type is abstract or not. Abstract types do not need to be complete, they are also not added to the final specification. They are templates for other classes.
- Raises:¶
SpecificationBuilderException – Raised if the node type is redefined.
- add_node_type_additional_data(name: str, additionaldata: Any)¶
Adds additional data to the node, in JSON-like format.
-
add_node_type_as_category(categoryname: str, categoryparent: str =
'', layer: str | None =None, extends: str | list[str] | None =None)¶ Adds a node type “as category” to the specification.
- add_node_type_category(name: str, category: str)¶
Adds a category to the given node. Raises an error if different category is already added.
-
add_node_type_interface(name: str, interfacename: str, interfacetype: str | list[str] | None =
None, direction: str ='inout', dynamic: bool | list[int] =False, side: str | None =None, maxcount: int | None =None, override: bool | None =None, array: list[int] | None =None)¶ Adds interface to the node type.
- Parameters:¶
- name : str¶
Name of the node type
- interfacename : str¶
Name of the interface
- interfacetype : Optional[Union[str, List[str]]]¶
List of matching types for interfaces
- direction : str¶
Direction of the connection, by default “inout”.
- dynamic : Union[bool, List[int]]¶
Determine whether a number of interfaces may be dynamically adjusted. By default, False.
- side : Optional[str]¶
On which side the interface should be placed by default
- maxcount : Optional[int]¶
The maximum connections to the given interface
- override : Optional[bool]¶
Determines whether interface should be overridden
- array : Optional[List[int]]¶
Creates an array of interfaces with given name. Accepts two integers - minimal and maximal value
- Raises:¶
SpecificationBuilderException – Raised when interface already exists.
- add_node_type_parent(name: str, parent_names: str | list[str])¶
Adds a parent class to the node type. Raises an exception if the base name does not exist.
-
add_node_type_property(name: str, propname: str, proptype: str, default: Any, description: str | None =
None, min: Any =None, max: Any =None, values: list[Any] | None =None, dtype: str | None =None, override: bool | None =None)¶ Adds property to the node.
- Parameters:¶
- name : str¶
Name of the node type
- propname : str¶
Name of the property
- proptype : str¶
Type of the property
- default : Any¶
Default value of the property
- description : Optional[str]¶
Optional description for the property
- min : Any¶
Minimal value
- max : Any¶
Maximal value
- values : Optional[List[Any]]¶
List of allowed values
- dtype : Optional[str]¶
Type of elements in property type is list
- override : Optional[bool]¶
Determines whether property should be overridden
- Raises:¶
SpecificationBuilderException – Raised when the property exists already.
-
add_node_type_property_group(name: str, propgroupname: str, propname: str, proptype: str, default: Any, description: str | None =
None, min: Any =None, max: Any =None, values: list[Any] | None =None, dtype: str | None =None, override: bool | None =None)¶ Adds a property to a property group.
- Parameters:¶
- name : str¶
Name of the node type
- propgroupname : str¶
Name of the group of property
- propname : str¶
Name of the property
- proptype : str¶
Type of the property
- default : Any¶
Default value of the property
- description : Optional[str]¶
Optional description for the property
- min : Any¶
Minimal value
- max : Any¶
Maximal value
- values : Optional[List[Any]]¶
List of allowed values
- dtype : Optional[str]¶
Type of elements in property type is list
- override : Optional[bool]¶
Determines whether property should be overridden
- Raises:¶
SpecificationBuilderException – Raised when no properties are found that could be bound to the group.
-
create_and_validate_spec(workspacedir: Path | None =
None, resolved_specification: Path | None =None, fail_on_warnings: bool =True, sort_spec: bool =False, dump_spec: Path | None =None) dict¶ Creates a specification and validates it using schema.
- Parameters:¶
- workspacedir : Optional[Path]¶
Path to the workspace directory for Pipeline Manager
- resolved_specification : Optional[Path]¶
Path to specification that has have resolved inheritance, i.e. resolved ‘extends’ attributes. If none then the specification is not resolved.
- fail_on_warnings : bool¶
Tells if the specification creation should fail on warnings
- sort_spec : bool¶
True if the entries in the specification should be sorted.
- dump_spec : Optional[Path]¶
Tells where the specification should be dumped to file before validation and resolving for debugging purposes.
- Returns:¶
Built specification, if successful
- Return type:¶
Dict
- Raises:¶
SpecificationBuilderException – Raised when specification is not valid or when warnings appeared.
-
create_property(propname: str, proptype: str, default: Any, description: str | None =
None, min: Any =None, max: Any =None, values: list[Any] | None =None, dtype: str | None =None, override: bool | None =None) dict¶ Creates and returns a property.
- Parameters:¶
- propname : str¶
Name of the property
- proptype : str¶
Type of the property
- default : Any¶
Default value of the property
- description : Optional[str]¶
Optional description for the property
- min : Any¶
Minimal value
- max : Any¶
Maximal value
- values : Optional[List[Any]]¶
List of allowed values
- dtype : Optional[str]¶
Type of elements in property type is list
- override : Optional[bool]¶
Determines whether property should be overridden
- Returns:¶
Creates a single property for the node type
- Return type:¶
dict
-
metadata_add_interface_styling(interfacename: str, interfacecolor: str | None =
None, interfaceconnpattern: str | None =None, interfaceconncolor: str | None =None)¶ Adds interface styling to metadata.
- interfacename: str
Name of the interface type
- interfacecolor: Optional[str]
Color of the interface
- interfaceconnpattern: Optional[str]
Interface connection line pattern
- interfaceconncolor: Optional[str]
Color of the interface connection line
-
metadata_add_layer(name: str, nodelayers: str | list[str] | None =
None, nodeinterfaces: str | list[str] | None =None)¶ Adds nodes’ layer to metadata.
-
metadata_add_param(paramname: str, paramvalue: Any, metadata: dict | None =
None)¶ Sets parameter in metadata. Modifies the metadata dictionary in place.
The following rules apply: * If the parameter is a list and the value is a list, the parameter is extended by the value. * If the parameter is a dictionary and the value is a dictionary, the parameter is recursively updated by the value. * Otherwise, the parameter is set to the value.
- metadata_add_url(groupname: str, displayname: str, icon: str, url: str)¶
Adds URL group to metadata.
- register_category(categorypath: str)¶
Adds a category to the set of available categories.
- reset()¶
Resets all fields for the specification.
- set_node_description(name: str, description: str)¶
Sets description for the node type.