How to set up flows

In Titan flows are set up and configured using a YAML configuration file. The flow configuration file consists of two sections, the lists Bricks: and Flows:. Here’s an example of a configuration file:

Bricks:
  - Name:   Generator
    Module: bricks/generator/generator.py
  - Name:   Increase
    Module: bricks/incrementer/incrementer.py
  - Name:   Display
    Module: bricks/display.py

Flows:
  -
    Name: Example Flow
    Bricks:
      -
        Brick: Generator
        Parameters:
          Values: 2
          Wait Between Values: 0.01
      -
        Name: Add42
        Brick: Increase
        Parameters:
          Addend: 42
      -
        Brick: Increase
        autoscale: True
        autoscale_queue_level: 2
        autoscale_max_instances: 3
      -
        Brick: Display

    Connections:
      -
        Source: Generator
        Target: Add42
        Ports:
          Source: A
      -
        Source: Add42
        Target: Increase
      -
        Source: Increase
        Target: Display

Bricks

Bricks: is the list of brick types available for usage in the flows.

• Name: brick_name : The name that identifies the brick (type) in the flow
• Module: e.g. brick/module/directory/file.py : The path to the brick module. This can be a path relative to the flow configuration file, an absolute path, or a python package name (e.g. mypackage) provided that it is resolvable via sys.path.

Flows

Flows: is the list of flows to be executed. A flow is described by:

• Name: flow_name : The name of the flow
• Bricks:[brick_list] : The list of individual bricks that constitute the flow, each specified by:
  • Name: brick_name : The (optional) unique name that identifies the brick in the flow
  • Brick: brick_type_name : The name of the brick as listed in the list of available bricks (Bricks:)
  • Parameters: {parameter_map} : A dictionary with parameters can be given if the brick (type) supports parameters. By default, parameter values given in the config.yml file of the brick are used.
  • autoscale: An (optional) boolean, that determines whether a brick can be scaled or not
  • autoscale_queue_level : An (optional) integer value, that defines at which output queue size of the preceding brick autoscaling is triggered.
  • autoscale_max_instances: An (optional) integer value, that sets the maximum number of instances that can be created.
• Connections: [list] : The list of connections between the bricks, specified by giving:
  • Source: source_brick_name : The name of the source brick (Name: if set in the flow’s Bricks: list, otherwise brick type name Brick:)
  • Target: target_brick_name : The name of the target brick (Name: if set in the flow’s Bricks: list, otherwise brick type name Brick:)
  • Ports: {port_map}: Optional dictionary of ports used to connect source and target brick. Ports are identified by characters.
    • Source: char: Output port of the source brick (defaults to ‘A’)