Custom templates
-----------------

Offcourse you can create your own templates. Basically a template is a yaml 
file and some jinja templating (https://jinja.palletsprojects.com/) magic to
make it configurable. There are two main levels defined for each template.

#. pipeline
#. scripts

We will make use of some yaml and jinja features to not have to define some of
the variables multiple times. We added those at the top of the file in a key
called basic-settings. All values (and keys) at the ``pipeline`` level are
available when rendering the pipeline config.xml. All key, values at the 
``scripts`` level are available when rendering script files.
Values that need to be defined at both levels could be added at the beginning 
of the template and referenced to from both ``pipeline`` and ``scripts`` levels. 

Creating a template
~~~~~~~~~~~~~~~~~~~

To do so we first create a folder ie: ``templates`` in the root
of our project folder. In this folder we create a new yaml file ie: 
``my_template.yaml``. First things to define are:

#. Version of the template defition file, this should be 1.
#. Location where CTP can store data like quarantine and root folders.
#. Define server properties like maxThreads and port number for the web interface.

This should look like this:

.. code-block:: yaml

  basic-settings:
    quarantine: &quarantine {{server.storage}}/quarantine/{{pipeline.name}}
    root: &root {{server.storage}}/root/{{pipeline.name}}
  version: 1
  pipeline:
    name: {{pipeline.name}}
    version: 1.0
    author: CTP Admin
    server:
      maxthreads: 20
      port: {{server.port}}

After defining the basic settings we can start defining the stages, and script
properties that are part of the pipeline. To keep the pipeline flexible we probably 
want to make the import and export stages flexible like in the default
templates. These should look like this:

.. code-block:: sls

  pipeline:
    stages:
      import:
        type: {{pipeline.import_type}}Import
        {% if ((pipeline.import_type == 'Dicom') or (pipeline.import_type == 'Http')) %}
        port: {{pipeline.import_port}}
        {%endif%}
        {% if ((pipeline.import_type == 'Directory') or (pipeline.import_type == 'Archive')) %}
        path: {{pipeline.import_path}}
        {%endif%}
        quarantine: *quarantine
        root: *root
      export:
        type: {{pipeline.export_type}}Export
        {% if ((pipeline.export_type == 'Dicom') or (pipeline.export_type == 'Http')) %}
        url: {{pipeline.export_url}}
        {% elif ((pipeline.export_type == 'File') or (pipeline.export_type == 'Directory')) %}
        path: {{pipeline.export_path}}
        {%endif%}
        quarantine: *quarantine
        root: *root

This gives the option to define an import and export stage with the following properties:

#. type: The type of import or export stage.
#. port: The port number the import stage should listen to. (Only for Dicom and Http)
#. path: The path where the import or export stage should read or write data to. (Only for Directory and Archive)
#. url: The url where the export stage should send data to. (Only for Http)

This already gives a working template that can be used to create a CTP
configuration. To add to option to enable/disable tracking of DICOM files we could add the following to the template:

.. code-block:: sls

  {% if pipeline.object_tracker %}
  object_input_tracker:
      type: ObjectTracker
      root: *root
  {% endif %}

Adding scripts
~~~~~~~~~~~~~~

Some stages requires scripts these can also be templated using CTP builder and
just lie the pipeline these can be made configurable using Jinja templating.
For example we could add a script to filter out DICOMS with series description
we know might contain personal information or are not of interest in this
study. So first let's create a new script file in the scripts folder ie:
``SeriesFilter.script``. The content of this file should look like this:

.. code-block:: jinja
  
  {%- for series_description in series_descriptions -%}
  {%- if loop.first %}
  ! SeriesDescription.equalsIgnoreCase("{{series_description}}")
  {%- else -%}
  * !SeriesDescription.equalsIgnoreCase("{{series_description}}")
  {%- endif -%}
  {%- endfor -%}

This will create a filter for all series_descriptions in the variable
series_descriptions. There a couple of acttions we have to do to make sure
this end up in the CTP configuration:

#. Add a stage to the pipeline.
#. Add the script to make sure it is included in the configuration.
#. Make sure the series descriptions are configurable.

The final pipeline template should look like:

.. code-block:: sls
  :emphasize-lines: 28, 29, 30, 31, 32, 33, 34, 44, 45, 46, 47, 48, 49, 50

  basic-settings:
    quarantine: &quarantine {{server.storage}}/quarantine/{{pipeline.name}}
    root: &root {{server.storage}}/root/{{pipeline.name}}
  version: 1
  pipeline:
    name: {{pipeline.name}}
    version: 1.0
    author: CTP Admin
    server:
      maxthreads: 20
      port: {{server.port}}
    stages:
      import:
        type: {{pipeline.import_type}}Import
        {% if ((pipeline.import_type == 'Dicom') or (pipeline.import_type == 'Http')) %}
        port: {{pipeline.import_port}}
        {%endif%}
        {% if ((pipeline.import_type == 'Directory') or (pipeline.import_type == 'Archive')) %}
        path: {{pipeline.import_path}}
        {%endif%}
        quarantine: *quarantine
        root: *root
      {% if pipeline.object_tracker %}
      object_input_tracker:
          type: ObjectTracker
          root: *root
      {% endif %}
      {% if scripts.series_descriptions %}
      series_filter:
        type: Filter
        script: scripts/SeriesFilter.script
        root: *root
        quarantine: *quarantine
      {% endif %}
      export:
        type: {{pipeline.export_type}}Export
        {% if ((pipeline.export_type == 'Dicom') or (pipeline.export_type == 'Http')) %}
        url: {{pipeline.export_url}}
        {% elif ((pipeline.export_type == 'File') or (pipeline.export_type == 'Directory')) %}
        path: {{pipeline.export_path}}
        {%endif%}
        quarantine: *quarantine
        root: *root
  {% if scripts.series_descriptions %}
  scripts:
    series_descriptions:
      {{ scripts.series_descriptions }}
    include:
      - SeriesFilter.script
  {% endif %}

This should finish the new template. Now we need to make a configuration to 
render the template into a configuration. The configuration looks like:

.. code-block:: yaml

  version: 2
  template: my_template
  server:
    port: 8080
    storage: /mnt/data
  pipeline:
    name: anonymization_project
    import_type: Dicom
    import_port: 1080
    export_type: Dicom
    export_url: https://xnat.example.com:8143
    object_tracker: True
  scripts:
    series_descriptions:
      - Report
      - Screenshot
      - Dose
      - Exam