Working with GDAL Vector Pipelines

Overview

MapServer can dynamically run a GDAL Vector Pipeline, and render its output - all through a simple Mapfile.

In this workshop we'll use the Tartu roads dataset used in the Line Styling exercise, dynamically buffer it using GDAL, and display the result in OpenLayers using a MapServer WMS.

This is a simple example of a pipeline, but additional steps can be chained together to create more complex workflows. MapServer reads a vector pipeline using the GDALG: GDAL Streamed Algorithm driver.

Checking the Pipelines with GDAL

Before configuring MapServer, it is often easier to test your pipelines directly with GDAL, to ensure they run correctly. Run the commands below to connect to the MapServer Docker container and use GDAL to get information about the pipelines.

# open a shell inside the MapServer container
docker exec -it mapserver /bin/bash

# check the dataset used in the pipeline
gdal vector info data/osm/roads.fgb

# inspect the a pipeline JSON file included in the container
gdal vector info roads.gdalg.json

# test an inline pipeline using a single-quoted string (recommended)
gdal vector info '{"type": "gdal_streamed_alg","command_line": "gdal vector pipeline ! read /etc/mapserver/data/osm/roads.fgb ! geom buffer --distance=0.0001"}'

# alternatively escape the double quotes
gdal vector info "{\"type\": \"gdal_streamed_alg\",\"command_line\": \"gdal vector pipeline ! read data/osm/roads.fgb ! geom buffer --distance=0.0001\"}"

The Mapfile

Embedding Pipelines in a Mapfile

The pipeline LAYER in this example uses CONNECTIONTYPE OGR and includes the GDAL pipeline "inline" - meaning the pipeline is defined directly in the Mapfile. Our example pipeline looks like this:

CONNECTION '{"type": "gdal_streamed_alg","command_line": "gdal vector pipeline ! read /etc/mapserver/data/osm/roads.fgb ! geom buffer --distance=0.0001"}'
DATA "0"

Key points to note:

DATA "0" tells MapServer to use the first (index 0) layer in the connection. Since the pipeline returns a single dataset, this will correspond to the buffered roads.
When using an inline GDAL pipeline, you must provide the absolute path to any datasets used by the pipeline.
GDAL requires a valid JSON string for the pipeline. All property names and string values must use double quotes.

In a Mapfile, you have two options for embedding JSON:

Wrap the JSON string in single quotes (as in the example above) - this is simpler and easier to read or copy/paste.
Escape the double quotes using \" as in the example below:

CONNECTION "{\"type\": \"gdal_streamed_alg\",\"command_line\": \"gdal vector pipeline ! read /etc/mapserver/data/osm/roads.fgb ! geom buffer --distance=0.0001\"}"

Referencing Pipelines in a JSON File

MapServer can also reference a JSON file containing a pipeline, which makes it easy to test and reuse the pipeline with GDAL. By convention GDALG files should use the .gdalg.json extension.

CONNECTION "roads.gdalg.json"
DATA "0"

The contents of pipeline JSON file roads.gdalg.json are shown below. Notice that the dataset path data/osm/roads.fgb is relative to the JSON file.

Using relative paths in a JSON file makes the pipeline more portable, because it can be moved to a different folder or system without changing the dataset paths.

{
    "type": "gdal_streamed_alg",
    "command_line": "gdal vector pipeline ! read data/osm/roads.fgb ! geom buffer --distance=0.0001"
}

Hatch Styling

Finally, we use a hatch symbol to style the buffered roads. Hatch symbols allow you to add patterned fills, and you can adjust their angle, width, and size in the STYLE block.

SYMBOL
  NAME "hatchsymbol"
  TYPE hatch
END
...
LAYER
  CLASS
    STYLE
      SYMBOL "hatchsymbol"
      COLOR "#78C8FF"
      WIDTH 0.1
      ANGLE 45
      SIZE 8
    END

Code

Example

gdalg.js

import '../css/style.css';
import ImageWMS from 'ol/source/ImageWMS.js';
import Map from 'ol/Map.js';
import View from 'ol/View.js';
import { Image as ImageLayer } from 'ol/layer.js';

const mapserverUrl = import.meta.env.VITE_MAPSERVER_BASE_URL;
const mapfilesPath = import.meta.env.VITE_MAPFILES_PATH;

const layers = [
    new ImageLayer({
        source: new ImageWMS({
            url: mapserverUrl + mapfilesPath + 'gdalg.map&',
            params: { 'LAYERS': 'buffered_roads,roads' },
            ratio: 1
        }),
    }),
];
const map = new Map({
    layers: layers,
    target: 'map',
    view: new View({
        center: [2975862.75916499, 8046369.8646329],
        zoom: 17,
    }),
});

gdalg.map

MAP
  NAME "GDALG"
  EXTENT 26.668678 58.339241 26.796582 58.409410
  SIZE 800 600
  IMAGECOLOR "#0A1E50"
  PROJECTION
    "init=epsg:4326"
  END
  WEB
    METADATA
      "ows_enable_request" "*"
      "ows_srs" "EPSG:4326 EPSG:3857"
    END
  END

  SYMBOL
    NAME "hatchsymbol"
    TYPE hatch
  END

  LAYER
    NAME "buffered_roads"
    TYPE POLYGON
    STATUS OFF
    CONNECTIONTYPE OGR
    # note we need to use the full path to the dataset using the inline pipeline
    CONNECTION '{"type": "gdal_streamed_alg","command_line": "gdal vector pipeline ! read /etc/mapserver/data/osm/roads.fgb ! geom buffer --distance=0.0001"}'
    DATA "0"
    # CONNECTION "roads.gdalg.json"
    # DATA "0"
    CLASS
      STYLE
        COLOR 50 100 180
      END
      STYLE
        SYMBOL "hatchsymbol"
        COLOR "#78C8FF"
        WIDTH 0.1
        ANGLE 45
        SIZE 8
      END
    END
  END

  # original unbuffered roads dataset
  LAYER
    NAME "roads"
    TYPE LINE
    STATUS OFF
    CONNECTIONTYPE FLATGEOBUF
    DATA "data/osm/roads.fgb"
    CLASS
      STYLE
        WIDTH 0.8
        COLOR "#DCF0FF"
      END
    END
  END

END

Exercises

Switch to using the JSON file - replace the inline GDAL pipeline with roads.gdalg.json in your Mapfile.
Extend the GDAL pipeline - add another processing step to the pipeline. Refer to the GDAL Vector Pipeline documentation for examples of available operations.
Experiment with hatch styling - adjust properties such as ANGLE and SIZE in your STYLE block to see how the hatch pattern changes.