MS RFC 137: Native FlatGeobuf support¶
- Last Updated:
This document is outdated, please follow the official document: FlatGeobuf
FlatGeobuf is an optimized binary encoding for vectors based on Flatbuffers, that focuses on read/query performance by leveraging a Packed Hilbert R-Tree spatial index, that enables fast bounding box spatial filtering. FlatGeobuf performance has helped make it become one of the chosen formats for streaming spatial data through HTTP / „the cloud”.
2. Current Status through OGR¶
Since GDAL 3.1.0, FlatGeobuf access is available in MapServer through OGR
CONNECTIONTYPE OGR CONNECTION "name.fgb" DATA "layername" CONNECTIONOPTIONS "VERIFY_BUFFERS" "NO" # for maximum performance END
There is some performance overhead connecting through OGR. The following rudimentary benchmark shows some overhead for shapefile through OGR
Shapefile (native) 0.010s FlatGeobuf (OGR) 0.013s Shapefile (OGR) 0.023s GeoPackage (OGR) 0.042s SpatiaLite (OGR) 0.045s PostGIS (native) 0.053s GeoJSON (OGR) 0.089s environment: MapServer-main + GDAL 3.4.3 (MS4W/Windows)
3. Proposed enhancement¶
add FlatGeobuf as a native format for MapServer
CONNECTIONTYPE flatgeobuf DATA "name.fgb"
FlatGeobuf C++ implementation is small and doesn’t require any dependencies
Flatbuffers source will be included
initial implementation will be read access only
3.1 Backwards Compatibility¶
3.2 Performance Implications¶
As mentioned above, this should give a performance boost by reading direct from the FlatGeobuf file. More benchmarks will be included here during the testing of this Pull Request. For example, here is an updated benchmark run on 2022-05-16 (note that this is for world extents, with country labels) :
FlatGeobuf (native) 0.008s Shapefile (native) 0.010s FlatGeobuf (OGR) 0.013s Shapefile (OGR) 0.023s GeoPackage (OGR) 0.042s SpatiaLite (OGR) 0.045s PostGIS (native) 0.053s GeoJSON (OGR) 0.089s environment: MapServer-main + GDAL 3.4.3 (MS4W/Windows)
This will be read access only, initially. This means that tricks such as setting an OUTPUTFORMAT for direct FlatGeobuf output from MapServer, will only work through OGR, for now.
The VERIFY_BUFFERS driver option of FlatGeobuf is not implemented through this native driver (it is exposed through the OGR connection in MapServer, if need be). Specifically this means that currently the value of VERIFY_BUFFERS is equivalent to NO for this native driver, at the moment.
3.4 Security Concerns¶
There are no known security concerns at the moment.
The new format has been exposed through layerObj->setConnectionType().
4. Implementation Details¶
4.1 Affected files¶
new files will be stored in: /flatgeobuf/
included files will be stored in: /flatgeobuf/include/
mapflatgeobuf.c: main logic
mapfile.c: add new CONNECTIONTYPE
test has added in /msautotest/misc/flatgeobuf.map
MapScript test has been added in /msautotest/php/layerObjTest.php
4.2 Ticket Reference¶
pull request: https://github.com/MapServer/MapServer/pull/6520
5. Voting History¶
+1 from JeromeB, TomK, SethG, MikeS, LarsS (community vote), JeffM 0 from EvenR