geothings - gdal

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

I’ve been trying to learn Rust lately. For some reason there is something that I find very appealing in this language. From all I could read, it’s very efficient in resources and memory, it’s fast, and it doesn’t let you mess around with memory. Regarding performances, it places itself next to C and C++, and when looking into the world of language performances, I learned that Python was kind of slow, around 100x slower according to some benchmarks. Obviously this value in itself doesn’t mean much, and many more aspects of a language must be taken account when choosing one for the task, but this information kind of planted its seed. It might be because I have a problem with inefficient and overly complicated process, but I find the idea to work with a system language very elegant.

But we’re not here to talk about the Rust language in itself. We’ll see how to read and write raster data using the Rust binding for GDAL. Also, this is about how I tried to compare the Rust and Python bindings for GDAL and how I learned that it doesn’t change a thing in terms of performance.

Wait what?

Well I started by thinking “Oh I can’t wait to see how faster GDAL in Rust is compared to Python”. So I started by reading a raster file and print some basic information. I use a band from a Sentinel-2 image.

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

I’m using a lot of .unwrap(), which in very short will cause the code to panic if a problem occurs, this is for a quick and dirty test. Within a “production” environment you would handle errors and stuff.

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

We will be using the Create() approach rather than the CreateCopy() (see the nuance in the doc). Geospatialized rasters come with series of metadata and overviews. The CreateCopy() automatically takes those from the source dataset and copy them into the destination. While this obviously simplifies the work if we don’t need to bring any change to these information, using Create() helps to better understand the structure of a raster. Also I have a quite important performance difference between the two and I’ve found Create() more efficient for my use case.

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

Both codes should create a new raster file that resembles our source, but in GeoTiff. And that’s it, you can convert a raster file into any format. We’re just scratching the surface, you can subset data, reproject, transpose it if you want etc.

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

So from my understanding, working with Rust or Python or any other language bindings is just working with the API that let us call the underlying code, for GDAL written in C and C++. Thus the overhead that we generate is only the interaction with the library, not during the process itself, since it’s running C/C++ code. The difference that we observe is probably the same that we were observing when we just opened a file.

Conclusion

GDAL raster processing is very similar in Rust and Python. When working with its bindings, it doesn’t change much which language you use to perform the GDAL operations in themself, since in any case you’re running C/C++ code from your program. But everything that is around need to be considered when choosing the right tool. If your program needs multi-threading then you’l probably chose Rust, if you need quick development without much optimization then Python might be a best fit. But it’s comfortable to see that the GDAL processing time won’t change significantly.

I’m not at all trying to say that Rust is not worth it, I’m excited to go further into it and work on projects, but it remains a tool and there should be no injunction to use it if it doesn’t bring much to the table for your use case.

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.

PMTiles exploration

2024-02-02T00:00:00+00:00

It’s a technology that I came across rather recently. Tile generation when maintaining a geographic oriented website can be a hassle: it can take time, generates a lot of files, and can be a critical process. PMTiles might have the potential to facilitate some of this work and we’ll try to see how.

What are PMTiles?

In itself, PMTiles is a specification for building single-file tile pyramids. It’s open source and part of a larger project called Protomaps, and it aims at mapping the World, just that. The major difference with the kind of tiles that I know is that they are embedded as a single file, like the MBTile format, which make the interaction very intuitive, I mean we all get the idea of what a file is. But what does it bring compared to MBTiles? Well from the Protomaps website:

A MBTiles file needs to be accessed as a SQLite database on disk, so requires a running server and disk storage for cloud hosting. PMTiles is designed to be cloud-native and readable in fine-grained parts from remote storage like Amazon S3.

It aims at simplifying the manipulation of tiles in a cloud native approach, for worldwide coverage.

How to create PMTiles?

The Protomaps website gives us some tools to create these tiles. Let’s try some possibilities.

Generate PMTiles using GDAL

Our Swiss Army Knife is back, starting version 3.8.0 GDAL has native support for PMTiles, so let’s try that.

First check the GDAL version you’re using, you’ll want 3.8.0 or higher:

gdalinfo --version
--> GDAL 3.8.3, released 2024/01/04 (debug build)

Let’s start from an OSM .pbf file (see Serving vector tiles with Martin). To convert this file into a PMTiles file you can enter:

ogr2ogr -f PMTiles out.pmtiles switzerland-latest.osm.pbf

The process might take some time if you’re working on a large dataset. By default the MAXZOOM is 5, which is rather low, so the run shouldn’t take to long, but if you start to increase the parameter to around 15, which allows to represent data from much closer, processing time increases.

But once the command is done, well that’s it!

Alternatively we can use tippecanoe to achieve the same result.

Generate PMTiles using tippecanoe

WARNING

Under construction

Visualize PMTiles

The easiest way is to go to the PMTiles Viewer Page and upload your newly created PMTiles file.

Obviously you can add it to your custom project. The easiest way of doing is by using the Leaflet plugin protomaps-leaflet.

Starting from complete scratch you can create an .html file with the following content:

<!DOCTYPE html>
<html>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"></script>
    <script src="https://unpkg.com/protomaps-leaflet@latest/dist/protomaps-leaflet.min.js"></script>
    <style>
        body,
        #map {
            height: 100vh;
            margin: 0px;
        }
    </style>
</head>

<body>

    <h2>Hey I'm a custom page</h2>
    <p>Let's add some Leaflet and PMTiles.</p>
    <div id="map"></div>

    <script>
        let PAINT_RULES = [
            {
                dataLayer: "multipolygons", //here we render only one layer, check your layers names 
                symbolizer: new protomapsL.PolygonSymbolizer({ fill: "steelblue" })
            }
        ];

        const map = L.map('map').setView([46.5, 6.6], 13); //chose a view that suits you

        var osm = L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png', {
            maxZoom: 19,
            attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
        }).addTo(map);

        var layer = protomapsL.leafletLayer(
            {
                url: 'http://127.0.0.1:8080/your_pmtile_file.pmtiles',
                paint_rules: PAINT_RULES
            }).addTo(map);

    </script>

</body>
</html>

For testing purposes you might have to serve you PMTiles through http. To do so you can do as the Protomaps website suggests:

npm install -g http-server
# Then from the folder containing your PMTiles.
http-server . --cors

Open your .html file on the browser of your choice and you should see the map with your PMTiles displayed.

When should PMTiles be used?

Again, the documentation on Protomaps website is quite complete and gives us some insights.

Cloud hosting

PMTiles are stored as a single file, and compression is so efficient that the size is ridiculous compared to classic png tiles. Number of requests necessary is drastically reduced and thus the cost of operation as well.

Static data

PMTiles don’t support localized updates so each time you’ll need to apply an update, the whole file needs to be rewritten. If your data don’t change that much PMTiles are a wise choice, but If you need to have a more dynamic rendering the documentation recommends PostGIS with the ST_As_MVT() function.

Simplicity

Here it’s more like a feeling about the philosophy of the specification. You have a single file, accessible without introduced complexity, directly from a library like Leaflet, which itself has simplicity as one of its core objectives. It looks like one of those project that allows you to make complicated-looking things so easily. I guess it’s a sign of a good project.

Final words

Well it was rather easy to discover. I mean we create a file, we basically load it within a project and we’re good to go. Obviously there are some customizations to do with styling and labels etc. But once you have the starting point and your tools, it takes minutes to create what you desire.

Here we’re just scratching the surface, remains to explore filtering, styling etc. But the concept in itself and the unreasonable performance of the compression make it very interesting if you’re in the situation where PMTiles could give you an advantage; the file created for OSM data Switzerland, with zooms from 0 to 14, is around 1GB, it costs pennies to upload and host on cloud services.

It’s a project to support and look for, I guess?

A little bit of GDAL/OGR appreciation

2024-01-19T00:00:00+00:00

If there had to be one single tool to work with spatial data, I think GDAL/OGR would be the first name to come to mind. Its usages range from raster to vector, it can be used as command line, imported as a library in many languages through bindings developed by the community, many applications provide UI to parameterize and run GDAL etc. Its versatility makes it an amazing tool to work with spatial data in a lot of different contexts, but it’s sometimes underestimated or even misunderstood.

We will quickly introduce GDAL/OGR, then give some examples of usage to help for some quick starts.

What is GDAL/OGR?

GDAL/OGR (Geospatial Data Abstraction Library/OpenGIS Simple Features Reference Implementation) is a library for working with raster and vector geospatial data formats, coded mainly in C++, C and Python. The first release dates from 2000, and the project has now more than 450 contributors at the time of writing. This is a very active project with regular releases.

The project is composed of two parts: the raster processing library GDAL and the feature processing library OGR, both merged during the release of GDAL 2.0. When working with raster data you will use programs prefixed with gdal (gdal_translate, gdalwarp etc.) while vector data will use ogr prefixed scripts (ogrinfo, ogr2ogr etc.). When talking aboutt GDAL, both libraries are included, and we’ll do the same from here.

As its name suggests, GDAL abstracts data in order to standardize its way of interacting with it. The driver creates the interface between the data source and the tools, allowing to focus on the logic of the process rather than how to read or write the data. This makes it a very extensible project; including a new format of data source requires to write a new driver, nothing more. You could see that the list of supported formats is quite something, either raster or vector.

GDAL describes itself as a translator, meaning that data come from a source and goes to its destination. Translations can occur at different moment during the process, during reading or by the script called. These translations can occur on multiple axis as well, we could for example:

Extract a subset of source data depending on a region of interest.
Extract a subset of source data based on an attribute.
Reproject data into a different coordinate system.
Merge a series of data files into a single one.
etc.

Source unknown, if it’s your image and you want to be cited please leave a comment

As an open source project with a permissive MIT license, GDAL is used by many projects, open source or proprietary, to process geospatial data. If you’ve used QGIS, ArcGIS, FME or PostGIS, you’ve used GDAL under the hood.

Why would I use it?

Weirdly, even though GDAL is basically everywhere there is geospatial data to be moved, it’s not so much acknowledged by GIS people, or at least it’s not really used on its own. If you read this, you definitely use GDAL, but maybe without noticing.

Ok but why would I complicate my life with some command lines when the same tool is being used by a software that provides me a user interface easier to use?

While it’s much intuitive to explore geospatial data through a user interface such as QGIS, where you can interactively load your data, chose the process you want to apply, its parameters, and visualize the results, things get a bit more complicated once you’ve done the exploratory part and want to reapply your newly defined process to new data. Repeating operations becomes quickly tedious and error-prone. You could use a tool such as the Model Designer from QGIS (amazing little tool, check it if you repeat a lot of operations in QGIS), but how do you schedule your process to be started when you’re not able to launch it manually?

If you use a heavy client ETL such as FME, try once to compare the difference in speed between a simple table transfer done by launching a FME workspace (using a command line) and a simple ogr2ogr command, there is at least an order of magnitude of difference.

It’s about using a tool designed for a purpose as efficiently as possible.

How to install?

We’ll assume you have no installation of GDAL yet, but just to make sure you can try:

gdalinfo --version

which gives:

GDAL 3.4.1, released 2021/12/27

If it’s not installed you can refer to the installation instructions from GDAL website.

For example, I’m working on Ubuntu 22.04, for which the GDAL package is the 3.4.1. The latest version can be installed by downloading the latest release, but I usually stick with the provided version for my platform, unless I need a specific update.

apt update
apt install gdal-bin

And you should be good!

WARNING

You might already have GDAL installed through another software like QGIS, be careful during the installation, multiple installations might interfere and create warnings or errors. On Windows you can use the OSGeo Shell to access any tool from the OSGeo ecosystem.

And how do we use it?

It’s easy, you just need to be comfortable with command lines. If you’ve mostly worked with user interfaces it might look scary, but it really is not complicated.

Access metadata

The first thing we usually do with data is to have some information about it. Let’s try to see the metadata of a random .jpeg:

gdalinfo gdal.jpeg

Driver: JPEG/JPEG JFIF
Files: gdal.jpeg
Size is 711, 882
Metadata:
  EXIF_Orientation=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  882.0)
Upper Right (  711.0,    0.0)
Lower Right (  711.0,  882.0)
Center      (  355.5,  441.0)
Band 1 Block=711x1 Type=Byte, ColorInterp=Red
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=711x1 Type=Byte, ColorInterp=Green
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=711x1 Type=Byte, ColorInterp=Blue
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG

Easy right? And it’s not even georeferenced, it’s a .jpeg, a driver exists, it opens.

We can try the same with some vector data. We’ll have a look at what is going on in one of the PostgreSQL database hosted on my computer.

ogrinfo -ro PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.
1: planet_osm_line (Line String)
2: planet_osm_polygon
3: planet_osm_roads (Line String)
4: planet_osm_point (Point)
5: natural_polygons

And as we can see I have 5 layers in my database. Notice how we used the ogrinfo command for questioning vector data. Here ogrinfo (like gdalinfo) requires a dataset source as input, the syntax will depend on each driver, make sure to have a look at the documentation. For a PostgreSQL connection, the line could have looked like:

ogrinfo -ro PG:"dbname='osm_suisse' host='localhost' port='5432' user='x' password='y'"

We can extend the information that we want to see for each layer by adding the -al flag to the line. The option -so will summarize the output.

ogrinfo -ro -so -al PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.

Layer name: planet_osm_line
Geometry: Line String
Feature Count: 2137192
Extent: (626211.331305, 5706647.213604) - (1253686.695428, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Geometry Column = way
osm_id: Integer64 (0.0)
access: String (0.0)
addr:housename: String (0.0)
addr:housenumber: String (0.0)
[...]

And now we have the full detail of our layers; name, type, feature count, etc.

Basic translations

Now that we know what were working, with we can try to move some things around. Let’s start by exporting our vector data into a geopackage.

ogr2ogr -f GPKG exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

Here we’re exporting the table planet_osm_point into the exported_data.gpkg geopackage. The format can be recognized by the extension of the output file, but it’s safer to ensure the format by passing the argument -f. Again, each driver documentation will have its driver name to be used.

Add a filter

We can export a subset of our data, based on an attribute for example. Let’s say we want to export the points where the attribute addr:housenumber is not null, and we want to save it in another table in our newly created geopackage. We’d do something like:

ogr2ogr -update -f GPKG -where '"addr:housenumber" is not null' -nln only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

We want to update our dataset (our geopackage) and not create a new one, so we use the -update option. The syntax of the -where parameter is a bit convoluted here since there is a special character in the column name, but with a simple name you’d just need a pair of quotes -where "housenumber" is not null".

Let’s have a look at our geopackage:

ogrinfo -ro -al -so exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: planet_osm_point
Geometry: Point
Feature Count: 2146246
Extent: (626415.435591, 5736097.599850) - (1189428.532375, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
[...]

Layer name: only_housenumbers
Geometry: Point
Feature Count: 366561
Extent: (663719.309401, 5751784.044673) - (1167785.106794, 6072459.331387)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
[...]

The new layer has been added to the geopackage. Has the query been respected?

ogrinfo -ro -al -so -sql 'select * from only_housenumbers where "addr:housenumber" is null' exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: SELECT
Geometry: Point
Feature Count: 0
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
[...]

Yes! No data without addr:housenumber.

TIP

The -sql option is another way of filtering data, it allows you to pass a sql query in any of the sql flavor supported by the driver, you can even save your queries in a text file and pass the file to the -sql parameter as -sql @path/to/my_file.

Geometric operations?

The -sql parameter can also be used to apply transformations to the data. We’ll request the same points, but this time we’ll create a buffer around them.

ogr2ogr -update -f GPKG -sql 'select ST_Buffer(way, 50) from planet_osm_point where "addr:housenumber" is not null' -nln buffered_only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

And that’s it, we have our buffer around the points of interest. Here GDAL makes use of the internal capabilities of PostgreSQL and PostGIS by passing directly the request to the database, but the same command using the only_housenumbers layer from our geopackage would also be valid.

ogr2ogr -update -f GPKG -sql "select ST_Buffer(way, 50) from only_housenumbers" -nln buffered_only_housenumbers exported_data.gpkg exported_data.gpkg

TIP

In the case of PostgreSQL/PostGIS you can write the -sql parameter like you would write your queries directly into the database. They are passed directly to the database by GDAL.

This should give you a small idea of the things you can do with GDAL and how to start experimenting.

Final words

Well that was a lot more than expected. But GDAL really is a very strong and important tool. Diving into the details of all the available options, both for the command AND the drivers, is not a small task, but understanding how to navigate around and find the information that you need to process your data as you wish can go a long way. Manual and repeated tasks can be replaced to help to focus on more pertinent work, process time can be (by experience) reduced drastically depending on the tools you’re initially using. And if you’re working on data hosted on a server without graphic interface, well you have a tool to get the info you need!

Obviously GDAL alone can’t do everything and has its imperfections; I’ve really struggled with installations in the past (Python binding might have bested me at some point), and the number of parameters is a bit overwhelming. But once you have it working it does the job, and really well.

If you can, use GDAL.