geothings - raster

Serve PDF and DXF files using qgis-server

2025-05-30T00:00:00+00:00

You might be faced with the need to distribute “plan” files. For some businesses, paper maps (pdf) and CAD drawings (dxf) are still the primary source of information, and sometimes you can’t propose anything else. If you’re working with QGIS there are simple ways to serve those files without much to do, and all with opensource solutions using qgis and qgis-server.

Setup

You’ll need… Well, qgis and qgis-server, and that’s it. First install qgis-server using instructions for you environment, you might have a look at the official doc or this short tuto. While working on this article, I followed the containerized deployment instruction from the QGIS documentation. I’ll link a repository with the resources for you to follow along.

The next step is the preparation of your exports using qgis. Your projects will serve as configuration, where you’ll define symbologies, renderings, services, etc. We’ll show some possibilities to configure pdf and dxf exports.

Export PDF files

For simple cases it’s the easiest setup, you’ll just need to configure a layout in your project. Define a map that suits your taste (or better your objectives) and save everything in a location where your server has the right to read your project and the linked data. And that’s it for the most part. You can then have a look at a resulting map by querying an url of the like:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&FORMAT=application/pdf

If we divide the query we have:

http://localhost:8080/qgis-server the url of your qgis-server.
SERVICE=WMS tells the server to use its WMS service.
VERSION=1.3.0 version of the service.
REQUEST=GetPrint run the GetPrint request.
MAP=project.qgz path to the qgis project to read.
TEMPLATE=layout the name of a template defined within the project that describes a layout, where you’ll define your map.
CRS=3857 the coordinate system to use, usually the one defined in your project.
map0:extent=xmin,ymin,xmax,ymax the bounding box defining the extent to print. In the CRS system.
FORMAT=application/pdf specify the output format, here pdf.

Your browser will show you something like the following:

In this simple case, we could imagine a service where you define an extraction extent, which then will be passed to your printing service in the form of a mapx:extent parameter. You’d receive your pdf as a response.

Export PDF files: atlas version

You can create atlases in QGIS which allows you to produce maps using predefined extents. Your qgis-server also supports those and you can request a specific map from an atlas by using the ATLAS_PK parameter in you query.

If we create a layout_atlas in our project, using a grid as overlay (see the example project), we could then query our atlas pages using the following:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout_atlas
&CRS=EPSG:3857
&ATLAS_PK=1

Which would then send back something like:

Here you’d need to get the atlas elements that intersect your region of interest and pass their id as ATLAS_PK. A query like the following could get you this information:

curl -X POST -H "Content-Type: application/xml" -d @BODY.xml localhost:8080/qgis-server?SERVICE=WFS&VERSION=1.3.0&REQUEST=GetFeature&MAP=project.qgs

Where BODY.xml contains the following body:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wfs:GetFeature maxFeatures="5000" version="1.1.0" service="WFS" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd" xmlns:wfs="http://www.opengis.net/wfs" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <wfs:Query typeName="grid">
    <ogc:Filter>
      <ogc:Intersects>
        <ogc:PropertyName>geom</ogc:PropertyName>
          <Polygon xmlns="http://www.opengis.net/gml" srsName="EPSG:3857">
            <exterior>
              <LinearRing>
                <posList srsDimension="2">1471131.73 1257643.13 2471131.7 1257643.13 2471131.7 257643.1 1471131.7 257643.1 1471131.73 1257643.1</posList>
              </LinearRing>
            </exterior>
          </Polygon>
      </ogc:Intersects>
    </ogc:Filter>
  </wfs:Query>
</wfs:GetFeature>

Extracting the ids from the response you could then query you atlas on the server with:

http://localhost:8080/qgis-server/?atlas_pk=44,45,46,52,53,54&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetPrint&MAP=project.qgz&TEMPLATE=layout_atlas&CRS=EPSG:3857&FORMAT=application/pdf

TIP

You’ll need to allow your server to print multiple pages. This can be done in Project > Properties > QGIS Server > WMS > Maximum page number for atlas queries.

Export DXF

At this point you have everything you need to export dxf files. Make sure you expose the layers you want to export through WFS (Project > Properties > QGIS Server > WFS > Tick Publish for the layers you want). You can then get your dxf using something like:

localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetMap
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&LAYERS=point,line,polygon
&FORMAT=application/dxf
&FILE_NAME=output.dxf

TIP

For dxf exports, the symbology is a bit trickier to get right in my opinion. You might have to play with units to get a configuration and a rendering that suits your needs. Also, Polylines3D don’t support linetypes and weight, so you might need to use the FORCE_2D parameter.

To conclude

PDF and DXF are, after all, rather easy to create with a qgis-server. No need for any other piece of software, potentially expensive. Make sure to play with the QGIS Server properties of your project and layers within QGIS to see how it influences the exports. Again, all the material used in this article is accessible in this repo.

Geoservices with qgis-server

2024-03-22T00:00:00+00:00

While the geospatial world has its own particularities that sometimes make it difficult to get on-board or communicate with (see the spatial exception), we have to admit that some tools and standards developed are both amazing and essential. Geoservices are crucial for the diffusion of georeferenced data, and they drastically changed the way we apprehend our localization; when was the last time you asked for direction?

Being WMS, WMTS, WFS, WCS or any W*S, geoservices are everywhere and sometimes are a bit mysterious. But even without understanding the fundamentals of their definition, it’s rather easy to get started.

We’ll see how to install a (geo)server that will be responsible for serving our data, quickly prepare some data, and even see how you can use these services to update your data safely. We’ll be using qgis-server here, because it’s the most easy to use and the quickest way to get started.

Some quick context

You’ve probably heard about the Open Geospatial Consortium (OGC), but to make it really short, if you hear about a standard regarding geospatial, the OGC is probably behind. At least it’s the case for the geoservices we’ll be diving into here.

And what are those geoservices? Well most of the time when we talk about “geoservice” we’re referring to WMS, WMTS, WFS, WCS. They are a standardized way of accessing georeferenced data through the internet. The same way you reach a website or an API using an URL, you can access some geographic data if you know some URL (and have the tools that go with it). You can then use the data for your use-case: displaying a background map in your app, showing queryable data for an interactive webmap etc. The most common are:

Web Map Service (WMS) –> probably the initial service. Used to display an image as .jpg or .png (see it as a background map)
Web Map Tiled Service (WMTS) –> the more efficient WMS. It’s the same, but images are tiled to restricted the amount of data to be transferred to the client based on the display localization and the zoom level.
Web Feature Service (WFS) –> the vector service. You can work on the data as vector, request features on the map, and even add, modify or delete features.
OGC API-Feature (OGC-APIF) –> the more “web standard” WFS, also known as WFS3. Features can be accessed individually through an URL. The OGC states that

The API building blocks specified in this standard are consistent with the architecture of the Web.
Web Coverage Service (WCS) –> at the time of writing I’ve never used this standard so I won’t go into details, but from my reading it’s a service close to the WMS standard but providing more information than just an image, allowing more complex interactions.

There are other existing services: vector tiles, WSC, WMC etc. But I either never encountered them or they usually are implemented by other tools than the common geoservers.

Ok let’s get started

It’s nice and all, but how do we serve a W*S. Well for starters we need a server. Here we’ll be using qgis-server with nginx. The choice for qgis-server is motivated by the simplicity of use, you serve a qgis project. The choice for nginx, well the doc mentions it and it seems very lightweigth. I’m by no mean a system administrator so bear with me as I’ll read the doc with you.

Install and configure qgis-server and nginx

I’m on Ubuntu so I’ll be using apt, and the installation includes the use of systemd. I’ll also be working with the ubuntugis ppa.

First add the ppa:

add-apt-repository ppa:ubuntugis

Then install qgis-server:

apt install qgis-server

Install nginx if not already present:

apt install nginx

Configure qgis-server

We’ll then follow instructions from the QGIS documentation for a qgis-server installation using systemd and nginx. The method defines a socket unit and a service unit as follow. The sockets will be the communication channel between our http server (nginx) and our application server (qgis-server). Create the socket file /etc/systemd/system/qgis-server@.socket with the following content:

[Unit]
Description=QGIS Server Listen Socket (instance %i)

[Socket]
Accept=false
ListenStream=/var/run/qgis-server-%i.sock
SocketUser=www-data
SocketGroup=www-data
SocketMode=0600

[Install]
WantedBy=sockets.target

You can start these sockets with the following command:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.socket; done

Which will start 4 sockets, through which our http server will be able to pass requests to our qgis-server.

Now for the services. Services are managed by systemd and in our case are responsible for the execution of the qgis-server. Create the file /etc/systemd/system/qgis-server@.service and add:

[Unit]
Description=QGIS Server Service (instance %i)

[Service]
User=www-data
Group=www-data
StandardOutput=null
StandardError=journal
StandardInput=socket
ExecStart=/usr/lib/cgi-bin/qgis_mapserv.fcgi
EnvironmentFile=/etc/qgis-server/env

[Install]
WantedBy=multi-user.target

Notice that the server is configured using a env file, here /etc/qgis-server/env. Finally, in the same manner as the sockets, start the 4 services with the following:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.service; done

Services are depending on sockets, you’ll need to start the sockets first then the services. {: .prompt-warning }

Configure nginx

We’ll then configure nginx for our future qgis-server using fastcgi by creating the file /etc/nginx/sites-enabled/qgis-server and adding the following:

upstream qgis-server_backend {
   server unix:/var/run/qgis-server-1.sock;
   server unix:/var/run/qgis-server-2.sock;
   server unix:/var/run/qgis-server-3.sock;
   server unix:/var/run/qgis-server-4.sock;
}

server {
   listen 8080;
   access_log  /var/log/nginx/qgis-server/access.log;
   error_log  /var/log/nginx/qgis-server/error.log;

   location /qgis-server {
       gzip off;
       include fastcgi_params;
       fastcgi_pass qgis-server_backend;
   }
}

This configuration will then be included by the nginx.conf file using the include /etc/nginx/sites-enabled/*; statement.

Create a QGIS project and configure it

Now it’s time to decide what we want to expose. I have some OSM data in a PostgreSQL database (see Serving vector tiles with Martin) so I’ll use that.

Create a project, add some data and apply the desired style. A quick example is shown below.

Data with some style

Once you’re satisfied with the look of your data, you’ll need to configure the project so it will tell qgis-server to proceed with the geoservice you want to serve.

Go to Project -> Properties -> QGIS Server and complete the configuration as you wish.
Under Service Capabilities, check the Enable Service Capabilities box.
Select the different services you want to enable under the different tabs.
Save your project under a folder accessible by the server user (www-data). If you’re using file data in your project, such as a geopackage, save them in the same place.

If you setup a WFS, the tab should look like something like below.

Data with some style

WARNING

Make sure that your tables have a primary key. Otherwise the update and delete operations won’t be possible.

WARNING

And also, be sure that you don’t have special characters in you column or table names. It messes up the WFS 1.0 and 1.1 requests.

You can then edit the env file with the corresponding environment variables. For me it looks like:

QGIS_PROJECT_FILE=/etc/qgis-server/my_project.qgz
QGIS_SERVER_LOG_STDERR=1
QGIS_SERVER_LOG_LEVEL=3

I use PostgreSQL data and have had troubles getting the server to reach the data, even on localhost but finally managed to have everything working with the following steps:

Make the project read the data using a PostgreSQL service (create a pg_service.conf if not existing).
Copy the pg_service.conf file somewhere accessible for the server.
Edit /etc/nginx/fastcgi_params and add fastcgi_param PGSERVICEFILE /path/to/service/pg_service.conf;.
Restart everything.
Bonus: touch the project file.

If everything works correctly, you should be able to reach localhost:8080/qgis-server/qgis_mapserv.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities for WMS capabilities or http://localhost:8080/qgis-server/wfs3/ for the OGC-APIF description and navigate the different endpoints defined in the OGC-APIF standard.

You should now be able to also call these URL from QGIS using the correct url. Try to add a WFS/OGC-APIF using the http://localhost:8080/qgis-server/wfs3/ url (qgis-server in this example, but you can setup your server with a different url).

Going further

We have the basics setup, now what can we do? Let’s start by alter data.

Updating data through a WFS

If at this point you have everything working, there shouldn’t be much to do to update data through your newly build WFS. Working with the OGC-APIF you’ll be able to add, delete and update data using URL like you would do with any REST API.

You’ll find the description of your API at the URL http://localhost:8080/qgis-server/wfs3/api (for this particular setup) and it should look something like this:

OpenAPI description of an OGC-APIF

This gives you an idea of the available endpoints to manipulate data.

To try this, let’s try to GET one of our feature. You can access a single feature through a collection and an item. For me, the URL http://localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736 will point to the Lausanne train station:

Getting an item using its dedicated URL

And the same can be done using curl or wget or any network utility you wish:

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736" | jq .

will print something like:

{
  "bbox": [
    6.627779,
    46.51671,
    6.630578,
    46.517298
  ],
  "geometry": {
    "coordinates": [
      [
        [
          6.627779,
          46.517099
        ],
        [
          6.628225,
          46.517038
        ],
        [
          6.628344,
          46.51702
        ],
[...]
  }
}

TIP

The piping into jq (| jq .) is just for printing purpose, it doesn’t participate in the query.

Now let’s delete this object:

curl --request "DELETE" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

And that’s it!

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

now returns:

[
  {
    "code": "Internal server error",
    "description": "Invalid feature [2102736]"
  }
]

And you can do exactly the same using QGIS. Add your WFS as a layer and work on it like any vector data: add, update ,delete features and then save.

TIP

I intended to use a PATCH request to update a single attribute, but couldn’t make it work. QGIS seems to systematically call a POST request, even for a single attribute change. If you now how to work with PATCH in this environment let me know.

Protect our service with authentication

Accessing information through an URL means that many client might have access to it as well. It’s important to keep in mind that authentication would be necessary if you expect to update information through this API.

Again, I’m not a system administration or a security expert, but I’ll try to show how to setup a very basic authentication directly using nginx. This is not how you secure an API and it won’t replace a modern and secure authentication setup.

I’ll follow the nginx documention for that.

First install apache2-utils if your on a Debian based distribution:

sudo apt install apache2-utils

Create a password file and a user:

sudo htpasswd -c /etc/apache2/.htpasswd user1

In the nginx configuration, under the location of your API, add a name and specify that you’ll be using a auth_basic_user_file.

location /qgis-server {
    [...]
    auth_basic           "Protected Area";
    auth_basic_user_file /etc/apache2/.htpasswd;
}

Restart nginx for good measure and that it, try to reach your WFS using an URL or with QGIS and you should see a connection popup like this:

The route now needs basic authentication to be reached

Conclusion

Services are a powerful tool to be aware of when working with geographical data. They allow to efficiently share them at a larger scale through the web and even, as we saw, update them. There are many ways to build these services, and qgis-server is one solution among others, but it has the advantage to work (almost) out of the box with potential QGIS projects that you could already have: a few tweaks and you can be going.

The possibility to give access to your data through an URL makes sharing them so easy; it reduces the amount of manipulation and tools required, you don’t need to maintain a data extractor or repositories. Just some data and a server.

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

I’ve been trying to learn Rust lately. For some reason there is something that I find very appealing in this language. From all I could read, it’s very efficient in resources and memory, it’s fast, and it doesn’t let you mess around with memory. Regarding performances, it places itself next to C and C++, and when looking into the world of language performances, I learned that Python was kind of slow, around 100x slower according to some benchmarks. Obviously this value in itself doesn’t mean much, and many more aspects of a language must be taken account when choosing one for the task, but this information kind of planted its seed. It might be because I have a problem with inefficient and overly complicated process, but I find the idea to work with a system language very elegant.

But we’re not here to talk about the Rust language in itself. We’ll see how to read and write raster data using the Rust binding for GDAL. Also, this is about how I tried to compare the Rust and Python bindings for GDAL and how I learned that it doesn’t change a thing in terms of performance.

Wait what?

Well I started by thinking “Oh I can’t wait to see how faster GDAL in Rust is compared to Python”. So I started by reading a raster file and print some basic information. I use a band from a Sentinel-2 image.

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

I’m using a lot of .unwrap(), which in very short will cause the code to panic if a problem occurs, this is for a quick and dirty test. Within a “production” environment you would handle errors and stuff.

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

We will be using the Create() approach rather than the CreateCopy() (see the nuance in the doc). Geospatialized rasters come with series of metadata and overviews. The CreateCopy() automatically takes those from the source dataset and copy them into the destination. While this obviously simplifies the work if we don’t need to bring any change to these information, using Create() helps to better understand the structure of a raster. Also I have a quite important performance difference between the two and I’ve found Create() more efficient for my use case.

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

Both codes should create a new raster file that resembles our source, but in GeoTiff. And that’s it, you can convert a raster file into any format. We’re just scratching the surface, you can subset data, reproject, transpose it if you want etc.

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

So from my understanding, working with Rust or Python or any other language bindings is just working with the API that let us call the underlying code, for GDAL written in C and C++. Thus the overhead that we generate is only the interaction with the library, not during the process itself, since it’s running C/C++ code. The difference that we observe is probably the same that we were observing when we just opened a file.

Conclusion

GDAL raster processing is very similar in Rust and Python. When working with its bindings, it doesn’t change much which language you use to perform the GDAL operations in themself, since in any case you’re running C/C++ code from your program. But everything that is around need to be considered when choosing the right tool. If your program needs multi-threading then you’l probably chose Rust, if you need quick development without much optimization then Python might be a best fit. But it’s comfortable to see that the GDAL processing time won’t change significantly.

I’m not at all trying to say that Rust is not worth it, I’m excited to go further into it and work on projects, but it remains a tool and there should be no injunction to use it if it doesn’t bring much to the table for your use case.

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

Wait what?

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

Conclusion

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.

A little bit of GDAL/OGR appreciation

2024-01-19T00:00:00+00:00

If there had to be one single tool to work with spatial data, I think GDAL/OGR would be the first name to come to mind. Its usages range from raster to vector, it can be used as command line, imported as a library in many languages through bindings developed by the community, many applications provide UI to parameterize and run GDAL etc. Its versatility makes it an amazing tool to work with spatial data in a lot of different contexts, but it’s sometimes underestimated or even misunderstood.

We will quickly introduce GDAL/OGR, then give some examples of usage to help for some quick starts.

What is GDAL/OGR?

GDAL/OGR (Geospatial Data Abstraction Library/OpenGIS Simple Features Reference Implementation) is a library for working with raster and vector geospatial data formats, coded mainly in C++, C and Python. The first release dates from 2000, and the project has now more than 450 contributors at the time of writing. This is a very active project with regular releases.

The project is composed of two parts: the raster processing library GDAL and the feature processing library OGR, both merged during the release of GDAL 2.0. When working with raster data you will use programs prefixed with gdal (gdal_translate, gdalwarp etc.) while vector data will use ogr prefixed scripts (ogrinfo, ogr2ogr etc.). When talking aboutt GDAL, both libraries are included, and we’ll do the same from here.

As its name suggests, GDAL abstracts data in order to standardize its way of interacting with it. The driver creates the interface between the data source and the tools, allowing to focus on the logic of the process rather than how to read or write the data. This makes it a very extensible project; including a new format of data source requires to write a new driver, nothing more. You could see that the list of supported formats is quite something, either raster or vector.

GDAL describes itself as a translator, meaning that data come from a source and goes to its destination. Translations can occur at different moment during the process, during reading or by the script called. These translations can occur on multiple axis as well, we could for example:

Extract a subset of source data depending on a region of interest.
Extract a subset of source data based on an attribute.
Reproject data into a different coordinate system.
Merge a series of data files into a single one.
etc.

Source unknown, if it’s your image and you want to be cited please leave a comment

As an open source project with a permissive MIT license, GDAL is used by many projects, open source or proprietary, to process geospatial data. If you’ve used QGIS, ArcGIS, FME or PostGIS, you’ve used GDAL under the hood.

Why would I use it?

Weirdly, even though GDAL is basically everywhere there is geospatial data to be moved, it’s not so much acknowledged by GIS people, or at least it’s not really used on its own. If you read this, you definitely use GDAL, but maybe without noticing.

Ok but why would I complicate my life with some command lines when the same tool is being used by a software that provides me a user interface easier to use?

While it’s much intuitive to explore geospatial data through a user interface such as QGIS, where you can interactively load your data, chose the process you want to apply, its parameters, and visualize the results, things get a bit more complicated once you’ve done the exploratory part and want to reapply your newly defined process to new data. Repeating operations becomes quickly tedious and error-prone. You could use a tool such as the Model Designer from QGIS (amazing little tool, check it if you repeat a lot of operations in QGIS), but how do you schedule your process to be started when you’re not able to launch it manually?

If you use a heavy client ETL such as FME, try once to compare the difference in speed between a simple table transfer done by launching a FME workspace (using a command line) and a simple ogr2ogr command, there is at least an order of magnitude of difference.

It’s about using a tool designed for a purpose as efficiently as possible.

How to install?

We’ll assume you have no installation of GDAL yet, but just to make sure you can try:

gdalinfo --version

which gives:

GDAL 3.4.1, released 2021/12/27

If it’s not installed you can refer to the installation instructions from GDAL website.

For example, I’m working on Ubuntu 22.04, for which the GDAL package is the 3.4.1. The latest version can be installed by downloading the latest release, but I usually stick with the provided version for my platform, unless I need a specific update.

apt update
apt install gdal-bin

And you should be good!

WARNING

You might already have GDAL installed through another software like QGIS, be careful during the installation, multiple installations might interfere and create warnings or errors. On Windows you can use the OSGeo Shell to access any tool from the OSGeo ecosystem.

And how do we use it?

It’s easy, you just need to be comfortable with command lines. If you’ve mostly worked with user interfaces it might look scary, but it really is not complicated.

Access metadata

The first thing we usually do with data is to have some information about it. Let’s try to see the metadata of a random .jpeg:

gdalinfo gdal.jpeg

Driver: JPEG/JPEG JFIF
Files: gdal.jpeg
Size is 711, 882
Metadata:
  EXIF_Orientation=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  882.0)
Upper Right (  711.0,    0.0)
Lower Right (  711.0,  882.0)
Center      (  355.5,  441.0)
Band 1 Block=711x1 Type=Byte, ColorInterp=Red
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=711x1 Type=Byte, ColorInterp=Green
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=711x1 Type=Byte, ColorInterp=Blue
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG

Easy right? And it’s not even georeferenced, it’s a .jpeg, a driver exists, it opens.

We can try the same with some vector data. We’ll have a look at what is going on in one of the PostgreSQL database hosted on my computer.

ogrinfo -ro PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.
1: planet_osm_line (Line String)
2: planet_osm_polygon
3: planet_osm_roads (Line String)
4: planet_osm_point (Point)
5: natural_polygons

And as we can see I have 5 layers in my database. Notice how we used the ogrinfo command for questioning vector data. Here ogrinfo (like gdalinfo) requires a dataset source as input, the syntax will depend on each driver, make sure to have a look at the documentation. For a PostgreSQL connection, the line could have looked like:

ogrinfo -ro PG:"dbname='osm_suisse' host='localhost' port='5432' user='x' password='y'"

We can extend the information that we want to see for each layer by adding the -al flag to the line. The option -so will summarize the output.

ogrinfo -ro -so -al PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.

Layer name: planet_osm_line
Geometry: Line String
Feature Count: 2137192
Extent: (626211.331305, 5706647.213604) - (1253686.695428, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Geometry Column = way
osm_id: Integer64 (0.0)
access: String (0.0)
addr:housename: String (0.0)
addr:housenumber: String (0.0)
[...]

And now we have the full detail of our layers; name, type, feature count, etc.

Basic translations

Now that we know what were working, with we can try to move some things around. Let’s start by exporting our vector data into a geopackage.

ogr2ogr -f GPKG exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

Here we’re exporting the table planet_osm_point into the exported_data.gpkg geopackage. The format can be recognized by the extension of the output file, but it’s safer to ensure the format by passing the argument -f. Again, each driver documentation will have its driver name to be used.

Add a filter

We can export a subset of our data, based on an attribute for example. Let’s say we want to export the points where the attribute addr:housenumber is not null, and we want to save it in another table in our newly created geopackage. We’d do something like:

ogr2ogr -update -f GPKG -where '"addr:housenumber" is not null' -nln only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

We want to update our dataset (our geopackage) and not create a new one, so we use the -update option. The syntax of the -where parameter is a bit convoluted here since there is a special character in the column name, but with a simple name you’d just need a pair of quotes -where "housenumber" is not null".

Let’s have a look at our geopackage:

ogrinfo -ro -al -so exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: planet_osm_point
Geometry: Point
Feature Count: 2146246
Extent: (626415.435591, 5736097.599850) - (1189428.532375, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
[...]

Layer name: only_housenumbers
Geometry: Point
Feature Count: 366561
Extent: (663719.309401, 5751784.044673) - (1167785.106794, 6072459.331387)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
[...]

The new layer has been added to the geopackage. Has the query been respected?

ogrinfo -ro -al -so -sql 'select * from only_housenumbers where "addr:housenumber" is null' exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: SELECT
Geometry: Point
Feature Count: 0
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
[...]

Yes! No data without addr:housenumber.

TIP

The -sql option is another way of filtering data, it allows you to pass a sql query in any of the sql flavor supported by the driver, you can even save your queries in a text file and pass the file to the -sql parameter as -sql @path/to/my_file.

Geometric operations?

The -sql parameter can also be used to apply transformations to the data. We’ll request the same points, but this time we’ll create a buffer around them.

ogr2ogr -update -f GPKG -sql 'select ST_Buffer(way, 50) from planet_osm_point where "addr:housenumber" is not null' -nln buffered_only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

And that’s it, we have our buffer around the points of interest. Here GDAL makes use of the internal capabilities of PostgreSQL and PostGIS by passing directly the request to the database, but the same command using the only_housenumbers layer from our geopackage would also be valid.

ogr2ogr -update -f GPKG -sql "select ST_Buffer(way, 50) from only_housenumbers" -nln buffered_only_housenumbers exported_data.gpkg exported_data.gpkg

TIP

In the case of PostgreSQL/PostGIS you can write the -sql parameter like you would write your queries directly into the database. They are passed directly to the database by GDAL.

This should give you a small idea of the things you can do with GDAL and how to start experimenting.

Final words

Well that was a lot more than expected. But GDAL really is a very strong and important tool. Diving into the details of all the available options, both for the command AND the drivers, is not a small task, but understanding how to navigate around and find the information that you need to process your data as you wish can go a long way. Manual and repeated tasks can be replaced to help to focus on more pertinent work, process time can be (by experience) reduced drastically depending on the tools you’re initially using. And if you’re working on data hosted on a server without graphic interface, well you have a tool to get the info you need!

Obviously GDAL alone can’t do everything and has its imperfections; I’ve really struggled with installations in the past (Python binding might have bested me at some point), and the number of parameters is a bit overwhelming. But once you have it working it does the job, and really well.

If you can, use GDAL.