geothings - opensource

Serve PDF and DXF files using qgis-server

2025-05-30T00:00:00+00:00

You might be faced with the need to distribute “plan” files. For some businesses, paper maps (pdf) and CAD drawings (dxf) are still the primary source of information, and sometimes you can’t propose anything else. If you’re working with QGIS there are simple ways to serve those files without much to do, and all with opensource solutions using qgis and qgis-server.

Setup

You’ll need… Well, qgis and qgis-server, and that’s it. First install qgis-server using instructions for you environment, you might have a look at the official doc or this short tuto. While working on this article, I followed the containerized deployment instruction from the QGIS documentation. I’ll link a repository with the resources for you to follow along.

The next step is the preparation of your exports using qgis. Your projects will serve as configuration, where you’ll define symbologies, renderings, services, etc. We’ll show some possibilities to configure pdf and dxf exports.

Export PDF files

For simple cases it’s the easiest setup, you’ll just need to configure a layout in your project. Define a map that suits your taste (or better your objectives) and save everything in a location where your server has the right to read your project and the linked data. And that’s it for the most part. You can then have a look at a resulting map by querying an url of the like:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&FORMAT=application/pdf

If we divide the query we have:

http://localhost:8080/qgis-server the url of your qgis-server.
SERVICE=WMS tells the server to use its WMS service.
VERSION=1.3.0 version of the service.
REQUEST=GetPrint run the GetPrint request.
MAP=project.qgz path to the qgis project to read.
TEMPLATE=layout the name of a template defined within the project that describes a layout, where you’ll define your map.
CRS=3857 the coordinate system to use, usually the one defined in your project.
map0:extent=xmin,ymin,xmax,ymax the bounding box defining the extent to print. In the CRS system.
FORMAT=application/pdf specify the output format, here pdf.

Your browser will show you something like the following:

In this simple case, we could imagine a service where you define an extraction extent, which then will be passed to your printing service in the form of a mapx:extent parameter. You’d receive your pdf as a response.

Export PDF files: atlas version

You can create atlases in QGIS which allows you to produce maps using predefined extents. Your qgis-server also supports those and you can request a specific map from an atlas by using the ATLAS_PK parameter in you query.

If we create a layout_atlas in our project, using a grid as overlay (see the example project), we could then query our atlas pages using the following:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout_atlas
&CRS=EPSG:3857
&ATLAS_PK=1

Which would then send back something like:

Here you’d need to get the atlas elements that intersect your region of interest and pass their id as ATLAS_PK. A query like the following could get you this information:

curl -X POST -H "Content-Type: application/xml" -d @BODY.xml localhost:8080/qgis-server?SERVICE=WFS&VERSION=1.3.0&REQUEST=GetFeature&MAP=project.qgs

Where BODY.xml contains the following body:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wfs:GetFeature maxFeatures="5000" version="1.1.0" service="WFS" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd" xmlns:wfs="http://www.opengis.net/wfs" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <wfs:Query typeName="grid">
    <ogc:Filter>
      <ogc:Intersects>
        <ogc:PropertyName>geom</ogc:PropertyName>
          <Polygon xmlns="http://www.opengis.net/gml" srsName="EPSG:3857">
            <exterior>
              <LinearRing>
                <posList srsDimension="2">1471131.73 1257643.13 2471131.7 1257643.13 2471131.7 257643.1 1471131.7 257643.1 1471131.73 1257643.1</posList>
              </LinearRing>
            </exterior>
          </Polygon>
      </ogc:Intersects>
    </ogc:Filter>
  </wfs:Query>
</wfs:GetFeature>

Extracting the ids from the response you could then query you atlas on the server with:

http://localhost:8080/qgis-server/?atlas_pk=44,45,46,52,53,54&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetPrint&MAP=project.qgz&TEMPLATE=layout_atlas&CRS=EPSG:3857&FORMAT=application/pdf

TIP

You’ll need to allow your server to print multiple pages. This can be done in Project > Properties > QGIS Server > WMS > Maximum page number for atlas queries.

Export DXF

At this point you have everything you need to export dxf files. Make sure you expose the layers you want to export through WFS (Project > Properties > QGIS Server > WFS > Tick Publish for the layers you want). You can then get your dxf using something like:

localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetMap
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&LAYERS=point,line,polygon
&FORMAT=application/dxf
&FILE_NAME=output.dxf

TIP

For dxf exports, the symbology is a bit trickier to get right in my opinion. You might have to play with units to get a configuration and a rendering that suits your needs. Also, Polylines3D don’t support linetypes and weight, so you might need to use the FORCE_2D parameter.

To conclude

PDF and DXF are, after all, rather easy to create with a qgis-server. No need for any other piece of software, potentially expensive. Make sure to play with the QGIS Server properties of your project and layers within QGIS to see how it influences the exports. Again, all the material used in this article is accessible in this repo.

How to build a single file QGIS project?

2024-09-28T00:00:00+00:00

We talked about how to collaborate on QGIS project at a “project management” level, but in practice, how do you make things easy for you to easily share a project. Well, one way is to minimize the amount of files by:

Storing your project in database
Storing your data in database
Storing your styles in database

Introducing Geopackage

Geopackage is basically a SQLite database with well defined rules for storing geospatial data. You get a single file that can contain all your data, stored in different layers. The standard has been extended to allow storage of QGIS projects, which allows to bundle a lot into a single file.

Storing your project in Geopackage

The first step would be to store your new project into a GPKG file. Open QGIS and create a new Geopackage database:

Layer > Create a new layer > New Geopackage layer

Then you can save your project in this newly created Geopackage:

Project > Save To > GeoPackage...

And that’s it, you can then open this project from QGIS:

Project > Open from > GeoPackage...

You now have a Geopackage containing your project.

GeoPackage
- Project

Storing your data in Geopackage

It’s rather self-explanatory, your can store all the data used in your project in the same Geopackage. If you use shapefile, please stop, it been making no sense for a while now.

Convert your data into Geopackage layer by right clicking on your layer of interest:

Export > Save features as... > GeoPackage

Select the Geopackage database containing your project as File name.

At this point you should already have your project and data in a single file, which is already way easier than managing multiple data sources.

GeoPackage
- Project
- Data
  - layer 1
  - layer 2
  - …

Storing styles

Having your data stored is nice, but styles are also important to add symbologies and capabilities to your layers. Geopackage also allow to store your style directly.

Define your layers styles (forms, symbologies, actions etc.) and then right click on your layer:

Properties... > Symbology > Style > Save Style... > In Database

And hey you can even go further! If you have .svg symbols in your style, you can load the symbol definition in the Geopackage. Like this you avoid broken links and unfound .svg file.

To do this, first create a new table in your Geopackage:

Layer > Create Layer > New GeoPackage Layer...

Select your Geopackage and fill the layer name (symbols or whatever you like). Then under New Field, add a field of type Binary Object (BLOB) and create the layer.

Add svg to Geopackage

Then add a new feature in your symbols and embed your .svg file in the BLOB column.

Finally, from your data layer symbology, select the SVG Marker symbol type and Edit the definition of the SVG with the rule of your choice, for example:

'base64:'|| to_base64(attribute(get_feature( 'symbols', 'fid', 1), 'svg'))

Here we’ll use the symbol with the fid value 1. You can create the rule that wou want, but make sure to encode the binary definition into base64.

TIP

You can also directly embed the file instead of creating the rule, you’ll just lose some flexibility in the choice of you symbol. A rule-based choice would allow you to use different SVG definitions based on a attribute for example.

At this point you should be in a situation like this:

GeoPackage
- Project
- Data
  - layer 1
    - Style 1
    - Style 2
    - …
  - layer 2
    - Style 1
    - Style 2
    - …
  - …

Which should be enough to cover the needs of most projects, all of this in a single file! Now it’s way easier to share, and there is no risk of breaking a link.

Conclusion

Easy right? And the best in all that, is that you can do exactly the same with a PostgreSQL database: storing your QGIS project, storing data (obviously) and storing your styles. It just take some “pre-procesing” by saving your data into your GPKG/PostgreSQL database, but you’ll want to do that anyway, and if you don’t, I assure you you want to.

Managing shared QGIS projects

2024-04-19T00:00:00+00:00

QGIS is an amazing tool. Being spatial analysis or digitizing data, it’s efficient, highly customable and extensible. There is so much you can do with it.

So it’s only natural that we want to share a project that we are working on and propose to other people to join our effort on our amazing data digitizing/spatial statistics calculating masterpiece. Well yes of course, things are better when they are shared (most of them at least), and we go further together than alone (wow very profound), but this is something that has to be given some thoughts beforehand to ensure that everything goes smoothly within the collaboration.

We’ll briefly have a look at what can help make collaborating with QGIS successful. Also, we’ll be talking a lot about projects, we are referring here to the literal QGIS file used to store the work environment (data, visualizations, etc.). It’s not project in the sense of a project management, but some things might be useful to apply from the project management world. And in any case, working together requires tools outside of the technical aspects.

When does this apply?

Obviously if you’re alone on your project or working with a handful of people, some points might feel over-the-top or even counter productive, but some questions are always useful to be asked and it can be helpful to use some “good” practice, even when working for ourselves.

For the last years I had to pick up some projects that were shared by tens of people, create new ones, and I can safely say that there need to be rules when more people are getting on board.

So if you work with spatial data and you need to have people collaborating using QGIS you might be interested

Ok let’s go, how to collaborate with QGIS?

This is obviously from my personal experience. It’s not exhaustive, or maybe not accurate, but if it can help..

tl;dr

Scope your project: have objectives for the project and keep it within controlled boundaries.
Have a data structure: define which data will be included and model them.
Give roles and responsibilities: have a project management in place.
Version control: track changes and decompose your project if it becomes too big.
Share it efficiently: have everything in a single place, keep it as simple as possible, save it in database.

Scope your project

It sometimes feels like it’s a bad thing to say in an Agile world, but giving some boundaries to the project helps staying on track. Maybe this is something that I feel due to unstructured work environments, but when working on a (QGIS) project, if there is no beginning and no end there is just no way to succeed.

Back to QGIS. When starting, think about what the project will be about: will it be used to digitize data? Should data processing and spatial statistics be in the project? Should this data X be included? QGIS is a powerful tool, and maps are powerful visualizations, when people unfamiliar with these notions start to see what can be done they tend to want to add a lot of information in a project. Hell you can do your taxes in QGIS if you want, should you? Probably not.

Let’s say you design your project for a certain team, to help them digitize certain data. Obviously, since they are the target of the project, their demands should be considered, analyzed and implemented. But let’s also imagine that the project has been shared with another team interested in visualizing the data. Should you take their demands into account as well? Well, has the work induced by these demands been considered when designing the project? If not then these demands should be redirected somewhere else.

Maybe try to have some ground rules: one project per data structure (see below), keep separate the digitization, processing and statistics, one project per team, etc.

Of course if your time is bound to be spent on a couple of project you can fit a lot of things and optimize them, having everything working together. But don’t forget that the more people start to use these projects, the more impactful any change becomes, and separation of concerns can help a lot to prevent that.

Structure your data before thinking about QGIS

Have a coherent data model to work with. If you have people around specialized in modeling data and structuring a database, listen to them. If they tell you that you should normalize your data because you’ll otherwise have problems when scaling up, please do it. I think that’s the most important advice, and it doesn’t even concern QGIS.

If you’re wondering what this is about, data modeling is, very simply put, how to structure/divide your data into blocks that make sense, allowing you to add, remove, update data in the most efficient way. For example, let’s imagine that I want to have information about trees in my city and the population of birds they are housing. I could have everything in a single table like below:

erDiagram
    tree {
        serial id_tree
        int age
        geometry geom
        string bird_species
        bool is_bird_cute
    }

Ok cool I can now create one tree that has a family of pigeons:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true

But wait, there is also a family of crow next branch:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true
2	125	POINT(x,y)	crow	true

You quickly see the problem appearing, we duplicate as much data as we create new one. That’s where a little bit of modeling helps. A potential alternative could look like this:

erDiagram
    tree ||--|| location : is_located
    tree {
        serial id PK
        int age
    }
    location {
        serial id_tree PK,FK
        geometry geom
    }
    tree ||--o{ bird : host
    bird {
        serial id_bird PK
        string species
        bool is_cute
        int id_tree FK
    }

Where trees have their own table, birds as well and they are connected using a foreign key (FK). We create a table for trees location as well; it’s not really necessary in that case since it is a 1:1 relationship, but it often appears that separating the information about an object and its geometries simplify many operations. It also shows that you can treat geometries like any other column in a data structure.

The above tables should now look something like:

tree

id	age
1	125

location

id_tree	geom
1	POINT(x,y)

bird

id_bird	bird_species	is_cute	id_tree
1	pigeon	true	1
2	crow	true	1

If migration season comes and 200 bird families choose this tree as their home, we just need to add elements in the bird table without duplicating any data regarding trees.

Your data model should be independent from the tools you’ll be using to manipulate it. A client limitation shouldn’t question a well structured data; if the tool is limited, update it, or maybe it’s not the one you need for the job. How many time have I seen weird database designs (Postgres triggers I see you) because it was not possible to work with certain situations in QGIS or because of performance issues with certain designs.

It’s a step for which you’ll be grateful when people ask to add some values, some attributes etc. Because if you’re data is well structured, you need to bring the change to a single place. But if the data structure didn’t follow some rules, you’ll probably have to change things in multiple locations, introducing much more space for error.

Give roles and responsibilities

Who is going to modify the project? Who is going to modify the data? Should you give read-only access? Who is going to do the long term maintenance? Who should be able to propose changes? Some questions should be answered before starting to build the project.

If we know who is responsible for what, decisions can be more efficient and implementation becomes most of the time a matter of minutes.

You should know who is allowed to ask for modifications in a certain project, who should implement them and who is allowed to reject the modifications.

But honestly, in short, have a project management.

Decompose, centralize and version control you project

Hun? Decompose and centralize? Sounds like there is a useless step here, but let me explain.

A QGIS project is basically a big xml file bringing many elements together. It’s nice because everything (almost) is in the same place, but it also means that any little change influences everything, which could provoke side effects. And when it comes to modifications brought to layers it can become cumbersome to track.

To mitigate that, QGIS allows to export many of its layer elements into separate files: symbols, forms, rendering, etc. Think about what will be modified regularly. You can export these elements as separate files to make sure that you focus on their changes and keep a specific history for them. Imagine you often have to modify the colors of certain layers; export the symbologies in .sld or .qml. Like this you will have the history of these files outside of the project and will be able to access older symbologies without changing the whole project.

Regarding version control, it seems obvious now but it’s not that easy when working with QGIS projects since the file is completely rewritten if you change anything and then save. But as said, a project is a combination of many things; there can be plugins, symbologies, profiles, etc. Making the effort of having everything in a single repository (even the result of your decomposition process) helps to understand what is needed and where, and by using a version control tool with it you can always come back to an earlier version in case of mistakes or breaking changes.

When working with a version control platform (such as GitLab or GitHub), you can also bundle your versions into releases, allowing to have everything needed in the project into a single archive, and accessible directly from an URL, which can help with the sharing process.

I find it bizarrely difficult to share a project with people. As said right above, a project is composed of multiple elements that work together: data, symbols, plugins, etc.

There are some ways of avoiding loosing information when sharing a project:

Keep all data in a single place: store your data in a database or a database file (e.g. Geopackage), it helps having everything in the same place to avoid Non valid layers or losing information.
Keep the number of plugins low: plugins are nice and they can help a lot, but any plugin added introduce uncertainty. They might not be updated, be usable with certain QGIS versions etc. In a more general sense, keep your project as simple and as clean as possible.
Save the project file in database or database file. QGIS allows you to store the project in database, which makes it so much easier to share. You theoretically can have a single Geopackage holding all the data AND the project. You then just need to share the file to others and they have basically everything they need to work with the project.
Maybe don’t share the project? Think outside of QGIS. If you start to do something in QGIS, you don’t have to stay in it. Do you really have to share the project or do you want to share the data inside it? Should the people you share with have a write access or not? There are so many tools available, maybe the task you’re trying to accomplish should not be done by sharing a QGIS project. Look into web-clients, file formats or any type of software that could reach the objective.

Conclusion

Now after rereading, I realize that I wrote a weird mix of technical aspects and project management. There might be something that I need to learn from that. But the main message remains that the tool shouldn’t guide your objectives and the underlying structure. Find a way to articulate your work that makes sense to you and your team, and that allows you to be efficient in order to answer request as well as possible.

Anyhow, QGIS is a very powerful tool: it’s easy to use, allows to accomplish a lot of things with spatial data and makes you feel like a genius. But the tool in itself shouldn’t replace a proper project management. Think about the objectives, the stakeholders, the responsibilities and make sure that everybody is on-board with the rules. It’s not because you can do anything with it that a project should host everything. Keep it clean and simple, one objective at a time, and find the method for sharing information that suits your need. With QGIS, a monolith project with a lot of people asking for modifications is a bad idea, at least from my experience.

But in any case, it’s a amazing tool, support it.

Cheers.

Geoservices with qgis-server

2024-03-22T00:00:00+00:00

While the geospatial world has its own particularities that sometimes make it difficult to get on-board or communicate with (see the spatial exception), we have to admit that some tools and standards developed are both amazing and essential. Geoservices are crucial for the diffusion of georeferenced data, and they drastically changed the way we apprehend our localization; when was the last time you asked for direction?

Being WMS, WMTS, WFS, WCS or any W*S, geoservices are everywhere and sometimes are a bit mysterious. But even without understanding the fundamentals of their definition, it’s rather easy to get started.

We’ll see how to install a (geo)server that will be responsible for serving our data, quickly prepare some data, and even see how you can use these services to update your data safely. We’ll be using qgis-server here, because it’s the most easy to use and the quickest way to get started.

Some quick context

You’ve probably heard about the Open Geospatial Consortium (OGC), but to make it really short, if you hear about a standard regarding geospatial, the OGC is probably behind. At least it’s the case for the geoservices we’ll be diving into here.

And what are those geoservices? Well most of the time when we talk about “geoservice” we’re referring to WMS, WMTS, WFS, WCS. They are a standardized way of accessing georeferenced data through the internet. The same way you reach a website or an API using an URL, you can access some geographic data if you know some URL (and have the tools that go with it). You can then use the data for your use-case: displaying a background map in your app, showing queryable data for an interactive webmap etc. The most common are:

Web Map Service (WMS) –> probably the initial service. Used to display an image as .jpg or .png (see it as a background map)
Web Map Tiled Service (WMTS) –> the more efficient WMS. It’s the same, but images are tiled to restricted the amount of data to be transferred to the client based on the display localization and the zoom level.
Web Feature Service (WFS) –> the vector service. You can work on the data as vector, request features on the map, and even add, modify or delete features.
OGC API-Feature (OGC-APIF) –> the more “web standard” WFS, also known as WFS3. Features can be accessed individually through an URL. The OGC states that

The API building blocks specified in this standard are consistent with the architecture of the Web.
Web Coverage Service (WCS) –> at the time of writing I’ve never used this standard so I won’t go into details, but from my reading it’s a service close to the WMS standard but providing more information than just an image, allowing more complex interactions.

There are other existing services: vector tiles, WSC, WMC etc. But I either never encountered them or they usually are implemented by other tools than the common geoservers.

Ok let’s get started

It’s nice and all, but how do we serve a W*S. Well for starters we need a server. Here we’ll be using qgis-server with nginx. The choice for qgis-server is motivated by the simplicity of use, you serve a qgis project. The choice for nginx, well the doc mentions it and it seems very lightweigth. I’m by no mean a system administrator so bear with me as I’ll read the doc with you.

Install and configure qgis-server and nginx

I’m on Ubuntu so I’ll be using apt, and the installation includes the use of systemd. I’ll also be working with the ubuntugis ppa.

First add the ppa:

add-apt-repository ppa:ubuntugis

Then install qgis-server:

apt install qgis-server

Install nginx if not already present:

apt install nginx

Configure qgis-server

We’ll then follow instructions from the QGIS documentation for a qgis-server installation using systemd and nginx. The method defines a socket unit and a service unit as follow. The sockets will be the communication channel between our http server (nginx) and our application server (qgis-server). Create the socket file /etc/systemd/system/qgis-server@.socket with the following content:

[Unit]
Description=QGIS Server Listen Socket (instance %i)

[Socket]
Accept=false
ListenStream=/var/run/qgis-server-%i.sock
SocketUser=www-data
SocketGroup=www-data
SocketMode=0600

[Install]
WantedBy=sockets.target

You can start these sockets with the following command:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.socket; done

Which will start 4 sockets, through which our http server will be able to pass requests to our qgis-server.

Now for the services. Services are managed by systemd and in our case are responsible for the execution of the qgis-server. Create the file /etc/systemd/system/qgis-server@.service and add:

[Unit]
Description=QGIS Server Service (instance %i)

[Service]
User=www-data
Group=www-data
StandardOutput=null
StandardError=journal
StandardInput=socket
ExecStart=/usr/lib/cgi-bin/qgis_mapserv.fcgi
EnvironmentFile=/etc/qgis-server/env

[Install]
WantedBy=multi-user.target

Notice that the server is configured using a env file, here /etc/qgis-server/env. Finally, in the same manner as the sockets, start the 4 services with the following:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.service; done

Services are depending on sockets, you’ll need to start the sockets first then the services. {: .prompt-warning }

Configure nginx

We’ll then configure nginx for our future qgis-server using fastcgi by creating the file /etc/nginx/sites-enabled/qgis-server and adding the following:

upstream qgis-server_backend {
   server unix:/var/run/qgis-server-1.sock;
   server unix:/var/run/qgis-server-2.sock;
   server unix:/var/run/qgis-server-3.sock;
   server unix:/var/run/qgis-server-4.sock;
}

server {
   listen 8080;
   access_log  /var/log/nginx/qgis-server/access.log;
   error_log  /var/log/nginx/qgis-server/error.log;

   location /qgis-server {
       gzip off;
       include fastcgi_params;
       fastcgi_pass qgis-server_backend;
   }
}

This configuration will then be included by the nginx.conf file using the include /etc/nginx/sites-enabled/*; statement.

Create a QGIS project and configure it

Now it’s time to decide what we want to expose. I have some OSM data in a PostgreSQL database (see Serving vector tiles with Martin) so I’ll use that.

Create a project, add some data and apply the desired style. A quick example is shown below.

Data with some style

Once you’re satisfied with the look of your data, you’ll need to configure the project so it will tell qgis-server to proceed with the geoservice you want to serve.

Go to Project -> Properties -> QGIS Server and complete the configuration as you wish.
Under Service Capabilities, check the Enable Service Capabilities box.
Select the different services you want to enable under the different tabs.
Save your project under a folder accessible by the server user (www-data). If you’re using file data in your project, such as a geopackage, save them in the same place.

If you setup a WFS, the tab should look like something like below.

Data with some style

WARNING

Make sure that your tables have a primary key. Otherwise the update and delete operations won’t be possible.

WARNING

And also, be sure that you don’t have special characters in you column or table names. It messes up the WFS 1.0 and 1.1 requests.

You can then edit the env file with the corresponding environment variables. For me it looks like:

QGIS_PROJECT_FILE=/etc/qgis-server/my_project.qgz
QGIS_SERVER_LOG_STDERR=1
QGIS_SERVER_LOG_LEVEL=3

I use PostgreSQL data and have had troubles getting the server to reach the data, even on localhost but finally managed to have everything working with the following steps:

Make the project read the data using a PostgreSQL service (create a pg_service.conf if not existing).
Copy the pg_service.conf file somewhere accessible for the server.
Edit /etc/nginx/fastcgi_params and add fastcgi_param PGSERVICEFILE /path/to/service/pg_service.conf;.
Restart everything.
Bonus: touch the project file.

If everything works correctly, you should be able to reach localhost:8080/qgis-server/qgis_mapserv.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities for WMS capabilities or http://localhost:8080/qgis-server/wfs3/ for the OGC-APIF description and navigate the different endpoints defined in the OGC-APIF standard.

You should now be able to also call these URL from QGIS using the correct url. Try to add a WFS/OGC-APIF using the http://localhost:8080/qgis-server/wfs3/ url (qgis-server in this example, but you can setup your server with a different url).

Going further

We have the basics setup, now what can we do? Let’s start by alter data.

Updating data through a WFS

If at this point you have everything working, there shouldn’t be much to do to update data through your newly build WFS. Working with the OGC-APIF you’ll be able to add, delete and update data using URL like you would do with any REST API.

You’ll find the description of your API at the URL http://localhost:8080/qgis-server/wfs3/api (for this particular setup) and it should look something like this:

OpenAPI description of an OGC-APIF

This gives you an idea of the available endpoints to manipulate data.

To try this, let’s try to GET one of our feature. You can access a single feature through a collection and an item. For me, the URL http://localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736 will point to the Lausanne train station:

Getting an item using its dedicated URL

And the same can be done using curl or wget or any network utility you wish:

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736" | jq .

will print something like:

{
  "bbox": [
    6.627779,
    46.51671,
    6.630578,
    46.517298
  ],
  "geometry": {
    "coordinates": [
      [
        [
          6.627779,
          46.517099
        ],
        [
          6.628225,
          46.517038
        ],
        [
          6.628344,
          46.51702
        ],
[...]
  }
}

TIP

The piping into jq (| jq .) is just for printing purpose, it doesn’t participate in the query.

Now let’s delete this object:

curl --request "DELETE" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

And that’s it!

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

now returns:

[
  {
    "code": "Internal server error",
    "description": "Invalid feature [2102736]"
  }
]

And you can do exactly the same using QGIS. Add your WFS as a layer and work on it like any vector data: add, update ,delete features and then save.

TIP

I intended to use a PATCH request to update a single attribute, but couldn’t make it work. QGIS seems to systematically call a POST request, even for a single attribute change. If you now how to work with PATCH in this environment let me know.

Protect our service with authentication

Accessing information through an URL means that many client might have access to it as well. It’s important to keep in mind that authentication would be necessary if you expect to update information through this API.

Again, I’m not a system administration or a security expert, but I’ll try to show how to setup a very basic authentication directly using nginx. This is not how you secure an API and it won’t replace a modern and secure authentication setup.

I’ll follow the nginx documention for that.

First install apache2-utils if your on a Debian based distribution:

sudo apt install apache2-utils

Create a password file and a user:

sudo htpasswd -c /etc/apache2/.htpasswd user1

In the nginx configuration, under the location of your API, add a name and specify that you’ll be using a auth_basic_user_file.

location /qgis-server {
    [...]
    auth_basic           "Protected Area";
    auth_basic_user_file /etc/apache2/.htpasswd;
}

Restart nginx for good measure and that it, try to reach your WFS using an URL or with QGIS and you should see a connection popup like this:

The route now needs basic authentication to be reached

Conclusion

Services are a powerful tool to be aware of when working with geographical data. They allow to efficiently share them at a larger scale through the web and even, as we saw, update them. There are many ways to build these services, and qgis-server is one solution among others, but it has the advantage to work (almost) out of the box with potential QGIS projects that you could already have: a few tweaks and you can be going.

The possibility to give access to your data through an URL makes sharing them so easy; it reduces the amount of manipulation and tools required, you don’t need to maintain a data extractor or repositories. Just some data and a server.

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

I’ve been trying to learn Rust lately. For some reason there is something that I find very appealing in this language. From all I could read, it’s very efficient in resources and memory, it’s fast, and it doesn’t let you mess around with memory. Regarding performances, it places itself next to C and C++, and when looking into the world of language performances, I learned that Python was kind of slow, around 100x slower according to some benchmarks. Obviously this value in itself doesn’t mean much, and many more aspects of a language must be taken account when choosing one for the task, but this information kind of planted its seed. It might be because I have a problem with inefficient and overly complicated process, but I find the idea to work with a system language very elegant.

But we’re not here to talk about the Rust language in itself. We’ll see how to read and write raster data using the Rust binding for GDAL. Also, this is about how I tried to compare the Rust and Python bindings for GDAL and how I learned that it doesn’t change a thing in terms of performance.

Wait what?

Well I started by thinking “Oh I can’t wait to see how faster GDAL in Rust is compared to Python”. So I started by reading a raster file and print some basic information. I use a band from a Sentinel-2 image.

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

I’m using a lot of .unwrap(), which in very short will cause the code to panic if a problem occurs, this is for a quick and dirty test. Within a “production” environment you would handle errors and stuff.

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

We will be using the Create() approach rather than the CreateCopy() (see the nuance in the doc). Geospatialized rasters come with series of metadata and overviews. The CreateCopy() automatically takes those from the source dataset and copy them into the destination. While this obviously simplifies the work if we don’t need to bring any change to these information, using Create() helps to better understand the structure of a raster. Also I have a quite important performance difference between the two and I’ve found Create() more efficient for my use case.

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

Both codes should create a new raster file that resembles our source, but in GeoTiff. And that’s it, you can convert a raster file into any format. We’re just scratching the surface, you can subset data, reproject, transpose it if you want etc.

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

So from my understanding, working with Rust or Python or any other language bindings is just working with the API that let us call the underlying code, for GDAL written in C and C++. Thus the overhead that we generate is only the interaction with the library, not during the process itself, since it’s running C/C++ code. The difference that we observe is probably the same that we were observing when we just opened a file.

Conclusion

GDAL raster processing is very similar in Rust and Python. When working with its bindings, it doesn’t change much which language you use to perform the GDAL operations in themself, since in any case you’re running C/C++ code from your program. But everything that is around need to be considered when choosing the right tool. If your program needs multi-threading then you’l probably chose Rust, if you need quick development without much optimization then Python might be a best fit. But it’s comfortable to see that the GDAL processing time won’t change significantly.

I’m not at all trying to say that Rust is not worth it, I’m excited to go further into it and work on projects, but it remains a tool and there should be no injunction to use it if it doesn’t bring much to the table for your use case.

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.

PMTiles exploration

2024-02-02T00:00:00+00:00

It’s a technology that I came across rather recently. Tile generation when maintaining a geographic oriented website can be a hassle: it can take time, generates a lot of files, and can be a critical process. PMTiles might have the potential to facilitate some of this work and we’ll try to see how.

What are PMTiles?

In itself, PMTiles is a specification for building single-file tile pyramids. It’s open source and part of a larger project called Protomaps, and it aims at mapping the World, just that. The major difference with the kind of tiles that I know is that they are embedded as a single file, like the MBTile format, which make the interaction very intuitive, I mean we all get the idea of what a file is. But what does it bring compared to MBTiles? Well from the Protomaps website:

A MBTiles file needs to be accessed as a SQLite database on disk, so requires a running server and disk storage for cloud hosting. PMTiles is designed to be cloud-native and readable in fine-grained parts from remote storage like Amazon S3.

It aims at simplifying the manipulation of tiles in a cloud native approach, for worldwide coverage.

How to create PMTiles?

The Protomaps website gives us some tools to create these tiles. Let’s try some possibilities.

Generate PMTiles using GDAL

Our Swiss Army Knife is back, starting version 3.8.0 GDAL has native support for PMTiles, so let’s try that.

First check the GDAL version you’re using, you’ll want 3.8.0 or higher:

gdalinfo --version
--> GDAL 3.8.3, released 2024/01/04 (debug build)

Let’s start from an OSM .pbf file (see Serving vector tiles with Martin). To convert this file into a PMTiles file you can enter:

ogr2ogr -f PMTiles out.pmtiles switzerland-latest.osm.pbf

The process might take some time if you’re working on a large dataset. By default the MAXZOOM is 5, which is rather low, so the run shouldn’t take to long, but if you start to increase the parameter to around 15, which allows to represent data from much closer, processing time increases.

But once the command is done, well that’s it!

Alternatively we can use tippecanoe to achieve the same result.

Generate PMTiles using tippecanoe

WARNING

Under construction

Visualize PMTiles

The easiest way is to go to the PMTiles Viewer Page and upload your newly created PMTiles file.

Obviously you can add it to your custom project. The easiest way of doing is by using the Leaflet plugin protomaps-leaflet.

Starting from complete scratch you can create an .html file with the following content:

<!DOCTYPE html>
<html>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"></script>
    <script src="https://unpkg.com/protomaps-leaflet@latest/dist/protomaps-leaflet.min.js"></script>
    <style>
        body,
        #map {
            height: 100vh;
            margin: 0px;
        }
    </style>
</head>

<body>

    <h2>Hey I'm a custom page</h2>
    <p>Let's add some Leaflet and PMTiles.</p>
    <div id="map"></div>

    <script>
        let PAINT_RULES = [
            {
                dataLayer: "multipolygons", //here we render only one layer, check your layers names 
                symbolizer: new protomapsL.PolygonSymbolizer({ fill: "steelblue" })
            }
        ];

        const map = L.map('map').setView([46.5, 6.6], 13); //chose a view that suits you

        var osm = L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png', {
            maxZoom: 19,
            attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
        }).addTo(map);

        var layer = protomapsL.leafletLayer(
            {
                url: 'http://127.0.0.1:8080/your_pmtile_file.pmtiles',
                paint_rules: PAINT_RULES
            }).addTo(map);

    </script>

</body>
</html>

For testing purposes you might have to serve you PMTiles through http. To do so you can do as the Protomaps website suggests:

npm install -g http-server
# Then from the folder containing your PMTiles.
http-server . --cors

Open your .html file on the browser of your choice and you should see the map with your PMTiles displayed.

When should PMTiles be used?

Again, the documentation on Protomaps website is quite complete and gives us some insights.

Cloud hosting

PMTiles are stored as a single file, and compression is so efficient that the size is ridiculous compared to classic png tiles. Number of requests necessary is drastically reduced and thus the cost of operation as well.

Static data

PMTiles don’t support localized updates so each time you’ll need to apply an update, the whole file needs to be rewritten. If your data don’t change that much PMTiles are a wise choice, but If you need to have a more dynamic rendering the documentation recommends PostGIS with the ST_As_MVT() function.

Simplicity

Here it’s more like a feeling about the philosophy of the specification. You have a single file, accessible without introduced complexity, directly from a library like Leaflet, which itself has simplicity as one of its core objectives. It looks like one of those project that allows you to make complicated-looking things so easily. I guess it’s a sign of a good project.

Final words

Well it was rather easy to discover. I mean we create a file, we basically load it within a project and we’re good to go. Obviously there are some customizations to do with styling and labels etc. But once you have the starting point and your tools, it takes minutes to create what you desire.

Here we’re just scratching the surface, remains to explore filtering, styling etc. But the concept in itself and the unreasonable performance of the compression make it very interesting if you’re in the situation where PMTiles could give you an advantage; the file created for OSM data Switzerland, with zooms from 0 to 14, is around 1GB, it costs pennies to upload and host on cloud services.

It’s a project to support and look for, I guess?

A little bit of GDAL/OGR appreciation

2024-01-19T00:00:00+00:00

If there had to be one single tool to work with spatial data, I think GDAL/OGR would be the first name to come to mind. Its usages range from raster to vector, it can be used as command line, imported as a library in many languages through bindings developed by the community, many applications provide UI to parameterize and run GDAL etc. Its versatility makes it an amazing tool to work with spatial data in a lot of different contexts, but it’s sometimes underestimated or even misunderstood.

We will quickly introduce GDAL/OGR, then give some examples of usage to help for some quick starts.

What is GDAL/OGR?

GDAL/OGR (Geospatial Data Abstraction Library/OpenGIS Simple Features Reference Implementation) is a library for working with raster and vector geospatial data formats, coded mainly in C++, C and Python. The first release dates from 2000, and the project has now more than 450 contributors at the time of writing. This is a very active project with regular releases.

The project is composed of two parts: the raster processing library GDAL and the feature processing library OGR, both merged during the release of GDAL 2.0. When working with raster data you will use programs prefixed with gdal (gdal_translate, gdalwarp etc.) while vector data will use ogr prefixed scripts (ogrinfo, ogr2ogr etc.). When talking aboutt GDAL, both libraries are included, and we’ll do the same from here.

As its name suggests, GDAL abstracts data in order to standardize its way of interacting with it. The driver creates the interface between the data source and the tools, allowing to focus on the logic of the process rather than how to read or write the data. This makes it a very extensible project; including a new format of data source requires to write a new driver, nothing more. You could see that the list of supported formats is quite something, either raster or vector.

GDAL describes itself as a translator, meaning that data come from a source and goes to its destination. Translations can occur at different moment during the process, during reading or by the script called. These translations can occur on multiple axis as well, we could for example:

Extract a subset of source data depending on a region of interest.
Extract a subset of source data based on an attribute.
Reproject data into a different coordinate system.
Merge a series of data files into a single one.
etc.

Source unknown, if it’s your image and you want to be cited please leave a comment

As an open source project with a permissive MIT license, GDAL is used by many projects, open source or proprietary, to process geospatial data. If you’ve used QGIS, ArcGIS, FME or PostGIS, you’ve used GDAL under the hood.

Why would I use it?

Weirdly, even though GDAL is basically everywhere there is geospatial data to be moved, it’s not so much acknowledged by GIS people, or at least it’s not really used on its own. If you read this, you definitely use GDAL, but maybe without noticing.

Ok but why would I complicate my life with some command lines when the same tool is being used by a software that provides me a user interface easier to use?

While it’s much intuitive to explore geospatial data through a user interface such as QGIS, where you can interactively load your data, chose the process you want to apply, its parameters, and visualize the results, things get a bit more complicated once you’ve done the exploratory part and want to reapply your newly defined process to new data. Repeating operations becomes quickly tedious and error-prone. You could use a tool such as the Model Designer from QGIS (amazing little tool, check it if you repeat a lot of operations in QGIS), but how do you schedule your process to be started when you’re not able to launch it manually?

If you use a heavy client ETL such as FME, try once to compare the difference in speed between a simple table transfer done by launching a FME workspace (using a command line) and a simple ogr2ogr command, there is at least an order of magnitude of difference.

It’s about using a tool designed for a purpose as efficiently as possible.

How to install?

We’ll assume you have no installation of GDAL yet, but just to make sure you can try:

gdalinfo --version

which gives:

GDAL 3.4.1, released 2021/12/27

If it’s not installed you can refer to the installation instructions from GDAL website.

For example, I’m working on Ubuntu 22.04, for which the GDAL package is the 3.4.1. The latest version can be installed by downloading the latest release, but I usually stick with the provided version for my platform, unless I need a specific update.

apt update
apt install gdal-bin

And you should be good!

WARNING

You might already have GDAL installed through another software like QGIS, be careful during the installation, multiple installations might interfere and create warnings or errors. On Windows you can use the OSGeo Shell to access any tool from the OSGeo ecosystem.

And how do we use it?

It’s easy, you just need to be comfortable with command lines. If you’ve mostly worked with user interfaces it might look scary, but it really is not complicated.

Access metadata

The first thing we usually do with data is to have some information about it. Let’s try to see the metadata of a random .jpeg:

gdalinfo gdal.jpeg

Driver: JPEG/JPEG JFIF
Files: gdal.jpeg
Size is 711, 882
Metadata:
  EXIF_Orientation=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  882.0)
Upper Right (  711.0,    0.0)
Lower Right (  711.0,  882.0)
Center      (  355.5,  441.0)
Band 1 Block=711x1 Type=Byte, ColorInterp=Red
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=711x1 Type=Byte, ColorInterp=Green
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=711x1 Type=Byte, ColorInterp=Blue
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG

Easy right? And it’s not even georeferenced, it’s a .jpeg, a driver exists, it opens.

We can try the same with some vector data. We’ll have a look at what is going on in one of the PostgreSQL database hosted on my computer.

ogrinfo -ro PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.
1: planet_osm_line (Line String)
2: planet_osm_polygon
3: planet_osm_roads (Line String)
4: planet_osm_point (Point)
5: natural_polygons

And as we can see I have 5 layers in my database. Notice how we used the ogrinfo command for questioning vector data. Here ogrinfo (like gdalinfo) requires a dataset source as input, the syntax will depend on each driver, make sure to have a look at the documentation. For a PostgreSQL connection, the line could have looked like:

ogrinfo -ro PG:"dbname='osm_suisse' host='localhost' port='5432' user='x' password='y'"

We can extend the information that we want to see for each layer by adding the -al flag to the line. The option -so will summarize the output.

ogrinfo -ro -so -al PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.

Layer name: planet_osm_line
Geometry: Line String
Feature Count: 2137192
Extent: (626211.331305, 5706647.213604) - (1253686.695428, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Geometry Column = way
osm_id: Integer64 (0.0)
access: String (0.0)
addr:housename: String (0.0)
addr:housenumber: String (0.0)
[...]

And now we have the full detail of our layers; name, type, feature count, etc.

Basic translations

Now that we know what were working, with we can try to move some things around. Let’s start by exporting our vector data into a geopackage.

ogr2ogr -f GPKG exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

Here we’re exporting the table planet_osm_point into the exported_data.gpkg geopackage. The format can be recognized by the extension of the output file, but it’s safer to ensure the format by passing the argument -f. Again, each driver documentation will have its driver name to be used.

Add a filter

We can export a subset of our data, based on an attribute for example. Let’s say we want to export the points where the attribute addr:housenumber is not null, and we want to save it in another table in our newly created geopackage. We’d do something like:

ogr2ogr -update -f GPKG -where '"addr:housenumber" is not null' -nln only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

We want to update our dataset (our geopackage) and not create a new one, so we use the -update option. The syntax of the -where parameter is a bit convoluted here since there is a special character in the column name, but with a simple name you’d just need a pair of quotes -where "housenumber" is not null".

Let’s have a look at our geopackage:

ogrinfo -ro -al -so exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: planet_osm_point
Geometry: Point
Feature Count: 2146246
Extent: (626415.435591, 5736097.599850) - (1189428.532375, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
[...]

Layer name: only_housenumbers
Geometry: Point
Feature Count: 366561
Extent: (663719.309401, 5751784.044673) - (1167785.106794, 6072459.331387)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
[...]

The new layer has been added to the geopackage. Has the query been respected?

ogrinfo -ro -al -so -sql 'select * from only_housenumbers where "addr:housenumber" is null' exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: SELECT
Geometry: Point
Feature Count: 0
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
[...]

Yes! No data without addr:housenumber.

TIP

The -sql option is another way of filtering data, it allows you to pass a sql query in any of the sql flavor supported by the driver, you can even save your queries in a text file and pass the file to the -sql parameter as -sql @path/to/my_file.

Geometric operations?

The -sql parameter can also be used to apply transformations to the data. We’ll request the same points, but this time we’ll create a buffer around them.

ogr2ogr -update -f GPKG -sql 'select ST_Buffer(way, 50) from planet_osm_point where "addr:housenumber" is not null' -nln buffered_only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

And that’s it, we have our buffer around the points of interest. Here GDAL makes use of the internal capabilities of PostgreSQL and PostGIS by passing directly the request to the database, but the same command using the only_housenumbers layer from our geopackage would also be valid.

ogr2ogr -update -f GPKG -sql "select ST_Buffer(way, 50) from only_housenumbers" -nln buffered_only_housenumbers exported_data.gpkg exported_data.gpkg

TIP

In the case of PostgreSQL/PostGIS you can write the -sql parameter like you would write your queries directly into the database. They are passed directly to the database by GDAL.

This should give you a small idea of the things you can do with GDAL and how to start experimenting.

Final words

Well that was a lot more than expected. But GDAL really is a very strong and important tool. Diving into the details of all the available options, both for the command AND the drivers, is not a small task, but understanding how to navigate around and find the information that you need to process your data as you wish can go a long way. Manual and repeated tasks can be replaced to help to focus on more pertinent work, process time can be (by experience) reduced drastically depending on the tools you’re initially using. And if you’re working on data hosted on a server without graphic interface, well you have a tool to get the info you need!

Obviously GDAL alone can’t do everything and has its imperfections; I’ve really struggled with installations in the past (Python binding might have bested me at some point), and the number of parameters is a bit overwhelming. But once you have it working it does the job, and really well.

If you can, use GDAL.

Serving vector tiles with Martin

2023-12-20T00:00:00+00:00

There are some amazing tools out there, and they are getting build fast. I haven’t had the opportunity to work with vector tiles during my day-to-day jobs, but I wanted to explore what it is and how to set some things up, just for the fun of it. We’ll see here how to start using Martin, a vector tile server built in Rust serving PostGIS data.

Vector tiles

The standard has been set by Mapbox. In the same manner a WMTS serves raster tiles, vector tiles allow for the visualization of large volumes of data over the web, but as their name suggests, they are vectors.

It comes with some advantages compared to raster formats:

It’s significantly lighter, about 20 to 50% of its raster equivalent according to MapTiler. Vector data are series of points and each pixel of the map doesn’t have to have information declared; a rectangle polygon covering thousands of square kilometers can still be defined with 4 points.
It is stylized dynamically. While raster tiles need to be generated with their style embedded, vector tiles style can just be modified on the fly, without going through the laborious process of tiles generation that can take easily several hours/days.
It’s fast. Moving through a vector tile map feels smooth, there is no apparent tile loading.
What you request is what you get. You directly query the tiles, not some sort of other service along your tiles.

It’s an amazing standard to visualize spatial data in a web map. Obviously it’s not suitable for all kinds of data, you have to have discrete data that make sense to be visualized as vector; I doubt that raw satellites images as vector tiles are a good idea.

Installing PostgreSQL and PostGIS

First thing first, we need to store data somewhere. Martin is designed to be used with PostGIS, PMTiles and MBTiles, we’ll explore the PostGIS alternative.

The following commands were used on an Ubuntu distribution.

Let’s start by installing PostgreSQL:

apt install postgresql

Then the PostGIS extension:

apt install postgis

We’ll already set things up in the database by creating a dedicated user, creating a database and creating the PostGIS extension in there. From psql, create a user:

create user martin_usr with password 'my_password';

Create a database:

create database martin_db;

Go to your new database (\c martin_db) and create the extension:

create extension postgis;

We won’t cover here the more extensive tuning of the database. You should be set on the database side for this example.

Downloading some data

Obviously you’ll want some data to serve as vector tiles. In a production environment you’ll probably want to serve your own data, so this step would probably not be necessary. For the example we don’t have any data yet, so we’ll use some from osm through GeoFabrik. Since I’m in Switzerland, let’s download data over the country:

curl https://download.geofabrik.de/europe/switzerland-latest.osm.pbf -O

We are using osm2pgsql to import data into the database, so the .pbf format is ideal.

You can then load your data (apt install osm2pgsql if you don’t have the tool):

osm2pgsql -d martin_db -U martin_usr switzerland-latest.osm.pbf

We now have osm data over a whole country, easy right?

Now install Martin

Here comes the best part, the vector tile server itself. There are multiple ways to start working with Martin, we’ll use rust and its package manager cargo. If you don’t want to install those on your server you might prefer the Docker installation, or simply using binaries.

As said earlier, Martin is build in rust, and it’s accessible as a crate using cargo. You can first install rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then install Martin:

cargo install martin
martin --help

And that’s it! Martin’s here.

Can’t be that simple right?

Finally, you can run your new server:

martin "postgresql://martin_usr:my_password@localhost:5432/martin_db"

Martin can discover geometric tables in the database from the given url, so by default everything is served. So yes it can be that simple! Obviously environment variables need to be set, passwords hidden etc. But you have an idea of the process.

You can try by yourself reading the catalog:

curl http://0.0.0.0:3000/catalog | json_pp

{
   "fonts" : {},
   "sprites" : {},
   "tiles" : {
      "planet_osm_line" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_line.way"
      },
      "planet_osm_point" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_point.way"
      },
      "planet_osm_polygon" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_polygon.way"
      },
      "planet_osm_roads" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_roads.way"
      }
   }
}

Visualize data using QGIS

Once the server is running we can easily visualize our data in a client. Starting QGIS 3.14,

Layer > Add Layer > Add Vector Tile Layer

And enter the url then OK:

http://localhost:3000/planet_osm_line,planet_osm_point,planet_osm_polygon/{z}/{x}/{y}

You should see a rather absurd amount of data.

To conclude

Here is how to get started with Martin, a rust vector tile server. It’s just the start, we don’t cover here the creation of symbology and more in depth configurations and optimization, but it gives an idea of how easy and powerful such a tool can be.

geothings - opensource

Serve PDF and DXF files using qgis-server

Setup

Export PDF files

Export PDF files: atlas version

Export DXF

To conclude

How to build a single file QGIS project?

Introducing Geopackage

Storing your project in Geopackage

Storing your data in Geopackage

Storing styles

Conclusion

Managing shared QGIS projects

When does this apply?

Ok let’s go, how to collaborate with QGIS?

tl;dr

Scope your project

Structure your data before thinking about QGIS

Give roles and responsibilities

Decompose, centralize and version control you project

Find an efficient way to share your project

Conclusion

Geoservices with qgis-server

Some quick context

Ok let’s get started

Install and configure qgis-server and nginx

Configure qgis-server

Configure nginx

Create a QGIS project and configure it

Going further

Updating data through a WFS

Protect our service with authentication

Conclusion

Opening and writing raster dataset with GDAL Rust

Wait what?

How to open a raster file?

Comparing performances

Let’s add some writing then

How to write a raster dataset with GDAL bindings?

But let’s compare performances again

Wait what? (bis)

Conclusion

PMTiles exploration

What are PMTiles?

How to create PMTiles?

Generate PMTiles using GDAL

Generate PMTiles using tippecanoe

Visualize PMTiles

When should PMTiles be used?

Cloud hosting

Static data

Simplicity

Final words

A little bit of GDAL/OGR appreciation

What is GDAL/OGR?

Why would I use it?

How to install?

And how do we use it?

Access metadata

Basic translations

Add a filter

Geometric operations?

Final words

Serving vector tiles with Martin

Vector tiles

Installing PostgreSQL and PostGIS

Downloading some data

Now install Martin

Can’t be that simple right?

Visualize data using QGIS

To conclude