geothings

Serve PDF and DXF files using qgis-server

2025-05-30T00:00:00+00:00

You might be faced with the need to distribute “plan” files. For some businesses, paper maps (pdf) and CAD drawings (dxf) are still the primary source of information, and sometimes you can’t propose anything else. If you’re working with QGIS there are simple ways to serve those files without much to do, and all with opensource solutions using qgis and qgis-server.

Setup

You’ll need… Well, qgis and qgis-server, and that’s it. First install qgis-server using instructions for you environment, you might have a look at the official doc or this short tuto. While working on this article, I followed the containerized deployment instruction from the QGIS documentation. I’ll link a repository with the resources for you to follow along.

The next step is the preparation of your exports using qgis. Your projects will serve as configuration, where you’ll define symbologies, renderings, services, etc. We’ll show some possibilities to configure pdf and dxf exports.

Export PDF files

For simple cases it’s the easiest setup, you’ll just need to configure a layout in your project. Define a map that suits your taste (or better your objectives) and save everything in a location where your server has the right to read your project and the linked data. And that’s it for the most part. You can then have a look at a resulting map by querying an url of the like:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&FORMAT=application/pdf

If we divide the query we have:

http://localhost:8080/qgis-server the url of your qgis-server.
SERVICE=WMS tells the server to use its WMS service.
VERSION=1.3.0 version of the service.
REQUEST=GetPrint run the GetPrint request.
MAP=project.qgz path to the qgis project to read.
TEMPLATE=layout the name of a template defined within the project that describes a layout, where you’ll define your map.
CRS=3857 the coordinate system to use, usually the one defined in your project.
map0:extent=xmin,ymin,xmax,ymax the bounding box defining the extent to print. In the CRS system.
FORMAT=application/pdf specify the output format, here pdf.

Your browser will show you something like the following:

In this simple case, we could imagine a service where you define an extraction extent, which then will be passed to your printing service in the form of a mapx:extent parameter. You’d receive your pdf as a response.

Export PDF files: atlas version

You can create atlases in QGIS which allows you to produce maps using predefined extents. Your qgis-server also supports those and you can request a specific map from an atlas by using the ATLAS_PK parameter in you query.

If we create a layout_atlas in our project, using a grid as overlay (see the example project), we could then query our atlas pages using the following:

http://localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetPrint
&MAP=project.qgz
&TEMPLATE=layout_atlas
&CRS=EPSG:3857
&ATLAS_PK=1

Which would then send back something like:

Here you’d need to get the atlas elements that intersect your region of interest and pass their id as ATLAS_PK. A query like the following could get you this information:

curl -X POST -H "Content-Type: application/xml" -d @BODY.xml localhost:8080/qgis-server?SERVICE=WFS&VERSION=1.3.0&REQUEST=GetFeature&MAP=project.qgs

Where BODY.xml contains the following body:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wfs:GetFeature maxFeatures="5000" version="1.1.0" service="WFS" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd" xmlns:wfs="http://www.opengis.net/wfs" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <wfs:Query typeName="grid">
    <ogc:Filter>
      <ogc:Intersects>
        <ogc:PropertyName>geom</ogc:PropertyName>
          <Polygon xmlns="http://www.opengis.net/gml" srsName="EPSG:3857">
            <exterior>
              <LinearRing>
                <posList srsDimension="2">1471131.73 1257643.13 2471131.7 1257643.13 2471131.7 257643.1 1471131.7 257643.1 1471131.73 1257643.1</posList>
              </LinearRing>
            </exterior>
          </Polygon>
      </ogc:Intersects>
    </ogc:Filter>
  </wfs:Query>
</wfs:GetFeature>

Extracting the ids from the response you could then query you atlas on the server with:

http://localhost:8080/qgis-server/?atlas_pk=44,45,46,52,53,54&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetPrint&MAP=project.qgz&TEMPLATE=layout_atlas&CRS=EPSG:3857&FORMAT=application/pdf

TIP

You’ll need to allow your server to print multiple pages. This can be done in Project > Properties > QGIS Server > WMS > Maximum page number for atlas queries.

Export DXF

At this point you have everything you need to export dxf files. Make sure you expose the layers you want to export through WFS (Project > Properties > QGIS Server > WFS > Tick Publish for the layers you want). You can then get your dxf using something like:

localhost:8080/qgis-server?
SERVICE=WMS
&VERSION=1.3.0
&REQUEST=GetMap
&MAP=project.qgz
&TEMPLATE=layout
&CRS=EPSG:3857
&map0:extent=-591992,-1689591,4318167,3574909
&LAYERS=point,line,polygon
&FORMAT=application/dxf
&FILE_NAME=output.dxf

TIP

For dxf exports, the symbology is a bit trickier to get right in my opinion. You might have to play with units to get a configuration and a rendering that suits your needs. Also, Polylines3D don’t support linetypes and weight, so you might need to use the FORCE_2D parameter.

To conclude

PDF and DXF are, after all, rather easy to create with a qgis-server. No need for any other piece of software, potentially expensive. Make sure to play with the QGIS Server properties of your project and layers within QGIS to see how it influences the exports. Again, all the material used in this article is accessible in this repo.

Building a geospatial strategy, sharing a real world experience - Part 1

2024-12-06T00:00:00+00:00

I’ve recently been given the responsibility at my job to implement a “geospatial strategy”, whatever this means to the people requesting. Nice, I guess, but I have two conflicting feelings:

I’m excited because it feels like it’s a very necessary thing, and I’m happy to be trusted and given the responsibility. From my perspective it’s something that should have been done years ago.
The motivations behind are unclear and I fear that I won’t be able to align what I think is necessary with what my hierarchy expects to achieve with it, but I’ll come back to that.

I’ll share my thoughts, discussions, tries and errors, successes and fails in these articles. It will be a series, but I don’t know how many episodes since the process my take a while before anything is achieved. The aim is to give feedbacks on what I’ll try, what succeeds, what fails, in hope that it could help someone else.

This first part will be dedicated to the starting thoughts and researches, the material that I’ll use to guide my though process, and my first discussions with different stakeholders.

Context

Let’s start with some context to understand the situation in which I operate.

I work in a medium sized (for swiss standards) local administration. The city for which I work employs around 5000 people, and is home for around 150’000 inhabitants.

My position is called “GIS specialist”, I’m engineer, I don’t manage people. I’m working in the cadaster department, within which we form a kind of hybrid unit of people working on a geospatial portal and a kind of EDM tool. We are not part of an IT department, but still have some relations. Our unit manage its own servers, not using the same standards as the IT department.

One important thing to note is that the environment I’m working in is very partitioned; departments communicate through hierarchy, but down in the organization chart you don’t get much information. It’s very true for objectives as well, when I’m trying to motivate collective work with other departments, the discussion is usually rapidly steering into: “How can our service position itself?”. But I feel like if we don’t think about the broader picture we won’t have anything to position ourself into.

Getting started

Once I received the “mandate” to setup a geospatial strategy, I started to go straight into literature, looking for feedbacks from other organizations, companies. I quickly realized that I wasn’t even sure what I was asked. What is the definition of a geospatial strategy? What are its objectives? I have to clarify that.

What is a geospatial strategy?

To get started I took inspiration from the ESRI document The value of a geospatial strategy.

A geospatial strategy is a business-oriented plan for how organizations will leverage their geospatial resources (people, processes, and technology) to produce the outcomes it’s trying to create.

Rather simple and generic enough, a plan for an organization around geospatial resources to reach objectives. We have here three concepts that need clarification:

organization –> define the organizational scope. Do we operate a the company scale? Within a department? Whose objectives do we listen to?
resources –> what do we leverage to make our strategy successful? Do we act on technologies, processes, all?
outcomes –> what are our objectives with the strategy?

These three points need to be defined before starting strategy definition, I’ll call this the broad vision for the strategy.

Align the visions

That’s a major step to validate. I made a first round of literature research but if everything I do is not what is expected then it’s all for nothing. To do so I’ll define a vision for the strategy. It will act as a rather abstract and large scale roadmap and define what will be its objectives, its scope and the potential stakeholders it will bring together. A meta-objective if you wish. Once I have this vision, I’ll motivate it and present it to executives commanding the strategy. We’ll have to meet on the terms of the strategy at this point, I hope that through a constructive discussion, we’ll find a way of developing an ambitious strategy, addressing multiple challenges and bringing a diverse set of people around the table. But most importantly that we will align our visions.

To illustrate diverging visions, here is an example:

My vision for the strategy would include the whole organization, the strategy would listen to most of the main stakeholders objectives and challenges. This would result in a methodology going from large to small scale, trying to define global objectives and building solutions to reach them.
My executives vision would focus on their organizational unit, defining its objectives and the added value it could bring to the organization. This strategy would start small and would hope to extend to other departments through exemplarity. The department would provide services that other units could request through a set of well defined governance.

Both are valid but one thing is for certain, if we don’t align our visions, the strategy will never answer the expected needs.

My initial proposition

My starting point for the discussion will look something like:

Organization –> I’ll motivate a strategy that includes the whole organization, in our case the communal administration. From my discussion with other departments, there is a need for guidance around geospatial information, regarding processes, technologies, roles, etc. I believe that including stakeholders from as many departments as possible would help to overcome the partitioned situation we are actually in. This would also allow better coordination of roles, processes or tools.
Resources –> The accent will be put on developing strong and clear governance. Going from large scale objectives to finer grained solutions, every aspects of the solution will need to be studied before implementation, including technical choices, stakeholders involved, their role and responsibilities.
Objectives –> We’ll start by looking at the large-scale objectives of the organization. Look at the broad situation and define a common goal for the organization, take this opportunity to bring together the departments by taking into account objectives/challenges from multiple services. It could sound like:
- Better management of underground networks.
- Better management and overview of urbanism.
- Improve collaboration between departments.
- Act as a local enabler by sharing open data.
- etc.

To conclude

And that’s it for now. This is really the preparatory part of the job, making sure that the objectives are clear and shared by everyone. Then we’ll be able to look into detailed objectives, planning and implementation of solutions.

For my part, I’ll prepare a broad presentation of the strategy objectives and motivate my vision. If not shared by my executives I’ll have to adapt to align with their view on the objectives.

The next step will then be to listen to the defined stakeholders objectives/problems and formalize the strategy.

I’ll refine these posts as I’m working on the project, but for now this is the state of the reflection.

How to build a single file QGIS project?

2024-09-28T00:00:00+00:00

We talked about how to collaborate on QGIS project at a “project management” level, but in practice, how do you make things easy for you to easily share a project. Well, one way is to minimize the amount of files by:

Storing your project in database
Storing your data in database
Storing your styles in database

Introducing Geopackage

Geopackage is basically a SQLite database with well defined rules for storing geospatial data. You get a single file that can contain all your data, stored in different layers. The standard has been extended to allow storage of QGIS projects, which allows to bundle a lot into a single file.

Storing your project in Geopackage

The first step would be to store your new project into a GPKG file. Open QGIS and create a new Geopackage database:

Layer > Create a new layer > New Geopackage layer

Then you can save your project in this newly created Geopackage:

Project > Save To > GeoPackage...

And that’s it, you can then open this project from QGIS:

Project > Open from > GeoPackage...

You now have a Geopackage containing your project.

GeoPackage
- Project

Storing your data in Geopackage

It’s rather self-explanatory, your can store all the data used in your project in the same Geopackage. If you use shapefile, please stop, it been making no sense for a while now.

Convert your data into Geopackage layer by right clicking on your layer of interest:

Export > Save features as... > GeoPackage

Select the Geopackage database containing your project as File name.

At this point you should already have your project and data in a single file, which is already way easier than managing multiple data sources.

GeoPackage
- Project
- Data
  - layer 1
  - layer 2
  - …

Storing styles

Having your data stored is nice, but styles are also important to add symbologies and capabilities to your layers. Geopackage also allow to store your style directly.

Define your layers styles (forms, symbologies, actions etc.) and then right click on your layer:

Properties... > Symbology > Style > Save Style... > In Database

And hey you can even go further! If you have .svg symbols in your style, you can load the symbol definition in the Geopackage. Like this you avoid broken links and unfound .svg file.

To do this, first create a new table in your Geopackage:

Layer > Create Layer > New GeoPackage Layer...

Select your Geopackage and fill the layer name (symbols or whatever you like). Then under New Field, add a field of type Binary Object (BLOB) and create the layer.

Add svg to Geopackage

Then add a new feature in your symbols and embed your .svg file in the BLOB column.

Finally, from your data layer symbology, select the SVG Marker symbol type and Edit the definition of the SVG with the rule of your choice, for example:

'base64:'|| to_base64(attribute(get_feature( 'symbols', 'fid', 1), 'svg'))

Here we’ll use the symbol with the fid value 1. You can create the rule that wou want, but make sure to encode the binary definition into base64.

TIP

You can also directly embed the file instead of creating the rule, you’ll just lose some flexibility in the choice of you symbol. A rule-based choice would allow you to use different SVG definitions based on a attribute for example.

At this point you should be in a situation like this:

GeoPackage
- Project
- Data
  - layer 1
    - Style 1
    - Style 2
    - …
  - layer 2
    - Style 1
    - Style 2
    - …
  - …

Which should be enough to cover the needs of most projects, all of this in a single file! Now it’s way easier to share, and there is no risk of breaking a link.

Conclusion

Easy right? And the best in all that, is that you can do exactly the same with a PostgreSQL database: storing your QGIS project, storing data (obviously) and storing your styles. It just take some “pre-procesing” by saving your data into your GPKG/PostgreSQL database, but you’ll want to do that anyway, and if you don’t, I assure you you want to.

FME is getting expensive? Stop using it

2024-08-16T00:00:00+00:00

I have trouble finding what happened at Safe Software at the beginning of 2024, but it seems that part of the company has been sold to an equity firm. No comment on that, but since then licenses prices seem to have taken a hit and customers are.. Well not very happy, especially regarding the communication around this increase:

Costs become unbearable for some companies and the look for alternatives is now a necessity. What can we do about it?

About FME and Safe

In itself I don’t have anything against FME, it does its job and if it allows more people to work with spatial data and create workflows then it’s cool. But there are things that I find, if not unacceptable, at least not very wise, and it’s not always Safe responsibility.

Safe as a company

I never had to negotiate any license or contract with Safe, but people around me always seemed happy about their relation with their resellers and the flexibility that was allowed with their licenses. There was no problem having a home/student license for free if you wanted to fiddle around with FME and prices were rather stable.

The rebranding and the sell at the beginning of 2024 has changed things, and some of the trust seems to be lost. The big problem is the way the company communicated the changes, which was apparently by giving no clear information. Increasing prices (reasonably) could hardly be considered a problem, that’s the world we’ve chosen, but at least having some transparency about it maintains the relation between seller and buyer. If you receive a x3 bill with a “go f“ck yourself if not happy”, this is at best disrespectful, at worst belittling, especially if you can’t act against this. And to me, the best answer to give is to gain independence and stop buying.

FME as a software

As I said, I don’t have anything against FME itself. It makes workflows easy to create and push to production, (spatial) data processing accessible to a broader audience and gives you time to focus on something else if you have a GIS role in your company (see On GIS and their integration for that part of the problem). But making yourself totally dependant on a proprietary solution, with a near monopoly on the field, IS a big problem, and honestly felt like a matter of time before it would happen. The situation is so problematic that some high level administrations (country level) are now negotiating terms with Safe because they are so dependant on FME that they can’t move easily from it, and will have no choice but to take the price hit if they don’t act. And maybe it’s common practice, I wouldn’t say I have enough experience to understand those kinds of negotiations, but it feels wrong.

That’s were digital sovereignty enter the game, regaining control over our processes, data, and costs. Some links :

If you work extensively with FME, I’d advise to take a step back and and have a look at what you need and how you could achieve this any other way, because most of the time, there are options, but we just don’t see them anymore. FME has become such an automatism that everything else is a blur, and worst, we lose skills and control.

What alternatives are there?

The first step is to have a look at what you need to achieve. Let’s see how FME defines itself from their website:

Expansive Data Solutions → well it doesn’t mean anything so next.
Data Transformation → Ok noted
Entreprise Automation → Ok noted

So we get data transformation and automation. Good.

Read, Write and Data transformation

Let’s start with the main reason for all of this. We read, transform and write data to solve problems and answer questions. Data need to be shaped into different models and linked with other data to bring additional knowledge. But how do we do that?

Transform with SQL

If you’re working with databases (PostgreSQL, MariaDB or even Oracle), you’re probably aware of their spatial capabilities. To me, SQL is a major skill, you can do a lot with a single SQL query. There is even a chance that your FME workspace can be entirely replaced by a single SQL query.

Now we want to Extract, Transform and Load data. Hum but wait, who said you can’t “transform” your data before even reading it? SQL allows you to do that; write your query, save it as a view if you wish, and you have your result ready to be read, in sync with your tables.

Let’s say you have regions (polygons), roads (lines) and bus stops (points), and you want:

road cut into portions by region
road portions length
to count bus stops on each road portion
- if a road has no bus stop you want to have “No bus stop” instead of 0
an attribute that says “Hopp Schwiiz !” if the road portion is located in a region where attribute name is equal to “Switzerland”, else “Oh, no Hopp Schwiitz…”.

You could have a FME workspace chaining writers, a clipper, a length calculator, aggregators, attributes managers etc. Or a single SQL query in the following manner (optimization not included):

SELECT 
    portion.name,
    CASE COUNT(b.id) 
        WHEN 0 THEN 'No bus stop'
        ELSE COUNT(b.id)::text
    END AS n_bus_stop,
    CASE portion.name
        WHEN 'Switzerland' THEN 'Hopp Schwiiz !'
        ELSE 'Oh, no Hopp Schwiitz...'
    END AS hop_suisse,
    St_Length(portion.geom) AS portion_length,
    portion.geom
FROM (SELECT r.name, St_Intersection(l.geom, r.geom) AS geom
        FROM road l
        JOIN region r ON St_Intersects(l.geom, r.geom)
        ) portion
LEFT JOIN bus_stop b ON St_Intersects(portion.geom, b.geom)
GROUP BY portion.name, portion.geom

And that’s it, you can read this data, export it, do whatever you want with it. Everything done within the query is visible at once, and if it takes some time to get used to the syntax and be comfortable with it, it becomes much easier to understand what happens in a well written SQL query than in the tidiest FME workspace (personal opinion included).

Another huge advantage of working with SQL is that you’re closer to a standard interoperable format (with a grain of salt since we use Postgis here, which is Postgres specific). If you ever need to change your system, the required effort to make the conversion will be minimal compared to translating a FME workspace.

You can import other data into your database to use it in your query, or use Foreign Data Wrappers (FDW) to read data from another database. You can combine your queries with tools such as dbt (Core) to have a more complete environment, or build your own processes, with GDAL or using the programming language of your choice.

Read and write with GDAL

Back at it with GDAL, but it’s so versatile that it has to be in your toolkit. Your can read and write basically any format, and apply transformations on the fly.

The series of command line tools that come with GDAL/OGR make it very complete and easy to use with raster or vector data.

You could reuse your previous SQL query with GDAL to extract your data and save it in GeoPackage for example. I could just save my query into a file named query_file.sql and run the following command:

ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Remember interoperability? Well that’s a nice example, have one standard way of solving a problem and you can connect it with other without changing anything. You have many possibilities.

Do everything with (Python) libraries

There are a bunch of libraries allowing you to read, transform and write data, using Python but not only. GDAL (omfg again) is also available in many languages through its bindings. If we focus on Python, a list (non-exhaustive of course) of libraries for working with spatial data:

Let’s try somethings with geopandas. If we consider that we are working with a single PostgreSQL database, we could reuse our earlier query to read the data and save it as a GeoPackage:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Open query file, apply and save
    with open("query.sql") as f:
        query = "".join(f.readlines())
        df = gpd.read_postgis(query, con)
        df.to_file("my_analysis.gpkg", layer="result", driver="GPKG")

        # Print the head for good measure
        print(df.head)


if __name__ == "__main__":
    run()

Will create a new file with your result saved in the result layer. It should also print something like:

0       France            2  Oh, no Hopp Schwiitz...      823.126128  LINESTRING (2537525.502 1152634.769, 2537254.4...
1       France            1  Oh, no Hopp Schwiitz...      118.753415  LINESTRING (2537290.714 1152264.433, 2537281.7...
2        Italy  No bus stop  Oh, no Hopp Schwiitz...      686.494161  LINESTRING (2538145.102 1151496.452, 2538283.1...
3  Switzerland  No bus stop           Hopp Schwiiz !      123.440260  LINESTRING (2536930.949 1152095.427, 2536884.0...
4  Switzerland            1           Hopp Schwiiz !     1102.779141  LINESTRING (2537281.727 1152146.02, 2537249.56...
5  Switzerland            1           Hopp Schwiiz !      206.632320  LINESTRING (2538019.283 1151332.542, 2538145.1...
6  Switzerland  No bus stop           Hopp Schwiiz !     1207.518986  LINESTRING (2537677.993 1151998.178, 2537874.0...

That was easy enough but we didn’t bring much since everything is done with SQL again. Let’s do the same thing but only using geopandas capabilities:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Read roads, regions and bus_stops
    roads = gpd.read_postgis("road", con)
    regions = gpd.read_postgis("region", con)
    bus_stops = gpd.read_postgis("bus_stop", con)

    # Intersection
    intersection = roads.overlay(regions, how="intersection")

    # Intersect bus stops
    intersect = intersection.sjoin(bus_stops, how="left", predicate="intersects")

    # Group by
    groupby = intersect.groupby(["name", "geometry"]).id.count()
    output = groupby.reset_index(name="n_bus_stop")

    # Hop Suisse
    output["hop_suisse"] = "Hopp Schwiiz !"
    output.loc[output["name"] != "Switzerland", "hop_suisse"] = (
        "Oh, no Hopp Schwiitz..."
    )

    # No bus stop
    output.loc[output["n_bus_stop"] == 0, "hop_suisse"] = "No bus stop"

    # Lost the Geo from GeoDataFrame at some point 
    output = gpd.GeoDataFrame(output, geometry="geometry")

    # Length
    output["portion_length"] = output["geometry"].length

    output.to_file("my_analysis.gpkg", layer="result_only_gpd", driver="GPKG")


if __name__ == "__main__":
    run()

Doesn’t take much more. Roads, regions and bus_stops could have been data from anywhere: another database, a csv file, a cloud hosted file, etc.

Working with a programming language allows you to easily read multiple data sources and combine them using your custom “transformers”, which will basically be functions that you would have written and tailored to your need.

But now let’s see what you can do to automate your process.

Automation

Not sure about what is so incredible about giving scheduling capabilities within FME when cron is available by default in many Linux distributions. If you’re working on Windows you can also schedule things with the Task Scheduler.

I feel like I’m oversimplifying things but I can’t see where. If you’re on Linux, you can schedule your data export task by adding such lines into your crontab (crontab -e to edit it):

# Everyday at 05:00
00 5 * * *  ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Run the task and send an email if it failed?

00 5 * * *  if ! ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"; then echo "your export failed" | mail -s "failing process" address@mail.me; fi

And that’s it. I doubt that FME brings so much on the table when it comes to scheduling tasks.

And that’s it?

I guess not, but I don’t think we will find many operations that are totally unfeasible without FME. If I’m missing something, please don’t hesitate to correct me.

Limitations

Unfortunately, there are some things that I don’t know how to do without FME, well at least one that I have no idea how to solve.

DWG and the closed formats problem

DWGs are ***** everywhere when your have people doing CAD around you. They could use DXF, which is an open source equivalent, but no, AutoCAD then DWG (which is no data btw, it’s just lines ffs). And I have no clue how to tackle that, except through a long process of mentality change around the tools used.

Earlier we saw that using standard formats can start a virtuous cycle, where you keep open the door for many solution to solve your problems. Well here we have the opposite, closed formats reduce the interoperability by making a handful of solutions your only options, and keep you from trying other things. If you need to have DWGs, then to my knowledge, FME becomes the only tool you can use to produce them programmatically from other data (even GDAL can’t all hope is lost). So from one license, you need “automatically” two.

And then one company decide to increase there prices, and there is nothing you can do about it.

In this case, the problem is to use a closed data format. To change that is to change deeply the habits of people using the tool creating this format, because, like with FME, there is a real addiction to the software. The skill becomes the software; not the technic, not the knowledge around the field, but just the software.

And that’s it for now

Fortunately, FME is not that essential. Maybe you won’t need to pay for the license for long… But I’m curious to read about what makes it a must-have for you. What can FME do that other tools cannot? Why is it necessary to you?

Conclusion

Feels good to rant a bit. To summarize:

Learn SQL
Don’t be afraid of command lines and programming languages
Have interoperability in mind
Get away from closed data formats

If you want to talk about all of this or if you need help converting some processes from FME to more open solutions, hit me up on Mastodon.

Managing shared QGIS projects

2024-04-19T00:00:00+00:00

QGIS is an amazing tool. Being spatial analysis or digitizing data, it’s efficient, highly customable and extensible. There is so much you can do with it.

So it’s only natural that we want to share a project that we are working on and propose to other people to join our effort on our amazing data digitizing/spatial statistics calculating masterpiece. Well yes of course, things are better when they are shared (most of them at least), and we go further together than alone (wow very profound), but this is something that has to be given some thoughts beforehand to ensure that everything goes smoothly within the collaboration.

We’ll briefly have a look at what can help make collaborating with QGIS successful. Also, we’ll be talking a lot about projects, we are referring here to the literal QGIS file used to store the work environment (data, visualizations, etc.). It’s not project in the sense of a project management, but some things might be useful to apply from the project management world. And in any case, working together requires tools outside of the technical aspects.

When does this apply?

Obviously if you’re alone on your project or working with a handful of people, some points might feel over-the-top or even counter productive, but some questions are always useful to be asked and it can be helpful to use some “good” practice, even when working for ourselves.

For the last years I had to pick up some projects that were shared by tens of people, create new ones, and I can safely say that there need to be rules when more people are getting on board.

So if you work with spatial data and you need to have people collaborating using QGIS you might be interested

Ok let’s go, how to collaborate with QGIS?

This is obviously from my personal experience. It’s not exhaustive, or maybe not accurate, but if it can help..

tl;dr

Scope your project: have objectives for the project and keep it within controlled boundaries.
Have a data structure: define which data will be included and model them.
Give roles and responsibilities: have a project management in place.
Version control: track changes and decompose your project if it becomes too big.
Share it efficiently: have everything in a single place, keep it as simple as possible, save it in database.

Scope your project

It sometimes feels like it’s a bad thing to say in an Agile world, but giving some boundaries to the project helps staying on track. Maybe this is something that I feel due to unstructured work environments, but when working on a (QGIS) project, if there is no beginning and no end there is just no way to succeed.

Back to QGIS. When starting, think about what the project will be about: will it be used to digitize data? Should data processing and spatial statistics be in the project? Should this data X be included? QGIS is a powerful tool, and maps are powerful visualizations, when people unfamiliar with these notions start to see what can be done they tend to want to add a lot of information in a project. Hell you can do your taxes in QGIS if you want, should you? Probably not.

Let’s say you design your project for a certain team, to help them digitize certain data. Obviously, since they are the target of the project, their demands should be considered, analyzed and implemented. But let’s also imagine that the project has been shared with another team interested in visualizing the data. Should you take their demands into account as well? Well, has the work induced by these demands been considered when designing the project? If not then these demands should be redirected somewhere else.

Maybe try to have some ground rules: one project per data structure (see below), keep separate the digitization, processing and statistics, one project per team, etc.

Of course if your time is bound to be spent on a couple of project you can fit a lot of things and optimize them, having everything working together. But don’t forget that the more people start to use these projects, the more impactful any change becomes, and separation of concerns can help a lot to prevent that.

Structure your data before thinking about QGIS

Have a coherent data model to work with. If you have people around specialized in modeling data and structuring a database, listen to them. If they tell you that you should normalize your data because you’ll otherwise have problems when scaling up, please do it. I think that’s the most important advice, and it doesn’t even concern QGIS.

If you’re wondering what this is about, data modeling is, very simply put, how to structure/divide your data into blocks that make sense, allowing you to add, remove, update data in the most efficient way. For example, let’s imagine that I want to have information about trees in my city and the population of birds they are housing. I could have everything in a single table like below:

erDiagram
    tree {
        serial id_tree
        int age
        geometry geom
        string bird_species
        bool is_bird_cute
    }

Ok cool I can now create one tree that has a family of pigeons:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true

But wait, there is also a family of crow next branch:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true
2	125	POINT(x,y)	crow	true

You quickly see the problem appearing, we duplicate as much data as we create new one. That’s where a little bit of modeling helps. A potential alternative could look like this:

erDiagram
    tree ||--|| location : is_located
    tree {
        serial id PK
        int age
    }
    location {
        serial id_tree PK,FK
        geometry geom
    }
    tree ||--o{ bird : host
    bird {
        serial id_bird PK
        string species
        bool is_cute
        int id_tree FK
    }

Where trees have their own table, birds as well and they are connected using a foreign key (FK). We create a table for trees location as well; it’s not really necessary in that case since it is a 1:1 relationship, but it often appears that separating the information about an object and its geometries simplify many operations. It also shows that you can treat geometries like any other column in a data structure.

The above tables should now look something like:

tree

id	age
1	125

location

id_tree	geom
1	POINT(x,y)

bird

id_bird	bird_species	is_cute	id_tree
1	pigeon	true	1
2	crow	true	1

If migration season comes and 200 bird families choose this tree as their home, we just need to add elements in the bird table without duplicating any data regarding trees.

Your data model should be independent from the tools you’ll be using to manipulate it. A client limitation shouldn’t question a well structured data; if the tool is limited, update it, or maybe it’s not the one you need for the job. How many time have I seen weird database designs (Postgres triggers I see you) because it was not possible to work with certain situations in QGIS or because of performance issues with certain designs.

It’s a step for which you’ll be grateful when people ask to add some values, some attributes etc. Because if you’re data is well structured, you need to bring the change to a single place. But if the data structure didn’t follow some rules, you’ll probably have to change things in multiple locations, introducing much more space for error.

Give roles and responsibilities

Who is going to modify the project? Who is going to modify the data? Should you give read-only access? Who is going to do the long term maintenance? Who should be able to propose changes? Some questions should be answered before starting to build the project.

If we know who is responsible for what, decisions can be more efficient and implementation becomes most of the time a matter of minutes.

You should know who is allowed to ask for modifications in a certain project, who should implement them and who is allowed to reject the modifications.

But honestly, in short, have a project management.

Decompose, centralize and version control you project

Hun? Decompose and centralize? Sounds like there is a useless step here, but let me explain.

A QGIS project is basically a big xml file bringing many elements together. It’s nice because everything (almost) is in the same place, but it also means that any little change influences everything, which could provoke side effects. And when it comes to modifications brought to layers it can become cumbersome to track.

To mitigate that, QGIS allows to export many of its layer elements into separate files: symbols, forms, rendering, etc. Think about what will be modified regularly. You can export these elements as separate files to make sure that you focus on their changes and keep a specific history for them. Imagine you often have to modify the colors of certain layers; export the symbologies in .sld or .qml. Like this you will have the history of these files outside of the project and will be able to access older symbologies without changing the whole project.

Regarding version control, it seems obvious now but it’s not that easy when working with QGIS projects since the file is completely rewritten if you change anything and then save. But as said, a project is a combination of many things; there can be plugins, symbologies, profiles, etc. Making the effort of having everything in a single repository (even the result of your decomposition process) helps to understand what is needed and where, and by using a version control tool with it you can always come back to an earlier version in case of mistakes or breaking changes.

When working with a version control platform (such as GitLab or GitHub), you can also bundle your versions into releases, allowing to have everything needed in the project into a single archive, and accessible directly from an URL, which can help with the sharing process.

I find it bizarrely difficult to share a project with people. As said right above, a project is composed of multiple elements that work together: data, symbols, plugins, etc.

There are some ways of avoiding loosing information when sharing a project:

Keep all data in a single place: store your data in a database or a database file (e.g. Geopackage), it helps having everything in the same place to avoid Non valid layers or losing information.
Keep the number of plugins low: plugins are nice and they can help a lot, but any plugin added introduce uncertainty. They might not be updated, be usable with certain QGIS versions etc. In a more general sense, keep your project as simple and as clean as possible.
Save the project file in database or database file. QGIS allows you to store the project in database, which makes it so much easier to share. You theoretically can have a single Geopackage holding all the data AND the project. You then just need to share the file to others and they have basically everything they need to work with the project.
Maybe don’t share the project? Think outside of QGIS. If you start to do something in QGIS, you don’t have to stay in it. Do you really have to share the project or do you want to share the data inside it? Should the people you share with have a write access or not? There are so many tools available, maybe the task you’re trying to accomplish should not be done by sharing a QGIS project. Look into web-clients, file formats or any type of software that could reach the objective.

Conclusion

Now after rereading, I realize that I wrote a weird mix of technical aspects and project management. There might be something that I need to learn from that. But the main message remains that the tool shouldn’t guide your objectives and the underlying structure. Find a way to articulate your work that makes sense to you and your team, and that allows you to be efficient in order to answer request as well as possible.

Anyhow, QGIS is a very powerful tool: it’s easy to use, allows to accomplish a lot of things with spatial data and makes you feel like a genius. But the tool in itself shouldn’t replace a proper project management. Think about the objectives, the stakeholders, the responsibilities and make sure that everybody is on-board with the rules. It’s not because you can do anything with it that a project should host everything. Keep it clean and simple, one objective at a time, and find the method for sharing information that suits your need. With QGIS, a monolith project with a lot of people asking for modifications is a bad idea, at least from my experience.

But in any case, it’s a amazing tool, support it.

Cheers.

Geoservices with qgis-server

2024-03-22T00:00:00+00:00

While the geospatial world has its own particularities that sometimes make it difficult to get on-board or communicate with (see the spatial exception), we have to admit that some tools and standards developed are both amazing and essential. Geoservices are crucial for the diffusion of georeferenced data, and they drastically changed the way we apprehend our localization; when was the last time you asked for direction?

Being WMS, WMTS, WFS, WCS or any W*S, geoservices are everywhere and sometimes are a bit mysterious. But even without understanding the fundamentals of their definition, it’s rather easy to get started.

We’ll see how to install a (geo)server that will be responsible for serving our data, quickly prepare some data, and even see how you can use these services to update your data safely. We’ll be using qgis-server here, because it’s the most easy to use and the quickest way to get started.

Some quick context

You’ve probably heard about the Open Geospatial Consortium (OGC), but to make it really short, if you hear about a standard regarding geospatial, the OGC is probably behind. At least it’s the case for the geoservices we’ll be diving into here.

And what are those geoservices? Well most of the time when we talk about “geoservice” we’re referring to WMS, WMTS, WFS, WCS. They are a standardized way of accessing georeferenced data through the internet. The same way you reach a website or an API using an URL, you can access some geographic data if you know some URL (and have the tools that go with it). You can then use the data for your use-case: displaying a background map in your app, showing queryable data for an interactive webmap etc. The most common are:

Web Map Service (WMS) –> probably the initial service. Used to display an image as .jpg or .png (see it as a background map)
Web Map Tiled Service (WMTS) –> the more efficient WMS. It’s the same, but images are tiled to restricted the amount of data to be transferred to the client based on the display localization and the zoom level.
Web Feature Service (WFS) –> the vector service. You can work on the data as vector, request features on the map, and even add, modify or delete features.
OGC API-Feature (OGC-APIF) –> the more “web standard” WFS, also known as WFS3. Features can be accessed individually through an URL. The OGC states that

The API building blocks specified in this standard are consistent with the architecture of the Web.
Web Coverage Service (WCS) –> at the time of writing I’ve never used this standard so I won’t go into details, but from my reading it’s a service close to the WMS standard but providing more information than just an image, allowing more complex interactions.

There are other existing services: vector tiles, WSC, WMC etc. But I either never encountered them or they usually are implemented by other tools than the common geoservers.

Ok let’s get started

It’s nice and all, but how do we serve a W*S. Well for starters we need a server. Here we’ll be using qgis-server with nginx. The choice for qgis-server is motivated by the simplicity of use, you serve a qgis project. The choice for nginx, well the doc mentions it and it seems very lightweigth. I’m by no mean a system administrator so bear with me as I’ll read the doc with you.

Install and configure qgis-server and nginx

I’m on Ubuntu so I’ll be using apt, and the installation includes the use of systemd. I’ll also be working with the ubuntugis ppa.

First add the ppa:

add-apt-repository ppa:ubuntugis

Then install qgis-server:

apt install qgis-server

Install nginx if not already present:

apt install nginx

Configure qgis-server

We’ll then follow instructions from the QGIS documentation for a qgis-server installation using systemd and nginx. The method defines a socket unit and a service unit as follow. The sockets will be the communication channel between our http server (nginx) and our application server (qgis-server). Create the socket file /etc/systemd/system/qgis-server@.socket with the following content:

[Unit]
Description=QGIS Server Listen Socket (instance %i)

[Socket]
Accept=false
ListenStream=/var/run/qgis-server-%i.sock
SocketUser=www-data
SocketGroup=www-data
SocketMode=0600

[Install]
WantedBy=sockets.target

You can start these sockets with the following command:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.socket; done

Which will start 4 sockets, through which our http server will be able to pass requests to our qgis-server.

Now for the services. Services are managed by systemd and in our case are responsible for the execution of the qgis-server. Create the file /etc/systemd/system/qgis-server@.service and add:

[Unit]
Description=QGIS Server Service (instance %i)

[Service]
User=www-data
Group=www-data
StandardOutput=null
StandardError=journal
StandardInput=socket
ExecStart=/usr/lib/cgi-bin/qgis_mapserv.fcgi
EnvironmentFile=/etc/qgis-server/env

[Install]
WantedBy=multi-user.target

Notice that the server is configured using a env file, here /etc/qgis-server/env. Finally, in the same manner as the sockets, start the 4 services with the following:

for i in 1 2 3 4; do systemctl enable --now qgis-server@$i.service; done

Services are depending on sockets, you’ll need to start the sockets first then the services. {: .prompt-warning }

Configure nginx

We’ll then configure nginx for our future qgis-server using fastcgi by creating the file /etc/nginx/sites-enabled/qgis-server and adding the following:

upstream qgis-server_backend {
   server unix:/var/run/qgis-server-1.sock;
   server unix:/var/run/qgis-server-2.sock;
   server unix:/var/run/qgis-server-3.sock;
   server unix:/var/run/qgis-server-4.sock;
}

server {
   listen 8080;
   access_log  /var/log/nginx/qgis-server/access.log;
   error_log  /var/log/nginx/qgis-server/error.log;

   location /qgis-server {
       gzip off;
       include fastcgi_params;
       fastcgi_pass qgis-server_backend;
   }
}

This configuration will then be included by the nginx.conf file using the include /etc/nginx/sites-enabled/*; statement.

Create a QGIS project and configure it

Now it’s time to decide what we want to expose. I have some OSM data in a PostgreSQL database (see Serving vector tiles with Martin) so I’ll use that.

Create a project, add some data and apply the desired style. A quick example is shown below.

Data with some style

Once you’re satisfied with the look of your data, you’ll need to configure the project so it will tell qgis-server to proceed with the geoservice you want to serve.

Go to Project -> Properties -> QGIS Server and complete the configuration as you wish.
Under Service Capabilities, check the Enable Service Capabilities box.
Select the different services you want to enable under the different tabs.
Save your project under a folder accessible by the server user (www-data). If you’re using file data in your project, such as a geopackage, save them in the same place.

If you setup a WFS, the tab should look like something like below.

Data with some style

WARNING

Make sure that your tables have a primary key. Otherwise the update and delete operations won’t be possible.

WARNING

And also, be sure that you don’t have special characters in you column or table names. It messes up the WFS 1.0 and 1.1 requests.

You can then edit the env file with the corresponding environment variables. For me it looks like:

QGIS_PROJECT_FILE=/etc/qgis-server/my_project.qgz
QGIS_SERVER_LOG_STDERR=1
QGIS_SERVER_LOG_LEVEL=3

I use PostgreSQL data and have had troubles getting the server to reach the data, even on localhost but finally managed to have everything working with the following steps:

Make the project read the data using a PostgreSQL service (create a pg_service.conf if not existing).
Copy the pg_service.conf file somewhere accessible for the server.
Edit /etc/nginx/fastcgi_params and add fastcgi_param PGSERVICEFILE /path/to/service/pg_service.conf;.
Restart everything.
Bonus: touch the project file.

If everything works correctly, you should be able to reach localhost:8080/qgis-server/qgis_mapserv.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities for WMS capabilities or http://localhost:8080/qgis-server/wfs3/ for the OGC-APIF description and navigate the different endpoints defined in the OGC-APIF standard.

You should now be able to also call these URL from QGIS using the correct url. Try to add a WFS/OGC-APIF using the http://localhost:8080/qgis-server/wfs3/ url (qgis-server in this example, but you can setup your server with a different url).

Going further

We have the basics setup, now what can we do? Let’s start by alter data.

Updating data through a WFS

If at this point you have everything working, there shouldn’t be much to do to update data through your newly build WFS. Working with the OGC-APIF you’ll be able to add, delete and update data using URL like you would do with any REST API.

You’ll find the description of your API at the URL http://localhost:8080/qgis-server/wfs3/api (for this particular setup) and it should look something like this:

OpenAPI description of an OGC-APIF

This gives you an idea of the available endpoints to manipulate data.

To try this, let’s try to GET one of our feature. You can access a single feature through a collection and an item. For me, the URL http://localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736 will point to the Lausanne train station:

Getting an item using its dedicated URL

And the same can be done using curl or wget or any network utility you wish:

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736" | jq .

will print something like:

{
  "bbox": [
    6.627779,
    46.51671,
    6.630578,
    46.517298
  ],
  "geometry": {
    "coordinates": [
      [
        [
          6.627779,
          46.517099
        ],
        [
          6.628225,
          46.517038
        ],
        [
          6.628344,
          46.51702
        ],
[...]
  }
}

TIP

The piping into jq (| jq .) is just for printing purpose, it doesn’t participate in the query.

Now let’s delete this object:

curl --request "DELETE" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

And that’s it!

curl --request "GET" "localhost:8080/qgis-server/wfs3/collections/planet_osm_polygon/items/2102736"

now returns:

[
  {
    "code": "Internal server error",
    "description": "Invalid feature [2102736]"
  }
]

And you can do exactly the same using QGIS. Add your WFS as a layer and work on it like any vector data: add, update ,delete features and then save.

TIP

I intended to use a PATCH request to update a single attribute, but couldn’t make it work. QGIS seems to systematically call a POST request, even for a single attribute change. If you now how to work with PATCH in this environment let me know.

Protect our service with authentication

Accessing information through an URL means that many client might have access to it as well. It’s important to keep in mind that authentication would be necessary if you expect to update information through this API.

Again, I’m not a system administration or a security expert, but I’ll try to show how to setup a very basic authentication directly using nginx. This is not how you secure an API and it won’t replace a modern and secure authentication setup.

I’ll follow the nginx documention for that.

First install apache2-utils if your on a Debian based distribution:

sudo apt install apache2-utils

Create a password file and a user:

sudo htpasswd -c /etc/apache2/.htpasswd user1

In the nginx configuration, under the location of your API, add a name and specify that you’ll be using a auth_basic_user_file.

location /qgis-server {
    [...]
    auth_basic           "Protected Area";
    auth_basic_user_file /etc/apache2/.htpasswd;
}

Restart nginx for good measure and that it, try to reach your WFS using an URL or with QGIS and you should see a connection popup like this:

The route now needs basic authentication to be reached

Conclusion

Services are a powerful tool to be aware of when working with geographical data. They allow to efficiently share them at a larger scale through the web and even, as we saw, update them. There are many ways to build these services, and qgis-server is one solution among others, but it has the advantage to work (almost) out of the box with potential QGIS projects that you could already have: a few tweaks and you can be going.

The possibility to give access to your data through an URL makes sharing them so easy; it reduces the amount of manipulation and tools required, you don’t need to maintain a data extractor or repositories. Just some data and a server.

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

I’ve been trying to learn Rust lately. For some reason there is something that I find very appealing in this language. From all I could read, it’s very efficient in resources and memory, it’s fast, and it doesn’t let you mess around with memory. Regarding performances, it places itself next to C and C++, and when looking into the world of language performances, I learned that Python was kind of slow, around 100x slower according to some benchmarks. Obviously this value in itself doesn’t mean much, and many more aspects of a language must be taken account when choosing one for the task, but this information kind of planted its seed. It might be because I have a problem with inefficient and overly complicated process, but I find the idea to work with a system language very elegant.

But we’re not here to talk about the Rust language in itself. We’ll see how to read and write raster data using the Rust binding for GDAL. Also, this is about how I tried to compare the Rust and Python bindings for GDAL and how I learned that it doesn’t change a thing in terms of performance.

Wait what?

Well I started by thinking “Oh I can’t wait to see how faster GDAL in Rust is compared to Python”. So I started by reading a raster file and print some basic information. I use a band from a Sentinel-2 image.

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

I’m using a lot of .unwrap(), which in very short will cause the code to panic if a problem occurs, this is for a quick and dirty test. Within a “production” environment you would handle errors and stuff.

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

We will be using the Create() approach rather than the CreateCopy() (see the nuance in the doc). Geospatialized rasters come with series of metadata and overviews. The CreateCopy() automatically takes those from the source dataset and copy them into the destination. While this obviously simplifies the work if we don’t need to bring any change to these information, using Create() helps to better understand the structure of a raster. Also I have a quite important performance difference between the two and I’ve found Create() more efficient for my use case.

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

Both codes should create a new raster file that resembles our source, but in GeoTiff. And that’s it, you can convert a raster file into any format. We’re just scratching the surface, you can subset data, reproject, transpose it if you want etc.

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

So from my understanding, working with Rust or Python or any other language bindings is just working with the API that let us call the underlying code, for GDAL written in C and C++. Thus the overhead that we generate is only the interaction with the library, not during the process itself, since it’s running C/C++ code. The difference that we observe is probably the same that we were observing when we just opened a file.

Conclusion

GDAL raster processing is very similar in Rust and Python. When working with its bindings, it doesn’t change much which language you use to perform the GDAL operations in themself, since in any case you’re running C/C++ code from your program. But everything that is around need to be considered when choosing the right tool. If your program needs multi-threading then you’l probably chose Rust, if you need quick development without much optimization then Python might be a best fit. But it’s comfortable to see that the GDAL processing time won’t change significantly.

I’m not at all trying to say that Rust is not worth it, I’m excited to go further into it and work on projects, but it remains a tool and there should be no injunction to use it if it doesn’t bring much to the table for your use case.

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.

PMTiles exploration

2024-02-02T00:00:00+00:00

It’s a technology that I came across rather recently. Tile generation when maintaining a geographic oriented website can be a hassle: it can take time, generates a lot of files, and can be a critical process. PMTiles might have the potential to facilitate some of this work and we’ll try to see how.

What are PMTiles?

In itself, PMTiles is a specification for building single-file tile pyramids. It’s open source and part of a larger project called Protomaps, and it aims at mapping the World, just that. The major difference with the kind of tiles that I know is that they are embedded as a single file, like the MBTile format, which make the interaction very intuitive, I mean we all get the idea of what a file is. But what does it bring compared to MBTiles? Well from the Protomaps website:

A MBTiles file needs to be accessed as a SQLite database on disk, so requires a running server and disk storage for cloud hosting. PMTiles is designed to be cloud-native and readable in fine-grained parts from remote storage like Amazon S3.

It aims at simplifying the manipulation of tiles in a cloud native approach, for worldwide coverage.

How to create PMTiles?

The Protomaps website gives us some tools to create these tiles. Let’s try some possibilities.

Generate PMTiles using GDAL

Our Swiss Army Knife is back, starting version 3.8.0 GDAL has native support for PMTiles, so let’s try that.

First check the GDAL version you’re using, you’ll want 3.8.0 or higher:

gdalinfo --version
--> GDAL 3.8.3, released 2024/01/04 (debug build)

Let’s start from an OSM .pbf file (see Serving vector tiles with Martin). To convert this file into a PMTiles file you can enter:

ogr2ogr -f PMTiles out.pmtiles switzerland-latest.osm.pbf

The process might take some time if you’re working on a large dataset. By default the MAXZOOM is 5, which is rather low, so the run shouldn’t take to long, but if you start to increase the parameter to around 15, which allows to represent data from much closer, processing time increases.

But once the command is done, well that’s it!

Alternatively we can use tippecanoe to achieve the same result.

Generate PMTiles using tippecanoe

WARNING

Under construction

Visualize PMTiles

The easiest way is to go to the PMTiles Viewer Page and upload your newly created PMTiles file.

Obviously you can add it to your custom project. The easiest way of doing is by using the Leaflet plugin protomaps-leaflet.

Starting from complete scratch you can create an .html file with the following content:

<!DOCTYPE html>
<html>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"></script>
    <script src="https://unpkg.com/protomaps-leaflet@latest/dist/protomaps-leaflet.min.js"></script>
    <style>
        body,
        #map {
            height: 100vh;
            margin: 0px;
        }
    </style>
</head>

<body>

    <h2>Hey I'm a custom page</h2>
    <p>Let's add some Leaflet and PMTiles.</p>
    <div id="map"></div>

    <script>
        let PAINT_RULES = [
            {
                dataLayer: "multipolygons", //here we render only one layer, check your layers names 
                symbolizer: new protomapsL.PolygonSymbolizer({ fill: "steelblue" })
            }
        ];

        const map = L.map('map').setView([46.5, 6.6], 13); //chose a view that suits you

        var osm = L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png', {
            maxZoom: 19,
            attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
        }).addTo(map);

        var layer = protomapsL.leafletLayer(
            {
                url: 'http://127.0.0.1:8080/your_pmtile_file.pmtiles',
                paint_rules: PAINT_RULES
            }).addTo(map);

    </script>

</body>
</html>

For testing purposes you might have to serve you PMTiles through http. To do so you can do as the Protomaps website suggests:

npm install -g http-server
# Then from the folder containing your PMTiles.
http-server . --cors

Open your .html file on the browser of your choice and you should see the map with your PMTiles displayed.

When should PMTiles be used?

Again, the documentation on Protomaps website is quite complete and gives us some insights.

Cloud hosting

PMTiles are stored as a single file, and compression is so efficient that the size is ridiculous compared to classic png tiles. Number of requests necessary is drastically reduced and thus the cost of operation as well.

Static data

PMTiles don’t support localized updates so each time you’ll need to apply an update, the whole file needs to be rewritten. If your data don’t change that much PMTiles are a wise choice, but If you need to have a more dynamic rendering the documentation recommends PostGIS with the ST_As_MVT() function.

Simplicity

Here it’s more like a feeling about the philosophy of the specification. You have a single file, accessible without introduced complexity, directly from a library like Leaflet, which itself has simplicity as one of its core objectives. It looks like one of those project that allows you to make complicated-looking things so easily. I guess it’s a sign of a good project.

Final words

Well it was rather easy to discover. I mean we create a file, we basically load it within a project and we’re good to go. Obviously there are some customizations to do with styling and labels etc. But once you have the starting point and your tools, it takes minutes to create what you desire.

Here we’re just scratching the surface, remains to explore filtering, styling etc. But the concept in itself and the unreasonable performance of the compression make it very interesting if you’re in the situation where PMTiles could give you an advantage; the file created for OSM data Switzerland, with zooms from 0 to 14, is around 1GB, it costs pennies to upload and host on cloud services.

It’s a project to support and look for, I guess?

A little bit of GDAL/OGR appreciation

2024-01-19T00:00:00+00:00

If there had to be one single tool to work with spatial data, I think GDAL/OGR would be the first name to come to mind. Its usages range from raster to vector, it can be used as command line, imported as a library in many languages through bindings developed by the community, many applications provide UI to parameterize and run GDAL etc. Its versatility makes it an amazing tool to work with spatial data in a lot of different contexts, but it’s sometimes underestimated or even misunderstood.

We will quickly introduce GDAL/OGR, then give some examples of usage to help for some quick starts.

What is GDAL/OGR?

GDAL/OGR (Geospatial Data Abstraction Library/OpenGIS Simple Features Reference Implementation) is a library for working with raster and vector geospatial data formats, coded mainly in C++, C and Python. The first release dates from 2000, and the project has now more than 450 contributors at the time of writing. This is a very active project with regular releases.

The project is composed of two parts: the raster processing library GDAL and the feature processing library OGR, both merged during the release of GDAL 2.0. When working with raster data you will use programs prefixed with gdal (gdal_translate, gdalwarp etc.) while vector data will use ogr prefixed scripts (ogrinfo, ogr2ogr etc.). When talking aboutt GDAL, both libraries are included, and we’ll do the same from here.

As its name suggests, GDAL abstracts data in order to standardize its way of interacting with it. The driver creates the interface between the data source and the tools, allowing to focus on the logic of the process rather than how to read or write the data. This makes it a very extensible project; including a new format of data source requires to write a new driver, nothing more. You could see that the list of supported formats is quite something, either raster or vector.

GDAL describes itself as a translator, meaning that data come from a source and goes to its destination. Translations can occur at different moment during the process, during reading or by the script called. These translations can occur on multiple axis as well, we could for example:

Extract a subset of source data depending on a region of interest.
Extract a subset of source data based on an attribute.
Reproject data into a different coordinate system.
Merge a series of data files into a single one.
etc.

Source unknown, if it’s your image and you want to be cited please leave a comment

As an open source project with a permissive MIT license, GDAL is used by many projects, open source or proprietary, to process geospatial data. If you’ve used QGIS, ArcGIS, FME or PostGIS, you’ve used GDAL under the hood.

Why would I use it?

Weirdly, even though GDAL is basically everywhere there is geospatial data to be moved, it’s not so much acknowledged by GIS people, or at least it’s not really used on its own. If you read this, you definitely use GDAL, but maybe without noticing.

Ok but why would I complicate my life with some command lines when the same tool is being used by a software that provides me a user interface easier to use?

While it’s much intuitive to explore geospatial data through a user interface such as QGIS, where you can interactively load your data, chose the process you want to apply, its parameters, and visualize the results, things get a bit more complicated once you’ve done the exploratory part and want to reapply your newly defined process to new data. Repeating operations becomes quickly tedious and error-prone. You could use a tool such as the Model Designer from QGIS (amazing little tool, check it if you repeat a lot of operations in QGIS), but how do you schedule your process to be started when you’re not able to launch it manually?

If you use a heavy client ETL such as FME, try once to compare the difference in speed between a simple table transfer done by launching a FME workspace (using a command line) and a simple ogr2ogr command, there is at least an order of magnitude of difference.

It’s about using a tool designed for a purpose as efficiently as possible.

How to install?

We’ll assume you have no installation of GDAL yet, but just to make sure you can try:

gdalinfo --version

which gives:

GDAL 3.4.1, released 2021/12/27

If it’s not installed you can refer to the installation instructions from GDAL website.

For example, I’m working on Ubuntu 22.04, for which the GDAL package is the 3.4.1. The latest version can be installed by downloading the latest release, but I usually stick with the provided version for my platform, unless I need a specific update.

apt update
apt install gdal-bin

And you should be good!

WARNING

You might already have GDAL installed through another software like QGIS, be careful during the installation, multiple installations might interfere and create warnings or errors. On Windows you can use the OSGeo Shell to access any tool from the OSGeo ecosystem.

And how do we use it?

It’s easy, you just need to be comfortable with command lines. If you’ve mostly worked with user interfaces it might look scary, but it really is not complicated.

Access metadata

The first thing we usually do with data is to have some information about it. Let’s try to see the metadata of a random .jpeg:

gdalinfo gdal.jpeg

Driver: JPEG/JPEG JFIF
Files: gdal.jpeg
Size is 711, 882
Metadata:
  EXIF_Orientation=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  882.0)
Upper Right (  711.0,    0.0)
Lower Right (  711.0,  882.0)
Center      (  355.5,  441.0)
Band 1 Block=711x1 Type=Byte, ColorInterp=Red
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=711x1 Type=Byte, ColorInterp=Green
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=711x1 Type=Byte, ColorInterp=Blue
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG

Easy right? And it’s not even georeferenced, it’s a .jpeg, a driver exists, it opens.

We can try the same with some vector data. We’ll have a look at what is going on in one of the PostgreSQL database hosted on my computer.

ogrinfo -ro PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.
1: planet_osm_line (Line String)
2: planet_osm_polygon
3: planet_osm_roads (Line String)
4: planet_osm_point (Point)
5: natural_polygons

And as we can see I have 5 layers in my database. Notice how we used the ogrinfo command for questioning vector data. Here ogrinfo (like gdalinfo) requires a dataset source as input, the syntax will depend on each driver, make sure to have a look at the documentation. For a PostgreSQL connection, the line could have looked like:

ogrinfo -ro PG:"dbname='osm_suisse' host='localhost' port='5432' user='x' password='y'"

We can extend the information that we want to see for each layer by adding the -al flag to the line. The option -so will summarize the output.

ogrinfo -ro -so -al PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.

Layer name: planet_osm_line
Geometry: Line String
Feature Count: 2137192
Extent: (626211.331305, 5706647.213604) - (1253686.695428, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Geometry Column = way
osm_id: Integer64 (0.0)
access: String (0.0)
addr:housename: String (0.0)
addr:housenumber: String (0.0)
[...]

And now we have the full detail of our layers; name, type, feature count, etc.

Basic translations

Now that we know what were working, with we can try to move some things around. Let’s start by exporting our vector data into a geopackage.

ogr2ogr -f GPKG exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

Here we’re exporting the table planet_osm_point into the exported_data.gpkg geopackage. The format can be recognized by the extension of the output file, but it’s safer to ensure the format by passing the argument -f. Again, each driver documentation will have its driver name to be used.

Add a filter

We can export a subset of our data, based on an attribute for example. Let’s say we want to export the points where the attribute addr:housenumber is not null, and we want to save it in another table in our newly created geopackage. We’d do something like:

ogr2ogr -update -f GPKG -where '"addr:housenumber" is not null' -nln only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

We want to update our dataset (our geopackage) and not create a new one, so we use the -update option. The syntax of the -where parameter is a bit convoluted here since there is a special character in the column name, but with a simple name you’d just need a pair of quotes -where "housenumber" is not null".

Let’s have a look at our geopackage:

ogrinfo -ro -al -so exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: planet_osm_point
Geometry: Point
Feature Count: 2146246
Extent: (626415.435591, 5736097.599850) - (1189428.532375, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
[...]

Layer name: only_housenumbers
Geometry: Point
Feature Count: 366561
Extent: (663719.309401, 5751784.044673) - (1167785.106794, 6072459.331387)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
[...]

The new layer has been added to the geopackage. Has the query been respected?

ogrinfo -ro -al -so -sql 'select * from only_housenumbers where "addr:housenumber" is null' exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: SELECT
Geometry: Point
Feature Count: 0
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
[...]

Yes! No data without addr:housenumber.

TIP

The -sql option is another way of filtering data, it allows you to pass a sql query in any of the sql flavor supported by the driver, you can even save your queries in a text file and pass the file to the -sql parameter as -sql @path/to/my_file.

Geometric operations?

The -sql parameter can also be used to apply transformations to the data. We’ll request the same points, but this time we’ll create a buffer around them.

ogr2ogr -update -f GPKG -sql 'select ST_Buffer(way, 50) from planet_osm_point where "addr:housenumber" is not null' -nln buffered_only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

And that’s it, we have our buffer around the points of interest. Here GDAL makes use of the internal capabilities of PostgreSQL and PostGIS by passing directly the request to the database, but the same command using the only_housenumbers layer from our geopackage would also be valid.

ogr2ogr -update -f GPKG -sql "select ST_Buffer(way, 50) from only_housenumbers" -nln buffered_only_housenumbers exported_data.gpkg exported_data.gpkg

TIP

In the case of PostgreSQL/PostGIS you can write the -sql parameter like you would write your queries directly into the database. They are passed directly to the database by GDAL.

This should give you a small idea of the things you can do with GDAL and how to start experimenting.

Final words

Well that was a lot more than expected. But GDAL really is a very strong and important tool. Diving into the details of all the available options, both for the command AND the drivers, is not a small task, but understanding how to navigate around and find the information that you need to process your data as you wish can go a long way. Manual and repeated tasks can be replaced to help to focus on more pertinent work, process time can be (by experience) reduced drastically depending on the tools you’re initially using. And if you’re working on data hosted on a server without graphic interface, well you have a tool to get the info you need!

Obviously GDAL alone can’t do everything and has its imperfections; I’ve really struggled with installations in the past (Python binding might have bested me at some point), and the number of parameters is a bit overwhelming. But once you have it working it does the job, and really well.

If you can, use GDAL.

Serving vector tiles with Martin

2023-12-20T00:00:00+00:00

There are some amazing tools out there, and they are getting build fast. I haven’t had the opportunity to work with vector tiles during my day-to-day jobs, but I wanted to explore what it is and how to set some things up, just for the fun of it. We’ll see here how to start using Martin, a vector tile server built in Rust serving PostGIS data.

Vector tiles

The standard has been set by Mapbox. In the same manner a WMTS serves raster tiles, vector tiles allow for the visualization of large volumes of data over the web, but as their name suggests, they are vectors.

It comes with some advantages compared to raster formats:

It’s significantly lighter, about 20 to 50% of its raster equivalent according to MapTiler. Vector data are series of points and each pixel of the map doesn’t have to have information declared; a rectangle polygon covering thousands of square kilometers can still be defined with 4 points.
It is stylized dynamically. While raster tiles need to be generated with their style embedded, vector tiles style can just be modified on the fly, without going through the laborious process of tiles generation that can take easily several hours/days.
It’s fast. Moving through a vector tile map feels smooth, there is no apparent tile loading.
What you request is what you get. You directly query the tiles, not some sort of other service along your tiles.

It’s an amazing standard to visualize spatial data in a web map. Obviously it’s not suitable for all kinds of data, you have to have discrete data that make sense to be visualized as vector; I doubt that raw satellites images as vector tiles are a good idea.

Installing PostgreSQL and PostGIS

First thing first, we need to store data somewhere. Martin is designed to be used with PostGIS, PMTiles and MBTiles, we’ll explore the PostGIS alternative.

The following commands were used on an Ubuntu distribution.

Let’s start by installing PostgreSQL:

apt install postgresql

Then the PostGIS extension:

apt install postgis

We’ll already set things up in the database by creating a dedicated user, creating a database and creating the PostGIS extension in there. From psql, create a user:

create user martin_usr with password 'my_password';

Create a database:

create database martin_db;

Go to your new database (\c martin_db) and create the extension:

create extension postgis;

We won’t cover here the more extensive tuning of the database. You should be set on the database side for this example.

Downloading some data

Obviously you’ll want some data to serve as vector tiles. In a production environment you’ll probably want to serve your own data, so this step would probably not be necessary. For the example we don’t have any data yet, so we’ll use some from osm through GeoFabrik. Since I’m in Switzerland, let’s download data over the country:

curl https://download.geofabrik.de/europe/switzerland-latest.osm.pbf -O

We are using osm2pgsql to import data into the database, so the .pbf format is ideal.

You can then load your data (apt install osm2pgsql if you don’t have the tool):

osm2pgsql -d martin_db -U martin_usr switzerland-latest.osm.pbf

We now have osm data over a whole country, easy right?

Now install Martin

Here comes the best part, the vector tile server itself. There are multiple ways to start working with Martin, we’ll use rust and its package manager cargo. If you don’t want to install those on your server you might prefer the Docker installation, or simply using binaries.

As said earlier, Martin is build in rust, and it’s accessible as a crate using cargo. You can first install rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then install Martin:

cargo install martin
martin --help

And that’s it! Martin’s here.

Can’t be that simple right?

Finally, you can run your new server:

martin "postgresql://martin_usr:my_password@localhost:5432/martin_db"

Martin can discover geometric tables in the database from the given url, so by default everything is served. So yes it can be that simple! Obviously environment variables need to be set, passwords hidden etc. But you have an idea of the process.

You can try by yourself reading the catalog:

curl http://0.0.0.0:3000/catalog | json_pp

{
   "fonts" : {},
   "sprites" : {},
   "tiles" : {
      "planet_osm_line" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_line.way"
      },
      "planet_osm_point" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_point.way"
      },
      "planet_osm_polygon" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_polygon.way"
      },
      "planet_osm_roads" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_roads.way"
      }
   }
}

Visualize data using QGIS

Once the server is running we can easily visualize our data in a client. Starting QGIS 3.14,

Layer > Add Layer > Add Vector Tile Layer

And enter the url then OK:

http://localhost:3000/planet_osm_line,planet_osm_point,planet_osm_polygon/{z}/{x}/{y}

You should see a rather absurd amount of data.

To conclude

Here is how to get started with Martin, a rust vector tile server. It’s just the start, we don’t cover here the creation of symbology and more in depth configurations and optimization, but it gives an idea of how easy and powerful such a tool can be.

On GIS and their integration

2023-12-15T00:00:00+00:00

Since I started working with spatial information, I’ve seen the term GIS used to define quite different things; ranging from mastering data input on QGIS to weird command line geeking. This variation of definition is very interesting, even though it results in misunderstandings and complicated situations at an organizational level. We’ll try to see where this can come from and maybe how to tackle it. This comes from my own personal experiences and the different working environments that I’ve observed.

Where does GIS come from?

We won’t re-write a post about GIS history here, other resources do this very well. You can have a look at ESRI’s website, this short introduction from the University of South Carolina, or GISGeography (first 3 search engine results huhu).

It’s interesting to note that GIS originate from environmental questions:

How can I link my natural resource inventory data to spatial information?
How can I visualize my resources?
How do I plan my exploitation?
etc.

Which makes sense, spatial is mostly about territory, and locating things is a very practical need, it’s also a very intuitive information. Even when we look at the origins of QGIS, which is the exact definition of a software, we see that it’s been initiated by Gary Sherman (interview), who was at the time a geologist in need of visualizing PostGIS data in a Linux environment.

At this point, GIS defines a tool, a software that helps you visualize, analyze, maybe process, spatial data. QGIS, ArcInfo/Map/GIS, MapInfo are examples of GIS in this context. You have some tables stored in files or a database, you connect your GIS to those data, and you have an information system that could manage the entirety of your company spatial information. It feels like this is still the most widespread definition of what GIS are. Even if this definition used to be accurate, the context we are in today is different, and I find it limiting. The amount of data we’re working with now is not the same as 20 years ago, and spatial data can’t be kept out of broader information systems.

The spatial exception

In spatial data there is spatial, and depending on the perspective, this can be a big deal. Coming from the applied side of data analysis, I used to think that spatial data had to be given a special treatment, as they are spatial. Obviously you can’t work with these data in the same way as you’d do with other types of normal data, right? Well they are spatial, but they are primarily data, and now that my head is most of the time in databases I’m focusing much more on that part of the expression. It feels weird to discover one day that the thing you thought special about what you are doing is basically a column in a table, and that it can be as simple as a text.

Obviously, geometries come with their very special and necessary set of tools and skills: input of spatial data, spatial indexing, spatial analysis and statistics, map creation and visualization etc. It does have its own set of challenges, and the developments of geospatial standards were necessary to respond to them. But now, the fact that it has its own protocols, catalogs, specialized management, all parallel to the world of data, makes it a type of information difficult to merge with the rest. It can even be very concrete, I’ve observed that in some companies, geospatial data units will exist out of the more global ‘data team’, using their own tools, standards and strategies, resulting in two worlds doing essentially the same job, but somehow uncoordinated. What is the point of using two different tools for transferring data between two tables?

If we considered spatial data separately, GIS could be an entire field, with its own unique and distinct rules. This definition, unlike the previous one, feels to broad and almost pretentious. Spatial data is not that special. It had to have its own standards to solve problems in its time, but with the general improvement of tools and computational capacities, the spatial field will probably gain from getting closer to the broader data community. The growth of the open source community and the speed and robustness of developed tools could benefit the spatial community.

A shift seems to be occurring though, the spatial world is finally colliding with what the rest of the domain has been doing (e.g. WFS3 following REST principles). Interoperability is becoming more and more present, and spatial standards joining broader standards will facilitate that.

OK so what?

Someone asked me recently, what is the role of a GIS specialist, and this stirred up some thoughts. Purely by experience, I can say that during recruitment processes and in day-to-day work, being specialized in GIS means a lot of things. I happened to set up Linux servers, install services, develop and schedule data pipelines, change map symbology in QGIS projects, all within the same job, with the same title: GIS specialist. To me, it doesn’t make much sense. GIS can’t be reduced to a single software like QGIS (amazing tool though). Neither can it realistically be its own independent network with specific rules.

Maybe we are forgetting a simple part, the information system part. If we take the Wikipedia definition we get:

An information system (IS) is a formal, sociotechnical, organizational system designed to collect, process, store, and distribute information.

Prefixing it with a G doesn’t change the essence of an information system. Many roles and skills are necessary to make an information system work, being geographic or not. From the definition it should make possible to collect, process, store and distribute data, geographic in our case. There need to be user interfaces developed for entering data, servers to be administered, databases to be architected, data to be analyzed, visualized, distributed etc.

GIS are information systems and should be treated as such. There is no reason to keep them out of more general policies and rules when it comes to organizational or technical decisions within a company.

What to do

I happened to be in the situation where, as a GIS specialist, I was completely out of the IT department, and I mean organizationally, physically and numerically, not even on the same network within the company. Trying to get closer to the IT seemed to be necessary in that case. Making sure to be aware of data policies, standards in data storage and processes, knowing about conventions, rules, tools etc. I tried to reach out for services responsible for data management, and it’s been very well received, and even encouraged. By doing so I could align on guidelines and finally start to take advantage from larger means.

It’s a simple question of using resources efficiently. GIS is now a completely digital field, while companies are more and more sensible to the importance of IT infrastructure, data management. It makes sense to bring spatial data into the equation. Resources will be given for the IT architecture, and GIS should be integrated.

Spatial data add something essential that other type of data don’t, but forgetting that they are firstly data is a mistake. I think a good example/metaphor of spatial integration would be how PostGIS works within PostgreSQL, it’s just an extension. Spatial data can make use of all the broader features of Postgres, and additionally they have their own set of tools for their special aspects, without being completely isolated.

geothings

Serve PDF and DXF files using qgis-server

Setup

Export PDF files

Export PDF files: atlas version

Export DXF

To conclude

Building a geospatial strategy, sharing a real world experience - Part 1

Context

Getting started

What is a geospatial strategy?

Align the visions

My initial proposition

To conclude

How to build a single file QGIS project?

Introducing Geopackage

Storing your project in Geopackage

Storing your data in Geopackage

Storing styles

Conclusion

FME is getting expensive? Stop using it

About FME and Safe

Safe as a company

FME as a software

What alternatives are there?

Read, Write and Data transformation

Transform with SQL

Read and write with GDAL

Do everything with (Python) libraries

Automation

And that’s it?

Limitations

DWG and the closed formats problem

And that’s it for now

Conclusion

Managing shared QGIS projects

When does this apply?

Ok let’s go, how to collaborate with QGIS?

tl;dr

Scope your project

Structure your data before thinking about QGIS

Give roles and responsibilities

Decompose, centralize and version control you project

Find an efficient way to share your project

Conclusion

Geoservices with qgis-server

Some quick context

Ok let’s get started

Install and configure qgis-server and nginx

Configure qgis-server

Configure nginx

Create a QGIS project and configure it

Going further

Updating data through a WFS

Protect our service with authentication

Conclusion

Opening and writing raster dataset with GDAL Rust

Wait what?

How to open a raster file?

Comparing performances

Let’s add some writing then

How to write a raster dataset with GDAL bindings?

But let’s compare performances again

Wait what? (bis)

Conclusion

PMTiles exploration

What are PMTiles?

How to create PMTiles?

Generate PMTiles using GDAL

Generate PMTiles using tippecanoe

Visualize PMTiles

When should PMTiles be used?

Cloud hosting

Static data

Simplicity

Final words

A little bit of GDAL/OGR appreciation

What is GDAL/OGR?

Why would I use it?

How to install?