geothings - tutorial

How to build a single file QGIS project?

2024-09-28T00:00:00+00:00

We talked about how to collaborate on QGIS project at a “project management” level, but in practice, how do you make things easy for you to easily share a project. Well, one way is to minimize the amount of files by:

Storing your project in database
Storing your data in database
Storing your styles in database

Introducing Geopackage

Geopackage is basically a SQLite database with well defined rules for storing geospatial data. You get a single file that can contain all your data, stored in different layers. The standard has been extended to allow storage of QGIS projects, which allows to bundle a lot into a single file.

Storing your project in Geopackage

The first step would be to store your new project into a GPKG file. Open QGIS and create a new Geopackage database:

Layer > Create a new layer > New Geopackage layer

Then you can save your project in this newly created Geopackage:

Project > Save To > GeoPackage...

And that’s it, you can then open this project from QGIS:

Project > Open from > GeoPackage...

You now have a Geopackage containing your project.

GeoPackage
- Project

Storing your data in Geopackage

It’s rather self-explanatory, your can store all the data used in your project in the same Geopackage. If you use shapefile, please stop, it been making no sense for a while now.

Convert your data into Geopackage layer by right clicking on your layer of interest:

Export > Save features as... > GeoPackage

Select the Geopackage database containing your project as File name.

At this point you should already have your project and data in a single file, which is already way easier than managing multiple data sources.

GeoPackage
- Project
- Data
  - layer 1
  - layer 2
  - …

Storing styles

Having your data stored is nice, but styles are also important to add symbologies and capabilities to your layers. Geopackage also allow to store your style directly.

Define your layers styles (forms, symbologies, actions etc.) and then right click on your layer:

Properties... > Symbology > Style > Save Style... > In Database

And hey you can even go further! If you have .svg symbols in your style, you can load the symbol definition in the Geopackage. Like this you avoid broken links and unfound .svg file.

To do this, first create a new table in your Geopackage:

Layer > Create Layer > New GeoPackage Layer...

Select your Geopackage and fill the layer name (symbols or whatever you like). Then under New Field, add a field of type Binary Object (BLOB) and create the layer.

Add svg to Geopackage

Then add a new feature in your symbols and embed your .svg file in the BLOB column.

Finally, from your data layer symbology, select the SVG Marker symbol type and Edit the definition of the SVG with the rule of your choice, for example:

'base64:'|| to_base64(attribute(get_feature( 'symbols', 'fid', 1), 'svg'))

Here we’ll use the symbol with the fid value 1. You can create the rule that wou want, but make sure to encode the binary definition into base64.

TIP

You can also directly embed the file instead of creating the rule, you’ll just lose some flexibility in the choice of you symbol. A rule-based choice would allow you to use different SVG definitions based on a attribute for example.

At this point you should be in a situation like this:

GeoPackage
- Project
- Data
  - layer 1
    - Style 1
    - Style 2
    - …
  - layer 2
    - Style 1
    - Style 2
    - …
  - …

Which should be enough to cover the needs of most projects, all of this in a single file! Now it’s way easier to share, and there is no risk of breaking a link.

Conclusion

Easy right? And the best in all that, is that you can do exactly the same with a PostgreSQL database: storing your QGIS project, storing data (obviously) and storing your styles. It just take some “pre-procesing” by saving your data into your GPKG/PostgreSQL database, but you’ll want to do that anyway, and if you don’t, I assure you you want to.

Serving vector tiles with Martin

2023-12-20T00:00:00+00:00

There are some amazing tools out there, and they are getting build fast. I haven’t had the opportunity to work with vector tiles during my day-to-day jobs, but I wanted to explore what it is and how to set some things up, just for the fun of it. We’ll see here how to start using Martin, a vector tile server built in Rust serving PostGIS data.

Vector tiles

The standard has been set by Mapbox. In the same manner a WMTS serves raster tiles, vector tiles allow for the visualization of large volumes of data over the web, but as their name suggests, they are vectors.

It comes with some advantages compared to raster formats:

It’s significantly lighter, about 20 to 50% of its raster equivalent according to MapTiler. Vector data are series of points and each pixel of the map doesn’t have to have information declared; a rectangle polygon covering thousands of square kilometers can still be defined with 4 points.
It is stylized dynamically. While raster tiles need to be generated with their style embedded, vector tiles style can just be modified on the fly, without going through the laborious process of tiles generation that can take easily several hours/days.
It’s fast. Moving through a vector tile map feels smooth, there is no apparent tile loading.
What you request is what you get. You directly query the tiles, not some sort of other service along your tiles.

It’s an amazing standard to visualize spatial data in a web map. Obviously it’s not suitable for all kinds of data, you have to have discrete data that make sense to be visualized as vector; I doubt that raw satellites images as vector tiles are a good idea.

Installing PostgreSQL and PostGIS

First thing first, we need to store data somewhere. Martin is designed to be used with PostGIS, PMTiles and MBTiles, we’ll explore the PostGIS alternative.

The following commands were used on an Ubuntu distribution.

Let’s start by installing PostgreSQL:

apt install postgresql

Then the PostGIS extension:

apt install postgis

We’ll already set things up in the database by creating a dedicated user, creating a database and creating the PostGIS extension in there. From psql, create a user:

create user martin_usr with password 'my_password';

Create a database:

create database martin_db;

Go to your new database (\c martin_db) and create the extension:

create extension postgis;

We won’t cover here the more extensive tuning of the database. You should be set on the database side for this example.

Downloading some data

Obviously you’ll want some data to serve as vector tiles. In a production environment you’ll probably want to serve your own data, so this step would probably not be necessary. For the example we don’t have any data yet, so we’ll use some from osm through GeoFabrik. Since I’m in Switzerland, let’s download data over the country:

curl https://download.geofabrik.de/europe/switzerland-latest.osm.pbf -O

We are using osm2pgsql to import data into the database, so the .pbf format is ideal.

You can then load your data (apt install osm2pgsql if you don’t have the tool):

osm2pgsql -d martin_db -U martin_usr switzerland-latest.osm.pbf

We now have osm data over a whole country, easy right?

Now install Martin

Here comes the best part, the vector tile server itself. There are multiple ways to start working with Martin, we’ll use rust and its package manager cargo. If you don’t want to install those on your server you might prefer the Docker installation, or simply using binaries.

As said earlier, Martin is build in rust, and it’s accessible as a crate using cargo. You can first install rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then install Martin:

cargo install martin
martin --help

And that’s it! Martin’s here.

Can’t be that simple right?

Finally, you can run your new server:

martin "postgresql://martin_usr:my_password@localhost:5432/martin_db"

Martin can discover geometric tables in the database from the given url, so by default everything is served. So yes it can be that simple! Obviously environment variables need to be set, passwords hidden etc. But you have an idea of the process.

You can try by yourself reading the catalog:

curl http://0.0.0.0:3000/catalog | json_pp

{
   "fonts" : {},
   "sprites" : {},
   "tiles" : {
      "planet_osm_line" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_line.way"
      },
      "planet_osm_point" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_point.way"
      },
      "planet_osm_polygon" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_polygon.way"
      },
      "planet_osm_roads" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_roads.way"
      }
   }
}

Visualize data using QGIS

Once the server is running we can easily visualize our data in a client. Starting QGIS 3.14,

Layer > Add Layer > Add Vector Tile Layer

And enter the url then OK:

http://localhost:3000/planet_osm_line,planet_osm_point,planet_osm_polygon/{z}/{x}/{y}

You should see a rather absurd amount of data.

To conclude

Here is how to get started with Martin, a rust vector tile server. It’s just the start, we don’t cover here the creation of symbology and more in depth configurations and optimization, but it gives an idea of how easy and powerful such a tool can be.