geothings - tiles

PMTiles exploration

2024-02-02T00:00:00+00:00

It’s a technology that I came across rather recently. Tile generation when maintaining a geographic oriented website can be a hassle: it can take time, generates a lot of files, and can be a critical process. PMTiles might have the potential to facilitate some of this work and we’ll try to see how.

What are PMTiles?

In itself, PMTiles is a specification for building single-file tile pyramids. It’s open source and part of a larger project called Protomaps, and it aims at mapping the World, just that. The major difference with the kind of tiles that I know is that they are embedded as a single file, like the MBTile format, which make the interaction very intuitive, I mean we all get the idea of what a file is. But what does it bring compared to MBTiles? Well from the Protomaps website:

A MBTiles file needs to be accessed as a SQLite database on disk, so requires a running server and disk storage for cloud hosting. PMTiles is designed to be cloud-native and readable in fine-grained parts from remote storage like Amazon S3.

It aims at simplifying the manipulation of tiles in a cloud native approach, for worldwide coverage.

How to create PMTiles?

The Protomaps website gives us some tools to create these tiles. Let’s try some possibilities.

Generate PMTiles using GDAL

Our Swiss Army Knife is back, starting version 3.8.0 GDAL has native support for PMTiles, so let’s try that.

First check the GDAL version you’re using, you’ll want 3.8.0 or higher:

gdalinfo --version
--> GDAL 3.8.3, released 2024/01/04 (debug build)

Let’s start from an OSM .pbf file (see Serving vector tiles with Martin). To convert this file into a PMTiles file you can enter:

ogr2ogr -f PMTiles out.pmtiles switzerland-latest.osm.pbf

The process might take some time if you’re working on a large dataset. By default the MAXZOOM is 5, which is rather low, so the run shouldn’t take to long, but if you start to increase the parameter to around 15, which allows to represent data from much closer, processing time increases.

But once the command is done, well that’s it!

Alternatively we can use tippecanoe to achieve the same result.

Generate PMTiles using tippecanoe

WARNING

Under construction

Visualize PMTiles

The easiest way is to go to the PMTiles Viewer Page and upload your newly created PMTiles file.

Obviously you can add it to your custom project. The easiest way of doing is by using the Leaflet plugin protomaps-leaflet.

Starting from complete scratch you can create an .html file with the following content:

<!DOCTYPE html>
<html>

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" />
    <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js"></script>
    <script src="https://unpkg.com/protomaps-leaflet@latest/dist/protomaps-leaflet.min.js"></script>
    <style>
        body,
        #map {
            height: 100vh;
            margin: 0px;
        }
    </style>
</head>

<body>

    <h2>Hey I'm a custom page</h2>
    <p>Let's add some Leaflet and PMTiles.</p>
    <div id="map"></div>

    <script>
        let PAINT_RULES = [
            {
                dataLayer: "multipolygons", //here we render only one layer, check your layers names 
                symbolizer: new protomapsL.PolygonSymbolizer({ fill: "steelblue" })
            }
        ];

        const map = L.map('map').setView([46.5, 6.6], 13); //chose a view that suits you

        var osm = L.tileLayer('https://tile.openstreetmap.org/{z}/{x}/{y}.png', {
            maxZoom: 19,
            attribution: '&copy; <a href="http://www.openstreetmap.org/copyright">OpenStreetMap</a>'
        }).addTo(map);

        var layer = protomapsL.leafletLayer(
            {
                url: 'http://127.0.0.1:8080/your_pmtile_file.pmtiles',
                paint_rules: PAINT_RULES
            }).addTo(map);

    </script>

</body>
</html>

For testing purposes you might have to serve you PMTiles through http. To do so you can do as the Protomaps website suggests:

npm install -g http-server
# Then from the folder containing your PMTiles.
http-server . --cors

Open your .html file on the browser of your choice and you should see the map with your PMTiles displayed.

When should PMTiles be used?

Again, the documentation on Protomaps website is quite complete and gives us some insights.

Cloud hosting

PMTiles are stored as a single file, and compression is so efficient that the size is ridiculous compared to classic png tiles. Number of requests necessary is drastically reduced and thus the cost of operation as well.

Static data

PMTiles don’t support localized updates so each time you’ll need to apply an update, the whole file needs to be rewritten. If your data don’t change that much PMTiles are a wise choice, but If you need to have a more dynamic rendering the documentation recommends PostGIS with the ST_As_MVT() function.

Simplicity

Here it’s more like a feeling about the philosophy of the specification. You have a single file, accessible without introduced complexity, directly from a library like Leaflet, which itself has simplicity as one of its core objectives. It looks like one of those project that allows you to make complicated-looking things so easily. I guess it’s a sign of a good project.

Final words

Well it was rather easy to discover. I mean we create a file, we basically load it within a project and we’re good to go. Obviously there are some customizations to do with styling and labels etc. But once you have the starting point and your tools, it takes minutes to create what you desire.

Here we’re just scratching the surface, remains to explore filtering, styling etc. But the concept in itself and the unreasonable performance of the compression make it very interesting if you’re in the situation where PMTiles could give you an advantage; the file created for OSM data Switzerland, with zooms from 0 to 14, is around 1GB, it costs pennies to upload and host on cloud services.

It’s a project to support and look for, I guess?

Serving vector tiles with Martin

2023-12-20T00:00:00+00:00

There are some amazing tools out there, and they are getting build fast. I haven’t had the opportunity to work with vector tiles during my day-to-day jobs, but I wanted to explore what it is and how to set some things up, just for the fun of it. We’ll see here how to start using Martin, a vector tile server built in Rust serving PostGIS data.

Vector tiles

The standard has been set by Mapbox. In the same manner a WMTS serves raster tiles, vector tiles allow for the visualization of large volumes of data over the web, but as their name suggests, they are vectors.

It comes with some advantages compared to raster formats:

It’s significantly lighter, about 20 to 50% of its raster equivalent according to MapTiler. Vector data are series of points and each pixel of the map doesn’t have to have information declared; a rectangle polygon covering thousands of square kilometers can still be defined with 4 points.
It is stylized dynamically. While raster tiles need to be generated with their style embedded, vector tiles style can just be modified on the fly, without going through the laborious process of tiles generation that can take easily several hours/days.
It’s fast. Moving through a vector tile map feels smooth, there is no apparent tile loading.
What you request is what you get. You directly query the tiles, not some sort of other service along your tiles.

It’s an amazing standard to visualize spatial data in a web map. Obviously it’s not suitable for all kinds of data, you have to have discrete data that make sense to be visualized as vector; I doubt that raw satellites images as vector tiles are a good idea.

Installing PostgreSQL and PostGIS

First thing first, we need to store data somewhere. Martin is designed to be used with PostGIS, PMTiles and MBTiles, we’ll explore the PostGIS alternative.

The following commands were used on an Ubuntu distribution.

Let’s start by installing PostgreSQL:

apt install postgresql

Then the PostGIS extension:

apt install postgis

We’ll already set things up in the database by creating a dedicated user, creating a database and creating the PostGIS extension in there. From psql, create a user:

create user martin_usr with password 'my_password';

Create a database:

create database martin_db;

Go to your new database (\c martin_db) and create the extension:

create extension postgis;

We won’t cover here the more extensive tuning of the database. You should be set on the database side for this example.

Downloading some data

Obviously you’ll want some data to serve as vector tiles. In a production environment you’ll probably want to serve your own data, so this step would probably not be necessary. For the example we don’t have any data yet, so we’ll use some from osm through GeoFabrik. Since I’m in Switzerland, let’s download data over the country:

curl https://download.geofabrik.de/europe/switzerland-latest.osm.pbf -O

We are using osm2pgsql to import data into the database, so the .pbf format is ideal.

You can then load your data (apt install osm2pgsql if you don’t have the tool):

osm2pgsql -d martin_db -U martin_usr switzerland-latest.osm.pbf

We now have osm data over a whole country, easy right?

Now install Martin

Here comes the best part, the vector tile server itself. There are multiple ways to start working with Martin, we’ll use rust and its package manager cargo. If you don’t want to install those on your server you might prefer the Docker installation, or simply using binaries.

As said earlier, Martin is build in rust, and it’s accessible as a crate using cargo. You can first install rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then install Martin:

cargo install martin
martin --help

And that’s it! Martin’s here.

Can’t be that simple right?

Finally, you can run your new server:

martin "postgresql://martin_usr:my_password@localhost:5432/martin_db"

Martin can discover geometric tables in the database from the given url, so by default everything is served. So yes it can be that simple! Obviously environment variables need to be set, passwords hidden etc. But you have an idea of the process.

You can try by yourself reading the catalog:

curl http://0.0.0.0:3000/catalog | json_pp

{
   "fonts" : {},
   "sprites" : {},
   "tiles" : {
      "planet_osm_line" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_line.way"
      },
      "planet_osm_point" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_point.way"
      },
      "planet_osm_polygon" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_polygon.way"
      },
      "planet_osm_roads" : {
         "content_type" : "application/x-protobuf",
         "description" : "public.planet_osm_roads.way"
      }
   }
}

Visualize data using QGIS

Once the server is running we can easily visualize our data in a client. Starting QGIS 3.14,

Layer > Add Layer > Add Vector Tile Layer

And enter the url then OK:

http://localhost:3000/planet_osm_line,planet_osm_point,planet_osm_polygon/{z}/{x}/{y}

You should see a rather absurd amount of data.

To conclude

Here is how to get started with Martin, a rust vector tile server. It’s just the start, we don’t cover here the creation of symbology and more in depth configurations and optimization, but it gives an idea of how easy and powerful such a tool can be.