geothings - python

FME is getting expensive? Stop using it

2024-08-16T00:00:00+00:00

I have trouble finding what happened at Safe Software at the beginning of 2024, but it seems that part of the company has been sold to an equity firm. No comment on that, but since then licenses prices seem to have taken a hit and customers are.. Well not very happy, especially regarding the communication around this increase:

Costs become unbearable for some companies and the look for alternatives is now a necessity. What can we do about it?

About FME and Safe

In itself I don’t have anything against FME, it does its job and if it allows more people to work with spatial data and create workflows then it’s cool. But there are things that I find, if not unacceptable, at least not very wise, and it’s not always Safe responsibility.

Safe as a company

I never had to negotiate any license or contract with Safe, but people around me always seemed happy about their relation with their resellers and the flexibility that was allowed with their licenses. There was no problem having a home/student license for free if you wanted to fiddle around with FME and prices were rather stable.

The rebranding and the sell at the beginning of 2024 has changed things, and some of the trust seems to be lost. The big problem is the way the company communicated the changes, which was apparently by giving no clear information. Increasing prices (reasonably) could hardly be considered a problem, that’s the world we’ve chosen, but at least having some transparency about it maintains the relation between seller and buyer. If you receive a x3 bill with a “go f“ck yourself if not happy”, this is at best disrespectful, at worst belittling, especially if you can’t act against this. And to me, the best answer to give is to gain independence and stop buying.

FME as a software

As I said, I don’t have anything against FME itself. It makes workflows easy to create and push to production, (spatial) data processing accessible to a broader audience and gives you time to focus on something else if you have a GIS role in your company (see On GIS and their integration for that part of the problem). But making yourself totally dependant on a proprietary solution, with a near monopoly on the field, IS a big problem, and honestly felt like a matter of time before it would happen. The situation is so problematic that some high level administrations (country level) are now negotiating terms with Safe because they are so dependant on FME that they can’t move easily from it, and will have no choice but to take the price hit if they don’t act. And maybe it’s common practice, I wouldn’t say I have enough experience to understand those kinds of negotiations, but it feels wrong.

That’s were digital sovereignty enter the game, regaining control over our processes, data, and costs. Some links :

If you work extensively with FME, I’d advise to take a step back and and have a look at what you need and how you could achieve this any other way, because most of the time, there are options, but we just don’t see them anymore. FME has become such an automatism that everything else is a blur, and worst, we lose skills and control.

What alternatives are there?

The first step is to have a look at what you need to achieve. Let’s see how FME defines itself from their website:

Expansive Data Solutions → well it doesn’t mean anything so next.
Data Transformation → Ok noted
Entreprise Automation → Ok noted

So we get data transformation and automation. Good.

Read, Write and Data transformation

Let’s start with the main reason for all of this. We read, transform and write data to solve problems and answer questions. Data need to be shaped into different models and linked with other data to bring additional knowledge. But how do we do that?

Transform with SQL

If you’re working with databases (PostgreSQL, MariaDB or even Oracle), you’re probably aware of their spatial capabilities. To me, SQL is a major skill, you can do a lot with a single SQL query. There is even a chance that your FME workspace can be entirely replaced by a single SQL query.

Now we want to Extract, Transform and Load data. Hum but wait, who said you can’t “transform” your data before even reading it? SQL allows you to do that; write your query, save it as a view if you wish, and you have your result ready to be read, in sync with your tables.

Let’s say you have regions (polygons), roads (lines) and bus stops (points), and you want:

road cut into portions by region
road portions length
to count bus stops on each road portion
- if a road has no bus stop you want to have “No bus stop” instead of 0
an attribute that says “Hopp Schwiiz !” if the road portion is located in a region where attribute name is equal to “Switzerland”, else “Oh, no Hopp Schwiitz…”.

You could have a FME workspace chaining writers, a clipper, a length calculator, aggregators, attributes managers etc. Or a single SQL query in the following manner (optimization not included):

SELECT 
    portion.name,
    CASE COUNT(b.id) 
        WHEN 0 THEN 'No bus stop'
        ELSE COUNT(b.id)::text
    END AS n_bus_stop,
    CASE portion.name
        WHEN 'Switzerland' THEN 'Hopp Schwiiz !'
        ELSE 'Oh, no Hopp Schwiitz...'
    END AS hop_suisse,
    St_Length(portion.geom) AS portion_length,
    portion.geom
FROM (SELECT r.name, St_Intersection(l.geom, r.geom) AS geom
        FROM road l
        JOIN region r ON St_Intersects(l.geom, r.geom)
        ) portion
LEFT JOIN bus_stop b ON St_Intersects(portion.geom, b.geom)
GROUP BY portion.name, portion.geom

And that’s it, you can read this data, export it, do whatever you want with it. Everything done within the query is visible at once, and if it takes some time to get used to the syntax and be comfortable with it, it becomes much easier to understand what happens in a well written SQL query than in the tidiest FME workspace (personal opinion included).

Another huge advantage of working with SQL is that you’re closer to a standard interoperable format (with a grain of salt since we use Postgis here, which is Postgres specific). If you ever need to change your system, the required effort to make the conversion will be minimal compared to translating a FME workspace.

You can import other data into your database to use it in your query, or use Foreign Data Wrappers (FDW) to read data from another database. You can combine your queries with tools such as dbt (Core) to have a more complete environment, or build your own processes, with GDAL or using the programming language of your choice.

Read and write with GDAL

Back at it with GDAL, but it’s so versatile that it has to be in your toolkit. Your can read and write basically any format, and apply transformations on the fly.

The series of command line tools that come with GDAL/OGR make it very complete and easy to use with raster or vector data.

You could reuse your previous SQL query with GDAL to extract your data and save it in GeoPackage for example. I could just save my query into a file named query_file.sql and run the following command:

ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Remember interoperability? Well that’s a nice example, have one standard way of solving a problem and you can connect it with other without changing anything. You have many possibilities.

Do everything with (Python) libraries

There are a bunch of libraries allowing you to read, transform and write data, using Python but not only. GDAL (omfg again) is also available in many languages through its bindings. If we focus on Python, a list (non-exhaustive of course) of libraries for working with spatial data:

Let’s try somethings with geopandas. If we consider that we are working with a single PostgreSQL database, we could reuse our earlier query to read the data and save it as a GeoPackage:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Open query file, apply and save
    with open("query.sql") as f:
        query = "".join(f.readlines())
        df = gpd.read_postgis(query, con)
        df.to_file("my_analysis.gpkg", layer="result", driver="GPKG")

        # Print the head for good measure
        print(df.head)


if __name__ == "__main__":
    run()

Will create a new file with your result saved in the result layer. It should also print something like:

0       France            2  Oh, no Hopp Schwiitz...      823.126128  LINESTRING (2537525.502 1152634.769, 2537254.4...
1       France            1  Oh, no Hopp Schwiitz...      118.753415  LINESTRING (2537290.714 1152264.433, 2537281.7...
2        Italy  No bus stop  Oh, no Hopp Schwiitz...      686.494161  LINESTRING (2538145.102 1151496.452, 2538283.1...
3  Switzerland  No bus stop           Hopp Schwiiz !      123.440260  LINESTRING (2536930.949 1152095.427, 2536884.0...
4  Switzerland            1           Hopp Schwiiz !     1102.779141  LINESTRING (2537281.727 1152146.02, 2537249.56...
5  Switzerland            1           Hopp Schwiiz !      206.632320  LINESTRING (2538019.283 1151332.542, 2538145.1...
6  Switzerland  No bus stop           Hopp Schwiiz !     1207.518986  LINESTRING (2537677.993 1151998.178, 2537874.0...

That was easy enough but we didn’t bring much since everything is done with SQL again. Let’s do the same thing but only using geopandas capabilities:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Read roads, regions and bus_stops
    roads = gpd.read_postgis("road", con)
    regions = gpd.read_postgis("region", con)
    bus_stops = gpd.read_postgis("bus_stop", con)

    # Intersection
    intersection = roads.overlay(regions, how="intersection")

    # Intersect bus stops
    intersect = intersection.sjoin(bus_stops, how="left", predicate="intersects")

    # Group by
    groupby = intersect.groupby(["name", "geometry"]).id.count()
    output = groupby.reset_index(name="n_bus_stop")

    # Hop Suisse
    output["hop_suisse"] = "Hopp Schwiiz !"
    output.loc[output["name"] != "Switzerland", "hop_suisse"] = (
        "Oh, no Hopp Schwiitz..."
    )

    # No bus stop
    output.loc[output["n_bus_stop"] == 0, "hop_suisse"] = "No bus stop"

    # Lost the Geo from GeoDataFrame at some point 
    output = gpd.GeoDataFrame(output, geometry="geometry")

    # Length
    output["portion_length"] = output["geometry"].length

    output.to_file("my_analysis.gpkg", layer="result_only_gpd", driver="GPKG")


if __name__ == "__main__":
    run()

Doesn’t take much more. Roads, regions and bus_stops could have been data from anywhere: another database, a csv file, a cloud hosted file, etc.

Working with a programming language allows you to easily read multiple data sources and combine them using your custom “transformers”, which will basically be functions that you would have written and tailored to your need.

But now let’s see what you can do to automate your process.

Automation

Not sure about what is so incredible about giving scheduling capabilities within FME when cron is available by default in many Linux distributions. If you’re working on Windows you can also schedule things with the Task Scheduler.

I feel like I’m oversimplifying things but I can’t see where. If you’re on Linux, you can schedule your data export task by adding such lines into your crontab (crontab -e to edit it):

# Everyday at 05:00
00 5 * * *  ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Run the task and send an email if it failed?

00 5 * * *  if ! ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"; then echo "your export failed" | mail -s "failing process" address@mail.me; fi

And that’s it. I doubt that FME brings so much on the table when it comes to scheduling tasks.

And that’s it?

I guess not, but I don’t think we will find many operations that are totally unfeasible without FME. If I’m missing something, please don’t hesitate to correct me.

Limitations

Unfortunately, there are some things that I don’t know how to do without FME, well at least one that I have no idea how to solve.

DWG and the closed formats problem

DWGs are ***** everywhere when your have people doing CAD around you. They could use DXF, which is an open source equivalent, but no, AutoCAD then DWG (which is no data btw, it’s just lines ffs). And I have no clue how to tackle that, except through a long process of mentality change around the tools used.

Earlier we saw that using standard formats can start a virtuous cycle, where you keep open the door for many solution to solve your problems. Well here we have the opposite, closed formats reduce the interoperability by making a handful of solutions your only options, and keep you from trying other things. If you need to have DWGs, then to my knowledge, FME becomes the only tool you can use to produce them programmatically from other data (even GDAL can’t all hope is lost). So from one license, you need “automatically” two.

And then one company decide to increase there prices, and there is nothing you can do about it.

In this case, the problem is to use a closed data format. To change that is to change deeply the habits of people using the tool creating this format, because, like with FME, there is a real addiction to the software. The skill becomes the software; not the technic, not the knowledge around the field, but just the software.

And that’s it for now

Fortunately, FME is not that essential. Maybe you won’t need to pay for the license for long… But I’m curious to read about what makes it a must-have for you. What can FME do that other tools cannot? Why is it necessary to you?

Conclusion

Feels good to rant a bit. To summarize:

Learn SQL
Don’t be afraid of command lines and programming languages
Have interoperability in mind
Get away from closed data formats

If you want to talk about all of this or if you need help converting some processes from FME to more open solutions, hit me up on Mastodon.

Opening and writing raster dataset with GDAL Rust

2024-03-08T00:00:00+00:00

I’ve been trying to learn Rust lately. For some reason there is something that I find very appealing in this language. From all I could read, it’s very efficient in resources and memory, it’s fast, and it doesn’t let you mess around with memory. Regarding performances, it places itself next to C and C++, and when looking into the world of language performances, I learned that Python was kind of slow, around 100x slower according to some benchmarks. Obviously this value in itself doesn’t mean much, and many more aspects of a language must be taken account when choosing one for the task, but this information kind of planted its seed. It might be because I have a problem with inefficient and overly complicated process, but I find the idea to work with a system language very elegant.

But we’re not here to talk about the Rust language in itself. We’ll see how to read and write raster data using the Rust binding for GDAL. Also, this is about how I tried to compare the Rust and Python bindings for GDAL and how I learned that it doesn’t change a thing in terms of performance.

Wait what?

Well I started by thinking “Oh I can’t wait to see how faster GDAL in Rust is compared to Python”. So I started by reading a raster file and print some basic information. I use a band from a Sentinel-2 image.

How to open a raster file?

Let’s do it. We’ll be working with the GDAL crate from crates.io (cargo add gdal) and the GDAL package prom PyPi (pip install GDAL).

In Rust:

use gdal::Dataset;

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Read band
let band = ds.rasterband(1).unwrap();

// Print statistics
if let Some(stats) = band.get_statistics(true, true).unwrap() {
    println!(
        "[ STATS ] =  Minimum={}, Maximum={}, Mean={}, StdDev={}",
        stats.min, stats.max, stats.mean, stats.std_dev
    );
};

WARNING

I’m using a lot of .unwrap(), which in very short will cause the code to panic if a problem occurs, this is for a quick and dirty test. Within a “production” environment you would handle errors and stuff.

And in Python:

from osgeo import gdal
gdal.UseExceptions()

# Open dataset
dataset = gdal.Open('IMG.jp2')
nband = dataset.RasterCount
print(f'This raster contains {nband} bands.') 

# Read band
srcband = dataset.GetRasterBand(1)

# Print statistics
stats = srcband.GetStatistics( True, True )
print(f'[ STATS ] =  Minimum=%.3f, Maximum=%.3f, Mean=%.3f, StdDev=%.3f' 
    % ( stats[0], stats[1], stats[2], stats[3] ))

File opened and information read. We can compare their runtime.

Comparing performances

Building the Rust program, then running and timing these two codes multiple times, we get results like follows:

Rust	Python
real 0m0.072s	real 0m0.156s
user 0m0.154s	user 0m0.752s
sys 0m0.275s	sys 0m1.689s

WARNING

This is a naive approach on testing and comparing performances, not a full-fledged benchmark.

Twice as fast without any processing or multi-threading, at that point I though “Oh it’s gonna be good”.

Let’s add some writing then

It’s only the start, we’ll now read the data in our raster dataset and, since my source is in .jp2, write it in a new raster dataset in the GeoTiff format.

How to write a raster dataset with GDAL bindings?

We will be using the Create() approach rather than the CreateCopy() (see the nuance in the doc). Geospatialized rasters come with series of metadata and overviews. The CreateCopy() automatically takes those from the source dataset and copy them into the destination. While this obviously simplifies the work if we don’t need to bring any change to these information, using Create() helps to better understand the structure of a raster. Also I have a quite important performance difference between the two and I’ve found Create() more efficient for my use case.

So the code now becomes as follow.

In Rust:

use gdal::{Dataset, DriverManager};

// Open dataset
let ds = Dataset::open("IMG.jp2").unwrap();
println!("This raster has {} bands.", ds.raster_count());

// Get source dataset metadata
let projection = ds.projection();
let spatial_ref = ds.spatial_ref().unwrap();
let geo_transform = ds.geo_transform().unwrap();
let (x, y) = ds.raster_size();

// Read band
let band = ds.rasterband(1).unwrap();

// Get GeoTiff driver  
let d = DriverManager::get_driver_by_name("GTiff").unwrap();

// Create the destination dataset as mutable with 1 band
let mut dst_ds = d.create_with_band_type::<u16, _>("IMG_as_tiff.gtiff", 
    x as isize, y as isize, 1).unwrap();

// Set metadata from source dataset
dst_ds.set_geo_transform(&geo_transform).unwrap();
dst_ds.set_projection(&projection).unwrap();
dst_ds.set_spatial_ref(&spatial_ref).unwrap();

// Get the destination band as mutable
let mut dst_band = dst_ds.rasterband(1).unwrap();

// Read the source data as Buffer<u16>
let buf = band.read_as::<u16>((0,0), (x, y), (x, y), None).unwrap();

// Write the data in the destination band
dst_band.write((0,0), (x, y), &buf).unwrap();

// Close dataset for good conscience
ds.close().unwrap();
dst_ds.close().unwrap();

And in Python:

# Open dataset
ds = gdal.Open('IMG.jp2')
nband = ds.RasterCount
print(f'This raster has {nband} bands.')

# Get dataset metadata
srcprojection = ds.GetProjection()
srcspatialref = ds.GetSpatialRef()
srcgeotransform = ds.GetGeoTransform()
(xsize, ysize) = (ds.RasterXSize, ds.RasterYSize)

# Get GeoTiff driver
fileformat = "GTiff"
driver = gdal.GetDriverByName(fileformat)

# Create the destination dataset with 1 band
dst_ds = driver.Create('IMG_as_tiff.gtiff', xsize=xsize, ysize=ysize,
                    bands=nband, eType=gdal.GDT_UInt16)

# Set metadata from source dataset
dst_ds.SetProjection(srcprojection)
dst_ds.SetGeoTransform(srcgeotransform)
dst_ds.SetSpatialRef(srcspatialref)

# Get the destination band
srcband = ds.GetRasterBand(1)

# Read the source data as Array
srcdata = srcband.ReadAsArray()

# Write the data in the destination band
dst_ds.GetRasterBand(1).WriteArray(srcdata)

# Close dataset for good conscience
ds = None
dst_ds = None

Both codes should create a new raster file that resembles our source, but in GeoTiff. And that’s it, you can convert a raster file into any format. We’re just scratching the surface, you can subset data, reproject, transpose it if you want etc.

But let’s compare performances again

Now as “la cerise sur le gâteau”, let’s see how powerful Rust can be.

Rust	Python
real 0m1.024s	real 0m1.133s
user 0m17.150s	user 0m17.335s
sys 0m1.629s	sys 0m2.089s

Wait let me rerun.

Rust	Python
real 0m1.053s	real 0m1.141s
user 0m17.076s	user 0m17.324s
sys 0m1.608s	sys 0m2.134s

Hmm.

Wait what? (bis)

Well that’s not what I expected.

In the process I forgot something important, we are working with bindings. Asking Wikipedia what a binding is:

In programming and software design, binding is an application programming interface (API) that provides glue code specifically made to allow a programming language to use a foreign library or operating system service (one that is not native to that language).

So from my understanding, working with Rust or Python or any other language bindings is just working with the API that let us call the underlying code, for GDAL written in C and C++. Thus the overhead that we generate is only the interaction with the library, not during the process itself, since it’s running C/C++ code. The difference that we observe is probably the same that we were observing when we just opened a file.

Conclusion

GDAL raster processing is very similar in Rust and Python. When working with its bindings, it doesn’t change much which language you use to perform the GDAL operations in themself, since in any case you’re running C/C++ code from your program. But everything that is around need to be considered when choosing the right tool. If your program needs multi-threading then you’l probably chose Rust, if you need quick development without much optimization then Python might be a best fit. But it’s comfortable to see that the GDAL processing time won’t change significantly.

I’m not at all trying to say that Rust is not worth it, I’m excited to go further into it and work on projects, but it remains a tool and there should be no injunction to use it if it doesn’t bring much to the table for your use case.

Also it’s totally possible that I totally misunderstood something and that those comparisons are meaningless, happy to learn more about it.