geothings - fme

FME is getting expensive? Stop using it

2024-08-16T00:00:00+00:00

I have trouble finding what happened at Safe Software at the beginning of 2024, but it seems that part of the company has been sold to an equity firm. No comment on that, but since then licenses prices seem to have taken a hit and customers are.. Well not very happy, especially regarding the communication around this increase:

Costs become unbearable for some companies and the look for alternatives is now a necessity. What can we do about it?

About FME and Safe

In itself I don’t have anything against FME, it does its job and if it allows more people to work with spatial data and create workflows then it’s cool. But there are things that I find, if not unacceptable, at least not very wise, and it’s not always Safe responsibility.

Safe as a company

I never had to negotiate any license or contract with Safe, but people around me always seemed happy about their relation with their resellers and the flexibility that was allowed with their licenses. There was no problem having a home/student license for free if you wanted to fiddle around with FME and prices were rather stable.

The rebranding and the sell at the beginning of 2024 has changed things, and some of the trust seems to be lost. The big problem is the way the company communicated the changes, which was apparently by giving no clear information. Increasing prices (reasonably) could hardly be considered a problem, that’s the world we’ve chosen, but at least having some transparency about it maintains the relation between seller and buyer. If you receive a x3 bill with a “go f“ck yourself if not happy”, this is at best disrespectful, at worst belittling, especially if you can’t act against this. And to me, the best answer to give is to gain independence and stop buying.

FME as a software

As I said, I don’t have anything against FME itself. It makes workflows easy to create and push to production, (spatial) data processing accessible to a broader audience and gives you time to focus on something else if you have a GIS role in your company (see On GIS and their integration for that part of the problem). But making yourself totally dependant on a proprietary solution, with a near monopoly on the field, IS a big problem, and honestly felt like a matter of time before it would happen. The situation is so problematic that some high level administrations (country level) are now negotiating terms with Safe because they are so dependant on FME that they can’t move easily from it, and will have no choice but to take the price hit if they don’t act. And maybe it’s common practice, I wouldn’t say I have enough experience to understand those kinds of negotiations, but it feels wrong.

That’s were digital sovereignty enter the game, regaining control over our processes, data, and costs. Some links :

If you work extensively with FME, I’d advise to take a step back and and have a look at what you need and how you could achieve this any other way, because most of the time, there are options, but we just don’t see them anymore. FME has become such an automatism that everything else is a blur, and worst, we lose skills and control.

What alternatives are there?

The first step is to have a look at what you need to achieve. Let’s see how FME defines itself from their website:

Expansive Data Solutions → well it doesn’t mean anything so next.
Data Transformation → Ok noted
Entreprise Automation → Ok noted

So we get data transformation and automation. Good.

Read, Write and Data transformation

Let’s start with the main reason for all of this. We read, transform and write data to solve problems and answer questions. Data need to be shaped into different models and linked with other data to bring additional knowledge. But how do we do that?

Transform with SQL

If you’re working with databases (PostgreSQL, MariaDB or even Oracle), you’re probably aware of their spatial capabilities. To me, SQL is a major skill, you can do a lot with a single SQL query. There is even a chance that your FME workspace can be entirely replaced by a single SQL query.

Now we want to Extract, Transform and Load data. Hum but wait, who said you can’t “transform” your data before even reading it? SQL allows you to do that; write your query, save it as a view if you wish, and you have your result ready to be read, in sync with your tables.

Let’s say you have regions (polygons), roads (lines) and bus stops (points), and you want:

road cut into portions by region
road portions length
to count bus stops on each road portion
- if a road has no bus stop you want to have “No bus stop” instead of 0
an attribute that says “Hopp Schwiiz !” if the road portion is located in a region where attribute name is equal to “Switzerland”, else “Oh, no Hopp Schwiitz…”.

You could have a FME workspace chaining writers, a clipper, a length calculator, aggregators, attributes managers etc. Or a single SQL query in the following manner (optimization not included):

SELECT 
    portion.name,
    CASE COUNT(b.id) 
        WHEN 0 THEN 'No bus stop'
        ELSE COUNT(b.id)::text
    END AS n_bus_stop,
    CASE portion.name
        WHEN 'Switzerland' THEN 'Hopp Schwiiz !'
        ELSE 'Oh, no Hopp Schwiitz...'
    END AS hop_suisse,
    St_Length(portion.geom) AS portion_length,
    portion.geom
FROM (SELECT r.name, St_Intersection(l.geom, r.geom) AS geom
        FROM road l
        JOIN region r ON St_Intersects(l.geom, r.geom)
        ) portion
LEFT JOIN bus_stop b ON St_Intersects(portion.geom, b.geom)
GROUP BY portion.name, portion.geom

And that’s it, you can read this data, export it, do whatever you want with it. Everything done within the query is visible at once, and if it takes some time to get used to the syntax and be comfortable with it, it becomes much easier to understand what happens in a well written SQL query than in the tidiest FME workspace (personal opinion included).

Another huge advantage of working with SQL is that you’re closer to a standard interoperable format (with a grain of salt since we use Postgis here, which is Postgres specific). If you ever need to change your system, the required effort to make the conversion will be minimal compared to translating a FME workspace.

You can import other data into your database to use it in your query, or use Foreign Data Wrappers (FDW) to read data from another database. You can combine your queries with tools such as dbt (Core) to have a more complete environment, or build your own processes, with GDAL or using the programming language of your choice.

Read and write with GDAL

Back at it with GDAL, but it’s so versatile that it has to be in your toolkit. Your can read and write basically any format, and apply transformations on the fly.

The series of command line tools that come with GDAL/OGR make it very complete and easy to use with raster or vector data.

You could reuse your previous SQL query with GDAL to extract your data and save it in GeoPackage for example. I could just save my query into a file named query_file.sql and run the following command:

ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Remember interoperability? Well that’s a nice example, have one standard way of solving a problem and you can connect it with other without changing anything. You have many possibilities.

Do everything with (Python) libraries

There are a bunch of libraries allowing you to read, transform and write data, using Python but not only. GDAL (omfg again) is also available in many languages through its bindings. If we focus on Python, a list (non-exhaustive of course) of libraries for working with spatial data:

Let’s try somethings with geopandas. If we consider that we are working with a single PostgreSQL database, we could reuse our earlier query to read the data and save it as a GeoPackage:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Open query file, apply and save
    with open("query.sql") as f:
        query = "".join(f.readlines())
        df = gpd.read_postgis(query, con)
        df.to_file("my_analysis.gpkg", layer="result", driver="GPKG")

        # Print the head for good measure
        print(df.head)


if __name__ == "__main__":
    run()

Will create a new file with your result saved in the result layer. It should also print something like:

0       France            2  Oh, no Hopp Schwiitz...      823.126128  LINESTRING (2537525.502 1152634.769, 2537254.4...
1       France            1  Oh, no Hopp Schwiitz...      118.753415  LINESTRING (2537290.714 1152264.433, 2537281.7...
2        Italy  No bus stop  Oh, no Hopp Schwiitz...      686.494161  LINESTRING (2538145.102 1151496.452, 2538283.1...
3  Switzerland  No bus stop           Hopp Schwiiz !      123.440260  LINESTRING (2536930.949 1152095.427, 2536884.0...
4  Switzerland            1           Hopp Schwiiz !     1102.779141  LINESTRING (2537281.727 1152146.02, 2537249.56...
5  Switzerland            1           Hopp Schwiiz !      206.632320  LINESTRING (2538019.283 1151332.542, 2538145.1...
6  Switzerland  No bus stop           Hopp Schwiiz !     1207.518986  LINESTRING (2537677.993 1151998.178, 2537874.0...

That was easy enough but we didn’t bring much since everything is done with SQL again. Let’s do the same thing but only using geopandas capabilities:

import geopandas as gpd
from sqlalchemy import create_engine


def run():
    # Connection
    db_connection_url = "postgresql://username:password@localhost/dbname"
    con = create_engine(db_connection_url)

    # Read roads, regions and bus_stops
    roads = gpd.read_postgis("road", con)
    regions = gpd.read_postgis("region", con)
    bus_stops = gpd.read_postgis("bus_stop", con)

    # Intersection
    intersection = roads.overlay(regions, how="intersection")

    # Intersect bus stops
    intersect = intersection.sjoin(bus_stops, how="left", predicate="intersects")

    # Group by
    groupby = intersect.groupby(["name", "geometry"]).id.count()
    output = groupby.reset_index(name="n_bus_stop")

    # Hop Suisse
    output["hop_suisse"] = "Hopp Schwiiz !"
    output.loc[output["name"] != "Switzerland", "hop_suisse"] = (
        "Oh, no Hopp Schwiitz..."
    )

    # No bus stop
    output.loc[output["n_bus_stop"] == 0, "hop_suisse"] = "No bus stop"

    # Lost the Geo from GeoDataFrame at some point 
    output = gpd.GeoDataFrame(output, geometry="geometry")

    # Length
    output["portion_length"] = output["geometry"].length

    output.to_file("my_analysis.gpkg", layer="result_only_gpd", driver="GPKG")


if __name__ == "__main__":
    run()

Doesn’t take much more. Roads, regions and bus_stops could have been data from anywhere: another database, a csv file, a cloud hosted file, etc.

Working with a programming language allows you to easily read multiple data sources and combine them using your custom “transformers”, which will basically be functions that you would have written and tailored to your need.

But now let’s see what you can do to automate your process.

Automation

Not sure about what is so incredible about giving scheduling capabilities within FME when cron is available by default in many Linux distributions. If you’re working on Windows you can also schedule things with the Task Scheduler.

I feel like I’m oversimplifying things but I can’t see where. If you’re on Linux, you can schedule your data export task by adding such lines into your crontab (crontab -e to edit it):

# Everyday at 05:00
00 5 * * *  ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"

Run the task and send an email if it failed?

00 5 * * *  if ! ogr2ogr -f GPKG -sql @query_file.sql output.gpkg "PG:service=input_db"; then echo "your export failed" | mail -s "failing process" address@mail.me; fi

And that’s it. I doubt that FME brings so much on the table when it comes to scheduling tasks.

And that’s it?

I guess not, but I don’t think we will find many operations that are totally unfeasible without FME. If I’m missing something, please don’t hesitate to correct me.

Limitations

Unfortunately, there are some things that I don’t know how to do without FME, well at least one that I have no idea how to solve.

DWG and the closed formats problem

DWGs are ***** everywhere when your have people doing CAD around you. They could use DXF, which is an open source equivalent, but no, AutoCAD then DWG (which is no data btw, it’s just lines ffs). And I have no clue how to tackle that, except through a long process of mentality change around the tools used.

Earlier we saw that using standard formats can start a virtuous cycle, where you keep open the door for many solution to solve your problems. Well here we have the opposite, closed formats reduce the interoperability by making a handful of solutions your only options, and keep you from trying other things. If you need to have DWGs, then to my knowledge, FME becomes the only tool you can use to produce them programmatically from other data (even GDAL can’t all hope is lost). So from one license, you need “automatically” two.

And then one company decide to increase there prices, and there is nothing you can do about it.

In this case, the problem is to use a closed data format. To change that is to change deeply the habits of people using the tool creating this format, because, like with FME, there is a real addiction to the software. The skill becomes the software; not the technic, not the knowledge around the field, but just the software.

And that’s it for now

Fortunately, FME is not that essential. Maybe you won’t need to pay for the license for long… But I’m curious to read about what makes it a must-have for you. What can FME do that other tools cannot? Why is it necessary to you?

Conclusion

Feels good to rant a bit. To summarize:

Learn SQL
Don’t be afraid of command lines and programming languages
Have interoperability in mind
Get away from closed data formats

If you want to talk about all of this or if you need help converting some processes from FME to more open solutions, hit me up on Mastodon.

A little bit of GDAL/OGR appreciation

2024-01-19T00:00:00+00:00

If there had to be one single tool to work with spatial data, I think GDAL/OGR would be the first name to come to mind. Its usages range from raster to vector, it can be used as command line, imported as a library in many languages through bindings developed by the community, many applications provide UI to parameterize and run GDAL etc. Its versatility makes it an amazing tool to work with spatial data in a lot of different contexts, but it’s sometimes underestimated or even misunderstood.

We will quickly introduce GDAL/OGR, then give some examples of usage to help for some quick starts.

What is GDAL/OGR?

GDAL/OGR (Geospatial Data Abstraction Library/OpenGIS Simple Features Reference Implementation) is a library for working with raster and vector geospatial data formats, coded mainly in C++, C and Python. The first release dates from 2000, and the project has now more than 450 contributors at the time of writing. This is a very active project with regular releases.

The project is composed of two parts: the raster processing library GDAL and the feature processing library OGR, both merged during the release of GDAL 2.0. When working with raster data you will use programs prefixed with gdal (gdal_translate, gdalwarp etc.) while vector data will use ogr prefixed scripts (ogrinfo, ogr2ogr etc.). When talking aboutt GDAL, both libraries are included, and we’ll do the same from here.

As its name suggests, GDAL abstracts data in order to standardize its way of interacting with it. The driver creates the interface between the data source and the tools, allowing to focus on the logic of the process rather than how to read or write the data. This makes it a very extensible project; including a new format of data source requires to write a new driver, nothing more. You could see that the list of supported formats is quite something, either raster or vector.

GDAL describes itself as a translator, meaning that data come from a source and goes to its destination. Translations can occur at different moment during the process, during reading or by the script called. These translations can occur on multiple axis as well, we could for example:

Extract a subset of source data depending on a region of interest.
Extract a subset of source data based on an attribute.
Reproject data into a different coordinate system.
Merge a series of data files into a single one.
etc.

Source unknown, if it’s your image and you want to be cited please leave a comment

As an open source project with a permissive MIT license, GDAL is used by many projects, open source or proprietary, to process geospatial data. If you’ve used QGIS, ArcGIS, FME or PostGIS, you’ve used GDAL under the hood.

Why would I use it?

Weirdly, even though GDAL is basically everywhere there is geospatial data to be moved, it’s not so much acknowledged by GIS people, or at least it’s not really used on its own. If you read this, you definitely use GDAL, but maybe without noticing.

Ok but why would I complicate my life with some command lines when the same tool is being used by a software that provides me a user interface easier to use?

While it’s much intuitive to explore geospatial data through a user interface such as QGIS, where you can interactively load your data, chose the process you want to apply, its parameters, and visualize the results, things get a bit more complicated once you’ve done the exploratory part and want to reapply your newly defined process to new data. Repeating operations becomes quickly tedious and error-prone. You could use a tool such as the Model Designer from QGIS (amazing little tool, check it if you repeat a lot of operations in QGIS), but how do you schedule your process to be started when you’re not able to launch it manually?

If you use a heavy client ETL such as FME, try once to compare the difference in speed between a simple table transfer done by launching a FME workspace (using a command line) and a simple ogr2ogr command, there is at least an order of magnitude of difference.

It’s about using a tool designed for a purpose as efficiently as possible.

How to install?

We’ll assume you have no installation of GDAL yet, but just to make sure you can try:

gdalinfo --version

which gives:

GDAL 3.4.1, released 2021/12/27

If it’s not installed you can refer to the installation instructions from GDAL website.

For example, I’m working on Ubuntu 22.04, for which the GDAL package is the 3.4.1. The latest version can be installed by downloading the latest release, but I usually stick with the provided version for my platform, unless I need a specific update.

apt update
apt install gdal-bin

And you should be good!

WARNING

You might already have GDAL installed through another software like QGIS, be careful during the installation, multiple installations might interfere and create warnings or errors. On Windows you can use the OSGeo Shell to access any tool from the OSGeo ecosystem.

And how do we use it?

It’s easy, you just need to be comfortable with command lines. If you’ve mostly worked with user interfaces it might look scary, but it really is not complicated.

Access metadata

The first thing we usually do with data is to have some information about it. Let’s try to see the metadata of a random .jpeg:

gdalinfo gdal.jpeg

Driver: JPEG/JPEG JFIF
Files: gdal.jpeg
Size is 711, 882
Metadata:
  EXIF_Orientation=1
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  882.0)
Upper Right (  711.0,    0.0)
Lower Right (  711.0,  882.0)
Center      (  355.5,  441.0)
Band 1 Block=711x1 Type=Byte, ColorInterp=Red
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=711x1 Type=Byte, ColorInterp=Green
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=711x1 Type=Byte, ColorInterp=Blue
  Overviews: 356x441, 178x221
  Image Structure Metadata:
    COMPRESSION=JPEG

Easy right? And it’s not even georeferenced, it’s a .jpeg, a driver exists, it opens.

We can try the same with some vector data. We’ll have a look at what is going on in one of the PostgreSQL database hosted on my computer.

ogrinfo -ro PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.
1: planet_osm_line (Line String)
2: planet_osm_polygon
3: planet_osm_roads (Line String)
4: planet_osm_point (Point)
5: natural_polygons

And as we can see I have 5 layers in my database. Notice how we used the ogrinfo command for questioning vector data. Here ogrinfo (like gdalinfo) requires a dataset source as input, the syntax will depend on each driver, make sure to have a look at the documentation. For a PostgreSQL connection, the line could have looked like:

ogrinfo -ro PG:"dbname='osm_suisse' host='localhost' port='5432' user='x' password='y'"

We can extend the information that we want to see for each layer by adding the -al flag to the line. The option -so will summarize the output.

ogrinfo -ro -so -al PG:dbname=osm_suisse

INFO: Open of `PG:dbname=osm_suisse'
      using driver `PostgreSQL' successful.

Layer name: planet_osm_line
Geometry: Line String
Feature Count: 2137192
Extent: (626211.331305, 5706647.213604) - (1253686.695428, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            MEMBER["World Geodetic System 1984 (G2139)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["Popular Visualisation Pseudo-Mercator",
        METHOD["Popular Visualisation Pseudo Mercator",
            ID["EPSG",1024]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Web mapping and visualisation."],
        AREA["World between 85.06°S and 85.06°N."],
        BBOX[-85.06,-180,85.06,180]],
    ID["EPSG",3857]]
Data axis to CRS axis mapping: 1,2
Geometry Column = way
osm_id: Integer64 (0.0)
access: String (0.0)
addr:housename: String (0.0)
addr:housenumber: String (0.0)
[...]

And now we have the full detail of our layers; name, type, feature count, etc.

Basic translations

Now that we know what were working, with we can try to move some things around. Let’s start by exporting our vector data into a geopackage.

ogr2ogr -f GPKG exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

Here we’re exporting the table planet_osm_point into the exported_data.gpkg geopackage. The format can be recognized by the extension of the output file, but it’s safer to ensure the format by passing the argument -f. Again, each driver documentation will have its driver name to be used.

Add a filter

We can export a subset of our data, based on an attribute for example. Let’s say we want to export the points where the attribute addr:housenumber is not null, and we want to save it in another table in our newly created geopackage. We’d do something like:

ogr2ogr -update -f GPKG -where '"addr:housenumber" is not null' -nln only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

We want to update our dataset (our geopackage) and not create a new one, so we use the -update option. The syntax of the -where parameter is a bit convoluted here since there is a special character in the column name, but with a simple name you’d just need a pair of quotes -where "housenumber" is not null".

Let’s have a look at our geopackage:

ogrinfo -ro -al -so exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: planet_osm_point
Geometry: Point
Feature Count: 2146246
Extent: (626415.435591, 5736097.599850) - (1189428.532375, 6117386.570796)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
[...]

Layer name: only_housenumbers
Geometry: Point
Feature Count: 366561
Extent: (663719.309401, 5751784.044673) - (1167785.106794, 6072459.331387)
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
    BASEGEOGCRS["WGS 84",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
[...]

The new layer has been added to the geopackage. Has the query been respected?

ogrinfo -ro -al -so -sql 'select * from only_housenumbers where "addr:housenumber" is null' exported_data.gpkg

INFO: Open of `exported_data.gpkg'
      using driver `GPKG' successful.

Layer name: SELECT
Geometry: Point
Feature Count: 0
Layer SRS WKT:
PROJCRS["WGS 84 / Pseudo-Mercator",
[...]

Yes! No data without addr:housenumber.

TIP

The -sql option is another way of filtering data, it allows you to pass a sql query in any of the sql flavor supported by the driver, you can even save your queries in a text file and pass the file to the -sql parameter as -sql @path/to/my_file.

Geometric operations?

The -sql parameter can also be used to apply transformations to the data. We’ll request the same points, but this time we’ll create a buffer around them.

ogr2ogr -update -f GPKG -sql 'select ST_Buffer(way, 50) from planet_osm_point where "addr:housenumber" is not null' -nln buffered_only_housenumbers exported_data.gpkg "PG:dbname=osm_suisse tables=planet_osm_point"

And that’s it, we have our buffer around the points of interest. Here GDAL makes use of the internal capabilities of PostgreSQL and PostGIS by passing directly the request to the database, but the same command using the only_housenumbers layer from our geopackage would also be valid.

ogr2ogr -update -f GPKG -sql "select ST_Buffer(way, 50) from only_housenumbers" -nln buffered_only_housenumbers exported_data.gpkg exported_data.gpkg

TIP

In the case of PostgreSQL/PostGIS you can write the -sql parameter like you would write your queries directly into the database. They are passed directly to the database by GDAL.

This should give you a small idea of the things you can do with GDAL and how to start experimenting.

Final words

Well that was a lot more than expected. But GDAL really is a very strong and important tool. Diving into the details of all the available options, both for the command AND the drivers, is not a small task, but understanding how to navigate around and find the information that you need to process your data as you wish can go a long way. Manual and repeated tasks can be replaced to help to focus on more pertinent work, process time can be (by experience) reduced drastically depending on the tools you’re initially using. And if you’re working on data hosted on a server without graphic interface, well you have a tool to get the info you need!

Obviously GDAL alone can’t do everything and has its imperfections; I’ve really struggled with installations in the past (Python binding might have bested me at some point), and the number of parameters is a bit overwhelming. But once you have it working it does the job, and really well.

If you can, use GDAL.