geothings - organization

Managing shared QGIS projects

2024-04-19T00:00:00+00:00

QGIS is an amazing tool. Being spatial analysis or digitizing data, it’s efficient, highly customable and extensible. There is so much you can do with it.

So it’s only natural that we want to share a project that we are working on and propose to other people to join our effort on our amazing data digitizing/spatial statistics calculating masterpiece. Well yes of course, things are better when they are shared (most of them at least), and we go further together than alone (wow very profound), but this is something that has to be given some thoughts beforehand to ensure that everything goes smoothly within the collaboration.

We’ll briefly have a look at what can help make collaborating with QGIS successful. Also, we’ll be talking a lot about projects, we are referring here to the literal QGIS file used to store the work environment (data, visualizations, etc.). It’s not project in the sense of a project management, but some things might be useful to apply from the project management world. And in any case, working together requires tools outside of the technical aspects.

When does this apply?

Obviously if you’re alone on your project or working with a handful of people, some points might feel over-the-top or even counter productive, but some questions are always useful to be asked and it can be helpful to use some “good” practice, even when working for ourselves.

For the last years I had to pick up some projects that were shared by tens of people, create new ones, and I can safely say that there need to be rules when more people are getting on board.

So if you work with spatial data and you need to have people collaborating using QGIS you might be interested

Ok let’s go, how to collaborate with QGIS?

This is obviously from my personal experience. It’s not exhaustive, or maybe not accurate, but if it can help..

tl;dr

Scope your project: have objectives for the project and keep it within controlled boundaries.
Have a data structure: define which data will be included and model them.
Give roles and responsibilities: have a project management in place.
Version control: track changes and decompose your project if it becomes too big.
Share it efficiently: have everything in a single place, keep it as simple as possible, save it in database.

Scope your project

It sometimes feels like it’s a bad thing to say in an Agile world, but giving some boundaries to the project helps staying on track. Maybe this is something that I feel due to unstructured work environments, but when working on a (QGIS) project, if there is no beginning and no end there is just no way to succeed.

Back to QGIS. When starting, think about what the project will be about: will it be used to digitize data? Should data processing and spatial statistics be in the project? Should this data X be included? QGIS is a powerful tool, and maps are powerful visualizations, when people unfamiliar with these notions start to see what can be done they tend to want to add a lot of information in a project. Hell you can do your taxes in QGIS if you want, should you? Probably not.

Let’s say you design your project for a certain team, to help them digitize certain data. Obviously, since they are the target of the project, their demands should be considered, analyzed and implemented. But let’s also imagine that the project has been shared with another team interested in visualizing the data. Should you take their demands into account as well? Well, has the work induced by these demands been considered when designing the project? If not then these demands should be redirected somewhere else.

Maybe try to have some ground rules: one project per data structure (see below), keep separate the digitization, processing and statistics, one project per team, etc.

Of course if your time is bound to be spent on a couple of project you can fit a lot of things and optimize them, having everything working together. But don’t forget that the more people start to use these projects, the more impactful any change becomes, and separation of concerns can help a lot to prevent that.

Structure your data before thinking about QGIS

Have a coherent data model to work with. If you have people around specialized in modeling data and structuring a database, listen to them. If they tell you that you should normalize your data because you’ll otherwise have problems when scaling up, please do it. I think that’s the most important advice, and it doesn’t even concern QGIS.

If you’re wondering what this is about, data modeling is, very simply put, how to structure/divide your data into blocks that make sense, allowing you to add, remove, update data in the most efficient way. For example, let’s imagine that I want to have information about trees in my city and the population of birds they are housing. I could have everything in a single table like below:

erDiagram
    tree {
        serial id_tree
        int age
        geometry geom
        string bird_species
        bool is_bird_cute
    }

Ok cool I can now create one tree that has a family of pigeons:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true

But wait, there is also a family of crow next branch:

id_tree	age	geom	bird_species	is_bird_cute
1	125	POINT(x,y)	pigeon	true
2	125	POINT(x,y)	crow	true

You quickly see the problem appearing, we duplicate as much data as we create new one. That’s where a little bit of modeling helps. A potential alternative could look like this:

erDiagram
    tree ||--|| location : is_located
    tree {
        serial id PK
        int age
    }
    location {
        serial id_tree PK,FK
        geometry geom
    }
    tree ||--o{ bird : host
    bird {
        serial id_bird PK
        string species
        bool is_cute
        int id_tree FK
    }

Where trees have their own table, birds as well and they are connected using a foreign key (FK). We create a table for trees location as well; it’s not really necessary in that case since it is a 1:1 relationship, but it often appears that separating the information about an object and its geometries simplify many operations. It also shows that you can treat geometries like any other column in a data structure.

The above tables should now look something like:

tree

id	age
1	125

location

id_tree	geom
1	POINT(x,y)

bird

id_bird	bird_species	is_cute	id_tree
1	pigeon	true	1
2	crow	true	1

If migration season comes and 200 bird families choose this tree as their home, we just need to add elements in the bird table without duplicating any data regarding trees.

Your data model should be independent from the tools you’ll be using to manipulate it. A client limitation shouldn’t question a well structured data; if the tool is limited, update it, or maybe it’s not the one you need for the job. How many time have I seen weird database designs (Postgres triggers I see you) because it was not possible to work with certain situations in QGIS or because of performance issues with certain designs.

It’s a step for which you’ll be grateful when people ask to add some values, some attributes etc. Because if you’re data is well structured, you need to bring the change to a single place. But if the data structure didn’t follow some rules, you’ll probably have to change things in multiple locations, introducing much more space for error.

Give roles and responsibilities

Who is going to modify the project? Who is going to modify the data? Should you give read-only access? Who is going to do the long term maintenance? Who should be able to propose changes? Some questions should be answered before starting to build the project.

If we know who is responsible for what, decisions can be more efficient and implementation becomes most of the time a matter of minutes.

You should know who is allowed to ask for modifications in a certain project, who should implement them and who is allowed to reject the modifications.

But honestly, in short, have a project management.

Decompose, centralize and version control you project

Hun? Decompose and centralize? Sounds like there is a useless step here, but let me explain.

A QGIS project is basically a big xml file bringing many elements together. It’s nice because everything (almost) is in the same place, but it also means that any little change influences everything, which could provoke side effects. And when it comes to modifications brought to layers it can become cumbersome to track.

To mitigate that, QGIS allows to export many of its layer elements into separate files: symbols, forms, rendering, etc. Think about what will be modified regularly. You can export these elements as separate files to make sure that you focus on their changes and keep a specific history for them. Imagine you often have to modify the colors of certain layers; export the symbologies in .sld or .qml. Like this you will have the history of these files outside of the project and will be able to access older symbologies without changing the whole project.

Regarding version control, it seems obvious now but it’s not that easy when working with QGIS projects since the file is completely rewritten if you change anything and then save. But as said, a project is a combination of many things; there can be plugins, symbologies, profiles, etc. Making the effort of having everything in a single repository (even the result of your decomposition process) helps to understand what is needed and where, and by using a version control tool with it you can always come back to an earlier version in case of mistakes or breaking changes.

When working with a version control platform (such as GitLab or GitHub), you can also bundle your versions into releases, allowing to have everything needed in the project into a single archive, and accessible directly from an URL, which can help with the sharing process.

I find it bizarrely difficult to share a project with people. As said right above, a project is composed of multiple elements that work together: data, symbols, plugins, etc.

There are some ways of avoiding loosing information when sharing a project:

Keep all data in a single place: store your data in a database or a database file (e.g. Geopackage), it helps having everything in the same place to avoid Non valid layers or losing information.
Keep the number of plugins low: plugins are nice and they can help a lot, but any plugin added introduce uncertainty. They might not be updated, be usable with certain QGIS versions etc. In a more general sense, keep your project as simple and as clean as possible.
Save the project file in database or database file. QGIS allows you to store the project in database, which makes it so much easier to share. You theoretically can have a single Geopackage holding all the data AND the project. You then just need to share the file to others and they have basically everything they need to work with the project.
Maybe don’t share the project? Think outside of QGIS. If you start to do something in QGIS, you don’t have to stay in it. Do you really have to share the project or do you want to share the data inside it? Should the people you share with have a write access or not? There are so many tools available, maybe the task you’re trying to accomplish should not be done by sharing a QGIS project. Look into web-clients, file formats or any type of software that could reach the objective.

Conclusion

Now after rereading, I realize that I wrote a weird mix of technical aspects and project management. There might be something that I need to learn from that. But the main message remains that the tool shouldn’t guide your objectives and the underlying structure. Find a way to articulate your work that makes sense to you and your team, and that allows you to be efficient in order to answer request as well as possible.

Anyhow, QGIS is a very powerful tool: it’s easy to use, allows to accomplish a lot of things with spatial data and makes you feel like a genius. But the tool in itself shouldn’t replace a proper project management. Think about the objectives, the stakeholders, the responsibilities and make sure that everybody is on-board with the rules. It’s not because you can do anything with it that a project should host everything. Keep it clean and simple, one objective at a time, and find the method for sharing information that suits your need. With QGIS, a monolith project with a lot of people asking for modifications is a bad idea, at least from my experience.

But in any case, it’s a amazing tool, support it.

Cheers.

On GIS and their integration

2023-12-15T00:00:00+00:00

Since I started working with spatial information, I’ve seen the term GIS used to define quite different things; ranging from mastering data input on QGIS to weird command line geeking. This variation of definition is very interesting, even though it results in misunderstandings and complicated situations at an organizational level. We’ll try to see where this can come from and maybe how to tackle it. This comes from my own personal experiences and the different working environments that I’ve observed.

Where does GIS come from?

We won’t re-write a post about GIS history here, other resources do this very well. You can have a look at ESRI’s website, this short introduction from the University of South Carolina, or GISGeography (first 3 search engine results huhu).

It’s interesting to note that GIS originate from environmental questions:

How can I link my natural resource inventory data to spatial information?
How can I visualize my resources?
How do I plan my exploitation?
etc.

Which makes sense, spatial is mostly about territory, and locating things is a very practical need, it’s also a very intuitive information. Even when we look at the origins of QGIS, which is the exact definition of a software, we see that it’s been initiated by Gary Sherman (interview), who was at the time a geologist in need of visualizing PostGIS data in a Linux environment.

At this point, GIS defines a tool, a software that helps you visualize, analyze, maybe process, spatial data. QGIS, ArcInfo/Map/GIS, MapInfo are examples of GIS in this context. You have some tables stored in files or a database, you connect your GIS to those data, and you have an information system that could manage the entirety of your company spatial information. It feels like this is still the most widespread definition of what GIS are. Even if this definition used to be accurate, the context we are in today is different, and I find it limiting. The amount of data we’re working with now is not the same as 20 years ago, and spatial data can’t be kept out of broader information systems.

The spatial exception

In spatial data there is spatial, and depending on the perspective, this can be a big deal. Coming from the applied side of data analysis, I used to think that spatial data had to be given a special treatment, as they are spatial. Obviously you can’t work with these data in the same way as you’d do with other types of normal data, right? Well they are spatial, but they are primarily data, and now that my head is most of the time in databases I’m focusing much more on that part of the expression. It feels weird to discover one day that the thing you thought special about what you are doing is basically a column in a table, and that it can be as simple as a text.

Obviously, geometries come with their very special and necessary set of tools and skills: input of spatial data, spatial indexing, spatial analysis and statistics, map creation and visualization etc. It does have its own set of challenges, and the developments of geospatial standards were necessary to respond to them. But now, the fact that it has its own protocols, catalogs, specialized management, all parallel to the world of data, makes it a type of information difficult to merge with the rest. It can even be very concrete, I’ve observed that in some companies, geospatial data units will exist out of the more global ‘data team’, using their own tools, standards and strategies, resulting in two worlds doing essentially the same job, but somehow uncoordinated. What is the point of using two different tools for transferring data between two tables?

If we considered spatial data separately, GIS could be an entire field, with its own unique and distinct rules. This definition, unlike the previous one, feels to broad and almost pretentious. Spatial data is not that special. It had to have its own standards to solve problems in its time, but with the general improvement of tools and computational capacities, the spatial field will probably gain from getting closer to the broader data community. The growth of the open source community and the speed and robustness of developed tools could benefit the spatial community.

A shift seems to be occurring though, the spatial world is finally colliding with what the rest of the domain has been doing (e.g. WFS3 following REST principles). Interoperability is becoming more and more present, and spatial standards joining broader standards will facilitate that.

OK so what?

Someone asked me recently, what is the role of a GIS specialist, and this stirred up some thoughts. Purely by experience, I can say that during recruitment processes and in day-to-day work, being specialized in GIS means a lot of things. I happened to set up Linux servers, install services, develop and schedule data pipelines, change map symbology in QGIS projects, all within the same job, with the same title: GIS specialist. To me, it doesn’t make much sense. GIS can’t be reduced to a single software like QGIS (amazing tool though). Neither can it realistically be its own independent network with specific rules.

Maybe we are forgetting a simple part, the information system part. If we take the Wikipedia definition we get:

An information system (IS) is a formal, sociotechnical, organizational system designed to collect, process, store, and distribute information.

Prefixing it with a G doesn’t change the essence of an information system. Many roles and skills are necessary to make an information system work, being geographic or not. From the definition it should make possible to collect, process, store and distribute data, geographic in our case. There need to be user interfaces developed for entering data, servers to be administered, databases to be architected, data to be analyzed, visualized, distributed etc.

GIS are information systems and should be treated as such. There is no reason to keep them out of more general policies and rules when it comes to organizational or technical decisions within a company.

What to do

I happened to be in the situation where, as a GIS specialist, I was completely out of the IT department, and I mean organizationally, physically and numerically, not even on the same network within the company. Trying to get closer to the IT seemed to be necessary in that case. Making sure to be aware of data policies, standards in data storage and processes, knowing about conventions, rules, tools etc. I tried to reach out for services responsible for data management, and it’s been very well received, and even encouraged. By doing so I could align on guidelines and finally start to take advantage from larger means.

It’s a simple question of using resources efficiently. GIS is now a completely digital field, while companies are more and more sensible to the importance of IT infrastructure, data management. It makes sense to bring spatial data into the equation. Resources will be given for the IT architecture, and GIS should be integrated.

Spatial data add something essential that other type of data don’t, but forgetting that they are firstly data is a mistake. I think a good example/metaphor of spatial integration would be how PostGIS works within PostgreSQL, it’s just an extension. Spatial data can make use of all the broader features of Postgres, and additionally they have their own set of tools for their special aspects, without being completely isolated.