Aquileo | Jochen Topf

Aquileo | Spyglass Improvements

Wed, 03 Jun 2026 17:14:10 +0200

Whenever I have some time, I keep improving OSM Spyglass. A very useful addition is that Spyglass now shows suggestions for the key and value filter fields. Spyglass gets possible keys/values from the data currently loaded and will show all keys/values matching what you have typed so far.

There are many other small improvements in the UI, one is a new menu option in the location control: Switch lat/lon. Very useful when you have a system that shows the coordinates the other way around from what OSM Spyglass expects (for instance QGIS). Just copy and paste the coordinates, Spyglass will jump to the wrong location, but one click later you are at the right spot.

There are many internal changes also. There is support now for storing OSM object attributes (version, changeset, timestamp) with the data. (No support for user ids or user names so far.) Attributes are currently not enabled on the version running on spyglass.jochentopf.com to save some disk space and import time, and they are not shown in the UI anyways. More work for the future…

There are also plenty of changes with the database tables, how I am organizing the data internally. I am trying out various formats to minimize storage and make queries more efficient and/or easy to use. The biggest change is that I switched from using JSONB for the tags to that old classic, the hstore format. JSONB and hstore are quite similar for what we are doing here, but there is one big difference: It is possible to create a GIST index that combines geometry and tags when using hstore making queries with key/value filter somewhat faster.

Another area I have been experimenting with is how to detect and display “unusual data”. Unusual data in this case are things like ways that contain only a single node, or ways having duplicate nodes in them, relations without type tag or without members and so on. So we are talking about some low-level errors in the data, that I want to detect and show to the user in a useful way, so that they can figure out why some object is not showing up properly in the map. Single-node ways, for instance, would usually not be visible, because you can’t have a line with only one point in it. So this case is detected and the way shown as black point with an orange center. When selected, the way is shown with a message “Way contains only a single point!”.

Some other “unusual” cases are currently detected and the information is in the database but now shown yet.

Aquileo | OSM Spyglass

Thu, 19 Feb 2026 21:32:50 +0100

Two years ago or so I started the OSM XRAY project, later I wrote about it in this blog post. Since then I have renamed this project to “OSM Spyglass” and I have kept working on it on and off.

At the State of the Map Europe 2025 in Dundee I gave a talk with the title “Everything Everywhere All At Once” about this project. You can see the video on Youtube. This got some people excited about the project, there is even some talk about putting the tool on OSMF infrastructure. Until this comes about the tool is now hosted at spyglass.jochentopf.com.

I am finally getting around to writing some more about what’s been happening since my first announcement and since the talk.

User Interface

I keep fiddling with the user interface. Optional globe view (not much to do for me now that Maplibre supports that out of the box), map is now resizable (horizontally), display of city names in some zoom levels, improved pop-up menus for keys and tags, and much more. Generally the UI has been getting faster and more reliable.

There are still some bugs to fix and plenty of possible improvements. And I’d be happy about feedback and ideas. Its quite a lot of information we are trying to show here in limited space, so good ideas on how to do that are needed.

Caching

In the first blog post I wrote about some caching that I implemented in the database. That did work but it turns out it is pretty useless. The user wants to access the newest data anway and we can keep up with minutely updates (at least in larger zoom levels), so I removed the caching completely for vector tiles and for high zoom rasters. Only raster images at zoom levels up to 10 are cached. Currently we can not deliver them fast enough otherwise.

Map updates

The database is updated from OSM using minutely diffs. We are usually about 3 to 5 minutes behind the OSM data, that’s just how long it takes the OSM servers to create the minutely diffs, push them out to their server and for our update job to download the data and to apply it to the database. It is unlikely we can improve on that much further. Spyglass shows the timestamp of the latest data it has in the bottom right corner. This timestamp is updated whenever new data is loaded, i.e. when you move the map or so.

Vector tiles are always generated on the fly from the current database, for higher zoom levels they contain all data, for medium zoom levels only “larger” objects are shown, i.e. long ways and larger areas. In small and medium zoom levels raster tiles are shown. They always contain all data. So for the medium zoom levels raster data in gray is overlayed with vector data in black (nodes and ways) or blue (relations). So you can see everything, but only click on the larger items.

Raster tiles in small zoom levels are only updated once per day, for zoom 0 to 7 this happens by taking the zoom level 8 tiles, and merging and rescaling them. I have spent quite some time on optimizing this. The first version happened in the database but only generated black-and-white tiles, the current version uses code written in Go which creates grayscale images which are much better than the black-and-white images. And it is much faster than the gdal tools I tried for this task. Gdal is a great tool, but, as an “all purpose tool”, it has to cope with all sorts of different data sources, projections etc. which makes it much slower than a specialized tool for a specific use case. It only takes a few minutes now to create the low zoom tiles from the zoom level 8 tiles. And they are not stored in the database any more but on disk which is easier and they are faster to use that way, too.

Rasters are still generated in the database from the data. That is, unfortunately, not as efficient as one might think. We don’t need to copy the data from the database into another process, and the cost of actually getting the data seems to be not that huge, but the rasterizing costs time. This is probably something that could be improved inside PostGIS, or maybe we have to get rid of this idea alltogether and move rendering outside the database. There is plenty of space to experiment and improve performance here.

Server

Originally I used pg_tileserv as server to create the vector tiles from the database on the fly. It could also be tricked into creating the raster tiles. But I also needed GeoJSON output and some other API endpoints. I experimented with pg_featureserv which did work, but having two servers with lots of specialized PL/pgSQL functions in the database plus an ever growing configuration for nginx (used as reverse proxy) became too complicated and error prone. So I decided to rewrite the server from scratch in Go. Turns out it is really easy to write robust and featureful HTTP servers in Go, it comes with everything you need; the only external library I am using is for accessing the database. And deployment is really easy: Just copy over one Go binary and restart the server, no extra configuration files or functions to update in the database etc.

Filters

Everything is done three times for nodes, ways, and relations. There are three sets of raster tiles, 3 sets of vector tiles. It is easy to switch those layers on and off in the UI. And then there is the key or tag filter. The vector tiles in higher zoom levels contain all the data, the filter is applied on the client, which is very fast. For raster tiles the filtering has to be done on the server which takes somewhat more time. Filtering is (silently) disabled on the small zoom levels, so you always see all data there. This isn’t great as a user experience, I’ll still have to figure out a way to make this transition more user friendly. Or, ideally, allow filtering on all zoom levels.

It is a lot of fun to zip around the map and look at far away places and how they are mapped. Try it out!. And if you have any problems or ideas, open an issue on Codeberg.

Aquileo | More taginfo news

Wed, 08 Jan 2025 12:55:21 +0100

Once a year or so I set aside some time to work on taginfo. Then I start working on it and find more and more things that I want to work on and invariably it takes longer than planned. And there is always much more to do that I don’t get to. But I did get some interesting things done this time around…

Tag status

OSM wiki pages about keys and tags show a Tag status for many tags. The status is somewhat fuzzy, which one of the about 10 common values a tag should have is not always so clear, and sometimes wiki pages for different languages show different values. But thanks to a lot of work from a lot of people the state of the wiki documentation has improved a lot over the years and this information is now better maintained and quite useful.

Taginfo has always parsed out this status information from the wiki pages but not done anything with it. But now it does. On key pages the Overview tab now shows a “Status” box which shows the status from the wiki, or it might tell you that the status is unknown or ambiguous (if there is a disagreement between different language wiki pages). I created a new “look” for the status badge which uses the same colors as the wiki for some of the status values.

There are probably more places where we want to show that status, not only on the “key” pages, but also the “tag” pages and some others. But I’ll leave that for some later time.

You can also get an overview about all the keys and tags and their status on the Tag status page on taginfo. This shows statistics about all the status values, lists of keys and tags, and also special tables showing which keys and tags have inconsistent status values in different languages.

Data from wiki

Which brings me to the next topic: There is a lot of information in the wiki. Taginfo can’t known what information is correct or not, but it can flag some problems with the data, inconsistencies it finds, or, for instance, all the images used to document keys and tags that are quite large and so need a long time to load.

The Wiki source section on taginfo now collects a lot of such information to help wiki maintainers with their job. Some of what’s shown there is new, some has been available in reports or taginfo-internal pages before. I hope this will be useful for somebody.

Discardable tags

There is one somewhat special tag status called “discardable”. Tags with this status have been found not to be useful and some editors remove these tags automatically and silently when OSM objects are edited. That way the data will sooner or later be removed from OSM without any huge automated edit.

Taginfo now reads the configuration from the iD and JOSM editors to find out which tags they think of as discardable as well as the tag status information from the wiki. A new report shows this information which could be especially useful for editor developers and wiki maintainers.

And in the mentioned “Status” box on the key pages, this information will also be highlighted.

CSV export

Most tables shown on taginfo have a “JSON” button in the header which links to a URL for downloading the data shown in the table. You don’t have to read the API docs, just use that link, maybe change the parameters a bit and get to the data that’s in that table.

Using JSON is great for many use cases, but in some cases it is still somewhat cumbersome. A user had asked for CSV export, so I implemented that also. Most tables now also have a “CSV” button. So it is really easy to, say export some data to CSV, load it into LibreOffice and create some chart.

Keyboard shortcuts

There probably aren’t many people using the keyboard shortcuts that taginfo provides. But they can be quite useful for experienced users. But who can remember them all?

There is now a new feature: You can press and hold the question mark (?) key on any page and taginfo will show you which shortcuts there are and what links on that page they are a shortcut for. Not the most important thing, but it was fun to work on that, fiddle around and see how to best do that.

Aquileo | Taginfo gets a new layout

Mon, 09 Dec 2024 17:58:02 +0100

Some of the most often seen pages on taginfo are the pages describing a key. To be more specific, the “Overview” tab on that page. And it has never really looked that great. The layout is strange and a lot of prime screen real estate is wasted.

And to make matters worse, the layout totally breaks on small screens:

So it was high time we did something about it. The new design is built around fixed size boxes containing different information about that key, as you can see here:

The boxes will wrap as needed, so the whole thing is still reasonably pretty on small screens:

To fit the values distribution chart into a box, it had to change. It uses a 3D layout now which shows a lot more information. I threw in a small version of the chronology chart and instead of just showing the image from the wiki, a list of wiki pages is shown that link directly to those pages. So there is a lot more information right there on the overview page.

Oh, and all of this goes for the tag pages and the relation type pages, too. They have also been updated with the new layout.

An added benefit of those boxes is that we can have more or less of them as needed. For instance when there is no wiki page about a key, the wiki box will not show up. For some tags (like website or wikidata) an extra box will be shown with links. I am already thinking about more boxes to add, but this will come at another time.

The new layout is now online, try it out at taginfo.openstreetmap.org.

Aquileo | Introducing OSM XRAY

Mon, 30 Sep 2024 23:17:11 +0200

This blog post is about the new OSM XRAY service. [Update April 2026: This service has since been renamed to Spyglass]. You can find a demo here. It uses vector tiles with all OSM data to really show all of OSM, mostly for debugging. But it is also interesting to just explore what’s there.

Vector Tiles with ALL Data

Every time there is talk about vector tiles of OpenStreetMap data the question arises whether we can have vector tiles with “all of OSM”. Not just data with this or that tag suitable for rendering this or that type of map but everything. All of it. All objects from the OSM database.

It is possible to do that. Nine years ago Mapbox/HOT already did that with something called QA tiles. The project is long defunct though. And we have a lot more data by now. More recently Wat Mildon published his WAMap (See also this thread on community.openstreetmap.org) which also shows that this is totally possible.

So it is possible. And it turns out it is actually not that complicated. But is it useful?

There are two problems with vector tiles containing “everything”:

“Everything from OSM” is quite a lot of data. Vector tiles work best if they are at most a few kilobytes each. If you include everything in OSM, vector tiles will often need a few megabytes. This data takes longer to transmit and the browser will struggle to render it. Everything will become sluggish.
Rendering OSM data into a usable map is difficult. You need to take many tags and tag combinations into account, handle bad data and handle OSM specialties like ways that can be lines or polygons or complex relations. And you need to do that with a rendering configuration that is rather limited and hard to use, like the styling language of Maplibre.

These problems become much more pronounced for smaller zoom levels. It might be slow, but on zoom level 14 you can download and show everything. But on zoom level 0 you would have to download the whole planet file with more than 70GB and process that in the browser just to show the outlines of the continents. Not really feasible.

The solution for all these problems is usually to preprocess the data on the server, decide which parts you need on which zoom level and convert the data into a format that the renderer can work with more easily.

Use Cases

Okay, so most use cases don’t need the “full OSM data” in vector tiles and they are better of doing things differently. But maybe there are use case which do need it. I can think of two: One is for experimentation. Sometimes you want to create a map but don’t know yet what OSM data there is and how best to transform it into something more useful and faster to render. Having all data available might make the map slow to be created, but it is still possible. So you can show the data, tweak the display and when you are done and know what you want, go back to the server, convert the data there and then have a fast map. This is a definite use case and I have used this approach many times, but usually using QGIS and not vector tiles. It is much easier to play around with a GUI tool.

The other use case I can think of is as a debug view. Similiar to what the “Map Data” button on openstreetmap.org does. Or similar to what you see in an OSM editor where you can click on everything and see the tags etc. In fact the existence of OSM editors in the browser shows that this is totally feasible and does make sense.

Now having said all of this, the question of what would be possible is still open. How far can we go?

How to do it?

Just creating vector tiles with everything in them is pretty easy. Use the flex output of osm2pgsql to create a database with all OSM objects having a “tags” JSONB column that contains all the tags and we are already half there. Using osm2pgsql gives me something extra that nobody has done yet with “full” vector tiles: The database can be updated continuously from minutely diffs, so we can show a map that always shows all current OSM data.

From the data in the database we need to create the vector tiles and we need to do it fast. There are several vector tile servers that can do that. For this project I am using pg_tileserv, because it will automatically convert all the data from the JSONB tags column into vector tile attributes and is pretty easy and flexibly configured using PostgreSQL functions. Creating vector tiles on something like zoom level 12 or 14 is very straightforward that way. In a densly mapped urban area the browser might take a while to download and render that tile, but it does work.

Of course if you want to do complex rendering this will become more and more expensive with every rendering rule you have. But for a debug site a wire-frame view is enough. So we only need a few rules for rendering black dots and lines.

For smaller zoom levels this isn’t a solution though. We need to somehow reduce the amount of data. I played around with several approaches for a while, simplifying the geometries. But sooner or later this approach breaks down. So I reverted back to “ye olde raster tile” approach for smaller zoom levels. I could have gone with a renderer like Mapnik, but just for the fun of it I decided to try the raster capabilities of the PostGIS extension. Turns out you can render vectors into raster tiles completely inside the database and after some experimentation I got it to be reasonably performant, too. There aren’t many styling options, though, so this approach will not work for most use cases. But in my case I am interested only in, as I said, drawing some black dots and lines, and PostGIS can do that.

Okay, I have the outline of a solution here. Vector tiles for zoom level 14 and above, raster tiles for zoom levels 0 to 13.

Caching and Prerendering

For the higher zoom levels the database query is actually quite performant. And because this service is not intended for massive use, just as a debugging tool for a few mappers, we could probably get by without any kind of caching. But just for the fun of it I added some caching inside the database. For a high traffic site I would not consider this, the overhead for caching inside PostgreSQL compared to about any other solution must be quite large. And the database is the bottleneck for the whole operation, so we should not add any load to it. But I realized I could add caching inside the database without any additional software component, just using the abilities of pg_tileserv and osm2pgsql for caching and expiry, so I wanted to try it. And at least I created the tile_cache table as UNLOGGED to reduce the overhead somewhat. We’ll see how that goes.

The smaller the zoom levels gets, the more data needs to be processed for each tile. And that takes time. So from zoom level 10 down I pre-render all tiles. Some tiles are rendered from the data and for the smallest zoom levels they are just resized bitmaps of the middle zoom levels. Currently these are rendered once on import and never rerendered, but it should be possible to do that once a week or so. It takes a few hours to do this.

Creating all of this took a few hours here and there on a few weekends. But I needed more, I needed some way to show all of this in a browser.

The Application

Actually creating the tiles was easy compared to figuring out how to show the data and how to interact with it. I wanted a fast and easy to use tool that allows exploring the OSM data. It took me quit a while and many experiments to get the interface to where it is now. How do you show tags? How do you handle selecting objects? What if there are multiple objects under the mouse cursor? How do you show relations? All at once or one member at a time?

The solution I came up with is certainly not perfect, I am sure we’ll improve it over time. There are some clear limitations, for instance selections only work in zoom level 14 and above. But it is already fun to explore OSM that way. So I am happy to release it in the state it is in. I am excited to see what improvements we’ll come up with over time.

Also it turns out that there is so much OSM data that it is sometimes difficult to navigate this map full of black dots and lines that cover the background map shown for reference. So I added an overlay of OSM boundaries. This was straightforward to do because XRAY uses the osm2pgsql Themepark framework which makes it easy to mix in layers from different configurations, in this case I am using the boundaries layer from the Shortbread configuration.

What’s next?

Currently this is only running as a test/demo on the osm2pgsql test machine sponsored by FOSSGIS. It can’t stay there forever because we need that maschine for testing new osm2pgsql versions etc. We’ll see where we can find a permanent host for this.

There are plenty of things that need improving. For some of them I have already opened issues in the OSM XRAY repository and I am sure you’ll come up with more ideas and find some bugs. The two most important issues in my mind are the question of how to handle selected objects in low zoom levels and how to handle filtering in low zoom levels. I am open to ideas.

Aquileo | Out of memory

Thu, 15 Aug 2024 13:19:57 +0200

I have to import a lot of OSM planet files to test things due to my work on osm2pgsql. I have a test machine for that graciously provided by FOSSGIS e.V., the OSM Foundation local chapter for Germany. A few days ago I imported a planet file and osm2pgsql ran out of memory.

I was doing a non-slim import, which means that all node locations and all way nodes are stored in memory. Until recently this worked okay, but it seems the planet has grown beyond what fits into the 128 GB RAM that this machine has. This does not come as a big surprise, it was going to happen sooner or later — I wrote about this issue in my 2022 paper on the “Evolution of the OSM Data Model” (PDF).

I don’t have a bigger machine and I am not going to get one anytime soon. And other people are probably in the same position, so I added support for the -F/--flat-nodes option in non-slim mode to osm2pgsql. This means we can move nearly 100 GB worth of data into a file on disk instead of having it in memory. The PR was just merged. (You still need lots of memory, though, because without enough memory for caching, processing will become painfully slow anway.)

Aquileo | Minutely updated map

Mon, 25 Sep 2023 12:39:04 +0200

Most OpenStreetMap-based maps you see these days on the Internet are only updated every so often. Some may be updated daily, most are probably only updated every few weeks or months. And that is totally okay for most use cases. But there are some maps that are updated continuously so that changes in the OSM data appear as fast as possible, which is important to motivate OSM contributors and helps with quickly detecting and fixing problems. The most important map with continuous updates is the main map you see on www.openstreetmap.org.

Minutely Diffs

To enable everybody to keep up with changes in OSM data, the OSMF publishes minutely diffs, i.e. files containing the OSM updates for a single minute. There are several challenges when keeping a map up to date from those diffs. The first is that the diffs don’t contain enough information. Only the immediate object that changed is in the diff, but not other objects that also need data from the changed object to draw them correctly. This means that you need a database with all the OSM data for the whole planet locally available to correctly interpret the diffs and update the map.

I am working with osm2pgsql which has solved this problem a long time ago and recently got some improvements on that front.

Data Conversion and Cleanup

The second problem is that even if you have an up-to-date OSM database available, you still need to draw the map from that data. And the OSM data format isn’t a good fit for that job. You can directly draw a map from OSM data (the JOSM editor does this for instance), but usually you want other data formats in between (such as PostgreSQL/PostGIS tables or vector tiles).

Osm2pgsql was invented years ago to solve this problem by doing conversions from OSM data into database tables with geometries which can be rendered into maps directly by many tools out there. With the flex output I have been working on for several years now, this conversion has become much more powerful, so that we are getting better and better at converting the data into formats that are easier to work with on the rendering side. In this step we can also clean up the data which makes it considerably easier to work with when rendering.

Generalization

The third problem is the problem of generalization. The conversions described above usually only work on one OSM object at a time (or a few like a way and its node members). That works well enough for large zoom levels where you want to render features on the same level of detail as they are modelled in the OSM database. But for smaller zoom levels you need to accumulate hundreds, sometimes tens of thousands of OSM objects into something larger to show on the map. You can, sort of, do without generalization, but that means rendering can become very slow. Which makes the continuously updated map not updating so quickly any more.

I have been working on that problem in the generalization project, adding code to osm2pgsql to quickly calculate generalized data from the OSM data already in the database when something changes and write the data back into the database.

Technology Demo

To tie all of this together and to show what we can do today (and what we can’t do yet) I have created [this technology demo] (Update: not available any more). The map on that page is updated continuously from OSM data using the minutely diffs. For most data this includes updates of all the lower zoom levels. The only exception are the roads on lower zoom levels which are only updated every once in a while, because I haven’t yet figured out a way to do this really quickly. And the ocean data is also only updated once daily from osmdata.openstreetmap.de.

Tiles are rendered on the fly from the PostgreSQL/PostGIS database. There is no caching involved, each tile is created for each request. Of course this is not something fit for a production map where some caching will be needed, but I didn’t do it for this map on purpose to demonstrate how fast this setup still is at producing those tiles.

The tiles use the Shortbread tiles format. It is a somewhat simple format, especially compared to the wealth of detail you see on the OSM Carto map on openstreetmap.org, but it shows the most important stuff and we have to start somewhere.

Want to try this yourself?

My setup runs on a 8 core (16 with HT) machine with 128 GB RAM and the database needs about 800 GB plus about 80 GB for the flat node file. Setup takes about 20 hours including the initial import, initial calculation of generalized data and then catchup with the changes that happened in OSM in the mean time.

You’ll need osm2pgsql and osm2pgsql-themepark, use the current master. I am using the T-Rex tileserver in my current setup and the osm2pgsql-themepark framework will generate a configuration for that, but you can also use a different tileserver. (In fact development on T-Rex has stopped, so you might want to look into alternatives.)

What’s next?

This map is only a prototype. There is more work to be done to turn this into something that could be used for real. To make this operational the whole stack including web server, tile server, and caching has to be set up properly. Because I am not caching tiles, I didn’t have to set up cache expiry either.

The generalization code we have now is already working quite well for lots of layers, but it is kind of simplistic. And it has to become faster and more robust and needs to be extended to other feature types. It currently takes about a minute to re-generate all the data for the smaller zoom levels, that’s kind of tight for something that should update every minute. And, as mentioned, the road layers can only be updated less often. (They currently need about 15 or 20 minutes to update for the whole planet.)

Over a decade ago OSM was the first map out there that not only everybody could change, but that also would show those changes for everybody to see in minutes. But we have become stuck at that point. Changes in higher zoom levels are shown quickly, but in lower zoom levels it takes more time. And, arguably, we have also been moving backwards, because typical vector tile maps don’t support minutely updates. With this demo and blog post I hope to get some more people interested in this work so that together we can make a better, continously updated, always current OSM map. My focus will be to keep working on the building blocks in osm2pgsql to make this possible.

Aquileo | Improving the middle of osm2pgsql

Tue, 25 Jul 2023 16:46:57 +0200

In osm2pgsql there is that code that stores the raw data it gets from the input files and makes it available to other parts of osm2pgsql for further processing. This code is called, in osm2pgsql-speak, the “middle”. There are two implementations of this “middle”, one stores the data in RAM and doesn’t support updates. The other stores the data in the database and supports updates. Back in 2020 I started looking into that code to see what could be improved. In 2021 we released version 1.5.0 which contained a basically rewritten RAM middle. And now, in 2023, I finally got around to improving the database middle.

There were several things in the old database middle which weren’t that great that we wanted to address.

Database table format

First, the format osm2pgsql uses in the database is rather strange. Tags are stored as an array with alternating keys and values, relation members are stored in an even stranger format. That’s done for historical reasons and it works well enough for osm2pgsql itself, but it made it rather awkward to access this data from SQL. With modern versions of PostgreSQL we have better options for storing this data, in both these cases we can use the JSONB type to efficiently store and access this data.

This change reduces the database size by nearly 10% and only slows down the import by a few minutes for the whole planet.

Storing attributes

The old format stores OSM object attributes (timestamp, version, changeset, user id, and user name) as pseudo-tags in the same place the other tags are stored. This is not only rather inefficient, it also means we have to disentangle real and pseudo-tags when we read the data again.

The new code simply stores the attributes in their own database columns. If an attribute isn’t there, a NULL is stored. The exception is the user name. Because the user id and user name duplicates information it is much more efficient to just store the user id with each OSM object and have an extra lookup table for the user names. This has the added benefit that when a user changes their user name, the lookup table will be updated the next time the user changes an object, this way the data is always consistent.

Because most users don’t need those attributes, the attribute storage is only enabled with a specific command line option. That did’t change.

Storing the attributes used to make the database more than 50% larger. With these changes the added storage needed is negligable. Processing time has improved somewhat but is still much larger compared to processing time without attributes. (This is not because of anything in the “middle”, but because with attributes enabled osm2pgsql will send all nodes through the output processing, not just nodes with tags.)

Node tags

The old format never stored node tags in the database, because historically this was not needed for osm2pgsql. Way and relation tags need to be stored so that when a change in a node or way comes in, all parent ways and relations can also be updated. But because nodes are never the parent of another object the node tags are never needed.

But we want to do more complex processing today. For a while now we have the so-called two-stage processing which allows special processing of relation members. But that only works for way members and not for node members, because we were missing those node tags.

The new code can now store the node tags, too. You have to ask for this using a command line option, because not everybody will need this and storage costs quite a bit of disk space (as I write this about 30GB plus 5GB index for the whole planet). With the new format you can now access this data from SQL. The code to tie this in with two-stage processing isn’t written yet though, more work to do…

Performance

There were several problems with the way osm2pgsql was processing the data and accessing the database which were fixed in this effort. Some of these we have known about for a long time, other issues only were diagnosed now that we were looking into all the details.

For each changed node or changed way, osm2pgsql asks the database for all ways and relations the changed objects are in. The new code asks only once for all nodes and once for all ways. This change alone can mean a 10x speed increase!
We used to process some ways and relations twice if a member node or way changed and also the object itself changed. That’s just wasteful. Saves about 10% processing time to not do this.
When deleting objects, the database would in some cases not realize that it should use an existing index to figure out what to delete. So it used a full table scan which turns an operation that should basically take no time at all into one that can take 10 seconds or more. And we have many of those queries. Simply forcing the database to use the index fixes this.
For each object read from a change file we have to delete a possibly existing earlier version of that object from the database. But many objects are brand new so they can’t possibly exist in the database. So we now get the maximum node, way, and relation ids that exist in the database and check all incoming objects against those. For every object with larger than the known max id, we omit the delete, which gets us a slight performance boost.

There are more places where performance can probably be improved, but it needs a lot of testing with different scenarios to figure out which change actually improves performance and which makes it worse. Quite often the database (or more often the complex interaction between database and osm2pgsql) behaves differently than what the user (or developer) would expect. It needs a lot of time to figure out exactly what’s happening and why and how the different pieces interact.

Our test server (graciously sponsored by FOSSGIS e.V.) spent many days and weeks doing planet imports and updates in different configurations to help figure out what works and what doesn’t.

The Properties table

There is another “piece of infrastructure” that we recently added. The osm2pgsql_properties table stores some information about how osm2pgsql was called and about the input data. This means that when you are running updates (with osm2pgsql --append) osm2pgsql already knows some of the settings it needs and you don’t have to specify them on the command line any more.

This feature comes in handy now that we have the old “legacy” database format and the shiny “new” database format. For some time osm2pgsql needs to support both formats so that users can move over when it is convenient for them to do a re-import. The format is stored in that properties table so that it is automatically used for updates.

So what have we gained?

For the first time we now have a documented database format so that users can “officially” use it instead of treating it as some internal thing that nobody is allowed to touch. This allows more specialized processing of OSM data involving several related objects, for instance barriers on roads or something like that.

For the first time it is now possible to easily store all OSM data with all tags and all attributes in a database using osm2pgsql. This opens up new use case for users who don’t just want a rendering database but do statistics or special analytics on raw OSM data. Before now you had to use Osmosis (or some other specialized tools) to do these things, now you can use osm2pgsql (and get all the other features of osm2pgsql at the same time).

While the new database format takes slightly more processing time than the old format this is more than made up with the performance improvements we have done. Users running minutely updates will probably only see slight improvements if at all, but if you import a larger update, processing time might only be a tenth of what is was before.

Status

All this great new stuff is available in the “master” branch at the moment, but we’ll do a release soon. The new database format is still marked as experimental, because we want to have the option to change the format if user testing should reveal some bigger flaw. You need to activate it using a command line option. See the Documentation for all the details. In further releases we’ll finalize the format and sooner or later make it the default.

Many thanks to Thunderforest and Geofabrik for supporting this work!

Aquileo | Linking to external tools from taginfo

Sat, 15 Jul 2023 19:43:25 +0200

Taginfo has as its goal to find and collect all useful knowledge about OSM tags and make it accessible. But taginfo can’t do everything. That’s why it has always linked to tools that “do something with OSM tags”, the most popular one is probably Overpass Turbo.

Taginfo does have maps showing an overview of where certains tags are used, but the maps are static and not available for all tags. The link to Overpass gives you more option of exploring the actual OSM data on a map (and much more).

Similarly taginfo can show you in a graph how often a key or tag was used over time. But you can only see this graph for a single key or tag and all it does is count the uses. But there are tools out there that can do more. In the last weeks I have integrated links to two such tools to taginfo: Martin Raifer’s OSM Tag history tool and the ohsome dashboard from HeiGIT. Both allow you do explore the OSM tag history, but in different ways.

To use these tools look for the tool buttons on the top right of all key, tag, and relation pages. The filter setting right above the tool buttons will affect those link (except on the relation pages where a filter on relations is always used).

The OSM Tag history tool is also available from the comparison page (example). In this case it shows the curves for all compared keys and tags on a single graph making it easy to see the differences and similarities.

Do you know of any other tools that can (generically) handle OSM tags, keys, and/or relation types and that might be useful to integrate into taginfo? Open an issue on the taginfo issue tracker!

Aquileo | Modernizing taginfo

Sat, 22 Apr 2023 18:07:54 +0200

About once a year I set aside a week to work on taginfo, one of my many projects that doesn’t get enough attention. And inevitably once I get started there are so many things I want to do that it takes two weeks instead of the originally allotted week. So this time I thought I plan with the two weeks from the start. You probably know where this goes: I spend more like three or four weeks on taginfo…

This time I worked on something I wanted to do for a long time: Modernize the Javascript and CSS. The web technologies HTML, CSS, and Javascript have matured quite a lot in the last decade. Browsers are pretty good at keeping backwards compatibility and the old code still worked, but it was showing its age and newer and better solutions for many problems are available today.

Back in 2010 when I started taginfo, jQuery was a popular library. It still exists today but browsers have gained so much native functionality that it isn’t as important. And most of the jQuery plugins I was using have not been updated in years, some you can’t even find on the Internet any more. So I decided to get rid of jQuery and most external Javascript libraries altogether and write the code I need myself.

The taginfo site doesn’t have much Javascript on it. The most complex piece is the code for the dynamic tables you see on many pages. They were generated by an unmaintained jQuery plugin called Flexigrid in which I have fixed some bugs and changed some things here or there. I looked around for a Javascript library that could give me the same functionality in more modern code, but couldn’t find anything not tied to some complex framework. So I decided to write that myself. This has the added advantage that I can write exactly the functionality I want and don’t have to add some external warts to some existing Javascript library. The tables get their content through asynchronous API calls, support sorting by different columns, filter queries, paging (with buttons or the keyboard, or — new — with the mouse wheel (press Shift to use that)) and resizing of columns.

The rest of the Javascript is mostly pretty simple. The “tab” functionality, autocomplete for the search, some mouse and keyboard interaction. A lot of the code is simple formatting of key and tag links, numbers and so on which was already there and basically unchanged.

The only external library I am still using is D3 for the tag cloud on the home page and the charts and graphs. That library is pretty stable and it does some clever and useful stuff. And it doesn’t have any dependencies on jQuery or anything else.

You might wonder why I am not using some kind of modern Javascript framework. The reason is simple: They are changing too fast. Taginfo has been around for a long time and I expect it to be around for a long time. But, as mentioned, I can only work on it for small amounts of time here or there. So it is important it keeps working without any intervention and that, when I find the time to work on it, I can actually work on it and not spend my time updating to the newest framework version or getting NodeJS updated so that the build system works before I can do this one little change. Most of the Javascript I am writing is in one file, the same with the CSS. I don’t build the CSS or use Typescript or something like that. I have to do some things myself that a framework could solve for me, but taginfo isn’t so complex that this isn’t possible.

So, after all that work, what do I have to show for it? Most people will not even notice the changes. The site looks and feels more or less the same as before. That’s intentional. I have ideas for more changes, but they have to wait for the next time. But the big changes under the hood make the site already work better on smaller screens. Many bug reports I get are about the site not working correctly on mobile phones. There is still a lot to be done in that regard, but because of the cleaner CSS and Javascript it is now much easier to work on that.

Inevitably I broke some things in the process. Some issues I found myself, some were found by users. And more will probably be found in the next weeks or months. But I am pretty happy with the state the web site is in right now. Maybe the next time I work on taginfo I can work on the content side.