Can I download OSM data and save as a file geodatabase?

9995
15
Jump to solution
05-20-2020 02:43 PM
NataliaGutierrez1
New Contributor III

Hello,

I am trying to figure out how to work with OSM data. 

My goal is to get building height and building footprints for the USA from OpenStreetMaps.

The problem is that if I download the state level shapefiles from Geofabrik (https://download.geofabrik.de/ ) they don't have building height data. The only way I've been able to see building height data is if I download the OSM.pdf file format and then convert it to shapefile with a third party converter software such as MyGeodata Cloud - GIS Data Warehouse, Converter, Maps .

I currently have ArcGIS Pro and from other posts I've read, I can only see OSM.pdf data in ArcMap. 

Will I be able to import OSM.pdf data into ArcMap, convert it to file geodatabase and then take it to ArcGIS Pro?

I need to run calculations, manipulate this data, and integrate it into some work I am doing.

I would appreciate your help,

Thank you,

Natalia

0 Kudos
15 Replies
MarcoBoeringa
MVP Regular Contributor

Natalia Gutierrez‌.

Am I right you used the "*.osm.bz2" file directly as input?

If so, that would explain the problem, because you need to "UN-bz2" (unzip / decompress) the file. Unfortunately, this cannot be done by the default unzip functionality of Windows, because *.bz2 files are in a different compression format.

I use the free 7-Zip tool for that, other generic compress / decompress utilities may be able to do it as well if they support the format.

After decompression, you should have a file with only the *.osm extension, which is in fact a plain text XML file, just with a different extension. If the file is small though (< 50MB), you can simply open it in Wordpad and see it's XML content as plain text.

It is this *.osm file that needs to be used as input for the tool, not the bz2 zipped one.

By the way, I am actually surprised you were able to sort of run the tool. I would expect the tool to error out immediately with a compressed file as input...

MervynLotter
Occasional Contributor III

Hi Marco

I was able to follow your instructions and import the point, line and polygon features into a geodatabase. Thank you.

However I am unable to symbolise any of these three using any of the .lyr files that came with the ArcGIS Editor for OpenStreetMap extension. Do you know where i can download layer filesthat will work? Or better still, download any .lyrx files for the Open Street Map data?

Thanks

NataliaGutierrez1
New Contributor III

Hi Marco thank you and yes I was able to figure this out and I finally imported the OSM data into GIS. 

But I have a new problem... 😕 this seems to never end..

I do not see any building height data in the attribute table. I only see one field with building data but it seems to be just the building use not the height. 

I first ran a test to see if OSM data contained building height data. I downloaded an extract directly from OSM, which I then converted to shp and then opened it in ArcGIS and I saw a field with building height data.

I expected to see this field as well when downloading the osm.xml file from Geofabrik but I don't see it.

Do you know if there are any other data providers that offer this field? I haven't found any documentation whatsoever on this other than this document from Geofabrik saying they sell the building height attribute data for shapefiles.

I am not sure how to move forward with this.

Your help is much appreciated!

0 Kudos
MarcoBoeringa
MVP Regular Contributor

Natalia Gutierrez

Be aware that actual building height information is only scarcely available. Some cities like e.g. New York may have pretty extensive building height information, and in some other cases, there may be information about the number of stories / floors / levels of a building, but certainly DO NOT expect anywhere near global consistent coverage of this information in OpenStreetMap. It is a volunteer project after all, and only the things users find interesting or have available source for, will be added.

To get some impression if your area of interest has actual building heights, you can visit the osmbuildings or F4map websites, that display OpenStreetMap 3D data based on actual OSM derived data. Note though that F4Map also adds in some more advanced 3D models including texture, that is not in OSM. osmbuildings is in that respect more faithful to the original data.

Note that, in order see specific attributes that are not part of default set of fields added, you can run the "OSM Attribute Selector" tool to add new fields. This tool is part of the toolbox. Note that this tool may be relatively slow though on really large datasets, as it needs to extract the information from a binary tag storage field that is added by default.

The other option is to specify a custom set of field on initial import using the "OSM File Loader (Load Only)" tool.

For your specific question, you probably need to think about these OpenStreetMap keys:

- height=x

- levels=x

- building:levels=x

As documented e.g. here: Simple 3D buildings - OpenStreetMap Wiki 

NataliaGutierrez1
New Contributor III

thank you Marco this is very helpful.

Will try specifying the specific fields with the initial import using the same tool I've been using which is the OSM File Loader (Load Only) Tool. 

This tool is taking a very long time even to load very small testing areas so I don't want to use even a slower tool.  But it is good to know that there is the OSM Attribute Selector tool as well.

Will let you know how that goes ! 

0 Kudos
MarcoBoeringa
MVP Regular Contributor

Natalia Gutierrez

What is your baseline for saying "This tool is taking a very long time even to load very small testing areas" (it shouldn't!), what do you compare it to, and do you have concrete figures for this (x MB *.osm file imports in y minutes)?

Yes, there are considerably faster tools, like the command line tools reliant on PostGIS like osm2pgsql and imposm 3, but those are not "ArcGIS friendly", and require quite a bit of IT proficiency and familiarity with Linux to be able to work with them.

By the way, you haven't forgot to set the Parallel Processing Factor geoprocessing environment setting when running the tool? Only when you set it, will the tool be able to use multiple cores.

Also, be aware that I consider slow HDD based network drives utterly dead for use with OpenStreetMap. To import OSM data, you need fast random access to data on drive. This requires SSD storage. If you have slow network drives, and only an HDD on your local computer, I strongly recommend to get a fast USB 3.x SSD based external drive, and use that to import the data. After import, you can copy the File Geodatabase to your slow network drive for usage.

Why do you need fast random access?

OpenStreetMap data has its own geometry datamodel, that resembles classic CAD files (or if you have been around long enough, ESRI's ArcInfo "coverage" format), where an explicit Polygon data type does not exist. You may be astounded to hear this, but OSM at its basis only knows about nodes (points / vertex) and ways (lines that may or may not form a closed loop). Polygons are only secondarily defined by 1) either ways forming a closed loop and having been tagged with a tag that is generally recognized to be a polygon (e.g. building=x, a closed way with the tag building=house will be imported as Polygon by every tool I know of), 2) closed ways additionally tagged with an explicit area=yes, and 3) if the way is part of a multipolygon relation, otherwise the way is recognized and imported as Line geometry.

There a Pro and Cons to this datamodel:

- Pro is that it is a very flexible and efficient data format: the nodes / vertexes of any shared line between polygons are only stored once, and the two ways bordering each other, simply reference them. This makes for a very efficient storage.

- Con is that any way / Line, or way / Polygon needs to be reconstructed from its referenced nodes and ways.

It is this latter Con that is causing the need for fast random access to data. For any way or multipolygon relation, the import tools need to look up the corresponding nodes and ways that make it up. Only then, is the tool able to create a Simple Features compliant geometry (Line / Polygon), that can be stored in a File Geodatabase feature class, shapefile or PostGIS database spatial column.