Can I download OSM data and save as a file geodatabase?

9997
15
Jump to solution
05-20-2020 02:43 PM
NataliaGutierrez1
New Contributor III

Hello,

I am trying to figure out how to work with OSM data. 

My goal is to get building height and building footprints for the USA from OpenStreetMaps.

The problem is that if I download the state level shapefiles from Geofabrik (https://download.geofabrik.de/ ) they don't have building height data. The only way I've been able to see building height data is if I download the OSM.pdf file format and then convert it to shapefile with a third party converter software such as MyGeodata Cloud - GIS Data Warehouse, Converter, Maps .

I currently have ArcGIS Pro and from other posts I've read, I can only see OSM.pdf data in ArcMap. 

Will I be able to import OSM.pdf data into ArcMap, convert it to file geodatabase and then take it to ArcGIS Pro?

I need to run calculations, manipulate this data, and integrate it into some work I am doing.

I would appreciate your help,

Thank you,

Natalia

0 Kudos
1 Solution

Accepted Solutions
MarcoBoeringa
MVP Regular Contributor

If you download the ArcGIS Editor for OpenStreetMap extension (which currently only exists for up to version 10.7.x of ArcMap, and not for ArcGIS Pro. The 10.8 version for ArcMap still needs releasing), you can use the "OSM File Loader (Load Only)" tool to import an OSM XML file (not PBF!) and create a File Geodatabase. Geofabrik also still offers XML as download format, although PBF is preferred due to size and speed, but cannot be handled by this import tool, so is of no use to you.

Note that this import process will be a lengthy process for something the size of the entire US. Even though the tool is multi-core enabled via the Parallel Processing Factor environment setting for some stages of the processing, and the tool is thus quite capable, it is still not nearly as fast as some of the other specialized open source tools out there as developed by the OpenStreetMap community (osm2pgsql, imposm3).

However, the nice thing about the Editor is that it is entirely "ArcGIS", and doesn't require setting up a PostgreSQL database, nor running a Linux environment to call command line tools, so it is easy and familiar to get "up and running".

Note also that this tool will load ALL information in the OSM XML file, including non-building data, which will significantly extent the time needed to import the data. If you are only interested in buildings, you should filter the OSM XML file and write a new XML file with only buildings from the Geofabrik extract using, as your likely best bet for this kind of filtering and re-writing work, the command line Osmium Tool. After writing a new, much smaller OSM XML file only containing buildings, you can then use the "OSM File Loader (Load Only)" tool to import the data into the File Geodatabase.

After having imported the data, you can simply access the Feature Classes in ArcGIS Pro, no problem.

By the way, I would strongly discourage you to use the "Download, Extract and Symbolize OSM Data" tool to attempt to download the entire US. It will fail. It is not designed to handle large data download loads. Only the "OSM File Loader (Load Only)" tool has the power to handle larger country sized extracts.

View solution in original post

15 Replies
MarcoBoeringa
MVP Regular Contributor

If you download the ArcGIS Editor for OpenStreetMap extension (which currently only exists for up to version 10.7.x of ArcMap, and not for ArcGIS Pro. The 10.8 version for ArcMap still needs releasing), you can use the "OSM File Loader (Load Only)" tool to import an OSM XML file (not PBF!) and create a File Geodatabase. Geofabrik also still offers XML as download format, although PBF is preferred due to size and speed, but cannot be handled by this import tool, so is of no use to you.

Note that this import process will be a lengthy process for something the size of the entire US. Even though the tool is multi-core enabled via the Parallel Processing Factor environment setting for some stages of the processing, and the tool is thus quite capable, it is still not nearly as fast as some of the other specialized open source tools out there as developed by the OpenStreetMap community (osm2pgsql, imposm3).

However, the nice thing about the Editor is that it is entirely "ArcGIS", and doesn't require setting up a PostgreSQL database, nor running a Linux environment to call command line tools, so it is easy and familiar to get "up and running".

Note also that this tool will load ALL information in the OSM XML file, including non-building data, which will significantly extent the time needed to import the data. If you are only interested in buildings, you should filter the OSM XML file and write a new XML file with only buildings from the Geofabrik extract using, as your likely best bet for this kind of filtering and re-writing work, the command line Osmium Tool. After writing a new, much smaller OSM XML file only containing buildings, you can then use the "OSM File Loader (Load Only)" tool to import the data into the File Geodatabase.

After having imported the data, you can simply access the Feature Classes in ArcGIS Pro, no problem.

By the way, I would strongly discourage you to use the "Download, Extract and Symbolize OSM Data" tool to attempt to download the entire US. It will fail. It is not designed to handle large data download loads. Only the "OSM File Loader (Load Only)" tool has the power to handle larger country sized extracts.

NataliaGutierrez1
New Contributor III

Hi Marco,

I really appreciate your detailed response on this matter.

I will make sure to download version 10.7.x of ArcMap. Do you know if I can select the version of ArcMap I need? I am in the process of buying it since I only have ArcGIS Pro.

Being able to filter the OSM XML file and write a new XML file with only buildings sounds ideal to expedite this process. The only problem with this is that it seemed quite difficult to me since I don't know a lot of coding. I will try to do more research to see if I can learn how to do it with the Osmium Tool. 

Thank you!

Natalia

0 Kudos
MarcoBoeringa
MVP Regular Contributor

There is one more option that I should have mentioned, and that you may be able to run from ArcGIS Pro: use the Data Interoperability extension. Personally, I have had mixed experiences with this tool for importing OSM data, but it does support PBF directly also if I remember well.

The output of the data is not the same as that of "OSM File Loader (Load Only)" tool: it will already break up the data in thematic layers (e.g. your desired buildings!), instead of outputting a Point/Line/Polygon feature class containing all data.

 

This may actually be desirable for you though. Try it out and see if it works for you.

One BIG warning though: once you enable Data Interoperability, ArcGIS will start to support "direct read" of OSM XML and PBF files in the Catalog window. That is: it will show apparent Feature Classes in the Catalog window, that you can select and add to your map.

While this sounds great, it is a nightmare when selecting a huge extract like the US: the Data Interoperability "direct read" will start automatic dynamic indexing and reading the entire huge XML or PBF file once you click it in the Catalog window, which is very likely to block your application for ever (hours at least), while it attempts to dynamically index and read the data. This just won't work for large dataset.

So: never select or click on large OSM extracts in the Catalog window once Data Interoperability is enabled (it does kind of work with tiny extracts, still I think this is a bad non-scalable implementation).

Tags (1)
0 Kudos
RenatoSalvaleon3
Esri Contributor

Direct read in the Catalog pane only applies to ArcMap. Data Interop for Prextension does not have Interoperability Connection by design. There is a way around this by using an ETL tool configured in the Model builder. We have a blog with a sample workspace in the Data Interop Community space that demonstrates this.

Additionally, you can always launch Data Interoperability independent of ArcGIS Pro although it requires a license enabled in the Pro backstage.


0 Kudos
MarcoBoeringa
MVP Regular Contributor

And one other last thing, you are aware of this recent ESRI blog post?:

https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/live-openstreetmap-data-in-arc... 

It is probably not going to offer you a solution for almost continental size processing as you desire with the US, but it is good to be aware it is available.

0 Kudos
NataliaGutierrez1
New Contributor III

thank you, I haven't checked it out. Will do now

Thank you so much for your help in this matter Marco. 

Much appreciated! 

0 Kudos
NataliaGutierrez1
New Contributor III

Hi Marco,

I am having trouble with the OSM File Loader (Load Only) tool. 

I've run the tool a couple of times with different files and the tool runs but in the end I don't see any of the data from OSM inside my gdb.

The osm.xml file from OSM that I chose is at the state level. I downloaded the .osm.bz2 option because if I understand correctly, when decompressed this yields an OSM XML file. This is where I downloaded the data from: https://download.geofabrik.de/north-america/us.html 

 

I wanted to attach an image showing you my inputs for the tool but for some reason it is not letting me attach anything to this response.

These are my inputs for the tool:

OSM File: new-jersey-latest.osm.bz2

OpenStreetMap point features: the gdb and the name of the point features

OpenStreetMap line features: the gdb and the name of the line features

OpenStreetMap polygon features: the gdb and the name of the polygon features

After the tool runs I expect to have 3 feature classes inside my gdb each one with all of the point, line and polygon data from OSM.

I don't see anything...

I am going to keep trying but would really appreciate your help. 

Thank you!

Natalia

0 Kudos
ALI_AZZOUHRI
New Contributor III

Hi Mr . Marco

Thanks for the explanation about osm data.

I'm using  ArcMap 10.8.2 and I just installed ArcGIS Editor for OpenStreetMap 10.8.x but it seems like is not working with me any ideas, please?

Hamed_Abdulgader_DowAlgassi
New Contributor

I had similar problems there are any solutions

 

Hamed_Abdulgader_DowAlgassi_0-1675799635517.png

 

 

0 Kudos