nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Can I download OSM data and save as a file geodatabase?

NataliaGutierrez1

Hello,

I am trying to figure out how to work with OSM data.

My goal is to get building height and building footprints for the USA from OpenStreetMaps.

The problem is that if I download the state level shapefiles from Geofabrik (https://download.geofabrik.de/ ) they don't have building height data. The only way I've been able to see building height data is if I download the OSM.pdf file format and then convert it to shapefile with a third party converter software such as MyGeodata Cloud - GIS Data Warehouse, Converter, Maps .

I currently have ArcGIS Pro and from other posts I've read, I can only see OSM.pdf data in ArcMap.

Will I be able to import OSM.pdf data into ArcMap, convert it to file geodatabase and then take it to ArcGIS Pro?

I need to run calculations, manipulate this data, and integrate it into some work I am doing.

I would appreciate your help,

Thank you,

Natalia

Najděte více příspěvků označených jako

Accepted answers

MarcoBoeringa

If you download the ArcGIS Editor for OpenStreetMap extension (which currently only exists for up to version 10.7.x of ArcMap, and not for ArcGIS Pro. The 10.8 version for ArcMap still needs releasing), you can use the "OSM File Loader (Load Only)" tool to import an OSM XML file (not PBF!) and create a File Geodatabase. Geofabrik also still offers XML as download format, although PBF is preferred due to size and speed, but cannot be handled by this import tool, so is of no use to you.

Note that this import process will be a lengthy process for something the size of the entire US. Even though the tool is multi-core enabled via the Parallel Processing Factor environment setting for some stages of the processing, and the tool is thus quite capable, it is still not nearly as fast as some of the other specialized open source tools out there as developed by the OpenStreetMap community (osm2pgsql, imposm3).

However, the nice thing about the Editor is that it is entirely "ArcGIS", and doesn't require setting up a PostgreSQL database, nor running a Linux environment to call command line tools, so it is easy and familiar to get "up and running".

Note also that this tool will load ALL information in the OSM XML file, including non-building data, which will significantly extent the time needed to import the data. If you are only interested in buildings, you should filter the OSM XML file and write a new XML file with only buildings from the Geofabrik extract using, as your likely best bet for this kind of filtering and re-writing work, the command line Osmium Tool. After writing a new, much smaller OSM XML file only containing buildings, you can then use the "OSM File Loader (Load Only)" tool to import the data into the File Geodatabase.

After having imported the data, you can simply access the Feature Classes in ArcGIS Pro, no problem.

By the way, I would strongly discourage you to use the "Download, Extract and Symbolize OSM Data" tool to attempt to download the entire US. It will fail. It is not designed to handle large data download loads. Only the "OSM File Loader (Load Only)" tool has the power to handle larger country sized extracts.

All comments

MarcoBoeringa

After having imported the data, you can simply access the Feature Classes in ArcGIS Pro, no problem.

NataliaGutierrez1

Hi Marco,

I really appreciate your detailed response on this matter.

I will make sure to download version 10.7.x of ArcMap. Do you know if I can select the version of ArcMap I need? I am in the process of buying it since I only have ArcGIS Pro.

Being able to filter the OSM XML file and write a new XML file with only buildings sounds ideal to expedite this process. The only problem with this is that it seemed quite difficult to me since I don't know a lot of coding. I will try to do more research to see if I can learn how to do it with the Osmium Tool.

Thank you!

Natalia

MarcoBoeringa

There is one more option that I should have mentioned, and that you may be able to run from ArcGIS Pro: use the Data Interoperability extension. Personally, I have had mixed experiences with this tool for importing OSM data, but it does support PBF directly also if I remember well.

The output of the data is not the same as that of "OSM File Loader (Load Only)" tool: it will already break up the data in thematic layers (e.g. your desired buildings!), instead of outputting a Point/Line/Polygon feature class containing all data.

This may actually be desirable for you though. Try it out and see if it works for you.

One BIG warning though: once you enable Data Interoperability, ArcGIS will start to support "direct read" of OSM XML and PBF files in the Catalog window. That is: it will show apparent Feature Classes in the Catalog window, that you can select and add to your map.

While this sounds great, it is a nightmare when selecting a huge extract like the US: the Data Interoperability "direct read" will start automatic dynamic indexing and reading the entire huge XML or PBF file once you click it in the Catalog window, which is very likely to block your application for ever (hours at least), while it attempts to dynamically index and read the data. This just won't work for large dataset.

So: never select or click on large OSM extracts in the Catalog window once Data Interoperability is enabled (it does kind of work with tiny extracts, still I think this is a bad non-scalable implementation).

MarcoBoeringa

And one other last thing, you are aware of this recent ESRI blog post?:

https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/live-openstreetmap-data-in-arcgis/

It is probably not going to offer you a solution for almost continental size processing as you desire with the US, but it is good to be aware it is available.

NataliaGutierrez1

thank you, I haven't checked it out. Will do now

Thank you so much for your help in this matter Marco.

Much appreciated!

NataliaGutierrez1

Hi Marco,

I am having trouble with the OSM File Loader (Load Only) tool.

I've run the tool a couple of times with different files and the tool runs but in the end I don't see any of the data from OSM inside my gdb.

The osm.xml file from OSM that I chose is at the state level. I downloaded the .osm.bz2 option because if I understand correctly, when decompressed this yields an OSM XML file. This is where I downloaded the data from: https://download.geofabrik.de/north-america/us.html

I wanted to attach an image showing you my inputs for the tool but for some reason it is not letting me attach anything to this response.

These are my inputs for the tool:

OSM File: new-jersey-latest.osm.bz2

OpenStreetMap point features: the gdb and the name of the point features

OpenStreetMap line features: the gdb and the name of the line features

OpenStreetMap polygon features: the gdb and the name of the polygon features

After the tool runs I expect to have 3 feature classes inside my gdb each one with all of the point, line and polygon data from OSM.

I don't see anything...

I am going to keep trying but would really appreciate your help.

Thank you!

Natalia

MarcoBoeringa

Natalia Gutierrez‌.

Am I right you used the "*.osm.bz2" file directly as input?

If so, that would explain the problem, because you need to "UN-bz2" (unzip / decompress) the file. Unfortunately, this cannot be done by the default unzip functionality of Windows, because *.bz2 files are in a different compression format.

I use the free 7-Zip tool for that, other generic compress / decompress utilities may be able to do it as well if they support the format.

After decompression, you should have a file with only the *.osm extension, which is in fact a plain text XML file, just with a different extension. If the file is small though (< 50MB), you can simply open it in Wordpad and see it's XML content as plain text.

It is this *.osm file that needs to be used as input for the tool, not the bz2 zipped one.

By the way, I am actually surprised you were able to sort of run the tool. I would expect the tool to error out immediately with a compressed file as input...

MervynLotter

Hi Marco

I was able to follow your instructions and import the point, line and polygon features into a geodatabase. Thank you.

However I am unable to symbolise any of these three using any of the .lyr files that came with the ArcGIS Editor for OpenStreetMap extension. Do you know where i can download layer filesthat will work? Or better still, download any .lyrx files for the Open Street Map data?

Thanks

NataliaGutierrez1

Hi Marco thank you and yes I was able to figure this out and I finally imported the OSM data into GIS.

But I have a new problem... 😕 this seems to never end..

I do not see any building height data in the attribute table. I only see one field with building data but it seems to be just the building use not the height.

I first ran a test to see if OSM data contained building height data. I downloaded an extract directly from OSM, which I then converted to shp and then opened it in ArcGIS and I saw a field with building height data.

I expected to see this field as well when downloading the osm.xml file from Geofabrik but I don't see it.

Do you know if there are any other data providers that offer this field? I haven't found any documentation whatsoever on this other than this document from Geofabrik saying they sell the building height attribute data for shapefiles.

I am not sure how to move forward with this.

Your help is much appreciated!

MarcoBoeringa

Natalia Gutierrez‌

Be aware that actual building height information is only scarcely available. Some cities like e.g. New York may have pretty extensive building height information, and in some other cases, there may be information about the number of stories / floors / levels of a building, but certainly DO NOT expect anywhere near global consistent coverage of this information in OpenStreetMap. It is a volunteer project after all, and only the things users find interesting or have available source for, will be added.

To get some impression if your area of interest has actual building heights, you can visit the osmbuildings or F4map websites, that display OpenStreetMap 3D data based on actual OSM derived data. Note though that F4Map also adds in some more advanced 3D models including texture, that is not in OSM. osmbuildings is in that respect more faithful to the original data.

Note that, in order see specific attributes that are not part of default set of fields added, you can run the "OSM Attribute Selector" tool to add new fields. This tool is part of the toolbox. Note that this tool may be relatively slow though on really large datasets, as it needs to extract the information from a binary tag storage field that is added by default.

The other option is to specify a custom set of field on initial import using the "OSM File Loader (Load Only)" tool.

For your specific question, you probably need to think about these OpenStreetMap keys:

- height=x

- levels=x

- building:levels=x

As documented e.g. here: Simple 3D buildings - OpenStreetMap Wiki

NataliaGutierrez1

thank you Marco this is very helpful.

Will try specifying the specific fields with the initial import using the same tool I've been using which is the OSM File Loader (Load Only) Tool.

This tool is taking a very long time even to load very small testing areas so I don't want to use even a slower tool. But it is good to know that there is the OSM Attribute Selector tool as well.

Will let you know how that goes !

ALI_AZZOUHRI

Hi Mr . Marco

Thanks for the explanation about osm data.

I'm using ArcMap 10.8.2 and I just installed ArcGIS Editor for OpenStreetMap 10.8.x but it seems like is not working with me any ideas, please?

osm.JPG

RenatoSalvaleon3

Direct read in the Catalog pane only applies to ArcMap. Data Interop for Prextension does not have Interoperability Connection by design. There is a way around this by using an ETL tool configured in the Model builder. We have a blog with a sample workspace in the Data Interop Community space that demonstrates this.

Additionally, you can always launch Data Interoperability independent of ArcGIS Pro although it requires a license enabled in the Pro backstage.

algassiafricageoportalalgassie

I had similar problems there are any solutions

Připojte se ke konverzaci

Přihlášení členové mohou přispívat, sledovat aktualizace a další. Jste tu noví? Zaregistrujte si bezplatný účet.

Rychlé odkazy

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.