caching for ArcGIS Pro base maps? side loading?

2456
9
Jump to solution
05-12-2023 12:12 PM
davedoesgis
Occasional Contributor III

Background: I'm using ArcGIS Pro to create a ginormous map of the lower 48 US states. This will be room-sized, approximately 36 feet by 24 feet. It will be exported as a single PNG and sent to a custom printer.

Problem: When exporting the PNG, there will be missing tiles from the Esri basemaps. My best guess is it is making a zillion tile requests in the background and a couple of tiles get hung up from network hiccups. We did one of these in the past without a base map by creating our own hillshade, hydro, transportation, and land use layers, which was a hassle.

Objective: Use a pre-existing base map, rather than create our own layers that simulate a base map.

Idea 1: I like the default Pro "Topographic" map, which adds layers "World Topographic Map" and "World Hillshade" to my map. The hillshade layer has a cache property, which I've turned on, but the other does not have that property. Is there a way to force Pro to pre-download all the tiles and cache these?

Idea 2: Is there a good base map that I can download and use? In the mobile world, I'd call this side-loading. Whether it's a custom base map, or just a layer I can download, anyone know of something good for the lower 48 US states?

Also open to other strategies, so fire away! thanks!

0 Kudos
1 Solution

Accepted Solutions
davedoesgis
Occasional Contributor III

Here is what I ended up doing. It wasn't perfect, and it took a lot of figuring out, but the ultimate solution ended up being pretty simple. The source data were all on my local system so I wasn't making so many web requests.

Hillshade

The USGS's 3DEP (3D Elevation Program) 1 arc second DEM was about the right resolution for me. They have a map viewer where you can generate a list of URLs to download TIF tiles for your map extent. I had over 1600 tiles, so I wrote a Python script to download all these.

The Pro GP tool Mosaic To New Raster will create a seamless DEM. Then, I projected that and drew it with multi-directional hillshade symbology. I just used the layer transparency on my layout's white background to lighten it up. I wanted to include terrain, but dial it back since it wasn't central to the purpose of my map.

NOTE: I tried a few times to pre-generate a raster with hillshade symbology so Pro wasn't doing so much processing to symbolize it on the fly. Everything I tried ended up looking blurry, so I just used the DEM rendered on the fly. It took several hours to export my layout, but it worked, so I guess pre-generating a hillshade raster wasn't necessary. This could probably be it's own forum post, but was a dead end for me.

Vector Tiles

I used the Vector Style Editor to create my own style based on the World Navigation basemap data and its default style. Mostly, I needed a few text labels and highway shields to be a larger font size. Here is a good tutorial on how to use it. I The editor saved my modified style to an AGOL item. AGOL says its type is "Tile layer". The size is only 1MB, so I'm assuming it's just the style definition files and maybe some icons (a.k.a. sprites), but not the actual data.

You can add that AGOL item as a map layer in Pro and you have everything from the basemap on your map symbolized the way you want. Note if I made subsequent changes to the AGOL style, I had to restart Pro to get it to pick them up. 

As far as I can tell, the style I used has all style classes set to 100% opacity. I'm not sure if there's a better way to do this, but I used the "Multiply" layer blend mode to draw it over my hillshade.

dcafdg_0-1686273998069.png

 

At 100% scale, the multiply layer blend looks fantastic. If you haven't played with it, it is sooooo much better than just setting transparency. That said, not every feature should blend. Zooming way in to a highway shield up in the mountains with a lot of terrain, you can see the hillshade subtly bleed through (it should just be white). Most roads and man-made features aren't running through this kind of terrain, and it's really subtle at 100% view, but it kind of annoyed me.

dcafdg_1-1686274225034.png

If I had time, I would create two basemap styles. The first style would include any feature that would use the layer blend transparency (mostly polygon layers like parks and land use). In the style editor, you can toggle the visibility of style classes, so I'd toggle off all other style classes. The second style would be anything that should be totally opaque, e.g.: roads, highway shields, text labels (turning off visibility for all other style classes from the first style). Then, these would be added as separate layers to my map, using the multiply layer blend on only the first style. This is more or less what the Imagery Hybrid basemap does with its reference layer:

dcafdg_2-1686274652176.png

 

One word of caution on this vector tile style approach is that you cannot edit the data. You can do a lot with the styles, but you're locked into the data Esri provides unless you want to create your own (which I didn't). I don't know if styles can include SQL-type queries to modify (or hide) specific features. In for a penny, in for a pound.

At this point, you're done for most normal map production. Because my map was so massive, I wanted to have all data locally on my hard drive. My first attempt at exporting this layout ran for days before I killed it, and I suspect the culprit was the zillions of requests to AGOL for the basemap. 

I used the Pro "Download Map" button to download my basemap layer to a local vector tile package (VTPK). That runs for quite a while and will save a vector tile package file under your Pro project's folder and create a new map layer for it. Just like the AGOL basemap, I used the Multiply layer blend to add in the hillshade.

The download map control has a checkbox to include the basemap and a scale drop-down list to choose the most detailed scale you want. For my purposes, the basemap had way too much detail at the scale I was using. I chose to use two scale intervals smaller than that, which cleaned up the map and greatly reduced the amount of data I was downloading. In the vector style editor, I had planned this out ahead of time. For print maps, figuring out the scale you need in advance simplified matters, since there's a lot of scale-dependent stuff in those styles, and I just needed to worry about one scale. Note that the scale level is in the editor's interface:

dcafdg_3-1686275742497.png

The other style editor tip I didn't discover soon enough was clicking on a feature on the map takes you right to the style class for it. This saves a lot of wading through Esri's lists of style classes. They're well organized, but still there's a lot to sift through, or at least there was for World Navigation.

NOTE: The download map tool will allow you to download a lot of data, but there is some restriction. As your map extent gets bigger, the larger scale intervals will not be available in the interface to select. My guess is this is designed to protect you from asking for too much data. In the case of my room-sized map, I was able to download it all in one go, so it's not that restrictive. On my first attempt, I was using a larger map scale, and it was not available in the drop-down list for scales in the download map tool. See my comment on this post if you need to download a larger map extent than what the GUI will let you. You have to do it in batches (tiles... of tiles) and the extents cannot have any gaps or overlaps. I put in a code sample to move the map extent to create a non-overlapping and seamless extents for downloading VTPKs.

I know this seems like a lot, but it wasn't that complicated in the end for the scale of what I'm trying to do.

View solution in original post

0 Kudos
9 Replies
DavidPike
MVP Frequent Contributor

Possibly read through all of this post and try 'Download Map' with no layers, and choose the nearest cache scale which would match your printed map https://community.esri.com/t5/arcgis-pro-questions/exporting-vector-tile-package/td-p/523007

there's a REST operation to export tiles but would be more fiddly than the Pro GUI.

davedoesgis
Occasional Contributor III

Thanks @DavidPike - The link you posted was broken for me (tried on two computers). Is this it?

https://community.esri.com/t5/arcgis-pro-questions/exporting-vector-tile-package/td-p/523007

 

MarcoBoeringa
MVP Regular Contributor

@davedoesgis wrote:

Also open to other strategies, so fire away! thanks!


Does that willingness to be "open to other strategies" also extent to seriously paying for a high quality / high resolution PDF or TIFF/PNG file fully prepared for you, or does that openness stop right there?

You are asking a lot here, at 36x24 feet final print size, but fortunately, I have potentially a lot to offer to you.

I am asking because I have been working on an un-released personal multi-year project to develop a sophisticated "ArcGIS Renderer for OpenStreetMap". This has been a gargantuan project and challenge to be honest, but I think the results are pretty astounding by now, and far surpass any of the existing well known map services like the ones from ESRI, Mapbox, Google or Bing in terms of cartography and detail (especially in the true topographic scales of >= 1:50k).

I have presented a poster of this work at last year's "State Of The Map" OpenStreetMap conference in Florence, Italy. A screenshot of that poster is attached here in the post. If you want to see the poster at its true size and in full glory, you can download it from the SOTM 2022 poster gallery here:

https://2022.stateofthemap.org/posters/

It is "Poster 22 ArcGIS Renderer for OpenStreeMap". Click on the (unfortunately garbled) poor quality preview to open the full PDF from the gallery and save it locally if you want.

About your project:

36x24 feet gives me a nice 1:500k scale map in UTM zone 14N centered on the middle of the US (lower 48 states).

That would give the detail you can view in the other two attached images. I actually developed two styles: one loosely based on the existing OpenStreetMap "Default" map style (but with significant enhancements), and a true topographic style similar to some of world's most well known National Mapping agencies. These styles do not match ESRI's "Topographic" style, they are my own custom ArcGIS Pro styles. Also, the actual style format is my own propriety design, not a Mapbox vector style derivative. It actually uses Pro layer files (*.lyrx) as part of the file format.

It is a massive amount of data for the United States at the desired scale, but fortunately the Renderer does employ sophisticated generalization to handle this amount of data, without the typical crude "minimum area" filtering & Douglas Peucker generalization causing random ugly losses in e.g. forest polygons, as seen in many vector tile maps at small scales.

As you can see from the poster, I actually currently run a 1.5 TB spatial database with the entire Planet's worth of OpenStreetMap data in true spatial vector format, so exporting the lower 48 states in the topographic style is no problem. You do have to realize though that the current map content of the US in OpenStreetMap, although significant, is not at the level of some of the European countries, and also doesn't perfectly match the content of ESRI's "Topographic" map that is likely enhanced with Natural Earth and USGS or other local government agencies data. The data related to especially the landuse, natural features like mountain ranges, and hydrography is kind of patchy and incomplete in the US in OpenStreetMap, which will influence the final map image. That said, the final result map will still be impressive at the scale and size you desire with the entire major road network of the US, cities and the hillshade blended in.

If you want to review the attached images, I recommend you to download them. Unfortunately, the "Document Preview" option here on the ESRI forums seems to forcefully re-scale the images causing blurry low quality images, and doesn't allow you to set exact 100% viewing scale.

0 Kudos
davedoesgis
Occasional Contributor III

Hey @MarcoBoeringa - This looks awesome! I'd love to pay you for this, but I work for the government and it might take years, which is more time than I have. If you need an early adopter and testimonial, I might be your guy.

0 Kudos
MarcoBoeringa
MVP Regular Contributor

Hi @davedoesgis ,

Well, some budget must at least be available, as you don't do a 36x24 feet wall print for free at a printing shop....

But I understand the "budget" and outsourcing problem.

Although this Renderer is a wholly personal project, in the past, working for a mid-size GIS company here in the Netherlands, I have been dispatched on long term consultancy jobs at multiple Dutch government agencies, sometimes working for years internally, so I have some sense of how these things work.

Clearly this is going to be a one-off project considering the output.

Anyway, could you share some more details about this project and yourself? If you do not want to share this information on a public forum like this one, drop me a private message here on the "ESRI Community", or send me an e-mail via the mail address included on the SOTM poster.

- What government agency are you working for?

- What is the occasion for putting up this wall print: temporary or semi-permanent?

- Where is it going to be displayed, what office?

- Could you share a photo of the wall where it is going to be installed?

- Do you have more technical details about your exact requirements for the file format, resolution and size of the needed data file?

- Is the content of the print "The lower 48 states" already fixed, or would you potentially consider other options? Surely, the 1:500k hillshaded print of the US will be impressive, but the 1:25k scale of the Renderer's output, as visible in the SOTM poster, is really beautiful as well in print. That would of course require a more localized subject choice, e.g. part of a single state which preferably has good landuse and building coverage in OpenStreetMap.

- What is the timeline for this project? When do you need to have this finished?

- Is the map itself the final product, or are you going to use it as a backdrop to overlay other data on it before sending it of for print? If so, what is it going to display?

- What cartographic projection did you have in mind? I sometimes simply choose a central UTM zone for large regions, as per the suggested UTM zone 14N, as I like the more-or-less equal area property of UTM, simply accepting more shape distortion to the sides, but the Renderer can in fact output to any projection ArcGIS supports, e.g. Equal Earth.

0 Kudos
davedoesgis
Occasional Contributor III

Without knowing the cost of this, I couldn't say if we can afford it, but as I said, the issue is time. Contracting with or selling to the government here is a long, slow process.

The plan is to lay it out on a floor and have chess-type pieces we can move around to simulate equipment, supplies, and personnel in interactive exercises. I don't really know more than that, since I'm just creating the map but not too involved in the exercise planning.

 

0 Kudos
MarcoBoeringa
MVP Regular Contributor

Ah, I misunderstood. I thought it would be put up against a wall, but it is going to be walked on for a one time exercise.

I may be wrong, but this sounds a bit like FEMA.gov type work? At least, I could see FEMA doing it.

Anyway, the idea sounds kind of fun. If it is indeed for something related to disaster management, maybe I can help you out no cost, but I would sure like to see a bit more info, if not here, via a Private Message or mail.

You also haven't specified how quickly you need this. Days, week, month?

0 Kudos
davedoesgis
Occasional Contributor III

Here is what I ended up doing. It wasn't perfect, and it took a lot of figuring out, but the ultimate solution ended up being pretty simple. The source data were all on my local system so I wasn't making so many web requests.

Hillshade

The USGS's 3DEP (3D Elevation Program) 1 arc second DEM was about the right resolution for me. They have a map viewer where you can generate a list of URLs to download TIF tiles for your map extent. I had over 1600 tiles, so I wrote a Python script to download all these.

The Pro GP tool Mosaic To New Raster will create a seamless DEM. Then, I projected that and drew it with multi-directional hillshade symbology. I just used the layer transparency on my layout's white background to lighten it up. I wanted to include terrain, but dial it back since it wasn't central to the purpose of my map.

NOTE: I tried a few times to pre-generate a raster with hillshade symbology so Pro wasn't doing so much processing to symbolize it on the fly. Everything I tried ended up looking blurry, so I just used the DEM rendered on the fly. It took several hours to export my layout, but it worked, so I guess pre-generating a hillshade raster wasn't necessary. This could probably be it's own forum post, but was a dead end for me.

Vector Tiles

I used the Vector Style Editor to create my own style based on the World Navigation basemap data and its default style. Mostly, I needed a few text labels and highway shields to be a larger font size. Here is a good tutorial on how to use it. I The editor saved my modified style to an AGOL item. AGOL says its type is "Tile layer". The size is only 1MB, so I'm assuming it's just the style definition files and maybe some icons (a.k.a. sprites), but not the actual data.

You can add that AGOL item as a map layer in Pro and you have everything from the basemap on your map symbolized the way you want. Note if I made subsequent changes to the AGOL style, I had to restart Pro to get it to pick them up. 

As far as I can tell, the style I used has all style classes set to 100% opacity. I'm not sure if there's a better way to do this, but I used the "Multiply" layer blend mode to draw it over my hillshade.

dcafdg_0-1686273998069.png

 

At 100% scale, the multiply layer blend looks fantastic. If you haven't played with it, it is sooooo much better than just setting transparency. That said, not every feature should blend. Zooming way in to a highway shield up in the mountains with a lot of terrain, you can see the hillshade subtly bleed through (it should just be white). Most roads and man-made features aren't running through this kind of terrain, and it's really subtle at 100% view, but it kind of annoyed me.

dcafdg_1-1686274225034.png

If I had time, I would create two basemap styles. The first style would include any feature that would use the layer blend transparency (mostly polygon layers like parks and land use). In the style editor, you can toggle the visibility of style classes, so I'd toggle off all other style classes. The second style would be anything that should be totally opaque, e.g.: roads, highway shields, text labels (turning off visibility for all other style classes from the first style). Then, these would be added as separate layers to my map, using the multiply layer blend on only the first style. This is more or less what the Imagery Hybrid basemap does with its reference layer:

dcafdg_2-1686274652176.png

 

One word of caution on this vector tile style approach is that you cannot edit the data. You can do a lot with the styles, but you're locked into the data Esri provides unless you want to create your own (which I didn't). I don't know if styles can include SQL-type queries to modify (or hide) specific features. In for a penny, in for a pound.

At this point, you're done for most normal map production. Because my map was so massive, I wanted to have all data locally on my hard drive. My first attempt at exporting this layout ran for days before I killed it, and I suspect the culprit was the zillions of requests to AGOL for the basemap. 

I used the Pro "Download Map" button to download my basemap layer to a local vector tile package (VTPK). That runs for quite a while and will save a vector tile package file under your Pro project's folder and create a new map layer for it. Just like the AGOL basemap, I used the Multiply layer blend to add in the hillshade.

The download map control has a checkbox to include the basemap and a scale drop-down list to choose the most detailed scale you want. For my purposes, the basemap had way too much detail at the scale I was using. I chose to use two scale intervals smaller than that, which cleaned up the map and greatly reduced the amount of data I was downloading. In the vector style editor, I had planned this out ahead of time. For print maps, figuring out the scale you need in advance simplified matters, since there's a lot of scale-dependent stuff in those styles, and I just needed to worry about one scale. Note that the scale level is in the editor's interface:

dcafdg_3-1686275742497.png

The other style editor tip I didn't discover soon enough was clicking on a feature on the map takes you right to the style class for it. This saves a lot of wading through Esri's lists of style classes. They're well organized, but still there's a lot to sift through, or at least there was for World Navigation.

NOTE: The download map tool will allow you to download a lot of data, but there is some restriction. As your map extent gets bigger, the larger scale intervals will not be available in the interface to select. My guess is this is designed to protect you from asking for too much data. In the case of my room-sized map, I was able to download it all in one go, so it's not that restrictive. On my first attempt, I was using a larger map scale, and it was not available in the drop-down list for scales in the download map tool. See my comment on this post if you need to download a larger map extent than what the GUI will let you. You have to do it in batches (tiles... of tiles) and the extents cannot have any gaps or overlaps. I put in a code sample to move the map extent to create a non-overlapping and seamless extents for downloading VTPKs.

I know this seems like a lot, but it wasn't that complicated in the end for the scale of what I'm trying to do.

0 Kudos
MarcoBoeringa
MVP Regular Contributor

After you posted your problem and question, I just couldn't resist doing an attempt to output a 1073x970 cm wall size map of the United States straight from ArcGIS Pro based on the 1:500k spec and the OpenStreetMap global database created with my ArcGIS Renderer for OpenStreetMap that I created.

The screenshot of the resulting giant 1GB(!) size output PDF at its full extent included below, took about 12 -18 hours to export if I remember well, including hillshading. I already had the result on the 13th of May, but since you showed no further interest and it was likely you were making further attempts to concoct your own version, I did not show it before.

The additional screenshots again show the actual 1:500k detail of the data, as viewed at 100% display scale in Adobe Reader, and as generalized and prepared by my Renderer from the original OpenStreetMap data.

The giant PDF can only be opened on a recent 64bit version of Adobe Reader, and it takes about 10-15 minutes to open on my admittedly powerful but slightly dated HP Z840 hardware. After that, browsing it at 100% is sluggish, but definitely not so bad to be unusable.

It was fun and exhilarating to see this gargantuan PDF open up with the tremendous amount of detail.

Again, its best to download these images due to the poor preview option of the ESRI Community website, not allowing you to view them at their original 100% scale.

I agree about your observations regarding the "Multiply" blend mode, it is the best of all available. I also recognize the issue you posted about vector layers of roads showing hillshade detail through.

This issue does not occur in my Renderer, due to the layers for roads being rendered and stacked on top of the hillshade with the blend mode, with the landuse layers below the hillshade in the TOC, that will show the blend and hillshading effect (but this is of course the desired thing). Due to the layers being stacked on top, the shields and road rendering isn't affected by the blending and hillshading (although in this particular "Topo" style I actually don't render shield, but only road ref labels parallel to the roads, I do have another style with a simple form of shields).

0 Kudos