Compression for Vector Tile Package VTPK

[NathanMellor]

VTPKs tend to be unforgiving if you try to modify them

There is a tool - Extract package - that will extract the tiles and other files.  

There is no way I see of reversing the process. Simply zipping it back up (uncompressed) will not do, because it doesn't want the tiles in exploded format

I found this link for packing files in your compactV2.  

GitHub - Esri/raster-tiles-compactcache: Compact Cache V2 is used by ArcGIS to store raster tiles. T... 

But is it meant to work on Vector tiles?  

The root.json of a vtpk will say:

"resourceInfo":{"styleVersion":8,"tileCompression":"gzip","cacheInfo":{"storageInfo":{"packetSize":128,"storageFormat":"compactV2"}}}}

It mentions "compactV2" which pointed me to this code. 

But some other things cause me to doubt. 

1. The set of files becomes 50 times bigger. Not very compact. 

2. ArcGIS Pro and ArcGISOnline cannot find any of the tiles in the new vtpk. 

I may have good reasons to use scripts to repackage tiles in vtpks. The vtpks often take days to weeks to generate and I might want to avoid redoing them just to get smaller areas. 

[DanPatterson_Retired]

the GitHub says ... raster... the only other thing that isn't deprecated is https://github.com/Esri/arcgis-vectortile-style-editor

[NathanMellor]

Yes. It is apparent that although it shares the same name, compactV2, the algorithm for vector tiles compression must be different somehow. 

BTW, I did look through the sample code in Vundler.py. I do not see a single reason why it would fail based on whether the file in question is a PNG or PBF. It only cares about the size of the file and writes it completely intact. 

Do they plan to provide guidance for compressing vector tiles and just haven't gotten around to it, or is being kept in the dark a matter of deliberate policy, because they don't want us to repackage tiles?

And does this whitepaper apply even if the sample code?

raster-tiles-compactcache/CompactCacheV2.md at master · Esri/raster-tiles-compactcache · GitHub 

I would like ESRI to comment if possible. 

This influences whether we use ArcGIS for some projects. 

[MarcoBoeringa]

I don't work for ESRI, so just my 2 cents worth, but a couple of things to note:

- Vector tile support, and vector tiles themselves, are still kind of in their infant stage. They haven't been around that long compared to other data formats.

- As a consequence, there a still some clear limitations, e.g. in the symbology that you can use.

- As far as I understood, yes, vector tile packages are not really "designed" or thought of to be "re-packagable" in the way you desire, like a random zip/unzip operation, with modification in between. I think ESRI, and many other companies like the original designer of the format Mapbox, that are supporting vector tiles, mainly think of them as a handy "create-once / style many" format. That is, you don't change the contents, just re-style by slapping on another JSON style file to get "another" map.

- I really think your current use-case is marginal (both in terms of current users even remotely attempting this, and what chances there are of ESRI supporting this in the near future).

- Then again, the vector tiles and support are likely still very much in active development, you never know what the future holds.

By the way, what is it exactly that you try to do? What modifications are you making to the unpackaged content?

I see you mentioning the need for "smaller areas", but is that it? And if so, why do you need it? I also think Vector Tiles are not designed as a distribution format if that is what you attempt to do, for that, create a Map Package (.mpkx - Map package—ArcGIS Pro | ArcGIS Desktop ) instead. Vector Tiles are designed to be deployed as a vector based webservice on e.g. ArcGIS Online, where your users generally should not care about "extents" of the data, as they simply zoom in / out to the area of interest.

If you do attempt to use it as a distribution format to send off to some other office / department, then remember the vector tiles do not contain the full quality of your data, nor content, and are cut in tiles, all of this may hamper subsequent usage by your "clients". Again, use a Map Package to distribute true original data including styling and all.

[NathanMellor]

Agreed on many points.

I forgot to mention that the context for this is using the ArcGIS Runtime libraries in an offline scenario.

In that context, the VTPK is very much a distribution format and is claimed as such in official documentation. 

In fact, it is the only format allowed by the runtime. 

Vector Tiles served from loose files on disk, in MBTiles format, or from any server that is not arcgis.com, are not allowed. 

MMPK and TPK formats do have tools that allow you to create a package that is a subset of the full package (like World Topographic Maps for instance). 

The VTPK does not, although that could change with some new runtime features. 

I'm okay with being the one to push the envelope. 

After all, the posted code says it welcomes contributions, and I may have some if I can figure this out. 

As a matter of fact, I found that this code may be misbehaving due to roundoff error, possibly due to the way later versions of python interpret types. With some modifications, I can get it to create compressed packages of vector tiles. 

However, there are still more hurdles to get it all to work correctly. 

In any case, it would be good to know if the trouble I am having is deliberate in order to stop people from using tiles from MapBox or somewhere else, or if it is simply that the tools haven't evolved enough yet.