ArcGIS Enterprise added support for the PBF output format in 10.7.1. We are having difficulty finding any documentation or examples on Esri's implementation of PBF in either the Developer's documentation or Esri's Github outside of Vector Tiles. We would like to look at using this format with some larger datasets. Any pointers would be appreciated?
I read the references you provided and also this article: FAQ: What is a root tile and how are they used to make a vector tile package with a local coordinate system? https://support.esri.com/en/technical-article/000022396
Using all the info, I tried to decode the a vector tile myself, using World Topographic Map (https://basemaps.arcgis.com/arcgis/rest/services/World_Basemap_v2/VectorTileServer). I downloaded this particular tile in pbf, https://basemaps.arcgis.com/arcgis/rest/services/World_Basemap_v2/VectorTileServer/tile/6/24/17, and start decoding in R language (sorry it's the only language I know).
However, as it turned out, for some unknown reason, the origin of the selected tile seems to be the bottom left corner (x axis positive to the right, y axis positive upward), instead of top left corner. It is shown in the following codes:
```{r} library(sf) librart(ggplot2) #the st_read() function automatically uses `MVT' driver, and convert the geometry data into tile coordinates test <- st_read("17.pbf", layer = "City small scale") #function that convert tile coordinates into the real mapping coordinates (EPSG: 3857) shift_tile_coord <- function(sf, level, row, column, crs = 3857) { if(crs != 3857) stop("currently only supports EPSG:3857 Web Mercator projection") origin_x <- -20037508.342787 origin_y <- 20037508.342787 maxscale <- 2.958287637957775E8 tile_size <- 512 #pixels of a row/ column dpi <- 96 inch2mapUnit <- 39.3700787 #1 meter = 39.3700787 inch tile_extent <- 4096 root_tile_width <- maxscale * (tile_size/(dpi * inch2mapUnit)) selected_tile_width <- root_tile_width/2^level sf_shifted <- sf #x = tile_x_coor*selected_tile_width/tile_extent + origin_x + selected_tile_width*column #y = tile_y_coor*selected_tile_width/tile_extent + origin_y - selected_tile_width*(row + 1) sf_shifted$geometry <- sf$geometry * matrix(c(selected_tile_width/tile_extent, selected_tile_width/tile_extent), ncol = 2) sf_shifted$geometry <- sf_shifted$geometry + matrix(c(origin_x + selected_tile_width * column, origin_y - selected_tile_width * (row + 1)), ncol = 2) sf_shifted <- st_set_crs(sf_shifted, crs) return(sf_shifted) } #visulaize the data test_shifted <- shift_tile_coord(test, level = 6, row = 24, column = 17) ggplot(data = test_shifted) + geom_sf_label(aes(label = X_name_en)) + geom_sf() + coord_sf(datum = st_crs(3857)) ```
Test Data
If you use map viewer to check (https://www.arcgis.com/apps/mapviewer/index.html?layers=7dc6cea0b1764a1f9af2e679f642f0f5#), the coordinates (in Web Mercator, EPSG: 3857) are correct. However, the conversion of a y coordinate = tile_y_coor*selected_tile_width/tile_extent + origin_y - selected_tile_width*(row + 1). This means a y coordinate first starts at the bottom of the tile (thus (row + 1)), and moves upward (positive sign in tile_y_coor*selected_tile_width/tile_extent). This seems to be against the documentation you provided. Is it designed to be this way, or is there something wrong?
@mgeorge
Hi @JINANYI, we release the FeatureTile Pbf spec here: https://github.com/Esri/arcgis-pbf. The VTL encoding we use is the same as mapbox/vector-tile-spec: Mapbox Vector Tile specification (github.com)
Basically what you will want to do is pull in some pbf parsing library which will then parse the format verbatim. After that, take a look at vector-tile-spec/README.md at master · mapbox/vector-tile-spec (github.com) to see what the properties of the tile mean. I.e., the geometry encoding for mapbox uses command integers that you need to decode what operation is happening.
Is there full documentation on how to de-serialize a vector tile pbf file yet? I hope to take a ESRI vector tile pbf and convert it into geojson. I tried to use the Mapbox spec (https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto) but it only transformed the data into another format that I do not know how to handle.
@mgeorge Do you have any update?
In c# you can deserialize: https://community.esri.com/t5/arcgis-enterprise-extensibility/soi-returning-pbf/td-p/1049024
I've added some info on how to parse the geometry bits over here. I've just reversed engineered things as best I could but it seems to be working, and hopefully in due course I'll publish a little npm module for doing it.
Hi all,
The FeatureCollection PBF spec can now be found here: https://github.com/Esri/arcgis-pbf
Matt
Hello, is there already a description how we can deserialize the pbf format for feature services?
Just a friendly bump, checking for an update on the FeatureService info. Would really like to replace json with pbf for feature download etc. Meaning, pbf to feature set or something similar. Any updates on this front?
I too would be interested in understanding the format/spec of the pbf of FeatureServices. Any examples of deserializing the features returned as pbf would be great. We are downloading data from the National Interagency Fire Center, and downloading a large payload in geojson is brutally slow. The pbf downloads super fast, but I have no good way to read it programmatically.
Kirk
Hi Tom,
For Vector Tiles, we do indeed use the Mapbox Vector Tile spec. The f=pbf format of FeatureServices however is an esri Feature Tile serialization spec. Feature Tiles are more for dynamic visualizations, whereas Vector Tiles are better for things like basemaps.
The documentation on this has been held up, but I think more details should be forthcoming. I'll check in again.
We've also worked to improve the performance of pbf encoded Feature Tiles in our upcoming 4.17 release. Whereas before we were doing some quite a bit of conversion to deserialize features, we've reworked things so that we can work directly against the compressed binary data.
It is my understanding that Esri has adopted the Mapbox Vector Tile Specification. Not sure which version, assuming 2.0 or earlier). Hopefully, Esri can confirm. PBF specification is at Specification | Vector tiles | Mapbox
Sorry for the lack of information on this, things are a bit hectic with UC. I'll follow-up internally.
Haven't heard anything.
Any update on this? Our organization is also looking for the pbf specs for ESRI's version.
Sorry Darren Breen, we are still planning to document the spec, but we want to at least have a ready built parser for folks to use that we can share. We need to do a little bit of refactoring to pull this out of the API.
Heya,
Sorry to hijack this question.
Is there any update on the spec for the pbf? Any chance we could have access to the proto file? Be keen to try to add it to koopjs featureserver.
Best regards,
Darren
Hi Tom, PBF is just a serialization format like JSON. The VectorTile pbf spec (Mapbox) is different from Esri Feature PBF spec in terms of how the payloads of vector/feature data are structured, though they both are PBF encoded specs. To request that a query use the PBF format, you just need to include f=pbf instead of f=json. This will return a binary payload. Currently only the /query endpoint I think has support for PBF, and this requires quantization.
To see some example usage, you can look at the queries being made by the 4.x JSAPI which will use PBF if the server supports it. For instance, take a look at this sample: ArcGIS API for JavaScript Sandbox
Example query:
https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/0/query?f=pbf&geomet…
We can find out that the service supports f=pbf by looking at the metadata for the associated service:
https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/0?f=json
{ ... "supportedQueryFormats": "JSON, geoJSON, PBF", ... }
Currently it's mostly just internal clients using this for now, but we have plans to document the spec itself.
Thank you for the responses.
We were directed by Esri Staff at the UC to look at the PBF format for some of the large Feature Services we were working with. However the documentation on Output formats—ArcGIS REST API: Services Directory | ArcGIS for Developers only states "f=pbf: The response is in the Protocol Buffers format. The PBF format is binary and is thus smaller and faster for clients to parse than JSON. It is used by newer versions of some Esri APIs and applications to improve efficiency. The PBF format is supported by some feature service operations in ArcGIS Enterprise 10.7 as well as hosted feature layers in ArcGIS Online."
This infers that PBF can be used by some feature service operations in ArcGIS Enterprise 10.7 and APIs but I have been unable to find any additional information or examples to its use outside of vector tiles.
We are actually currently exploring integrating a Quantum Ledger Database into our Web GIS to fulfill our needs. This resolves several major issues with the lack of modern datatypes (INT8/SERIAL and JSONB, specifically), ability to store various geometry type, and provide an immutable transaction log.
Tom Neer I know there were some plans to create a repository detailing the Esri feature tile pbf format (it's a bit different from vt-pbf), but I don't think it's been released yet. I'll check on the status and get back to you.
I don't know what PBF-formatted data ESRI support other than vector tiles and fontstacks. We implemented PBF tile generation by following the MapBox spec (https://docs.mapbox.com/vector-tiles/specification/ and https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto) and I believe (I didn't implement it myself) taking inspiration from https://github.com/mapbox/vt-pbf. The actual implementation was done in C++ using Google's Protocol Buffers library (https://developers.google.com/protocol-buffers/docs/reference/cpp)
Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.