topic Re: Protocolbuffer Binary Format (PBF) Documentation or Examples? in ArcGIS JavaScript Maps SDK Questions

Protocolbuffer Binary Format (PBF) Documentation or Examples?

TomNeer — Wed, 17 Jul 2019 16:10:50 GMT

ArcGIS Enterprise added support for the PBF output format in 10.7.1. We are having difficulty finding any documentation or examples on Esri's implementation of PBF in either the Developer's documentation or Esri's Github outside of Vector Tiles. We would like to look at using this format with some larger datasets. Any pointers would be appreciated?

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

RogerAustin — Tue, 24 Sep 2019 05:52:15 GMT

I don't know what PBF-formatted data ESRI support other than vector tiles and fontstacks. We implemented PBF tile generation by following the MapBox spec (https://docs.mapbox.com/vector-tiles/specification/ and https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto) and I believe (I didn't implement it myself) taking inspiration from https://github.com/mapbox/vt-pbf. The actual implementation was done in C++ using Google's Protocol Buffers library (https://developers.google.com/protocol-buffers/docs/reference/cpp)

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Tue, 24 Sep 2019 19:09:58 GMT

Tom Neer‌ I know there were some plans to create a repository detailing the Esri feature tile pbf format (it's a bit different from vt-pbf), but I don't think it's been released yet. I'll check on the status and get back to you.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

TomNeer — Tue, 24 Sep 2019 20:08:18 GMT

Thank you for the responses.

We were directed by Esri Staff at the UC to look at the PBF format for some of the large Feature Services we were working with. However the documentation on Output formats—ArcGIS REST API: Services Directory | ArcGIS for Developers only states "f=pbf: The response is in the Protocol Buffers format. The PBF format is binary and is thus smaller and faster for clients to parse than JSON. It is used by newer versions of some Esri APIs and applications to improve efficiency. The PBF format is supported by some feature service operations in ArcGIS Enterprise 10.7 as well as hosted feature layers in ArcGIS Online."

This infers that PBF can be used by some feature service operations in ArcGIS Enterprise 10.7 and APIs but I have been unable to find any additional information or examples to its use outside of vector tiles.

We are actually currently exploring integrating a Quantum Ledger Database into our Web GIS to fulfill our needs. This resolves several major issues with the lack of modern datatypes (INT8/SERIAL and JSONB, specifically), ability to store various geometry type, and provide an immutable transaction log.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Tue, 24 Sep 2019 22:14:54 GMT

Hi Tom, PBF is just a serialization format like JSON. The VectorTile pbf spec (Mapbox) is different from Esri Feature PBF spec in terms of how the payloads of vector/feature data are structured, though they both are PBF encoded specs. To request that a query use the PBF format, you just need to include f=pbf instead of f=json. This will return a binary payload. Currently only the /query endpoint I think has support for PBF, and this requires quantization.

To see some example usage, you can look at the queries being made by the 4.x JSAPI which will use PBF if the server supports it. For instance, take a look at this sample: ArcGIS API for JavaScript Sandbox

Example query:

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/0/query?f=pbf&geomet…

We can find out that the service supports f=pbf by looking at the metadata for the associated service:

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/0?f=json

{ ... "supportedQueryFormats": "JSON, geoJSON, PBF", ... }

Currently it's mostly just internal clients using this for now, but we have plans to document the spec itself.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

DarrenBreen — Tue, 22 Oct 2019 09:06:22 GMT

Heya,

Sorry to hijack this question.

Is there any update on the spec for the pbf? Any chance we could have access to the proto file? Be keen to try to add it to koopjs featureserver.

Best regards,

Darren

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Tue, 22 Oct 2019 16:50:56 GMT

Sorry Darren Breen‌, we are still planning to document the spec, but we want to at least have a ready built parser for folks to use that we can share. We need to do a little bit of refactoring to pull this out of the API.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

DarellStoick — Thu, 18 Jun 2020 21:44:04 GMT

Any update on this? Our organization is also looking for the pbf specs for ESRI's version.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

TomNeer — Mon, 29 Jun 2020 15:08:47 GMT

Haven't heard anything.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Mon, 29 Jun 2020 19:42:52 GMT

Hi all,

Sorry for the lack of information on this, things are a bit hectic with UC. I'll follow-up internally.

Matt

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

TomNeer — Tue, 29 Sep 2020 22:57:38 GMT

It is my understanding that Esri has adopted the Mapbox Vector Tile Specification. Not sure which version, assuming 2.0 or earlier). Hopefully, Esri can confirm. PBF specification is at Specification | Vector tiles | Mapbox

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Tue, 29 Sep 2020 23:13:55 GMT

Hi Tom,

For Vector Tiles, we do indeed use the Mapbox Vector Tile spec. The f=pbf format of FeatureServices however is an esri Feature Tile serialization spec. Feature Tiles are more for dynamic visualizations, whereas Vector Tiles are better for things like basemaps.

The documentation on this has been held up, but I think more details should be forthcoming. I'll check in again.

We've also worked to improve the performance of pbf encoded Feature Tiles in our upcoming 4.17 release. Whereas before we were doing some quite a bit of conversion to deserialize features, we've reworked things so that we can work directly against the compressed binary data.

Matt

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

KirkDybvik — Mon, 12 Oct 2020 17:34:56 GMT

I too would be interested in understanding the format/spec of the pbf of FeatureServices. Any examples of deserializing the features returned as pbf would be great. We are downloading data from the National Interagency Fire Center, and downloading a large payload in geojson is brutally slow. The pbf downloads super fast, but I have no good way to read it programmatically.

Kirk

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

SolidData — Sat, 13 Feb 2021 15:53:44 GMT

Just a friendly bump, checking for an update on the FeatureService info. Would really like to replace json with pbf for feature download etc. Meaning, pbf to feature set or something similar. Any updates on this front?

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

Cees — Wed, 10 Mar 2021 07:49:23 GMT

Hello, is there already a description how we can deserialize the pbf format for feature services?

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Fri, 26 Mar 2021 18:30:49 GMT

Hi all,

The FeatureCollection PBF spec can now be found here: https://github.com/Esri/arcgis-pbf

Matt

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

RowanWinsemius2 — Tue, 18 May 2021 04:44:02 GMT

I've added some info on how to parse the geometry bits over here. I've just reversed engineered things as best I could but it seems to be working, and hopefully in due course I'll publish a little npm module for doing it.

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

nicogis — Mon, 07 Jun 2021 08:01:58 GMT

In c# you can deserialize: https://community.esri.com/t5/arcgis-enterprise-extensibility/soi-returning-pbf/td-p/1049024

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

JINANYI — Tue, 18 Jan 2022 09:20:56 GMT

Is there full documentation on how to de-serialize a vector tile pbf file yet? I hope to take a ESRI vector tile pbf and convert it into geojson. I tried to use the Mapbox spec (https://github.com/mapbox/vector-tile-spec/blob/master/2.1/vector_tile.proto) but it only transformed the data into another format that I do not know how to handle.

@mgeorge Do you have any update?

Re: Protocolbuffer Binary Format (PBF) Documentation or Examples?

mgeorge — Tue, 18 Jan 2022 17:42:56 GMT

Hi @JINANYI, we release the FeatureTile Pbf spec here: https://github.com/Esri/arcgis-pbf. The VTL encoding we use is the same as mapbox/vector-tile-spec: Mapbox Vector Tile specification (github.com)

Basically what you will want to do is pull in some pbf parsing library which will then parse the format verbatim. After that, take a look at vector-tile-spec/README.md at master · mapbox/vector-tile-spec (github.com) to see what the properties of the tile mean. I.e., the geometry encoding for mapbox uses command integers that you need to decode what operation is happening.