Protocolbuffer Binary Format (PBF) Documentation or Examples?

[TomNeer]

ArcGIS Enterprise added support for the PBF output format in 10.7.1. We are having difficulty finding any documentation or examples on Esri's implementation of PBF in either the Developer's documentation or Esri's Github outside of Vector Tiles. We would like to look at using this format with some larger datasets. Any pointers would be appreciated? 

[mgeorge]

Bit of a rabbit hole - never used R before. But it looks like the sf library uses GDAL, and the MVT driver for GDAL appears to flip the y on decoding: 

gdal/ogrmvtdataset.cpp at master · OSGeo/gdal (github.com)

Are you seeing any difference between how mbox and esri vtls are handled, and that's causing this question? 

[mgeorge]

Whoops didn't see your response @JINANYI. These forums get a little borked when reply chains get long. I posted above what I think the reason is. 

[JINANYI]

@mgeorge 

Starting at line 817 (https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/mvt/ogrmvtdataset.cpp#L817), this piece of code could indeed be the problem, but I don't have enough knowledge in C++ to make a judgement

/************************************************************************/
/*                              GetXY()                                 */
/************************************************************************/

void OGRMVTLayer::GetXY(int nX, int nY, double& dfX, double& dfY)
{
    if( m_poDS->m_bGeoreferenced )
    {
        dfX = m_poDS->m_dfTopX + nX * m_poDS->m_dfTileDimX / m_nExtent;
        dfY = m_poDS->m_dfTopY - nY * m_poDS->m_dfTileDimY / m_nExtent;
    }
    else
    {
        dfX = nX;
        dfY = static_cast<double>(m_nExtent) - nY;
    }
}

Looking at this, the "dfY = static_cast<double>(m_nExtent) - nY" seems to be the flipping part causing this trouble. On the other hand, the "dfY = m_poDS->m_dfTopY - nY * m_poDS->m_dfTileDimY / m_nExtent" part seems to be the right way to do it (based on my interpretation on how vector tile works), so why does it not run this part of code for this dataset? What causes this "if( m_poDS->m_bGeoreferenced )" condition to be false?

Maybe the driver thinks this dataset  has no georeference?

Ah, such a rabbit hole. Please feel free to tell me if you think it is too much.

[mgeorge]

Sorry @JINANYI, that's probably as far as I can dive into this ... I think georeferencing in this context is probably something specific to GDAL -- e.g., maybe you can add an image and map vector tiles onto it. 

[mgeorge]

Also I don't believe spatial reference and what not gets encoded into VT payloads to begin with. That's in contrast to Feature Tiles where the transformation does actually get encoded into the FetaureSet pbf payload. 

arcgis-pbf/FeatureCollection.proto at main · Esri/arcgis-pbf (github.com)

[JINANYI]

@mgeorge  Thank you very much!

[mgeorge]

Tom Neer‌ I know there were some plans to create a repository detailing the Esri feature tile pbf format (it's a bit different from vt-pbf), but I don't think it's been released yet. I'll check on the status and get back to you.

[TomNeer]

Thank you for the responses.

We were directed by Esri Staff at the UC to look at the PBF format for some of the large Feature Services we were working with. However the documentation on Output formats—ArcGIS REST API: Services Directory | ArcGIS for Developers only states "f=pbf: The response is in the Protocol Buffers format. The PBF format is binary and is thus smaller and faster for clients to parse than JSON. It is used by newer versions of some Esri APIs and applications to improve efficiency. The PBF format is supported by some feature service operations in ArcGIS Enterprise 10.7 as well as hosted feature layers in ArcGIS Online."

This infers that PBF can be used by some feature service operations in ArcGIS Enterprise 10.7 and APIs but I have been unable to find any additional information or examples to its use outside of vector tiles.

We are actually currently exploring integrating a Quantum Ledger Database into our Web GIS to fulfill our needs. This resolves several major issues with the lack of modern datatypes (INT8/SERIAL and JSONB, specifically), ability to store various geometry type, and provide an immutable transaction log. 

[mgeorge]

Hi Tom, PBF is just a serialization format like JSON. The VectorTile pbf spec (Mapbox) is different from Esri Feature PBF spec in terms of how the payloads of vector/feature data are structured, though they both are PBF encoded specs. To request that a query use the PBF format, you just need to include f=pbf instead of f=json. This will return a binary payload. Currently only the /query endpoint I think has support for PBF, and this requires quantization.

To see some example usage, you can look at the queries being made by the 4.x JSAPI which will use PBF if the server supports it. For instance, take a look at this sample: ArcGIS API for JavaScript Sandbox 

Example query:

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/... 

We can find out that the service supports f=pbf by looking at the metadata for the associated service:

https://services.arcgis.com/V6ZHFr6zdgNZuVG0/arcgis/rest/services/NYC_Footprints_fgdb/FeatureServer/...

{ ... "supportedQueryFormats": "JSON, geoJSON, PBF", ... }

Currently it's mostly just internal clients using this for now, but we have plans to document the spec itself.

[DarrenBreen]

Heya,

Sorry to hijack this question. 

Is there any update on the spec for the pbf? Any chance we could have access to the proto file? Be keen to try to add it to koopjs featureserver.

Best regards,

Darren