topic Re: arcpy.FeatureSet() From Feature Service in Python Questions

arcpy.FeatureSet() From Feature Service

JamesCrandall — Sun, 12 Dec 2021 03:59:25 GMT

I'm not having luck with the GlobalID and ParentGlobalID columns when attempting to populate Feature Classes and Tables in a FGDB from a hosted feature service. It's all fairly straight forward and seems to work just fine, except for when specifying these two fields.

Issues:

1. The GlobalId field does not seem to throw any error, however the feature classes get populated with the value: {00000000-0000-0000-0000-000000000000}

2. There's also related feature classes that contain a ParentGlobalID column as well, these throw an error when attempting to load the feature set "RuntimeError: RecordSetObject: Cannot open table for Load". If I remove the "ParentGlobalID" column from my list of fields to download, then the process works fine without error (minus the ParentGlobalID column of course).

This is the important section:

fields = """objectid,globalid,SurveyStartTime,SurveyEndTime,CreationDate,Creator,EditDate,Editor"""
taburl =  <URLtoHostedFeatureService>/query/0
where = '1=1'
query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, '*', token)
queryfsURL = taburl + query
#try to convert the hosted service to a fgdb table
fs = arcpy.FeatureSet()
fs.load(queryfsURL)
fs.save(r'<pathToDirectory>\downloadTab.gdb\HostedFsNameHere)
#also attempted this
#arcpy.CopyRows_management(fs, r'<pathToDirectory>\downloadTab.gdb\HostedFsNameHere)‍‍‍‍‍‍‍‍‍‍‍

Re: arcpy.FeatureSet() From Feature Service

JoshuaBixby — Fri, 16 Feb 2018 22:18:34 GMT

What version of server and client are you running? I just tested your general steps using a feature service from ArcGIS Enterprise 10.5.1 and ArcGIS Pro 2.1 and the GlobalIDs are intact.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Sat, 17 Feb 2018 13:21:20 GMT

ArcGIS Server 10.4 (maybe 10.4.1)

ArcGIS Desktop 10.3.1

Good point -- I do have a 10.4 client I use to publish GP services but I didn't run this particular script on that box. I only ran (with my issues) on the 10.3 install.

I'll check that out!

Thank you.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Mon, 19 Feb 2018 14:36:41 GMT

Thanks again,

I ran the script from a client with ArcGIS Desktop10.4 installed but has the same issue with GlobalID -- it creates the feature class without error including all of the fields specified, but the globalId column is populated with {00000000-0000-0000-0000-000000000000}

Also: this is a hosted feature service on AGOL that I'm working with.

From the Feature Service end point:

globalid (type: esriFieldTypeGlobalID, alias: GlobalID, SQL Type: sqlTypeGUID, length: 38, nullable: false, editable: false)

Re: arcpy.FeatureSet() From Feature Service

JoshuaBixby — Mon, 19 Feb 2018 15:06:41 GMT

I only have access to an ArcMap 10.6 machine today, but I tested the same feature service I used to test with Pro 2.1, and the GlobalID field carries through fine. So, it is either a hosted-feature service issue or client version issue. I will test an older client and AGOL hosted feature service sometime soon.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Mon, 19 Feb 2018 15:11:03 GMT

Much appreciated.

Re: arcpy.FeatureSet() From Feature Service

RandyBurton — Tue, 20 Feb 2018 04:15:54 GMT

I was experimenting with your script and was able to get a feature from AGOL with the global ID intact. Since this is a URL request, I was wondering if the "where" clause might be part of the problem. Normally the 1=1 would be encoded as 1%3D1 (%3D would be the equals sign). Edit-- I did another try using where=1=1 (without escaping the equals sign) and was able to get the global IDs.

Have you examined the returned json before inserting it into your database to verify the global IDs?

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Sun, 12 Dec 2021 03:59:27 GMT

Good call, Randy. I did as you suggested and set a search cursor on the FeatureSet() after .load() to see what the GlobalID values are at that point (before converting to fgdb Feature Classes / tables).

You're test validates that they are {00000000-0000-0000-0000-000000000000}

When queried on the REST endpoint (WHERE 1=1, Fields = *) I see the globalid and parentglobalid values as expected.

with arcpy.da.SearchCursor(fs, ['objectid','globalid']) as fscur:
   for fsrow in fscur:
       print fsrow[1]

##result:
globalid
{00000000-0000-0000-0000-000000000000}
{00000000-0000-0000-0000-000000000000}
{00000000-0000-0000-0000-000000000000}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Tue, 20 Feb 2018 14:46:12 GMT

Have you examined the returned json before inserting it into your database to verify the global IDs?

Response on the request from Dev Tools Response tab is:

"features" : [
    {
      "attributes" : {
        "objectid" : 2,
        "globalid" : "9667a224-a732-49f3-b3c2-3eafb17a482d",

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Tue, 20 Feb 2018 15:27:45 GMT

I have published a new feature service from a different feature class source (containing globalid's) and re-running the script produces the same {00000000-0000-0000-0000-000000000000} globald values.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Sun, 12 Dec 2021 03:59:30 GMT

As a separate test to possibly isolate if the issues is at the fs.load(url), I populated a json object and inspected the "features" attribute. This actually shows the correct GlobalID values when printed in the python console.

fields = """objectid,globalid,SurveyStartTime,SurveyEndTime,CreationDate,Creator,EditDate,Editor"""
taburl =  <URLtoHostedFeatureService>/query/0
where = '1=1'
query = "?where={}&outFields={}&returnGeometry=true&f=json&token={}".format(where, '*', token)
queryfsURL = taburl + query

#this shows the globalid values when gfeats is printed in the python console
greq = urllib2.Request(queryfsURL)
gresp = urllib2.urlopen(greq)
gdata = json.load(gresp)
gfeats = gdata['features']
print gfeats‍‍‍‍‍‍‍‍‍‍‍‍

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Tue, 20 Feb 2018 16:52:01 GMT

Another test/confirmation: GlobalID values come across fine after FeatureSet.load() when script is run on a feature service published to ArcGIS Server.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Tue, 20 Feb 2018 19:54:39 GMT

Another test attempted: compare the toolbox "JSONToFeatures" tool with python scripted "arcpy.JSONToFeatures_conversion"

Using the exact same .json file in both instances, the toolbox tool's output contains the expected GlobalID values, the arcpy method does not (again, populated with {00000000-0000-0000-0000-000000000000}).

.json file source:

{"features": [{"geometry": {"y": 3302628.348359901, "x": -9080214.856802708}, "attributes": {"GlobalID": "3d7fd4db-8fd4-4150-a6fa-7c8d9f490028", "OBJECTID": 1}}, {"geometry": {"y": 2935905.840858899, "x": -8949481.954276735}, "attributes": {"GlobalID": "17b18396-7d70-4f7f-987b-dd3c869e3d13", "OBJECTID": 2}}, {"geometry": {"y": 3182365.217025425, "x": -8958169.568138303}, "attributes": {"GlobalID": "fd287aa8-c666-4ebb-973e-af42c9210555", "OBJECTID": 3}}, {"geometry": {"y": 3116365.997722158, "x": -8944001.624695944}, "attributes": {"GlobalID": "0ef2bc1a-7ab3-4eac-936d-53c59c6ad720", "OBJECTID": 4}}, {"geometry": {"y": 3266689.3323510713, "x": -9056080.682907999}, "attributes": {"GlobalID": "c17153b0-72c6-4a71-8c9c-d08fbca4f971", "OBJECTID": 5}}, {"geometry": {"y": 3032950.2018160364, "x": -9103563.008196728}, "attributes": {"GlobalID": "7822675c-67b9-43c2-9c6d-dd3686b7684c", "OBJECTID": 6}}, {"geometry": {"y": 2987973.085353658, "x": -8925101.388748247}, "attributes": {"GlobalID": "82eb8efe-0344-424b-aed3-40f521ca0096", "OBJECTID": 7}}, {"geometry": {"y": 3149190.0434092716, "x": -8943513.028244277}, "attributes": {"GlobalID": "0ca97eef-6b5b-4f8a-8de8-43e34b8e7352", "OBJECTID": 8}}, {"geometry": {"y": 2969635.1807482317, "x": -8980605.152716484}, "attributes": {"GlobalID": "78134e5e-1e33-46f7-943c-101453e01652", "OBJECTID": 9}}, {"geometry": {"y": 3300730.1157377195, "x": -9069763.544277236}, "attributes": {"GlobalID": "6c982a39-94f5-4249-9672-db7f04e46b1b", "OBJECTID": 10}}, {"geometry": {"y": 2916452.7305216496, "x": -8963985.064438965}, "attributes": {"GlobalID": "30680b80-3ec7-492f-85e6-95d2260e7331", "OBJECTID": 11}}, {"geometry": {"y": 3295387.3267123713, "x": -9088780.378074935}, "attributes": {"GlobalID": "d8309d5a-3844-4ba9-adb9-d31793ab26f2", "OBJECTID": 12}}, {"geometry": {"y": 3083860.600086069, "x": -9123405.091868907}, "attributes": {"GlobalID": "c04be547-1c72-47f8-894f-a9c6eef4ed32", "OBJECTID": 13}}, {"geometry": {"y": 3224239.209942657, "x": -9052199.086908078}, "attributes": {"GlobalID": "5bfc2427-4189-417e-bdad-8ff71f9d76d7", "OBJECTID": 14}}, {"geometry": {"y": 2956551.1227857363, "x": -8990878.972877821}, "attributes": {"GlobalID": "9e640c17-debc-457a-b695-a2962d9cd354", "OBJECTID": 15}}, {"geometry": {"y": 2925328.7789350976, "x": -8967704.78599132}, "attributes": {"GlobalID": "7d8c6a90-1cc3-4502-a0fd-6588c0ccca47", "OBJECTID": 16}}, {"geometry": {"y": 2999472.875453083, "x": -8921906.202857886}, "attributes": {"GlobalID": "4ce0f7f6-b5fd-4387-ade9-89cbc0a9f654", "OBJECTID": 17}}, {"geometry": {"y": 3009337.3731106035, "x": -8931162.126738088}, "attributes": {"GlobalID": "6723b6c6-9094-4316-8a9c-23a229a09ea8", "OBJECTID": 18}}, {"geometry": {"y": 3040321.612889241, "x": -9097337.725443132}, "attributes": {"GlobalID": "026ec2ad-4be0-4234-a2dd-f9070fec7270", "OBJECTID": 19}}, {"geometry": {"y": 2957907.2298183567, "x": -8939240.551882569}, "attributes": {"GlobalID": "a1718bc2-f54f-4a94-a06b-d880c861498b", "OBJECTID": 20}}, {"geometry": {"y": 3164780.4135744474, "x": -9044995.89939325}, "attributes": {"GlobalID": "a981bb05-ced4-48c4-9867-c16f21bf6f9b", "OBJECTID": 21}}, {"geometry": {"y": 3007557.9593651486, "x": -9094193.111578912}, "attributes": {"GlobalID": "c8a0bf73-9f4b-4217-821f-3c22ee35ea57", "OBJECTID": 22}}, {"geometry": {"y": 3040321.612889241, "x": -9097337.725443132}, "attributes": {"GlobalID": "881b6655-92ce-4902-b650-8ab80cf5cc0c", "OBJECTID": 23}}, {"geometry": {"y": 3055864.9450819413, "x": -9065465.833404854}, "attributes": {"GlobalID": "d34c1eb3-f244-42ae-b6a2-79fb5e35eeba", "OBJECTID": 24}}, {"geometry": {"y": 3224292.2406486287, "x": -9038813.640524628}, "attributes": {"GlobalID": "41f9a259-ef37-4d0c-a621-d50955f308c7", "OBJECTID": 25}}, {"geometry": {"y": 2927263.8650856293, "x": -8953185.156013427}, "attributes": {"GlobalID": "8a860774-c19c-4a5a-a925-c52e341d91ef", "OBJECTID": 26}}, {"geometry": {"y": 3062601.9988861633, "x": -9097594.207727797}, "attributes": {"GlobalID": "536a63d6-1f2d-4607-bcbc-ae8de0796a82", "OBJECTID": 27}}, {"geometry": {"y": 3070923.263292275, "x": -9132372.515240341}, "attributes": {"GlobalID": "df893039-0ec9-4ddc-a658-d6cda417f0ab", "OBJECTID": 28}}, {"geometry": {"y": 3041218.6997780814, "x": -9103491.233161572}, "attributes": {"GlobalID": "64aa8f72-d006-4cd0-995b-701ff6022e70", "OBJECTID": 29}}, {"geometry": {"y": 2935177.955242901, "x": -8968202.019990554}, "attributes": {"GlobalID": "5dd0a695-7da8-4901-b552-976d6e18e5b0", "OBJECTID": 30}}, {"geometry": {"y": 2917410.3796297098, "x": -8966413.374766212}, "attributes": {"GlobalID": "a83bb34d-a24d-435b-b275-fd7a5514c6a6", "OBJECTID": 31}}, {"geometry": {"y": 3119756.0254174476, "x": -8916007.179496944}, "attributes": {"GlobalID": "8fe4916a-acb9-45d2-86ac-78be46956794", "OBJECTID": 32}}, {"geometry": {"y": 3280828.470373303, "x": -9053413.521350337}, "attributes": {"GlobalID": "92f09d6b-513c-42e0-925c-044f43549f11", "OBJECTID": 33}}, {"geometry": {"y": 3062601.9988861633, "x": -9097594.207727797}, "attributes": {"GlobalID": "4bcbace2-3ec2-4133-9e2b-f3f52f9e70e8", "OBJECTID": 34}}, {"geometry": {"y": 3068641.422484618, "x": -9090565.776465254}, "attributes": {"GlobalID": "e75833c9-46d1-4be4-ad0d-b83e42810d80", "OBJECTID": 35}}, {"geometry": {"y": 3156160.1965427827, "x": -8945633.634291844}, "attributes": {"GlobalID": "029a444a-ce2c-4079-9638-9c6b26548106", "OBJECTID": 36}}, {"geometry": {"y": 2975152.3453906407, "x": -8938679.301698191}, "attributes": {"GlobalID": "4e9f8c83-12a4-4348-8cdc-b07e0ddf3465", "OBJECTID": 37}}, {"geometry": {"y": 3068261.150886741, "x": -9114097.527042406}, "attributes": {"GlobalID": "a56a9893-a6eb-4a16-a0ed-3d1b46a3c115", "OBJECTID": 38}}, {"geometry": {"y": 3047310.0172173204, "x": -8919962.125129992}, "attributes": {"GlobalID": "f44736fd-7cbb-4a61-b18e-e6ace08f2802", "OBJECTID": 39}}, {"geometry": {"y": 3030133.243087964, "x": -8916829.71315664}, "attributes": {"GlobalID": "df925e89-29c0-42e1-9f5d-53a04d1e6d6e", "OBJECTID": 40}}, {"geometry": {"y": 3038496.552766487, "x": -8918397.465804476}, "attributes": {"GlobalID": "58dc4684-2657-468d-af0e-85277746984d", "OBJECTID": 41}}], "fields": [{"domain": null, "name": "OBJECTID", "defaultValue": null, "alias": "OBJECTID", "sqlType": "sqlTypeOther", "type": "esriFieldTypeOID"}, {"domain": null, "name": "GlobalID", "defaultValue": null, "alias": "GlobalID", "length": 38, "sqlType": "sqlTypeOther", "type": "esriFieldTypeGlobalID"}], "spatialReference": {"wkid": 102100, "latestWkid": 3857}, "geometryType": "esriGeometryPoint", "objectIdFieldName": "OBJECTID", "globalIdFieldName": "GlobalID"}‍

Re: arcpy.FeatureSet() From Feature Service

RandyBurton — Tue, 20 Feb 2018 22:10:14 GMT

For my previous test I was working in an IDE outside ArcMap (version 10.5). I will do some experimenting with your last code sample later today.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Tue, 20 Feb 2018 22:13:49 GMT

fyi: I attempted in ArcGIS 10.3.1 and 10.4 client with pyScripter, IDLW and PythonWIn in combos of 32 & 64 bit all with the same result.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Wed, 21 Feb 2018 16:42:09 GMT

Hi Randy,

Could you test my hosted feature service?

Please follow me back so that I can send you python code in a message!

Re: arcpy.FeatureSet() From Feature Service

RandyBurton — Wed, 21 Feb 2018 17:49:57 GMT

Following. I did some additional testing last night and still couldn't duplicate the issue with the global ID. I was going to suggest a test of your hosted feature service, and I would be willing to try that later today.

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Wed, 21 Feb 2018 18:13:09 GMT

Sending a note now! Really do appreciate the assist.

Re: arcpy.FeatureSet() From Feature Service

RandyBurton — Sun, 12 Dec 2021 03:59:33 GMT

I was able to run the script you sent without issue. I retrieved 41 global IDs:

ObjectID: 1 GlobalID: {3D7FD4DB-8FD4-4150-A6FA-7C8D9F490028}
....
ObjectID: 41 GlobalID: {58DC4684-2657-468D-AF0E-85277746984D}
‍‍‍

I also pasted the URL into a web browser and was able to examine the json. All global IDs were intact.

Since you are using a URL to access the feature, my only suggestion would be to use something like urllib.urlencode to make sure any characters in the URL string that need special encoding are processed. The characters include = < >. But in these tests, it didn't seem to create a problem.

I did wonder about the field names and case sensitivity, but again this did not seem to be an issue.

Again, my tests were done using 10.5 and Python's 32 bit IDE 2.7.12

Re: arcpy.FeatureSet() From Feature Service

JamesCrandall — Wed, 21 Feb 2018 20:44:24 GMT

Thank you!