topic Re: From a FeatureSet to FeatureLayer for Overlay analysis in ArcGIS API for Python Questions

From a FeatureSet to FeatureLayer for Overlay analysis

GretaVega — Sun, 12 Dec 2021 16:55:06 GMT

Hello,

I am trying to use the analysis tool overlay_layers. Because my dataset is so large I am querying it so I have two FeatureSets. This tool requires FeatureLayers so I am using the information from here to have feature collections but when I run the overlay tool I get an error.

Many thanks to any feedback

Here is my code:

#checking I have FeatureCollections 
type(fc1)
arcgis.features.feature.FeatureCollection

type(fc2)
arcgis.features.feature.FeatureCollection

#Running the analysis
overlay_rf = overlay_layers(input_layer=fc1, overlay_layer=fc2, overlay_type='Intersect')

And here is the error I get:

---------------------------------------------------------------------------JSONDecodeError                           Traceback (most recent call last)<ipython-input-30-4a4a1668cec5> in <module>----> 1 overlay_rf_subfacet = overlay_layers(input_layer=wdpa_fc, overlay_layer=facet_fc, overlay_type='Intersect')~/anaconda3/lib/python3.7/site-packages/arcgis/features/manage_data.py in overlay_layers(input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, gis, estimate)    191         output_name,    192         context,--> 193         estimate=estimate)    194     195 def create_route_layers( ~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in overlay_layers(self, input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, estimate)   1529    1530 -> 1531         task_url, job_info, job_id = super()._analysis_job(task, params)   1532    1533         job_info = super()._analysis_job_status(task_url, job_info)~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in _analysis_job(self, task, params)    143         params["f"] = "json"    144 --> 145         resp = self._con.post(submit_url, params, token=self._token)    146         #print(resp)    147         return task_url, resp, resp['jobId']~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/connection.py in post(self, path, postdata, files, ssl, compress, is_retry, use_ordered_dict, add_token, verify_cert, token, try_json, out_folder, file_name, force_bytes, add_headers)   1141             resp_json = json.loads(resp_data, object_pairs_hook=OrderedDict)   1142         else:-> 1143             resp_json = json.loads(resp_data)   1144    1145  ~/anaconda3/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)    346             parse_int is None and parse_float is None and    347             parse_constant is None and object_pairs_hook is None and not kw):--> 348         return _default_decoder.decode(s)    349     if cls is None:    350         cls = JSONDecoder ~/anaconda3/lib/python3.7/json/decoder.py in decode(self, s, _w)    335     336         """--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())    338         end = _w(s, end).end()    339         if end != len(s):~/anaconda3/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)    353             obj, end = self.scan_once(s, idx)    354         except StopIteration as err:--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None    356         return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Re: From a FeatureSet to FeatureLayer for Overlay analysis

EarlMedina — Wed, 12 Jun 2019 22:09:21 GMT

Hi Greta,

From the little bit you shared the code looks correct. I tested on version 1.6.1 and I've got this working with Feature Collections or individual layers. Can you tell us more about the data types (point, polyline, polygon)? From visual inspection do you expect this to return a result?

The error seems to indicate that the response is invalid (or it is valid but it's not being decoded properly). If you add the below lines to the script we can get a better sense of what's going out and what's coming in:

import logging
logging.basicConfig(level=logging.DEBUG)

-Earl

Re: From a FeatureSet to FeatureLayer for Overlay analysis

GretaVega — Sun, 12 Dec 2021 16:55:09 GMT

Hi Earl,

Thank you so much for your quick reply. Apologies for not having given more information.

Can you tell us more about the data types (point, polyline, polygon)?

Both fc1 and fc2 are polygons.

From visual inspection do you expect this to return a result?

Yes. fc1 has many features of many different shapes (red) and fc2 is a grid that overlaps with them (blue).

Thanks for the lines of code. I have rerun the analyses and now this is the message I get:

IOPub data rate exceeded.

The notebook server will temporarily stop sending output

to the client in order to avoid crashing it.

To change this limit, set the config variable

`--NotebookApp.iopub_data_rate_limit`.




Current values:

NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)

NotebookApp.rate_limit_window=3.0 (secs)




DEBUG:arcgis._impl.connection:RESPONSE: https://analysis9.arcgis.com/arcgis/rest/services/tasks/GPServer/OverlayLayers/submitJob, <html>

<head>

<title></title>

<link href="https://community.esri.com/arcgis/rest/static/esri-rest.css"

rel="stylesheet" type="text/css">

<!-- Favorite Bookmark Icon -->

<link rel="shortcut icon" type="image/ico"

href="http://www.esri.com/favicon.ico" />

<link rel="icon" type="image/ico" href="http://www.esri.com/favicon.ico" />

<!-- End Favorite Bookmark Icon -->

</head>

<body>

<table id="userTable">

<tbody>

<tr>

<td id="titlecell">ArcGIS Server Administrator Directory</td>

<td align="right">

<a

href="https://community.esri.com/arcgis/rest/login">Login</a>

 | <a

href="https://www.arcgis.com/sharing/generateToken">Generate Token</a>

</td>

</tr>

</tbody>

</table>

<table id="navTable">

<tbody>

<tr valign="top">

<td id="breadcrumbs">

<a href="https://community.esri.com/arcgis/rest/">Home</a>

</td>

<!--

<td align="right" id="help">

<a href="#" target="_blank">Help</a> | <a href="#" target="_blank">API Reference</a>

</td>

-->

</tr>

</tbody>

</table>

<script type="text/javascript">

function trim(value) {

if (value) {

return value.replace(/^\s+|\s+$/g, "");

}

return "";

}

function validate() {

var value = document.getElementById("username");

if (value && trim(value.value).length < 1) {

alert("Username cannot be an empty string.");

value.focus();

return false;

}

value = document.getElementById("password");

if (value && trim(value.value).length < 1) {

alert("Password cannot be an empty string.");

value.value = "";

value.focus();

return false;

}

return true;

}

</script>

<div class="restBody">

<br />

You do not have permissions to use this resource.

<form name="frmLogin" onsubmit="return validate();"

action="/arcgis/rest/login"

method="post">

<table class="formTable">

<tbody>

<tr>

<td colspan="2"><b>Login</b></td>

</tr>

<tr>

<td>User Name</td>

<td><input id="username" name="username"

value="" /></td>

</tr>

<tr>

<td>Password</td>

<td><input id="password" name="password" type="password" /></td>

</tr>

<tr>

<td colspan="2"><input type="submit" value='Login' /></td>

</tr>

</tbody>

</table>

<input type="hidden" name="redirect" value="/arcgis/rest/services/tasks/GPServer/OverlayLayers/submitJob" />

</form>

</div>

</body>

</html>

---------------------------------------------------------------------------

JSONDecodeError                           Traceback (most recent call last)

<ipython-input-22-4a4a1668cec5> in <module>

----> 1 overlay_rf_subfacet = overlay_layers(input_layer=wdpa_fc, overlay_layer=facet_fc, overlay_type='Intersect')




~/anaconda3/lib/python3.7/site-packages/arcgis/features/manage_data.py in overlay_layers(input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, gis, estimate)

    191         output_name,

    192         context,

--> 193         estimate=estimate)

    194 

    195 def create_route_layers(




~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in overlay_layers(self, input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, estimate)

   1529 

   1530 

-> 1531         task_url, job_info, job_id = super()._analysis_job(task, params)

   1532 

   1533         job_info = super()._analysis_job_status(task_url, job_info)




~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in _analysis_job(self, task, params)

    143         params["f"] = "json"

    144 

--> 145         resp = self._con.post(submit_url, params, token=self._token)

    146         #print(resp)

    147         return task_url, resp, resp['jobId']




~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/connection.py in post(self, path, postdata, files, ssl, compress, is_retry, use_ordered_dict, add_token, verify_cert, token, try_json, out_folder, file_name, force_bytes, add_headers)

   1141             resp_json = json.loads(resp_data, object_pairs_hook=OrderedDict)

   1142         else:

-> 1143             resp_json = json.loads(resp_data)

   1144 

   1145 




~/anaconda3/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)

    346             parse_int is None and parse_float is None and

    347             parse_constant is None and object_pairs_hook is None and not kw):

--> 348         return _default_decoder.decode(s)

    349     if cls is None:

    350         cls = JSONDecoder




~/anaconda3/lib/python3.7/json/decoder.py in decode(self, s, _w)

    335 

    336         """

--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())

    338         end = _w(s, end).end()

    339         if end != len(s):




~/anaconda3/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)

    353             obj, end = self.scan_once(s, idx)

    354         except StopIteration as err:

--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None

    356         return obj, end




JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I have increased 10 times the limit of

--NotebookApp.iopub_data_rate_limit

But I keep getting the same error (with the updated limit number).
I am going to keep trying by increasing again the limit.

Thanks again.

Greta

Re: From a FeatureSet to FeatureLayer for Overlay analysis

GretaVega — Sun, 12 Dec 2021 16:55:12 GMT

UPDATE

Hello again,

By pushing the limit 100 times the message has changed. Now I get a huge message that I don't think it is worth to paste here, my Jupyter notebook kind of crashes when I scroll down the message.

I have thought that maybe fc1 has overlapping polygons and I have tried to use dissolve_boundaries but I got again a huge message and my Jupyter notebook froze when scrolling down.

I have managed to open the jupyter notebook with a text editor and at the end of the message there is this:

dissolveFields': [], 'summaryFields': [], 'f': 'json', 'token': 'VPyLIISPGHqc1TWSwiqNP6OKjsbVdDGepgxKtTaCkjWN1hUaL6DjkeU8wuIZ2DyBgQMw_AuoJYPyi7kG4uypL2wMbLN1RdLqe_DlOKQnDbkjekgBrXyI8ThqeMvCQy3BUvjNc1MabDU6MM3hjggbkMsuOhjWJyrm2I6KyxluwSQ.'}\n",

      "DEBUG:arcgis._impl.connection:RESPONSE: https://analysis9.arcgis.com/arcgis/rest/services/tasks/GPServer/DissolveBoundaries/submitJob, <html>\n",

      "<head>\n",

      "<title></title>\n",

      "<link href=\"/arcgis/rest/static/esri-rest.css\"\n",

      "rel=\"stylesheet\" type=\"text/css\">\n",

      "<!-- Favorite Bookmark Icon -->\n",

      "<link rel=\"shortcut icon\" type=\"image/ico\"\n",

      "href=\"http://www.esri.com/favicon.ico\" />\n",

      "<link rel=\"icon\" type=\"image/ico\" href=\"http://www.esri.com/favicon.ico\" />\n",

      "<!-- End Favorite Bookmark Icon -->\n",

      "</head>\n",

      "<body>\n",

      "<table id=\"userTable\">\n",

      "<tbody>\n",

      "<tr>\n",

      "<td id=\"titlecell\">ArcGIS Server Administrator Directory</td>\n",

      "<td align=\"right\">\n",

      "<a\n",

      "href=\"/arcgis/rest/login\">Login</a>\n",

      " | <a\n",

      "href=\"https://www.arcgis.com/sharing/generateToken\">Generate Token</a>\n",

      "</td>\n",

      "</tr>\n",

      "</tbody>\n",

      "</table>\n",

      "<table id=\"navTable\">\n",

      "<tbody>\n",

      "<tr valign=\"top\">\n",

      "<td id=\"breadcrumbs\">\n",

      "<a href=\"/arcgis/rest/\">Home</a>\n",

      "</td>\n",

      "<!--\n",

      "<td align=\"right\" id=\"help\">\n",

      "<a href=\"#\" target=\"_blank\">Help</a> | <a href=\"#\" target=\"_blank\">API Reference</a>\n",

      "</td>\n",

      "-->\n",

      "</tr>\n",

      "</tbody>\n",

      "</table>\n",

      "<script type=\"text/javascript\">\n",

      "function trim(value) {\n",

      "if (value) {\n",

      "return value.replace(/^\\s+|\\s+$/g, \"\");\n",

      "}\n",

      "return \"\";\n",

      "}\n",

      "function validate() {\n",

      "var value = document.getElementById(\"username\");\n",

      "if (value && trim(value.value).length < 1) {\n",

      "alert(\"Username cannot be an empty string.\");\n",

      "value.focus();\n",

      "return false;\n",

      "}\n",

      "value = document.getElementById(\"password\");\n",

      "if (value && trim(value.value).length < 1) {\n",

      "alert(\"Password cannot be an empty string.\");\n",

      "value.value = \"\";\n",

      "value.focus();\n",

      "return false;\n",

      "}\n",

      "return true;\n",

      "}\n",

      "</script>\n",

      "<div class=\"restBody\">\n",

      "<br />\n",

      "You do not have permissions to use this resource.\n",

      "<form name=\"frmLogin\" onsubmit=\"return validate();\"\n",

      "action=\"/arcgis/rest/login\"\n",

      "method=\"post\">\n",

      "<table class=\"formTable\">\n",

      "<tbody>\n",

      "<tr>\n",

      "<td colspan=\"2\"><b>Login</b></td>\n",

      "</tr>\n",

      "<tr>\n",

      "<td>User Name</td>\n",

      "<td><input id=\"username\" name=\"username\"\n",

      "value=\"\" /></td>\n",

      "</tr>\n",

      "<tr>\n",

      "<td>Password</td>\n",

      "<td><input id=\"password\" name=\"password\" type=\"password\" /></td>\n",

      "</tr>\n",

      "<tr>\n",

      "<td colspan=\"2\"><input type=\"submit\" value='Login' /></td>\n",

      "</tr>\n",

      "</tbody>\n",

      "</table>\n",

      "<input type=\"hidden\" name=\"redirect\" value=\"/arcgis/rest/services/tasks/GPServer/DissolveBoundaries/submitJob\" />\n",

      "</form>\n",

      "</div>\n",

      "</body>\n",

      "</html>\n"

I guess the key message is:

You do not have permissions to use this resource

So I am wondering, are there limits on the size of the feature collection.

So I have created a new fc3, which only has 3 features and does intercept with fc2 (see picture). I manage to dissolve fc3, but then the overlay returns an empty feature collection without an error message.

If I run the overlay analysis within arcgis.com I do get the expected result. The southernmost polygon is divided.

I am kind of starting to run out of ideas... Is there any limitation I should be aware of?

Thanks again for any feedback.

Cheers,

Greta

Re: From a FeatureSet to FeatureLayer for Overlay analysis

EarlMedina — Tue, 18 Jun 2019 14:09:14 GMT

Hi Greta,

Thank you for the additional information. The "IOPub data rate exceeded" error I've not encountered, but as far as I know that would only affect visualization of the output. What happens when you don't visualize the result immediately - does it this still fail?

A few more questions:

Can you tell us a bit more about the environment running the analysis? This looks to be a standalone anaconda installation on a Unix-like system.
Do you notice any issues with performance when the analysis is running? Is anything spiking (CPU/RAM usage)?
Approximately how many features are involved in the analysis? Is it feasible for you to share the source data for testing?

-Earl

Re: From a FeatureSet to FeatureLayer for Overlay analysis

GretaVega — Wed, 03 Jul 2019 15:56:46 GMT

Hi Earl,

I don't manage to visualize anything... Just errors...

I am using a mac, python 3.7 in JupyterLab. I haven't been able to identify any spikes.

Ideally I would like to run this workflow with global data. Unfortunately at the moment I am not able to share the data. The subset I am using (fc3) only has three features. In the web I use a global grid (fc2) and it works, but when I try using the API fc2 is a featureCollection of only one feature and the returned featureCollection is empty.

Thanks again for your input.

Greta

Re: From a FeatureSet to FeatureLayer for Overlay analysis

Nick_PascalJakuschona — Fri, 20 Sep 2019 08:39:09 GMT

Hi Greta,

if you transform the featureSet with

correctInput = arcgis.features.FeatureSet.to_dict('your_featureset').

You can use correctInput as Input variable for the overlay_layers function

Hope this helps

Nick

Re: From a FeatureSet to FeatureLayer for Overlay analysis

CDPHEPO — Wed, 24 Mar 2021 19:12:15 GMT

Hello all!

Has there been a resolution for this topic? I am getting the same error

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Input and overlay inputs created like this:

from arcgis.features import FeatureCollection
my_fc = FeatureCollection(my_featureset.to_dict())

Re: From a FeatureSet to FeatureLayer for Overlay analysis

SadanandacharB1 — Tue, 22 Mar 2022 02:27:25 GMT

how to set NotebookApp.iopub_data_rate_limit in Arcgis pro

Re: From a FeatureSet to FeatureLayer for Overlay analysis

MariadelCarmenPerez-Serrano — Tue, 15 Nov 2022 10:27:39 GMT

Did you find how to change it in Arcgis Pro??