From a FeatureSet to FeatureLayer for Overlay analysis
3686
9
06-12-2019 09:05 AM
by
Hello,
I am trying to use the analysis tool overlay_layers. Because my dataset is so large I am querying it so I have two FeatureSets. This tool requires FeatureLayers so I am using the information from here to have feature collections but when I run the overlay tool I get an error.
Many thanks to any feedback 
Here is my code:
#checking I have FeatureCollections type(fc1) arcgis.features.feature.FeatureCollection type(fc2) arcgis.features.feature.FeatureCollection #Running the analysis overlay_rf = overlay_layers(input_layer=fc1, overlay_layer=fc2, overlay_type='Intersect')
And here is the error I get:
---------------------------------------------------------------------------JSONDecodeError Traceback (most recent call last)<ipython-input-30-4a4a1668cec5> in <module>----> 1 overlay_rf_subfacet = overlay_layers(input_layer=wdpa_fc, overlay_layer=facet_fc, overlay_type='Intersect')~/anaconda3/lib/python3.7/site-packages/arcgis/features/manage_data.py in overlay_layers(input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, gis, estimate) 191 output_name, 192 context,--> 193 estimate=estimate) 194 195 def create_route_layers( ~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in overlay_layers(self, input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, estimate) 1529 1530 -> 1531 task_url, job_info, job_id = super()._analysis_job(task, params) 1532 1533 job_info = super()._analysis_job_status(task_url, job_info)~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in _analysis_job(self, task, params) 143 params["f"] = "json" 144 --> 145 resp = self._con.post(submit_url, params, token=self._token) 146 #print(resp) 147 return task_url, resp, resp['jobId']~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/connection.py in post(self, path, postdata, files, ssl, compress, is_retry, use_ordered_dict, add_token, verify_cert, token, try_json, out_folder, file_name, force_bytes, add_headers) 1141 resp_json = json.loads(resp_data, object_pairs_hook=OrderedDict) 1142 else:-> 1143 resp_json = json.loads(resp_data) 1144 1145 ~/anaconda3/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 346 parse_int is None and parse_float is None and 347 parse_constant is None and object_pairs_hook is None and not kw):--> 348 return _default_decoder.decode(s) 349 if cls is None: 350 cls = JSONDecoder ~/anaconda3/lib/python3.7/json/decoder.py in decode(self, s, _w) 335 336 """--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s):~/anaconda3/lib/python3.7/json/decoder.py in raw_decode(self, s, idx) 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err:--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0)
by
Hi Earl,
Thank you so much for your quick reply. Apologies for not having given more information.
Can you tell us more about the data types (point, polyline, polygon)?
Both fc1 and fc2 are polygons.
From visual inspection do you expect this to return a result?
Yes. fc1 has many features of many different shapes (red) and fc2 is a grid that overlaps with them (blue).
Thanks for the lines of code. I have rerun the analyses and now this is the message I get:
IOPub data rate exceeded. The notebook server will temporarily stop sending output to the client in order to avoid crashing it. To change this limit, set the config variable `--NotebookApp.iopub_data_rate_limit`. Current values: NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec) NotebookApp.rate_limit_window=3.0 (secs) DEBUG:arcgis._impl.connection:RESPONSE: https://analysis9.arcgis.com/arcgis/rest/services/tasks/GPServer/OverlayLayers/submitJob, <html> <head> <title></title> <link href="https://community.esri.com/arcgis/rest/static/esri-rest.css" rel="stylesheet" type="text/css"> <!-- Favorite Bookmark Icon --> <link rel="shortcut icon" type="image/ico" href="http://www.esri.com/favicon.ico" /> <link rel="icon" type="image/ico" href="http://www.esri.com/favicon.ico" /> <!-- End Favorite Bookmark Icon --> </head> <body> <table id="userTable"> <tbody> <tr> <td id="titlecell">ArcGIS Server Administrator Directory</td> <td align="right"> <a href="https://community.esri.com/arcgis/rest/login">Login</a> | <a href="https://www.arcgis.com/sharing/generateToken">Generate Token</a> </td> </tr> </tbody> </table> <table id="navTable"> <tbody> <tr valign="top"> <td id="breadcrumbs"> <a href="https://community.esri.com/arcgis/rest/">Home</a> </td> <!-- <td align="right" id="help"> <a href="#" target="_blank">Help</a> | <a href="#" target="_blank">API Reference</a> </td> --> </tr> </tbody> </table> <script type="text/javascript"> function trim(value) { if (value) { return value.replace(/^\s+|\s+$/g, ""); } return ""; } function validate() { var value = document.getElementById("username"); if (value && trim(value.value).length < 1) { alert("Username cannot be an empty string."); value.focus(); return false; } value = document.getElementById("password"); if (value && trim(value.value).length < 1) { alert("Password cannot be an empty string."); value.value = ""; value.focus(); return false; } return true; } </script> <div class="restBody"> <br /> You do not have permissions to use this resource. <form name="frmLogin" onsubmit="return validate();" action="/arcgis/rest/login" method="post"> <table class="formTable"> <tbody> <tr> <td colspan="2"><b>Login</b></td> </tr> <tr> <td>User Name</td> <td><input id="username" name="username" value="" /></td> </tr> <tr> <td>Password</td> <td><input id="password" name="password" type="password" /></td> </tr> <tr> <td colspan="2"><input type="submit" value='Login' /></td> </tr> </tbody> </table> <input type="hidden" name="redirect" value="/arcgis/rest/services/tasks/GPServer/OverlayLayers/submitJob" /> </form> </div> </body> </html> --------------------------------------------------------------------------- JSONDecodeError Traceback (most recent call last) <ipython-input-22-4a4a1668cec5> in <module> ----> 1 overlay_rf_subfacet = overlay_layers(input_layer=wdpa_fc, overlay_layer=facet_fc, overlay_type='Intersect') ~/anaconda3/lib/python3.7/site-packages/arcgis/features/manage_data.py in overlay_layers(input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, gis, estimate) 191 output_name, 192 context, --> 193 estimate=estimate) 194 195 def create_route_layers( ~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in overlay_layers(self, input_layer, overlay_layer, overlay_type, snap_to_input, output_type, tolerance, output_name, context, estimate) 1529 1530 -> 1531 task_url, job_info, job_id = super()._analysis_job(task, params) 1532 1533 job_info = super()._analysis_job_status(task_url, job_info) ~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/tools.py in _analysis_job(self, task, params) 143 params["f"] = "json" 144 --> 145 resp = self._con.post(submit_url, params, token=self._token) 146 #print(resp) 147 return task_url, resp, resp['jobId'] ~/anaconda3/lib/python3.7/site-packages/arcgis/_impl/connection.py in post(self, path, postdata, files, ssl, compress, is_retry, use_ordered_dict, add_token, verify_cert, token, try_json, out_folder, file_name, force_bytes, add_headers) 1141 resp_json = json.loads(resp_data, object_pairs_hook=OrderedDict) 1142 else: -> 1143 resp_json = json.loads(resp_data) 1144 1145 ~/anaconda3/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 346 parse_int is None and parse_float is None and 347 parse_constant is None and object_pairs_hook is None and not kw): --> 348 return _default_decoder.decode(s) 349 if cls is None: 350 cls = JSONDecoder ~/anaconda3/lib/python3.7/json/decoder.py in decode(self, s, _w) 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s): ~/anaconda3/lib/python3.7/json/decoder.py in raw_decode(self, s, idx) 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I have increased 10 times the limit of
--NotebookApp.iopub_data_rate_limit
But I keep getting the same error (with the updated limit number).
I am going to keep trying by increasing again the limit.
Thanks again.
Greta
by
UPDATE
Hello again,
By pushing the limit 100 times the message has changed. Now I get a huge message that I don't think it is worth to paste here, my Jupyter notebook kind of crashes when I scroll down the message.
I have thought that maybe fc1 has overlapping polygons and I have tried to use dissolve_boundaries but I got again a huge message and my Jupyter notebook froze when scrolling down.
I have managed to open the jupyter notebook with a text editor and at the end of the message there is this:
dissolveFields': [], 'summaryFields': [], 'f': 'json', 'token': 'VPyLIISPGHqc1TWSwiqNP6OKjsbVdDGepgxKtTaCkjWN1hUaL6DjkeU8wuIZ2DyBgQMw_AuoJYPyi7kG4uypL2wMbLN1RdLqe_DlOKQnDbkjekgBrXyI8ThqeMvCQy3BUvjNc1MabDU6MM3hjggbkMsuOhjWJyrm2I6KyxluwSQ.'}\n", "DEBUG:arcgis._impl.connection:RESPONSE: https://analysis9.arcgis.com/arcgis/rest/services/tasks/GPServer/DissolveBoundaries/submitJob, <html>\n", "<head>\n", "<title></title>\n", "<link href=\"/arcgis/rest/static/esri-rest.css\"\n", "rel=\"stylesheet\" type=\"text/css\">\n", "<!-- Favorite Bookmark Icon -->\n", "<link rel=\"shortcut icon\" type=\"image/ico\"\n", "href=\"http://www.esri.com/favicon.ico\" />\n", "<link rel=\"icon\" type=\"image/ico\" href=\"http://www.esri.com/favicon.ico\" />\n", "<!-- End Favorite Bookmark Icon -->\n", "</head>\n", "<body>\n", "<table id=\"userTable\">\n", "<tbody>\n", "<tr>\n", "<td id=\"titlecell\">ArcGIS Server Administrator Directory</td>\n", "<td align=\"right\">\n", "<a\n", "href=\"/arcgis/rest/login\">Login</a>\n", " | <a\n", "href=\"https://www.arcgis.com/sharing/generateToken\">Generate Token</a>\n", "</td>\n", "</tr>\n", "</tbody>\n", "</table>\n", "<table id=\"navTable\">\n", "<tbody>\n", "<tr valign=\"top\">\n", "<td id=\"breadcrumbs\">\n", "<a href=\"/arcgis/rest/\">Home</a>\n", "</td>\n", "<!--\n", "<td align=\"right\" id=\"help\">\n", "<a href=\"#\" target=\"_blank\">Help</a> | <a href=\"#\" target=\"_blank\">API Reference</a>\n", "</td>\n", "-->\n", "</tr>\n", "</tbody>\n", "</table>\n", "<script type=\"text/javascript\">\n", "function trim(value) {\n", "if (value) {\n", "return value.replace(/^\\s+|\\s+$/g, \"\");\n", "}\n", "return \"\";\n", "}\n", "function validate() {\n", "var value = document.getElementById(\"username\");\n", "if (value && trim(value.value).length < 1) {\n", "alert(\"Username cannot be an empty string.\");\n", "value.focus();\n", "return false;\n", "}\n", "value = document.getElementById(\"password\");\n", "if (value && trim(value.value).length < 1) {\n", "alert(\"Password cannot be an empty string.\");\n", "value.value = \"\";\n", "value.focus();\n", "return false;\n", "}\n", "return true;\n", "}\n", "</script>\n", "<div class=\"restBody\">\n", "<br />\n", "You do not have permissions to use this resource.\n", "<form name=\"frmLogin\" onsubmit=\"return validate();\"\n", "action=\"/arcgis/rest/login\"\n", "method=\"post\">\n", "<table class=\"formTable\">\n", "<tbody>\n", "<tr>\n", "<td colspan=\"2\"><b>Login</b></td>\n", "</tr>\n", "<tr>\n", "<td>User Name</td>\n", "<td><input id=\"username\" name=\"username\"\n", "value=\"\" /></td>\n", "</tr>\n", "<tr>\n", "<td>Password</td>\n", "<td><input id=\"password\" name=\"password\" type=\"password\" /></td>\n", "</tr>\n", "<tr>\n", "<td colspan=\"2\"><input type=\"submit\" value='Login' /></td>\n", "</tr>\n", "</tbody>\n", "</table>\n", "<input type=\"hidden\" name=\"redirect\" value=\"/arcgis/rest/services/tasks/GPServer/DissolveBoundaries/submitJob\" />\n", "</form>\n", "</div>\n", "</body>\n", "</html>\n"
I guess the key message is:
You do not have permissions to use this resource
So I am wondering, are there limits on the size of the feature collection.
So I have created a new fc3, which only has 3 features and does intercept with fc2 (see picture). I manage to dissolve fc3, but then the overlay returns an empty feature collection without an error message.
If I run the overlay analysis within arcgis.com I do get the expected result. The southernmost polygon is divided.
I am kind of starting to run out of ideas... Is there any limitation I should be aware of?
Thanks again for any feedback.
Cheers,
Greta
by
by
by
by
by
