topic Re: Extracting data from a public web layer in ArcGIS REST APIs and Services Questions

Extracting data from a public web layer

StefanStefanov — Fri, 08 Feb 2019 18:04:05 GMT

Hi,

Not sure if it's the right place to ask this question but here it is. There is an open-access GIS that hosts some data I'm interested in:

https://gis.wwf.bg/mobilz/en/

For some reason English and Bulgarian version have different layer structure but anyways.

I'm fairly new to ArcGIS (and GIS in general) and it's probably an overkill for my needs at this point so I'd like to start with something simple - extract data from "Protected areas in Bulgaria" and "Protected zones Natura 2000" sublayers and have each sublayer in a separate file - preferably KMZ or GPX. I'm aware some metadata will be lost but I don't care about it at this point. Not sure how much will be lost when going to KMZ but for GPX polygons will have to be converted to tracks which is fine for me (as long as I can identify the original object name from the track name). Accuracy is of little importance as it's good enough to know where object boundaries roughly lie on the map. Also, I'm trying to automate this extraction in a Linux shell script.

Couple of things I tried:

1) Retrieving (questionably) all objects within a layer via the rest API:

wget -O - "https://gis.trimbul.com/arcgis/rest/services/trees/zashtiteniteritorii/MapServer/find?f=json&searchText=резерват&layers=0"

However, two issues with this approach:

searchText is mandatory. While I expect certain words to be present in every layer it's not a foolproof solution.
I don't understand the result that I get. First object for example:

{"results":[{"layerId":0,"layerName":"Резервати","displayFieldName":"Name_ENG","foundFieldName":"Type_Name_","value":"Резерват Вълчи дол","attributes":{"FID":"0","Shape":"Polygon","Name_ENG":"Valchi Dol","SITE_CODE":"176839","DESIG_ABBR":"BG01","area":"767","Name_BG":"Вълчи дол","FID_1":"0","Type_Name_":"Резерват Вълчи дол","Purpose_BG":"Целта е да опази единствената в България колония на белоглави лешояди, както и местообитания на други редки и застрашени видове птици.","NAME_BG_1":" ","LINK":" ","NAME_BG_12":" ","LINK_1":" "},"geometryType":"esriGeometryPolygon","geometry":{"rings":[[[2848916.5844476218,5099207.6745737931],[2848891.1791850836,5099275.1015510419],[2848841.0894904374,5099379.3485733839],[2848795.297296484,5099446.2420776887]...

Comma-separated values must be the GPS coordinates of the polygon points but they are in a coordinate system that means nothing to me. If I get those in WGS-84 I might be able to create a GPX file from the output and live with the limitation of this method. Fetching the data via REST API is my number one target as it's easy to process it on CLI.

2) Importing the web layer to ArcGIS Online and exporting the data in KMZ. Doesn't happen due to lack of granted privileges on the server side I guess.

3) Trying the Rest API Python package created by Caleb Mackey‌ but hit a few brickwalls. Python is not in my good skills list unfortunately. I'm on Ubuntu 18.04 which has limited support for Python 2 as far as I'm aware so this might be a problem too.

https://community.esri.com/people/Caleb1987/blog/2015/05/08/restapi-python-package

4) Not tested yet - fetching data in a desktop product (ArcMap would be the correct one I guess) and trying to extract it there. Still waiting for my license. This might be useful if I ever get skilled enough to consolidate all my data in an ArcGIS software and make it public it via a web GIS one day.

Anywhere, my theory is - if a browser can fetch and visualize some data there must be a way to extract it and make use of it.

Any help appreciated.

Re: Extracting data from a public web layer

Anonymous User — Sun, 12 Dec 2021 02:08:20 GMT

You can accomplish this easily with restapi. I did just push up a few changes, so may want to do a `git pull` first.

To get a kmz, I just did the following:

url = 'https://gis.trimbul.com/arcgis/rest/services/trees/zashtiteniteritorii/MapServer/0'
lyr = restapi.MapServiceLayer(url)
where = u"Type_Name_ LIKE '%резерват%'"
out_kmz = r'D:\Users\calebma\Desktop\tester.kmz' # change this to your output path for kmz
fs = lyr.query(where=where, f='kmz', kmz=out_kmz)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Extracting data from a public web layer

JakeSkinner — Fri, 08 Feb 2019 18:54:15 GMT

Hi Stefan,

If you are using ArcGIS Desktop, you can use the following tool to download the data as a feature class:

Download ArcGIS Online Feature Service or ArcGIS Server Feature/Map Service

Ex:

Re: Extracting data from a public web layer

EarlMedina — Mon, 11 Feb 2019 04:38:47 GMT

I think you should be able to achieve the desired result using only wget:

wget "https://gis.trimbul.com/arcgis/rest/services/trees/zashtiteniteritorii/MapServer/0/query?where=Type_Name_+LIKE+'%резерват%'&f=kmz" -O test.kmz‍‍‍

Re: Extracting data from a public web layer

StefanStefanov — Mon, 11 Feb 2019 08:15:06 GMT

Holy cow, thanks a bunch guys! All your answers are useful to me cause now I got greedy and want this this stuff both dynamically loaded in a desktop GIS and as an offline version to use when no Internet connection is present !

Re: Extracting data from a public web layer

StefanStefanov — Mon, 11 Feb 2019 08:17:12 GMT

Caleb, am I missing something here, perhaps a library I have not imported?

>>> import os
>>> import restapi
open source: False
/usr/local/lib/python2.7/dist-packages/restapi/common_types.py:61: UserWarning: No Arcpy found, some limitations in functionality may apply.
warnings.warn('No Arcpy found, some limitations in functionality may apply.')
>>> url = 'https://gis.trimbul.com/arcgis/rest/services/trees/zashtiteniteritorii/MapServer/0'
>>> lyr = restapi.MapServiceLayer(url)
where = u"Type_Name_ LIKE '%резерват%'"
out_kmz = r'D:\Users\calebma\Desktop\tester.kmz' # change this to your output path for kmz
fs = lyr.query(where=where, f='kmz', kmz=out_kmz)>>> where = u"Type_Name_ LIKE '%резерват%'"
>>> out_kmz = r'D:\Users\calebma\Desktop\tester.kmz' # change this to your output path for kmz
>>> fs = lyr.query(where=where, f='kmz', kmz=out_kmz)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/restapi/common_types.py", line 1000, in query
r = self.request(query_url, params)
File "/usr/local/lib/python2.7/dist-packages/restapi/rest_utils.py", line 590, in request
return do_post(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/restapi/rest_utils.py", line 186, in do_post
_json = r.json()
File "/usr/local/lib/python2.7/dist-packages/restapi/requests/models.py", line 866, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Re: Extracting data from a public web layer

Anonymous User — Mon, 11 Feb 2019 14:36:26 GMT

You may need to do a `git pull` in the GitHub repo to pull the latest version of the code. You'll also want to change the output kmz path and name.

Re: Extracting data from a public web layer

StefanStefanov — Mon, 11 Feb 2019 16:42:46 GMT

Sorry, I was half asleep when I tried this Of course your code pulls the data correctly. It has the added benefit over the wget solution to fetch correctly the reserve names. To be fair most of the methods I tried couldn't export the names properly in a KMZ/KML.

It's a pity I can't "mark as correct" multiple answers as all your suggestions guys give the desired results. Thank you so much again!

Re: Extracting data from a public web layer

Forest_TeamTeam — Mon, 16 Nov 2020 16:18:22 GMT

Dear Stefan,

Instead loading our servers with your bulk requests and downloading data that you are not authorized for, easier is to ask for them. Many people try to get them and then sell them via mobile apps they develop. We are were informed about these activities in Bulgaria already.

Please, administrators here to delete this topic.

Best regards,

WWF Forest Team