topic Re: Using arcpy to copy a portal feature service to a fgdb feature class in Python Questions

Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Fri, 10 Dec 2021 20:10:21 GMT

ArcGIS Pro 2.4.3

In pro, you can add a feature service and manually export the features to a file geodatabase feature class.

How do you mimic that functionality in python? I have tried:

import arcpy
source = r'https://path/toOur/rest/services/MapServer/0'

arcpy.FeatureClassToFeatureClass_conversion(source,r'C:\pathTo\Test.gdb','newFC')
‍‍‍‍

and get :

ExecuteError: ERROR 000210: Cannot create output C:\pathTo\Test.gdb\newFC
Failed to execute (FeatureClassToFeatureClass).

I googled the error, and then tried to use the same python approach with an Egdb feature class as the source, and it worked splendidly. Is this a case for the python api?

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoshuaBixby — Sun, 12 Jan 2020 15:46:58 GMT

Joe, ArcPy isn't really built to directly deal with "web GIS," which is why the ArcGIS API for Python | ArcGIS for Developers was developed. Use arcgis.features module — arcgis 1.7.0 documentation to retrieve a FeatureLayer of the service layer, then query it to get a FeatureSet that has a save option to file geodatabase.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Sun, 12 Jan 2020 16:04:20 GMT

Mondays project... Thanks.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Mon, 13 Jan 2020 15:28:31 GMT

Joshua- esri tech support provided this link to me : How To: Create and download a file geodatabase replica from a hosted feature service using Python . This particular approach creates a replica and uses userLib2(); as I told them, I'm not interested in a replica and userlib2() is not available in 3.x. Maybe ESRI can get on board to provide us a more modern approach....

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoshuaBixby — Fri, 10 Dec 2021 20:10:23 GMT

Wow, that is an Esri Support #fail for sure. That technical article is ancient from a web GIS perspective. It was "last published" right around the time of the first release of the ArcGIS API for Python, which likely explains why it is not using that API and instead having people make HTTP calls themselves.

Here are some basic steps to download a feature service layer as a feature class in a file geodatabase:

from arcgis.gis import GIS
from arcgis.features import FeatureLayer

url_gis = # URL to AGOL or Portal
url_fl =  # URL for feature layer to download as feature class
user =  # AGOL or Portal username
pwd = # user password
fgdb = # path to file geodatabase
fc = # name of feature class

gis = GIS(url_gis, user, pwd)
fl = FeatureLayer(url_fl)
fs = fl.query()
fs.save(fgdb, fc)‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Mon, 13 Jan 2020 18:20:25 GMT

Thank you so much Joshua. I've been floundering about with Jupyter notebooks for the pas hour or so, but this looks just like what I need. Locally, we do have a little glitch with the GIS() method, but I'm working with the S.A to get past that.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Fri, 10 Dec 2021 20:10:26 GMT

Joshua- following the instructions given here: Working with different authentication schemes | ArcGIS for Developers I'm taking a slightly different approach in passing the user, and pwd variables; I need to pass a 'client_id' as well:

gis = GIS(url_gis,user,pwd,client_id = 'Ox05rrnllB1biFgW')‍‍

But I get an error:

  File "C:\EnvClones\arcgispro-py3-clone243\lib\site-packages\arcgis\_impl\connection.py", line 449, in oauth_authenticate
    from bs4 import BeautifulSoup

ModuleNotFoundError: No module named 'bs4'‍‍‍‍‍‍‍‍

Not having a whole lot of luck with dr.google trying to find a remedy for this: it seems as though with the python 3.6 install from Pro, bs4 is no where to be found...

edited to add: actually found a couple threads that suggest using pip3 to install bs4 into python 3.6: here is one from quora. Anaconda offers one as well. I guess I need to use one of these while in my cloned environment?

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoshuaBixby — Mon, 13 Jan 2020 19:59:38 GMT

It appears to be trying to load a Python package/module that isn't installed in your virtual env. Confirm that beautifulsoup4 is installed, and if not, install it. It might be a defect in ArcGIS API for Python where they aren't packaging a dependency.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Mon, 13 Jan 2020 21:11:51 GMT

From Esri Tech Support: BUG-000118107 - In ArcGIS API for Python, requests for installing a third-party module named bs4 when the User Authentication with OAuth 2.0 method is used to login into ArcGIS Online.

by the way:

Kory Kramer‌

Re: Using arcpy to copy a portal feature service to a fgdb feature class

KoryKramer — Mon, 13 Jan 2020 21:38:35 GMT

BUG-000118107: In ArcGIS API for Python, requests for installing a ..

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoshuaBixby — Mon, 13 Jan 2020 21:49:48 GMT

I must be clairvoyant. Nah, just worked with Esri for a long time. This bug is fairly simple to work around yourself, assuming there isn't yet another bug just waiting right around the corner once the package import is resolved.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoeBorgione — Mon, 13 Jan 2020 22:22:01 GMT

After going back and forth on this with the portal admin here, Joshua's approach works for me.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

davedoesgis — Tue, 04 Feb 2020 01:26:54 GMT

Joshua Bixby‌ - Back when I used to work with MapService type services, you had to set a maximum number of features that could be retrieved. The default was 1000 and you could set it higher, but a very large data set could always be more than whatever number you put in there. So a query like this always ran the risk of not getting all the features.

Is that an issue with your code querying FeatureService type services? If so, any suggestions for a work-around?

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JoshuaBixby — Tue, 04 Feb 2020 15:06:24 GMT

David, the ArcGIS API for Python is REST API-aware, so the underlying code addresses the feature limit of services and does the appropriate batching of records to retrieve the entire data set.

Re: Using arcpy to copy a portal feature service to a fgdb feature class

SoratoSouza_e_Silva — Thu, 09 Jul 2020 21:18:13 GMT

This code is very good, but my results is:

Can you help me?

Re: Using arcpy to copy a portal feature service to a fgdb feature class

KoriKurtzeborn — Fri, 07 Apr 2023 22:27:47 GMT

Hi Josh! Hoping to troubleshoot an error I'm getting in the query portion of my code. I think because of the number of features. Any tips on how I can adjust my code to accommodate the size of the feature layer I'm trying to download. The error I get is KeyError: 'count'

Re: Using arcpy to copy a portal feature service to a fgdb feature class

KatGIS — Tue, 18 Jul 2023 18:15:20 GMT

Thanks, this is helpful! What would be the most efficient way to include attachments with the feature layer?

Re: Using arcpy to copy a portal feature service to a fgdb feature class

JamesTurner2 — Tue, 26 Mar 2024 11:24:13 GMT

Is there an modification of this that would work with an enterprise geodatabase? I have a script uses the API to query a feature service and then export the data to a table (no spatial data) in my EGDB. I also use arcpy to set privileges'. I currently use the API and ArcPy, I'm wondering if there is a way to do something similar to the .save() method to eliminate the arcpy. I suppose I could do something like an append to get around overwriting the entire table and having to reset privileges.