topic Re: publish() to Enterprise Portal in stand alone script in ArcGIS API for Python Questions

publish() to Enterprise Portal in stand alone script

JaredPilbeam2 — Thu, 25 Mar 2021 16:25:23 GMT

Is it possible to use the API publish() method in a stand alone script? The following script runs from a Pro Notebook using version 1.8.4 of the API. But, if I run the same script as a stand alone I get a JSONDecodeError. I'm trying to publish a CSV to Enterprise Portal. When I put it in a try and except block it catches the error, but nothing publishes

Script that I'm running in Pro:

from arcgis.gis import GIS import os gis_dest = GIS('Home',verify_cert=False, trust_env=True) # data fp = r'C:\data.csv' #path to CSV # content manager cm = gis_dest.content # 1. add item = cm.add(item_properties={ 'type' : 'CSV', 'title' : 'c19_Vaccine_Current_test3', 'tags' : 'tags', 'typeKeywords' : "CSV" }, data=fp) # 2. analyze analyze_csv = cm.analyze(item=item, file_type='csv') pp = analyze_csv['publishParameters'] # 3. Modify the publish parameters then publish # For tables ensure the `locationType` is None pp['locationType'] = 'none' pitem = item.publish(publish_parameters=pp, file_type='csv')

Line 17 it's referring to is this line:

data=fp)

Traceback (most recent call last): File "pathto\test.py", line 17, in <module> data=fp) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\__init__.py", line 4265, in add owner_name, folder) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_portalpy.py", line 355, in add_item resp = self.con.post(path, postdata, files) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 720, in post force_bytes=kwargs.pop('force_bytes', False)) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\arcgis\gis\_impl\_con\_connection.py", line 504, in _handle_response data = resp.json() File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 900, in json return complexjson.loads(self.text, **kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\__init__.py", line 348, in loads return _default_decoder.decode(s) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Re: publish() to Enterprise Portal in stand alone script

JaredPilbeam2 — Fri, 26 Mar 2021 20:43:23 GMT

I'm posting my whole script in case it helps to have the whole picture. The first part (which writes requested JSON to a CSV) works fine. It only fails where I pointed out in the above post. NOTE: this is easily reproducible.

import json,requests from arcgis.gis import GIS import os,sys import csv from datetime import datetime ##-------------this part writes requested JSON to a CSV----------- #file name csv_file = 'blankcsv' #the directory you want the csv written to file_path = r'C:\Users\jpilbeam\Test' ##idph url goes here idph_data = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent' full_file = os.path.join(file_path, csv_file) + str(datetime.now().strftime('_%m_%d_%Y_%H%M')) + '.csv' #get json data from idph response = requests.get(idph_data,verify=True) #read the json response and keep the VaccineAdministration part data = response.json()['VaccineAdministration'] #write to file with open(full_file, 'w', newline='', encoding='UTF-8') as csvfile: f = csv.writer(csvfile) #write the headers of the csv file f.writerow(['County','AdminCount','AdminCountChange', 'RollAvg', 'AllocDoses', 'FullyVaccinated', 'FullyVaccinatedChange', 'ReportDate', 'Pop', 'PctVaccinated', 'LHDInventory', 'CommInventory','TotalInventory', 'InventoryDate']) for elem in data: #get the values for all the keys (i.e. CountyName, AdministeredCount, etc...) f.writerow([elem['CountyName'], elem['AdministeredCount'], elem['AdministeredCountChange'], elem['AdministeredCountRollAvg'], elem['AllocatedDoses'], elem['PersonsFullyVaccinated'], elem['PersonsFullyVaccinatedChange'], elem['Report_Date'], elem['Population'], elem['PctVaccinatedPopulation'], elem['LHDReportedInventory'], elem['CommunityReportedInventory'], elem['TotalReportedInventory'], elem['InventoryReportDate']]) ##---------This part (is supposed to) adds then publishes to Portal--------- #provide Enterprise Portal login info here gis_dest = GIS('portalURL', 'user', 'pass', verify_cert=False, trust_env=True) #content manager cm = gis_dest.content #add the CSV file item = cm.add(item_properties={ 'type' : 'CSV', 'title' : 'VaccineCurrent', 'tags' : 'test', 'typeKeywords' : "CSV" }, data=full_file) #analyze the CSV file to autogenerate the publish parameters analyze_csv = cm.analyze(item=item, file_type='csv') pp = analyze_csv['publishParameters'] #Modify the publish parameters #for tables ensure the `locationType` is None pp['locationType'] = 'none' #Publish pitem = item.publish(publish_parameters=pp, overwrite=True, file_type='csv') print("---done---")

Re: publish() to Enterprise Portal in stand alone script

MarkKinnaman — Wed, 31 Mar 2021 12:25:25 GMT

I saw your post on github (and the other post here) and was wondering if you got this to work. I am having the same issue when publishing to portal, but not when publishing to AGOL.

What enterprise/portal version are you trying to publish to? I am currently using 10.6.1.

I took a slightly different approach to you, but got the same result.

import pandas as pd import json from arcgis import GIS j = 'https://idph.illinois.gov/DPHPublicInformation/api/covidVaccine/getVaccineAdministrationCurrent' r = requests.get(j).json()['VaccineAdministration'] df = pd.json_normalize(r) #AGOL Connection gis = GIS(url='URL', username= 'username', password='password') #Enterprise Portal Connection gisE = GIS(url='URL', username='username', password='password') cm = gisE.content item = cm.add(item_properties={ 'type' : 'CSV', 'title' : 'VaccineCurrent', 'tags' : 'test', 'typeKeywords' : 'CSV'}, data='/media/WDB/Backup/Projects/data/IDataa.csv') cm.analyze(item=item, file_type='csv', location_type='none')

Re: publish() to Enterprise Portal in stand alone script

JaredPilbeam2 — Wed, 31 Mar 2021 13:59:57 GMT

Hi Mark,

I still don't have a surefire way. I'm using Enterprise Portal v. 10.81. I have a case going with ESRI Support and will report back whenever I find something out.

Re: publish() to Enterprise Portal in stand alone script

JaredPilbeam2 — Thu, 01 Apr 2021 14:12:08 GMT

You are not passing in the f=json parameter needed to get a JSON response from the REST API.

I got some help on my GitHub post, but I don't know enough to know exactly where the f=json parameter is supposed to go? It is in response to the JSONDecodeError on this line:

data=full_file)

Re: publish() to Enterprise Portal in stand alone script

MarkKinnaman — Thu, 01 Apr 2021 16:47:13 GMT

Alright I will post on Github as well.

If you look at my script below, the json parameter is being passed as a method. This is the same as passing the parameter 'f=json'. You are accomplishing the same thing by using 'response.json()'. At least that is how I understand it.

r = requests.get(j).json()['VaccineAdministration']