No response from addPart operation when trying to upload parts of tpk to AGOL, commit fails sayings no parts uploaded (python, requests)

170
1
04-16-2020 06:25 AM
CarlVricella
New Contributor II

Hi all,

Trying to automate the updating of a large tpk on AGOL. When the tpk is updated on my servers I do the following:

- Break into parts 

- Call updateItem opeartion with multipart = true

- make a request to upload each part using addPart

- make a request with the commit function to finalize the update to the tpk on AGOL

My issue is that I get no response when trying to use addPart, then when the commit is finally hit in my code it says there are no parts to commit:

Code:

update_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/update'.format(username,tpk_id)

up_data = {'f':'json',
'token':token,
'title': layer_name,
'itemId':tpk_id,
'type': 'Tile Package',
'overwrite': 'true', # existing item, overwrite is set to true
'async':'true',
'multipart':"true"}


update_response = requests.post(update_url, up_data=data, verify = cert).json()

print update_response


part_num = 1
part_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/addPart'.format(username,tpk_id)

for part in os.listdir(parts):

path = r"\\psegliny.com\oms\oms_gis_prd\fileshare_mapping\cache_extent\es_secondary_tpks\full_service_area"

part = path + "\{}".format(part)

print part
print(str(part_num))


##files_up = {"file": open(part, 'rb')}
with open(part, "rb") as f:
files_up = {"file":f}

data = {'f':'json',
'token':token,
'partNum':part_num}


part_response = requests.post(part_url,data = data,files=files_up,verify=cert)
part_num+=1
try:
jres = json.loads(part_response.text)
print jres
except:
print "no response"
##print add_part_response

commit_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/commit'.format(username,tpk_id)

data = {'f':'json',
'token':token}

commit_response = requests.post(commit_url,data=data,verify=cert).json()

print commit_response
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Anyone know what the issue is?

0 Kudos
1 Reply
MarkVolz1
New Contributor

There are a couple of problems in your code, and I have only found out by losing a couple of days to the same issue.

The first is that you can only send 5mb in the first 4 parts and then on the 5th you can send the remainder. The second is that you need to send the parameters (token, partNum, streamData, etc) in the URL of the addPart. And the final is the undocumented detail that you need to send the file through as a multipart form record.

Here is a snippet of the addPart call that works for me:

    CHUNK_SIZE = 5000000
partNum = 1
finalChunk = None
with open(itemFilePath,'rb') as f:
chunk = f.read(CHUNK_SIZE)
while chunk:
if (partNum < 5):
log.debug('sending the chunk' + str(partNum))
FilePartURL = siteRoot + "/sharing/rest/content/users/" + username + "/" + folderId + "/items/" + addResultJson['id'] + '/addPart?partNum=' + str(partNum) + '&streamData=true&f=json&token=' + token
multipart_form_data = {'file':(filename, chunk)}

addPartCurl = requests.post(url=FilePartURL, files=multipart_form_data)
if (addPartCurl.status_code !=200):
log.exception("An error occured calling " + addItemURL + " , response code was " + str(addPartCurl.status_code) + ", server message was: " + str(addPartCurl.content) )
addPartResult = "Error"
else:
addPartResult = addPartCurl.text
log.debug("No errors occured calling " + addItemURL + ". response as a string was: " + addPartCurl.text)
addPartCurl.close()
else:
finalChunk = finalChunk + chunk
log.debug('doing the final chunk')
partNum += 1
chunk = f.read(CHUNK_SIZE)
#send the final chunk
if (partNum > 4):
finalFilePartURL = siteRoot + "/sharing/rest/content/users/" + username + "/items/" + addResultJson['id'] + '/addPart?partNum=5&streamData=true&f=json&token=' + token
finalMultipart_form_data = {'file':(filename, finalChunk)}
FinalFilecurl = requests.post(url=finalFilePartURL, files=finalMultipart_form_data)
if (FinalFilecurl.status_code !=200):
log.exception("An error occured calling " + addItemURL + " , response code was " + str(FinalFilecurl.status_code) + ", server message was: " + str(FinalFilecurl.content) )
finalFileresult = "Error"
else:
finalFileresult = FinalFilecurl.text
log.debug("No errors occured calling " + addItemURL + ". response as a string was: " + FinalFilecurl.text)
FinalFilecurl.close()

Hope that helps.

0 Kudos