topic Re: No response from addPart operation when trying to upload parts of tpk to AGOL, commit fails sayings no parts uploaded (python, requests) in ArcGIS REST APIs and Services Questions

No response from addPart operation when trying to upload parts of tpk to AGOL, commit fails sayings no parts uploaded (python, requests)

CarlVricella — Fri, 10 Dec 2021 20:25:42 GMT

Hi all,

Trying to automate the updating of a large tpk on AGOL. When the tpk is updated on my servers I do the following:

- Break into parts

- Call updateItem opeartion with multipart = true

- make a request to upload each part using addPart

- make a request with the commit function to finalize the update to the tpk on AGOL

My issue is that I get no response when trying to use addPart, then when the commit is finally hit in my code it says there are no parts to commit:

Code:

update_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/update'.format(username,tpk_id)

up_data = {'f':'json',
    'token':token,
    'title': layer_name,
    'itemId':tpk_id,
    'type': 'Tile Package',
    'overwrite': 'true', # existing item, overwrite is set to true
    'async':'true',
    'multipart':"true"}
    

update_response = requests.post(update_url, up_data=data, verify = cert).json()

print update_response


part_num = 1
part_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/addPart'.format(username,tpk_id)

for part in os.listdir(parts):

    path = r"\\psegliny.com\oms\oms_gis_prd\fileshare_mapping\cache_extent\es_secondary_tpks\full_service_area"

    part = path + "\{}".format(part)

    print part
    print(str(part_num))


    ##files_up = {"file": open(part, 'rb')}
    with open(part, "rb") as f:
        files_up = {"file":f}

        data = {'f':'json',
                'token':token,
                'partNum':part_num}

    
        part_response = requests.post(part_url,data = data,files=files_up,verify=cert)
    part_num+=1
    try:
        jres = json.loads(part_response.text)
        print jres
    except:
        print "no response"
    ##print add_part_response

commit_url = 'https://lipa.maps.arcgis.com/sharing/rest/content/users/{}/items/{}/commit'.format(username,tpk_id)

data = {'f':'json',
'token':token}

commit_response = requests.post(commit_url,data=data,verify=cert).json()

print commit_response
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Anyone know what the issue is?

Re: No response from addPart operation when trying to upload parts of tpk to AGOL, commit fails sayings no parts uploaded (python, requests)

MarkVolz1 — Fri, 10 Dec 2021 20:25:45 GMT

There are a couple of problems in your code, and I have only found out by losing a couple of days to the same issue.

The first is that you can only send 5mb in the first 4 parts and then on the 5th you can send the remainder. The second is that you need to send the parameters (token, partNum, streamData, etc) in the URL of the addPart. And the final is the undocumented detail that you need to send the file through as a multipart form record.

Here is a snippet of the addPart call that works for me:

    CHUNK_SIZE = 5000000
    partNum = 1
    finalChunk = None
    with open(itemFilePath,'rb') as f:
        chunk = f.read(CHUNK_SIZE)
        while chunk:
            if (partNum < 5):
                log.debug('sending the chunk' + str(partNum))
                FilePartURL = siteRoot + "/sharing/rest/content/users/" + username + "/" + folderId + "/items/" + addResultJson['id'] + '/addPart?partNum=' + str(partNum) + '&streamData=true&f=json&token=' + token
                multipart_form_data = {'file':(filename, chunk)}
 
                addPartCurl = requests.post(url=FilePartURL, files=multipart_form_data)
                if (addPartCurl.status_code !=200):
                    log.exception("An error occured calling " + addItemURL + " , response code was " + str(addPartCurl.status_code) + ", server message was: " + str(addPartCurl.content) )
                    addPartResult = "Error"
                else:
                    addPartResult = addPartCurl.text
                    log.debug("No errors occured calling " + addItemURL + ". response as a string was: " + addPartCurl.text)
                addPartCurl.close()
            else:
                finalChunk = finalChunk + chunk
                log.debug('doing the final chunk')
            partNum += 1
            chunk = f.read(CHUNK_SIZE)
    #send the final chunk
    if (partNum > 4):
        finalFilePartURL = siteRoot + "/sharing/rest/content/users/" + username + "/items/" + addResultJson['id'] + '/addPart?partNum=5&streamData=true&f=json&token=' + token
        finalMultipart_form_data = {'file':(filename, finalChunk)}
        FinalFilecurl = requests.post(url=finalFilePartURL, files=finalMultipart_form_data)
        if (FinalFilecurl.status_code !=200):
            log.exception("An error occured calling " + addItemURL + " , response code was " + str(FinalFilecurl.status_code) + ", server message was: " + str(FinalFilecurl.content) )
            finalFileresult = "Error"
        else:
            finalFileresult = FinalFilecurl.text
            log.debug("No errors occured calling " + addItemURL + ". response as a string was: " + FinalFilecurl.text)
        FinalFilecurl.close()‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hope that helps.