GP Service Publish Fails in ArcGIS Desktop & ArcGIS Pro

[RachappaBellappa2]

This is the code snippet that was added and it's able to run the Toolbox but when ever i try to Publish as GP service, ArcMap/ArcCatalog crashes and not that much information is there in the dump file. It mentions stack overflow exception.

Main aim is to process huge set of data using chunks instead of querying all the data at once which was having some performance issues.

a) First Get set of ObjectID's (May be 100000)

b) Call the below generate_chunk --> Which passes list of 10000 Id's 10 time

c) Call the process_chunk which inturn calls the map service for 10000 Id's at a time and get the response JSON.

def generate_chunk(l, chunk_size):
    '''
        Build chunk
        l: Input Object Ids.
        chunk_size: the size of chunk.
    '''
    remainder = len(l) % chunk_size
    # print(remainder)
    if remainder != 0:
        chunk_length = int(len(l) / chunk_size)
        chunk = [l[i * chunk_size:i * chunk_size + chunk_size] for i in range(chunk_length)]
        chunk.append(l[-remainder:])
        return chunk
    else:
        chunk_length = int(len(l) / chunk_size)
        chunk = [l[i * chunk_size:i * chunk_size + chunk_size] for i in range(chunk_length)]
        return chunk


def process_chunk(chunk, serviceURL):
    '''
        Process the chunk and return map service json
    '''
    try:
        for item in chunk:
            objectIds = ','.join([str for i in item])
            url_data = urllib.parse.urlencode({'f': 'json',
                                         'where': '1=1',
                                         'outFields': '*',
                                         'returnZ': 'true',
                                         'objectIds': objectIds})
            url_data = url_data.encode('utf-8')
            req = urllib.request.Request(serviceURL, url_data)
            results = json.load(urllib.request.urlopen(req))
            yield results
    except Exception:
        tb = sys.exc_info()[2]
        tbinfo = traceback.format_tb(tb)[0]
        pymsg = "Error getting the JSON from ArcGIS Service:\n    " + \
                str(sys.exc_type) + ": " + str(sys.exc_value) + "\n"
        AddMsgAndPrint(pymsg)

This is causing StackOverFLowException and not only it crashes ArcGIS Desktop it also crashes ArcGIS PRO 2.4

[JayJohnson6]

Try chunks of 1000 records.  In some cases that's the most it can handle at once.

Large dataset upload and download considerations—Documentation | ArcGIS Enterprise