How to solve the error:out of memory

6258
14
01-10-2016 02:04 PM
HuaShang
New Contributor

I want to use the clip function in python, the code is as follow:

import os

from multiprocessing import Pool

import arcpy

from arcpy import env

def data_clip(shp):

    env.workspace = r"F:/GCM_CMIP5/FGOALS-g2(LASG-CESS)/mrro/ShpFile/f"

    clip_features = "D:/Research/boundry.shp"

    pre, suf = os.path.splitext(shp)

    new = pre + "_china" + suf

    in_mem = os.path.join("in_memory", pre)

    out_feature_class = os.path.join("F:/GCM_CMIP5/FGOALS-g2(LASG-CESS)/mrro/ChinaShpFile", new)

   

    arcpy.Clip_analysis(shp, clip_features, in_mem)

    arcpy.CopyFeatures_management(in_mem, out_feature_class)

    arcpy.DeleteFeatures_management(in_mem)

if __name__ == "__main__":

    env.overwriteOutput = True

    env.workspace = r"F:/GCM_CMIP5/FGOALS-g2(LASG-CESS)/mrro/ShpFile/f"

   

    p = Pool(7)

    fcs = arcpy.ListFeatureClasses()

    p.map(data_clip, fcs)

   

    print('Clip finish!')

But I got the "out of memory" error.

1.png

If anyone knows how to solve this problem?

Thank you so much!!!

0 Kudos
14 Replies
DanPatterson_Retired
MVP Emeritus

when it comes to multiprocessing, have you had a look at the links Curtis Price posted here... it is in response to another thread  https://community.esri.com/docs/DOC-3824#comment-11549

0 Kudos
HuaShang
New Contributor

Thanks! I'll check that. But when I didn't use multiprocessing, it comes out the same error.

0 Kudos
DanPatterson_Retired
MVP Emeritus

p = Pool(7)  this line confuses me then.  Perhaps the files are just too big then. ... and I did notice that you tried it in-memory

0 Kudos
RebeccaStrauch__GISP
MVP Emeritus

You may want to try to move your data to a simpler path.  Slight change that it doesn't like the _ and -  ( ) etc in the path name.  Even though the error doesn't point to this, it doesn't hurt to eliminate that as an issue.

Also, it would be easier to debug if you tried Posting Code blocks in the new GeoNet

0 Kudos
HuaShang
New Contributor

Thank you !

Yes , I tried the simpler path, it still got that error.

0 Kudos
ChrisSmith7
Frequent Contributor

Have you thought about smoothing/generalizing your polygons, if that's an option for you? Reducing the vertices will result in a smaller file, but at the expense of losing some spatial resolution. What about splitting-up some of your features? Have you considered increasing your memory? How much do you have installed?

0 Kudos
XanderBakker
Esri Esteemed Contributor

... and what if you take out the multiprocessing and the in_memory workspace. Does it work? If I run into memory problems it is often related to the in_memory workspace.

0 Kudos
ChrisSnyder
Regular Contributor III

Before you run the Clip tool, run the Dice tool (Dicing Godzillas (features with too many vertices) | ArcGIS Blog) on the layer you are trying to clip. The Dice tool is specifically for the case when the input geometry is "too big to process" (one of the symptoms being the 'out of memory' error). 

DanPatterson_Retired
MVP Emeritus

with the caveat that it requires an Advanced license Dice—Help | ArcGIS for Desktop