nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Ongoing drama with memory workspace

FelixPertziger

I use arcpy.da.FeatureClassToNumPyArray with option EXPLODE TO POINTS to extract vertices of polygons/polylines. It is followed by arcpy.da.FeatureClassToNumPyArray conversion to points.

This is what I see when output points are saved in geodatabase:

which is all good. But when output points saved in 'memory' I see this on a map :

There are massive 'gaps' for memory/points.

memory workspace is my favorite for intermediate outputs due to multiple reasons. Speed is one of them. For example it took 15.5 seconds to create 1.6M memory/POINTS and 1 minute 15 seconds for GDB/points (GDB on SSD ! disk). With now retired ArcGIS 'in_memory' workspace worked like a charm...

Story does not ends here. If I use memory/points with other tools in GUI or script, ArcPRO does not 'see' missing points. To sober it up, I need to run update cursor on it/ or dumb field calculation on any field without changing any of the values and ghost features will pop up. So, physically records are present, but are ignored/skipped/disregarded.

I can confirm that the same thing is happening with out of the box tool, e.g. Union when I try to fill many 100s of holes in polygons.

I played with rendering, cache, number of processes and multiple other suggestions from AI. Nothing is working …

I am suspecting this has happened after one of the updates. It is definitely not faulty 3.7.2 patch which I removed lately.

Anybody has the same experience with such erratic behavior, please?

And yes, category I've picked

is hardly relevant

Find more posts tagged with

Accepted answers

All comments

DanPatterson

@FelixPertziger did you mean that you use NumPyArrayToFeatureClass in your sentence

…. in followed by arcpy.da.FeatureClassToNumPyArray ….

Are any of the poly* features multipart ? If so does converting them (multipart to singlepart) affect anything either in a gdb or in memory?

AlfredBaldenweck

How does Feature Vertices To Points compare?

gprince

@FelixPertziger could you open a ticket with Esri Support, we'd be happy to investigate.

I just attempted to reproduce this with generic dataset and could not. thanks

FelixPertziger

@gprince

I tried to open ticket through my ESRI and failed

FelixPertziger

@DanPatterson yes indeed to ".. use NumPyArrayToFeatureClass" thank you.

FelixPertziger

I am on Basic license, can't compare

DanPatterson

a substitute for Feature Vertices to Points… but it uses NumPyArrayToFeatureClass which you are already using 😊Feature vertices to points ... for Basic ArcGIS Pro

And I can't replicate the issue either regardless whether the features are multipart and/or contain holes.

If I need a featureclass, I haven't used "memory" as a destination since I do most of the processing within numpy and gdb for final output.

Hope this finds a resolution and you can report back.

FelixPertziger

@gprince I tried to open ticket through my ESRI and failed. Let's try it from here please.

Attached file stores file geodatabase called 'memory_issues.gdb' with 2 feature classes:

POLYGON_2_TEST - single polygon
RECTANGLE buffered extent of polygon.

It also has Python script called 'script_2_run.py' renamed to script_2_run.txt for safety reasons.

May I suggest this workflow:

add 2 feature classes to empty map (note spatial reference)
add script to any toolbox with input parameter (feature layer) being POLYGON_2_TEST and output being memory/whatever (feature class)
run script which uses feature class to numpy array and numpy array to feature class methods. Script messages will say "Length of array = 396869" and "Length of output = 396869". This IS actual number of features in output class
Select by location (intersect) new feature layer 'whatever' using RECTANGLE as selectin feature. You'll see that there are few thousands points less than actual 396869 selected. You might try Intersect tool between RECTANGLE and output points to see the same reduced amount of points.

I greatly appreciate your help, hoping it's a memory leakage in my out of date PC:

Processor Intel(R) Xeon(R) CPU E5-1603 0 @ 2 .80GHz, 2800 Mhz, 4 Core(s), 4 Logical Processor(s)

Installed Physical Memory (RAM) 16.0 GB

OR some project options that I tweaked…

Please let me know if you need more info.

memory_issues.gdb.zip

FelixPertziger

@DanPatterson holes and or multiparts play no role in this. It is happenning with single part/multipart polylines as well, but seemingly with few 100K vertices. No issues noticed with small datasets. See my latest comment with test data attached.

PS: I looked at your tool… Well, inventing bicycle was fun, except of that non-sense with gaps

DanPatterson

slow night… so, I did some work using a notebook (attached in a zip file)

memory_issues.zip

unique points 396054
Input    : C:\Temp\memory_issues\memory_issues.gdb\POLYGON_2_TEST
Spat Ref : NZGD_2000_New_Zealand_Transverse_Mercator
Num pnts : 396869
Fields   : ('SHAPE_X', 'SHAPE_Y')

C:\Temp\memory_issues\memory_issues.gdb\POLYGON_2_TEST_pnts, memory\test_pnts
array pnts  : 396869
memory pnts : 396869
pairwise pnts  : 392239
unique points in array : 396054

"unique points " was the number of unique points in the structured array, duplicate start/end accounted for see the notebook.

I did a pairwise intersect with the rectangle file and I observed what you have noted (396869 real points 392239 for pairwise intersect)

I haven't got around to seeing where the missing ones are yet, or if there is any pattern

FelixPertziger

Bad luck. I was hoping it's just me. This gaps are not visible, moreover - they are nomadic and can disappear in a different place. To find them in a table one can use relate between intersected points and one with 'ghosts'. Switch selection to see these records in table view. Run some field calculator on any field without even changing original field value and wow! magic: missing points will become visible. Something to do with cursors and data access in memory space. My solution is to run update cursor inside script 'updating' every 10th or so value on a known field. What a waste, but many thanks for confirmation

gprince

Visual inspection of output lgtm

doing Select By Location on result return 396869

i also tried the PairwiseIntesect with RECTANGLE and put that result into GetCount and got 396869.

So doesn't seem to be a problem at 3.8. I can't speak to 3.7 yet.

FelixPertziger

Wonderful news. It seems my guess regarding faulty version was correct. I will mark your comment as solution as soon as 3.8.1 will become available. As for now I see this:

AlfredBaldenweck

In the meantime, try this code and see if it works for you?

import arcpy
inFeat = r"path to your polygons"
spatRef = arcpy.da.Describe(inFeat)["spatialReference"].factoryCode
spatRef = arcpy.SpatialReference(spatRef)

outF = arcpy.management.CreateFeatureclass("memory", "pts", geometry_type = "POINT", 
                                           template = inFeat, spatial_reference = spatRef)
arcpy.management.AddField(outF, "OriginalFID", "LONG")

pointList = []
with arcpy.da.SearchCursor(inFeat, ["OID@", "SHAPE@"]) as cursor:
    for row in cursor:
        for part in row[1]:
            for vert in part:
                pointList.append([row[0], vert])
del cursor

with arcpy.da.InsertCursor(outF, ["OriginalFID", "SHAPE@"]) as cursor:
    for point in pointList:
        cursor.insertRow(point)
del cursor

I don't think any of those things require a license.

I fed the output points the original table as a template just in case, but I don't use any of the fields. You can if you want.

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.