topic Re: Model Builder - How to remove selected data from a shapefile? in Geoprocessing Questions

Model Builder - How to remove selected data from a shapefile?

RussellQuintero — Wed, 09 Jan 2013 17:56:56 GMT

I am attempting to automate a process I have historically done manually using model builder.

So far I have managed to successfully convert one shapefile to points, and then iterate through multiple rasters to extract the values of those rasters to points, creating multiple shapefiles.

This is where I run into problems.

Some of those newly created shapefiles contain invalid data, with values of -9999 in a field showing the raster had no data there. I want to remove that data.

I have tried the Analysis toolbox "Select" tool with the SQL query set to "Field Name" <> -9999, but it returns an error 000229.

I have tried "Feature Class to Feature Class" (FC2FC) with the same SQL query, and it says the geometry is invalid. When I try and open the file outside of Model Builder using FC2FC, it says it is unusable. If I use "Add Data" to add the shapefile to my current map, and then use FC2FC, it can suddenly use the file just fine.

So my questions here are two-fold:

1) What is the proper way in Model Builder to remove selected data from a shapefile?
2) How do I create a tool in Model Builder that will run without requiring that intermediate data be added to the map?

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 18:27:39 GMT

How about modifying your query with, say, "Field Name" < -999 (or try '=' since that is what you want to delete, correct?), and test to see what selected features there are as a result, use Get Count...if a count returned is the entire feature class count, then you know the query didn't work.

Then you can use Delete Features to get rid of the features once the selected set meets your criteria.

Re: Model Builder - How to remove selected data from a shapefile?

RussellQuintero — Wed, 09 Jan 2013 18:38:23 GMT

How about modifying your query with, say, "Field Name" < -999 (or try '=' since that is what you want to delete, correct?), and test to see what selected features there are as a result, use Get Count...if a count returned is the entire feature class count, then you know the query didn't work.

Then you can use Delete Features to get rid of the features once the selected set meets your criteria.

When doing it manually, I've been opening the attribute table, selecting the invalid data, and deleting it.

When doing this in Model Builder, the two options select the data that matches the query, so I need everything except -9999 data. If I select the data "Field Name" = -9999, and then delete the selection, how do I keep the other data?

I did try changing it to >-9999 instead of <> but I get exactly the same error of no supported geometry, which appears to be what it has decided to call the error where it can't open the file when it isn't loaded into the map, as mentioned in my first post. A rather frustrating requirement for an intermediate data product.

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Sat, 11 Dec 2021 15:27:26 GMT

When you Delete Features with a selected set on the layer, it should only delete selected - the remainder should be untouched.
Still, it is a good idea to make sure that you have a selection with a count - if there's no selected features, the tool politely deletes the entire dataset and generates 'empty output'...that's kind of alarming!

In short, see this pseudocode:

if not Count < total count:
     print 'whoa, better stop here...tell someone something unexpected has occurred'
else:
     arcpy.deleteFeatures(features)

Results may vary, so always check what's actually going on in your code! Not sure what you mean about having invalid geometry - do you have points in a layer as you initially stated?...or are you trying to access an intermediate layer which is not persisted? One more thing, not sure how your gp environment is set up, but you have to have the overwriteoutputs property on.

EDIT- So if it is the case you cannot access these 'invalid' features, then try Repair Geometry.

Re: Model Builder - How to remove selected data from a shapefile?

RussellQuintero — Sat, 11 Dec 2021 15:27:29 GMT

When you Delete Features with a selected set on the layer, it should only delete selected - the remainder should be untouched.
Still, it is a good idea to make sure that you have a selection with a count - if there's no selected features, the tool politely deletes the entire dataset and generates 'empty output'...that's kind of alarming!

In short, see this pseudocode:
if not Count < total count:
     print 'whoa, better stop here...tell someone something unexpected has occurred'
else:
     arcpy.deleteFeatures(features)
Results may vary, so always check what's actually going on in your code! Not sure what you mean about having invalid geometry - do you have points in a layer as you initially stated?...or are you trying to access an intermediate layer which is not persisted? One more thing, not sure how your gp environment is set up, but you have to have the overwriteoutputs property on.

EDIT- So if it is the case you cannot access these 'invalid' features, then try Repair Geometry.

Thanks for all of the responses.

First, let's back up to the use of code at all. In Model Builder, I'm just dragging in tools. My intention is to output a tool I can use directly from the toolbox, not a python script.

How does one add code like that within model builder?

Now to move on to what I have:

Inputs:
A directory with rasters
A shapefile with line features

I pass the shapefile through "Feature Vertices to Points" to produce a set of point features. This serves as an input to the sub-model.

The submodel takes in the directory and the point features. An iterator opens each raster in turn, uses "Extract values to points" to extract the raster data onto the points. The output is then used as the input of the "Feature Class to Feature Class" tool.

So yes, the input is an intermediate data item, although it when it crashes it doesn't seem to clean up the intermediate data and the shapefile persists for me to open.

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 19:34:28 GMT

Okay, I think I see - and the pseudocode was just to 'present' the idea, you can add things like checking for preconditions to a model, also Delete Features is a tool you can drag into a model as well...but looks like you may have a problem with your iterator, and possibly when it crashes it is creating 'malformed' output, ghost output, if you will.... It would help if you posted your model, although I am not in the office now to take a look at it.

I suggest this though at least to try to isolate the error - your spatial analyst tool has the syntax:
ExtractValuesToPoints (in_point_features, in_raster, out_point_features, {interpolate_values}, {add_attributes})

...or, say this, if this makes better sense:
ExtractValuesToPoints(inPointFeatures, inRaster, outPointFeatures, "INTERPOLATE", "VALUE_ONLY")

So your iterator must be feeding the raster dataset in for the 2nd param inRaster, shown above -- your inPointFeatures can remain constant but that is where you must be getting 'no value' data, where there is nothing spatially coincident with the currently used raster dataset. It may be more efficient to 'preprocess' or 'qualify' points (possibly with the extent environment) with the extraction.

Also, you have to somehow provide a mechanism to rename the outPointFeatures on each iteration...how are you including that? ...could attempt overwriting the same temp fc, but append it or copy it to a permanent location before moving on in the loop; again, must have unique names if copying to a separate fc.

It would help greatly to 'see' your model. Include any additional error messages and shapefile output, if you can.

Re: Model Builder - How to remove selected data from a shapefile?

RussellQuintero — Wed, 09 Jan 2013 19:45:34 GMT

If I remove the last step, "Feature Class to Feature Class," the whole process runs fine and produces a stack of shapefiles with points.

I use the %Name% variable to change the name on each iteration, and for the output of FC2FC, I use %Name%_filtered.shp

So I think the iterator is working, it's a problem with the next step. Attached is an image of the model. Is there a way to upload the model itself?

[ATTACH=CONFIG]20612[/ATTACH]

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 19:53:28 GMT

You can zip the toolbox in which the model resides...zip it with a few output shapefiles if you can. ...or if too big, zip separately and post, no need to post your entire output...just a sample.
I may not be able to pinpoint this today (as I explained I am not in the office), but I bet once you post your zip, there are ModelBuilder experts that can fix it for you quickly! I still suspect your iterator -- are you sure it is 'aware' of the FCtoFC part of the process??

(oops, I meant go to Windows Explorer, select and zip the toolbox [tbx extension], etc.)

Re: Model Builder - How to remove selected data from a shapefile?

RussellQuintero — Wed, 09 Jan 2013 20:52:06 GMT

You can zip the toolbox in which the model resides...zip it with a few output shapefiles if you can. ...or if too big, zip separately and post, no need to post your entire output...just a sample.
I may not be able to pinpoint this today (as I explained I am not in the office), but I bet once you post your zip, there are ModelBuilder experts that can fix it for you quickly! I still suspect your iterator -- are you sure it is 'aware' of the FCtoFC part of the process??

(oops, I meant go to Windows Explorer, select and zip the toolbox [tbx extension], etc.)

All of the data was too big for the size limits. I uploaded it to google drive here:

https://docs.google.com/folder/d/0B_wIIORblfXkMENRMlNISm94Mm8/edit

shoreline.zip contains the shapefile input.
dem.zip contains two DEMs for use as the contents of a folder for the iterator's needed directory input.

I've been experimenting with Model Builder a bit, so there are some unneeded tools in that toolbox. The relevant two are:

"Lidar Shoreline Slope Step 1" and "DEM to Slope Points (Sub-model)"

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 21:11:48 GMT

OK, sorry about that, I didn't mean for you to load everything - what's most important is to examine the 'guts' of the model, for extra I wanted to see a couple of your output shapefiles......I'm fairly sure now it is linked to your iterator and how the Value produced from it is used for variable substitution. Just looking again at your last post, the graphic of the model, it doesn't appear you are substituting output names for FCtoFC sensitive to Value. When the model iterates, I think it crashes because the last part is attempting to be a part of the iteration, but the output naming convention doesn't make sense.

Just curious, what is the Value text typically...no file extension, correct, and just a name? If Value contains 'extra parts', their is a Parse Path tool to strip out what you need. I think you're okay, but have to be careful and check that when mixing data types as in your model (from raster output, then shapefile output)....to simplify presentation undoubtedly, many of the webhelp examples deal with the same datatype.

So I'd look carefully at your inline variable substitution.

I downloaded your tbx zip, will take a look at that and your shapefile...let me look at that 1st.

EDIT: I said Value but meant Name that you are using for in-line substitutions.

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 21:25:02 GMT

You need in-line substitution for your FC2FC output -- the output workspace can be constant, but the actually fc name must change, so make that %Name%_filtered as you said you tried, but I don't see it in the graphic. (I think the .shp will automatically be applied)

Can you make sure that you tried that?

Also, FC2FC has an optional where_clause parameter you are not using - is this not where you want to eliminate 'empty' or null records? I think this can be a 'static' query but I'd have to look closer at it.

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Wed, 09 Jan 2013 22:45:43 GMT

This illustrates what I was thinking, using Make Feature Layer and Delete Features - this way you aren't creating essentially 2 copies of the output...the in-line variable substitution looks fine, although I didn't test it. I would say this as well, to set:

1- the overwriteoutputs geoprocessing env to 'true' (gp options)
2- 'un-check' the intermediate output for your shp output before Make Feature layer (this is now your final output)
3- run a small sample if you can as a test...no need to run everything at once and have to wait, if it needs tweaking.

Notice the query in the 1st pic, may need to tweak that as well. Let me know if that works... interesting. If not, I'd like to 'dissect' it tomorrow when back at the office.

-Wayne

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Thu, 10 Jan 2013 18:05:34 GMT

My apologies, first of all, for the number of posts I've made...but bear with me, I've developed a keen interest in this part of the topic dealing with iterators and in-line var substitution.

You said your problem was isolated in the copy of features without data from that same source that satisfies a query...see my last post. The idea was to use FC2FC with a query to output to a in-line sub determined output. An alternate idea was to modify the point shapefile output with Delete Features with a query entered at the tool, Make Feature Layer.

Please see the attached model graphic (and toolbox zip containing that model). What I have done is 'similate' your Rasters iterator with a Feature Classes iterator (I do not have the Spatial Analyst extension, and didn't need to simulate that part of the model anyway).

Let me explain:
The Extract Values to Points in your model was substituted for Copy Features in my model to simulate feature class output (points in your case; polygons in mine). The output in-line var sub name for yours is %Name%_raw.shp; for mine it is %Name%_CF (CF stands for Copy Features).

I used Make Feature Layer and Delete Features to operate on the individual CF outputs, essentially removing the features from each those satisfying the query, "PARCELNO" LIKE '2166%', which is a static query (the '%' in this case is a SQL wildcard character).

This ran without error, making each copy for each fc found in the gdb, appending text to the input name, then proceeding to execute the same SQL query on each CF output via Make Feature Layer....once there is a feature layer 'handle' on those records to delete, this is in turn fed into Delete Features (no error trapping provided). The final outputs are the individual CF feature classes with the designated features removed - carried out by Make Feature Layer and Delete Features, 'moderated' or fed by the iterator. So a 2nd 'hard copy' output was not necessary.

Hope that helps...ordinarily I would script this type thing in Python, but this is an interesting way to do it too.
If you're still having problems, I would examine the 'intermediate' output and the query to make sure those are valid. Hope it isn't a bug somewhere....

Thanks,
Wayne

Re: Model Builder - How to remove selected data from a shapefile?

T__WayneWhitley — Thu, 10 Jan 2013 18:15:57 GMT

Forgot the toolbox, in case you want to examine it...see attached.