Model Builder - How to remove selected data from a shapefile?

[RussellQuintero]

I am attempting to automate a process I have historically done manually using model builder.

So far I have managed to successfully convert one shapefile to points, and then iterate through multiple rasters to extract the values of those rasters to points, creating multiple shapefiles.

This is where I run into problems.

Some of those newly created shapefiles contain invalid data, with values of -9999 in a field showing the raster had no data there.  I want to remove that data.

I have tried the Analysis toolbox "Select" tool with the SQL query set to "Field Name" <> -9999, but it returns an error 000229.

I have tried "Feature Class to Feature Class" (FC2FC) with the same SQL query, and it says the geometry is invalid.  When I try and open the file outside of Model Builder using FC2FC, it says it is unusable. If I use "Add Data" to add the shapefile to my current map, and then use FC2FC, it can suddenly use the file just fine.

So my questions here are two-fold:

1) What is the proper way in Model Builder to remove selected data from a shapefile?
2) How do I create a tool in Model Builder that will run without requiring that intermediate data be added to the map?

[T__WayneWhitley]

How about modifying your query with, say, "Field Name" < -999 (or try '=' since that is what you want to delete, correct?), and test to see what selected features there are as a result, use Get Count...if a count returned is the entire feature class count, then you know the query didn't work.

Then you can use Delete Features to get rid of the features once the selected set meets your criteria.

[RussellQuintero]

How about modifying your query with, say, "Field Name" < -999 (or try '=' since that is what you want to delete, correct?), and test to see what selected features there are as a result, use Get Count...if a count returned is the entire feature class count, then you know the query didn't work.

Then you can use Delete Features to get rid of the features once the selected set meets your criteria.

When doing it manually, I've been opening the attribute table, selecting the invalid data, and deleting it.

When doing this in Model Builder, the two options select the data that matches the query, so I need everything except -9999 data. If I select the data "Field Name" = -9999, and then delete the selection, how do I keep the other data?

I did try changing it to >-9999 instead of <> but I get exactly the same error of no supported geometry, which appears to be what it has decided to call the error where it can't open the file when it isn't loaded into the map, as mentioned in my first post. A rather frustrating requirement for an intermediate data product.

[T__WayneWhitley]

When you Delete Features with a selected set on the layer, it should only delete selected - the remainder should be untouched.
Still, it is a good idea to make sure that you have a selection with a count - if there's no selected features, the tool politely deletes the entire dataset and generates 'empty output'...that's kind of alarming!

In short, see this pseudocode:

if not Count < total count:
     print 'whoa, better stop here...tell someone something unexpected has occurred'
else:
     arcpy.deleteFeatures(features)

Results may vary, so always check what's actually going on in your code!  Not sure what you mean about having invalid geometry - do you have points in a layer as you initially stated?...or are you trying to access an intermediate layer which is not persisted?  One more thing, not sure how your gp environment is set up, but you have to have the overwriteoutputs property on.

EDIT-  So if it is the case you cannot access these 'invalid' features, then try Repair Geometry.

[RussellQuintero]

When you Delete Features with a selected set on the layer, it should only delete selected - the remainder should be untouched.
Still, it is a good idea to make sure that you have a selection with a count - if there's no selected features, the tool politely deletes the entire dataset and generates 'empty output'...that's kind of alarming!

In short, see this pseudocode:

if not Count < total count:
     print 'whoa, better stop here...tell someone something unexpected has occurred'
else:
     arcpy.deleteFeatures(features)

Results may vary, so always check what's actually going on in your code!  Not sure what you mean about having invalid geometry - do you have points in a layer as you initially stated?...or are you trying to access an intermediate layer which is not persisted?  One more thing, not sure how your gp environment is set up, but you have to have the overwriteoutputs property on.

EDIT-  So if it is the case you cannot access these 'invalid' features, then try Repair Geometry.

Thanks for all of the responses.

First, let's back up to the use of code at all.  In Model Builder, I'm just dragging in tools. My intention is to output a tool I can use directly from the toolbox, not a python script.

How does one add code like that within model builder?

Now to move on to what I have:

Inputs:
A directory with rasters
A shapefile with line features

I pass the shapefile through "Feature Vertices to Points" to produce a set of point features. This serves as an input to the sub-model.

The submodel takes in the directory and the point features.  An iterator opens each raster in turn, uses "Extract values to points" to extract the raster data onto the points. The output is then used as the input of the "Feature Class to Feature Class" tool.

So yes, the input is an intermediate data item, although it when it crashes it doesn't seem to clean up the intermediate data and the shapefile persists for me to open.

[T__WayneWhitley]

Okay, I think I see - and the pseudocode was just to 'present' the idea, you can add things like checking for preconditions to a model, also Delete Features is a tool you can drag into a model as well...but looks like you may have a problem with your iterator, and possibly when it crashes it is creating 'malformed' output, ghost output, if you will....  It would help if you posted your model, although I am not in the office now to take a look at it.

I suggest this though at least to try to isolate the error - your spatial analyst tool has the syntax:
ExtractValuesToPoints (in_point_features, in_raster, out_point_features, {interpolate_values}, {add_attributes})

...or, say this, if this makes better sense:
ExtractValuesToPoints(inPointFeatures, inRaster, outPointFeatures, "INTERPOLATE", "VALUE_ONLY")

So your iterator must be feeding the raster dataset in for the 2nd param inRaster, shown above -- your inPointFeatures can remain constant but that is where you must be getting 'no value' data, where there is nothing spatially coincident with the currently used raster dataset.  It may be more efficient to 'preprocess' or 'qualify' points (possibly with the extent environment) with the extraction.

Also, you have to somehow provide a mechanism to rename the outPointFeatures on each iteration...how are you including that?  ...could attempt overwriting the same temp fc, but append it or copy it to a permanent location before moving on in the loop; again, must have unique names if copying to a separate fc.

It would help greatly to 'see' your model.  Include any additional error messages and shapefile output, if you can.

[RussellQuintero]

If I remove the last step, "Feature Class to Feature Class," the whole process runs fine and produces a stack of shapefiles with points.

I use the %Name% variable to change the name on each iteration, and for the output of FC2FC, I use %Name%_filtered.shp

So I think the iterator is working, it's a problem with the next step.  Attached is an image of the model. Is there a way to upload the model itself?

[ATTACH=CONFIG]20612[/ATTACH]

[T__WayneWhitley]

You can zip the toolbox in which the model resides...zip it with a few output shapefiles if you can.  ...or if too big, zip separately and post, no need to post your entire output...just a sample.
I may not be able to pinpoint this today (as I explained I am not in the office), but I bet once you post your zip, there are ModelBuilder experts that can fix it for you quickly!  I still suspect your iterator -- are you sure it is 'aware' of the FCtoFC part of the process??

(oops, I meant go to Windows Explorer, select and zip the toolbox [tbx extension], etc.)

[RussellQuintero]

You can zip the toolbox in which the model resides...zip it with a few output shapefiles if you can.  ...or if too big, zip separately and post, no need to post your entire output...just a sample.
I may not be able to pinpoint this today (as I explained I am not in the office), but I bet once you post your zip, there are ModelBuilder experts that can fix it for you quickly!  I still suspect your iterator -- are you sure it is 'aware' of the FCtoFC part of the process??

(oops, I meant go to Windows Explorer, select and zip the toolbox [tbx extension], etc.)

All of the data was too big for the size limits. I uploaded it to google drive here:

https://docs.google.com/folder/d/0B_wIIORblfXkMENRMlNISm94Mm8/edit

shoreline.zip contains the shapefile input.
dem.zip contains two DEMs for use as the contents of a folder for the iterator's needed directory input.

I've been experimenting with Model Builder a bit, so there are some unneeded tools in that toolbox.  The relevant two are:

"Lidar Shoreline Slope Step 1" and "DEM to Slope Points (Sub-model)"

[T__WayneWhitley]

OK, sorry about that, I didn't mean for you to load everything - what's most important is to examine the 'guts' of the model, for extra I wanted to see a couple of your output shapefiles......I'm fairly sure now it is linked to your iterator and how the Value produced from it is used for variable substitution.  Just looking again at your last post, the graphic of the model, it doesn't appear you are substituting output names for FCtoFC sensitive to Value.  When the model iterates, I think it crashes because the last part is attempting to be a part of the iteration, but the output naming convention doesn't make sense.

Just curious, what is the Value text typically...no file extension, correct, and just a name?  If Value contains 'extra parts', their is a Parse Path tool to strip out what you need.  I think you're okay, but have to be careful and check that when mixing data types as in your model (from raster output, then shapefile output)....to simplify presentation undoubtedly, many of the webhelp examples deal with the same datatype.

So I'd look carefully at your inline variable substitution.

I downloaded your tbx zip, will take a look at that and your shapefile...let me look at that 1st.

EDIT:  I said Value but meant Name that you are using for in-line substitutions.