Select to view content in your preferred language

Forcing da.SearchCursor to retrieve new data?

1834
16
Jump to solution
10-31-2022 12:00 PM
AustinBachurski
Occasional Contributor

I put together a tool that runs in the system tray and compares an attribute table column for a feature class to information pulled from a website to let me know when there's new work to be done.  It works for the most part but I'm having issues with it not updating after I add new information to the table.  I'm using arcpy.da.SearchCursor() in a function to retrieve the data from the table, but each time I call the function, I get the data that was retrieved on the first call, I need it to retrieve the latest data.  I've tried calling the .reset() method on the object it returns, deleting the object, and calling arcpy.ResetEnvironments to no avail.  The only success I have with the tool or in a python terminal is closing the tool/terminal and reopening it.  Is there a way to force SearchCursor to pull new data?   Maybe there's another method that I should be using?  Thanks.

 

def get_mapped_permits():
    building_permits_2022 = "path.to.sde"
    mapped = [each for permit in arcpy.da.SearchCursor(building_permits_2022, "PERMIT_NUM") for each in permit]
    return mapped

 

0 Kudos
1 Solution

Accepted Solutions
AustinBachurski
Occasional Contributor

Well after trying everything suggested and then some including manually deleting the cursor outside of a comprehension, using a context manager, moving it into separate functions, etc. etc.  I finally got it to work the way I wanted it to.  I eventually found out it was something to do with calling it from a while loop.  I'm not 100% sure why, but if I got rid of that it worked as expected.  So, what I got rid of the while loop entirely, and just have the function call itself after sleeping.  From everything I've read it seems like this is an acceptable solution, and it actually works properly.

 

So instead of this.

 

def monitoring():
    while True:
        app.check_now("")
        # Check every 30 minutes.
        time.sleep(1800)

 

 

 

I'm using this.

 

def monitoring():
    """Checks for updates every 30 minutes."""
    app.check_now("auto")
    time.sleep(1800)
    threading.Thread(target=monitoring, daemon=True).start()

 

 

Which calls the searchcursor via.

 

def get_mapped_permits(self):
    """Returns a list of building permits found in the BuildingPermits
    feature class for the current year."""
    with arcpy.da.SearchCursor(self.feature_class, "PERMIT_NUM") as search:
        return [permit[0] for permit in search]

 

 

Thanks for all the help!

 

One last note on this in case it will help anyone in the future, I added another call to the function containing the SearchCursor() and started having the same problem again.  Was able to fix it by changing the function to target an instance variable rather than return the value, then wrapping the function call in a daemon thread (terminology may be incorrect, noob, sry...) this produced the expected result.

def get_mapped_permits(self):
    with arcpy.da.SearchCursor(self.feature_class, "PERMIT_NUM") as search:
        self.mapped_permits = [permit[0] for permit in search]

# called from

self.mapped_permits.clear()
t = threading.Thread(target=self.get_mapped_permits, daemon=True)
t.start()

 

View solution in original post

16 Replies
RhettZufelt
MVP Notable Contributor

Normally, the way to get SearchCursor to "pull new data" is to establish it again.

Not sure how you are calling it, but if I test your code in IDLE shell, it reports all additions to the FC each time I run it (as long as I saved the changes). Of course, I have to replace "path.to.sde" with the path/name to actual feature class).

R_

 

0 Kudos
AustinBachurski
Occasional Contributor

I'm still learning so I may not be answering your question correctly...  The code block in my first post (aside from the path, stored in a variable) is the exact function I'm using, have a while loop running in a thread that calls it every 30 minutes, the exact line I'm using for the call is: 

mapped_permits = get_mapped_permits()

 At first, I thought it was something goofy with the threading, but I tested it in PyCharm using the python console there and it won't update, I can delete all the variables I'm using in the console but still, every time I call it, it returns the same data.

0 Kudos
JoshuaBixby
MVP Esteemed Contributor

Have you tested it outside of an enterprise geodatabase?  Is the code your running and the changes to the data working on the same version of the data?  If I have users modifying data in version X and they don't reconcile and post, I won't see them in version Y.

RhettZufelt
MVP Notable Contributor

Was wondering what you meant "runs in the system tray" and how it is initiated.

However, if it still doesn't update in python console, then something else is going on.  Suspect maybe a version issue as @JoshuaBixby  suggests.

I did my testing on SDE FC with traditional moving edits to base, so any changes are represented on save.

R_

0 Kudos
BlakeTerhune
MVP Regular Contributor

I'm not sure this is the solution to your problem, but the list comprehension on line 3 seems wrong. It should be something like

def get_mapped_permits():
    building_permits_2022 = "path.to.sde"
    mapped = [permit[0] for permit in arcpy.da.SearchCursor(building_permits_2022, "PERMIT_NUM")]
    return mapped
0 Kudos
RhettZufelt
MVP Notable Contributor

I thought this at first also, but testing gives the same result with both list comprehension syntax.

R_

AustinBachurski
Occasional Contributor

The reasoning for the comprehension on line 3 is that the SearchCursor object is a list of tuples, that was what I came up with to get the contents of each tuple as a string, probably a better way to do it, but as stated I'm still learning...  If it helps, I put the code up on github fair warning, I'm new to coding in general so it's probably horrible, but I'd gladly take any criticism - gotta learn somehow.  As far as I'm aware I'm the only one who is editing this particular feature, our organization is fairly small.  There are a couple map services that reference the layer, but I guess I wouldn't expect that to affect this.

AustinBachurski
Occasional Contributor

In response to JoshuaBixby, I did just test it with a scratch database on my local machine and it updates as expected, at least in the console window.  Pretty much says it has to be something to do with the fact that it's on an enterprise SDE or whatever.  Just seems odd that closing/opening the tool would change things if that's the case.  Not sure what the deal is...

0 Kudos
by Anonymous User
Not applicable

Returning the whole row might be keeping the reference to the cursor iterator, which might cause it to not be released and the dataset (cached for speed) to be used again.  Could try moving it out of the list comprehension and explicitly deleting the cursor when the list is built.

I think converting some parts of the script to a Class would help manage the application state better, and then just instantiate a new class in each new cycle to process the data.