topic Calculate Accessibility Matrix failing on MakeFeatureLayer in Public Transit Questions

Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sun, 10 Oct 2021 13:43:47 GMT

I'm using the Calculate Accessibility Matrix python tool from version 0.5.1 of the tool box. I'm running this in ArcGIS Pro 2.8.3. I get an error that the parameters are not valid to the gp.MakeFeatureLayer_management method.

The python command I ran was:

arcpy.TransitNetworkAnalysisTools.CalculateAccessibilityMatrix("Origin", "Dest_rand", r"E:\Work\GIS_DB\ArcGIS\NetworkAnalyst\MDC_TPO\MDC_6\MDC_6.gdb\Origin_CalculateAccessibilityMatrix", "TransitNetwork_ND", "Public transit time", 50, "Minutes", "Wednesday", "08:00", "Wednesday", "09:00", 1, 1000, 4, None, None, True)

The Origin layer has just one point in it. The Dest_rand layer has 20 points. I have successfully run other network analyses on the network dataset like Route and Service Area using the Public transit time mode and this day/time.

I'm attaching the messages from the geoprocessing history. I thought it might have to do with a space in the name of one of the folders in the path to the output layer. But I've removed the space and still get the same result. Any thoughts or tips would be greatly appreciated.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Mon, 11 Oct 2021 15:04:05 GMT

Hi Eli. I think you're right that the issue you're having with Prepare Time Lapse Polygons described here is probably the same underlying problem as this issue. I'll try to answer them both here.

My guess is that your machine is having trouble spinning up the parallel processes for some reason. I'm not sure why, but my first question for you is how you're calling this from python. Are you using a standalone python script? The python window in Pro? I suspect this will get us to the heart of the matter.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Tue, 12 Oct 2021 01:58:49 GMT

Hi Melinda. I'm not actually running it on the a command line. I run the tools by double clicking the tool under catalog>toolboxes>Transit Network Analysis Tools>tool. Where tool is Prepare Time Lapse Polygons or Create Accessibility Matrix. I extracted the zipped folder in my ArcGIS project folder, then in the extracted folder I clicked to add the python toolbox to the project toolboxes. Then I run it from there with the nice geoprocessing dialog with the input fields. I just thought it would be easier to show you what I ran if I gave you the command since I can right click>copy python command to clipboard.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Tue, 12 Oct 2021 16:11:10 GMT

Oh, okay. That's fine. Just means my initial idea of where to look for the problem isn't actually the problem. 🙂

I am not sure what's going on. I cannot reproduce the problem. Here's my next suggestion for getting to the bottom of this:

When you run the tool, if you look at the messages, you'll see one that looks like "Intermediate outputs will be written to [blah]." In there will be a bunch of folders with names using strings of letters and numbers (just auto-generated unique names), and in each of those will be a log file. These are the logs for the individual parallel processes. Please take a look and see if you can find the log files and post the files or their contents. We should be able to get more information about that MakeFeatureLayer failure from that.

It looks like when you ran the tool, it finished one process successfully and then failed on the second one for some reason. That's kind of weird. Anyway, since it failed early in the parallel processing, there should only be a few of those folders and log files to look through to find the right one. I'm guessing no more than 4.

You might need to re-run the tool again to generate some fresh log files. I'm not sure if the old ones from your earlier failed run might have been automatically cleaned up with Pro's garbage collection at some point.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Tue, 12 Oct 2021 16:17:54 GMT

By the way, while looking into this issue, I identified and fixed a couple of other, unrelated minor issues. An updated version of the toolbox is available at https://www.arcgis.com/home/item.html?id=23cc1b720eb0418db4d0e61955b8fe43. Among them was fixing the Travel Mode parameter so it auto-populates after you select the network. I had inadvertently messed that up with my most recent update a few weeks ago.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Thu, 14 Oct 2021 01:04:16 GMT

I found the log files. There were 62 folders with one log in each. I've looked maybe 10 of them. Of the ones I've looked at tThey all look like this one except for one which had less info in it.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sat, 16 Oct 2021 17:37:45 GMT

Should I look for a version of the tools from before it started using parallel processing? Is there a way to run the latest code so it does not use parallel processing. I've tried 1, 2 and leaving default 4 for the max number of parallel processes.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Sun, 17 Oct 2021 16:31:03 GMT

Sorry you're having these problems. I was hoping the log files would be more enlightening, but they don't tell us much at all.

If you just want to move along and get your work done, which is completely understandable, you can download the older, ArcMap-compatible version of this toolbox here: https://arcg.is/1u8WKS The tools should work in Pro, and they don't do the multiprocessing thing at all. In fact, they run in a completely different way, so be sure to look at the user's guide to see how to set up your inputs.

If you're willing to keep experimenting and have the time for it, you could add a couple of lines of code to write more info to the log files.

Put these at line 426 of parallel_odcm.py. Be sure to close and reopen Pro (otherwise it won't pick up the changes) before re-running the tool. Then check the log files again. This just prints the input parameters to the logs and might tell us something more.

self.logger.debug(origins_where_clause) self.logger.debug(self.origins) self.logger.debug(self.input_origins_layer)

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sat, 30 Oct 2021 14:24:59 GMT

I just tried adding in the debug lines as shown in the screenshot and it made it so it cant run at all

I also tried adding the lines within the run_gp_tool(arcpy.management.MakeFeatureLayer... parenthesis. Similarly it didn't run and gave the same 9 error messages.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sat, 30 Oct 2021 15:11:25 GMT

New test that also wouldn't run. I tried to add the workspace parameter to the MakeFeatureLayer calls. Seems like the same reason. Meaning, the changes I tried didn't introduce another problem and also didn't fix whatever is going on.

So far this morning I was still using the previous version of the tools. I've downloaded the zip file again today and I'll begin using it for further testing.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sat, 30 Oct 2021 15:35:57 GMT

Downloaded and ran the Calc Accessibility Matrix as is from the 0.5.3 of the toolbox with the same results. Which I think was to be expected. But I did see the drop down list of travel modes for my Transit Network Dataset. At this point I am wondering if I could get a description of a working setup. As in, I'm wondering if by having my project/map folder on a separate drive/partition could be causing the issue.

In the output of the errors I see that the way I have my files set up, the python tools, my gdb etc are on one partition that is not the "C" drive. And some of the python scripts are getting called from C, eg C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\concurrent\futures\_base.py. This _base.py seems to be where I'm having issues.

Also, the ...AppData\Local\Temp\ArcGISProTemp984\scratch\CalcAccMtx... logs etc are going on the C drive.

My next test was to copy my project folder somewhere on C and the results were the same.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Mon, 01 Nov 2021 15:13:37 GMT

Oh, I think this might be my bad. Maybe try this instead:

self.logger.debug(str(origins_where_clause)) self.logger.debug(str(self.origins)) self.logger.debug(str(self.input_origins_layer))

Maybe the new failures are caused because LOGGER doesn't know how to convert those things to strings and print them.

At this point I'm not sure what else to check besides this. I don't think having your data on a different drive or partition is going to make any difference.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Mon, 01 Nov 2021 15:16:12 GMT

I think, based on the error message, there has to be something wrong with the origins_where_clause. Like somehow, the code has created a where clause that for whatever reason is invalid. I don't know how it would do that, but if we can see what the where clause is, that will help.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sat, 06 Nov 2021 22:21:23 GMT

OK. Yes those additional lines of code worked to add the where clause to the log files. The first time I tried it today with those new lines, I had only one origin point in the Origins layer. Because the where clause tried to get all object IDs >= and <= to 1, I thought I would add a couple more. It still crashed out. Logs attached in case it helps.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Sat, 06 Nov 2021 23:08:07 GMT

Sigh...I'm so sorry, Eli. I am completely befuddled by this problem. I have absolutely no idea what's going on.

Thank you for testing with more than one origin. The OBJECTID <= 1 AND OBJECTID >= 1 query expression definitely looked odd, so it was a good thought you had to test with more than one origin. Too bad that didn't end up being the issue.

Typically if there were a problem with one of the input parameters, the Make Feature Layer tool would generate a more specific error, like “ERROR 000732 Input Features: Dataset <blah> does not exist or is not supported” or “ERROR 000358: Invalid expression”. Your logs are showing that the tool returns “ERROR 000816: The tool is not valid”, which doesn't even make sense in this context. I sent a query to the person here at Esri who "owns" the Make Feature Layer tool. Maybe he can shed some light on what circumstances would cause the tool to return this error.

In the meantime, I think your best option is to revert back to using the slower ArcMap version. I'm so sorry!

One last thing to check on your end, if you haven't already: Can you run the Make Feature Layer tool on your origins at all, just normally within an ArcGIS Pro session? Does the tool work at all? If it doesn't, then at least we've found our issue. The best thing to do in that case is probably a) uninstall and reinstall Pro and try again, and if that doesn't help, call Esri Support and ask them why a core tool is not working on your machine.

Actually, for that matter, you could try uninstalling and reinstalling Pro anyway, just in case. Or hang on until Pro 2.9 is released in a week or two and see if that fixes it...?

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

EliBrossell — Sun, 07 Nov 2021 19:07:07 GMT

Hi Melinda, Thanks for all the efforts. Yes, I was able just now to just run MakeFeatureLayer on the Origins layer with no issues.

I wonder if it could be something I have misconfigured in my Transit Network Dataset.

Re: Calculate Accessibility Matrix failing on MakeFeatureLayer

MelindaMorang — Mon, 08 Nov 2021 16:03:45 GMT

No, that's almost certainly not the problem here. The place where the code is erroring out is before it even touches the network dataset. The code is just trying to select a subset of the origins to use for each chunk as is failing for some inexplicable reason.