Latest Contributions by Mark_Hotz

‎01-23-2020

Thanks Joshua: Admittedly I agree that I'm somewhat of a newbie to Python scripting and I do have a lot to learn. However, my issue here is really not with the hard coding...I can get this query to work using the physical values. The problem I'm having is getting the script to read the variable with those values that is being passed to that expression. When the syntax is correct I also run into type issues as the program will typically crash with the error message: ERROR 000358: Invalid expression This "might" have something to do with data type, so my problem, I think, is learning more about quotation and how it works when using variables, plus how to pass non-strings to a variable. To help illustrate where I'm at, I can't see the difference between: ("DateTime_2 = '201901' And MMSI = 316012834") - which works. Or fme_name_1b = '201901' record11 = 316012834 ("DateTime_2 = '" + str(fme_name_1b) + "' and MMSI = '" + str(record11) + "'") - which does not work the (record11) by itself won't work in a string with + symbology, as per: expression6 = ('MMSI = "'(record11)'"') The behavior of the script changes when one adds a variable...so how does one pass a numeric value in a variable to expression without adding quotes or converting the value to a string? This part of the code works: ("DateTime_2 = '" + str(fme_name_1b) + "'") The program crashes only on this part: ("'MMSI = '" + str(record11) + "'") - or variations of which: ExecuteError: ERROR 000358: Invalid expression Failed to execute (SelectLayerByAttribute). Suggestions?

‎01-22-2020

I recently came to this forum with the virtually the same question, but now I'm using an "AND" instead of an "OR". For the life of me I just cannot get this to work, and I don't know why. The help I received for my last issue resolved my problem perfectly. But now my query is only slightly more complex and it just won't go...so I'm wondering if a 2nd pair of eyes will help in my case. It seems that I can do this when I hard code everything, but as soon as I try and use a variable it crashes with a very non-descript error message (e.g. Invalid Expression). I have used 6 different expressions (listed and numbered below), and only expression2 and expression6 don't work...and these are the ones I need to work as this code is going to be transplanted into my main Python script once it's functioning. expression1 is what works now after the help I received here...but I thought I could easily get the AND operator working because I now have the code working with an "OR" operator. That didn't happen LOL. Ideally, I would like to use the "AND" Boolean operator in this case (e.g. DateTime_2 = x and MMSI = y)...but I continue to struggle with this. I try to keep my code very simple so it's easy for me to debug...and for the most part I'm able to do this (I'll continue to do this until I get better at Python scripting of course). Can anyone see my errors? Why can I not use expression2 or expression6? I thought expression2 was almost an exact replica of expression1, but I guess not. I use the exact same variable and value "record11" by itself in another script and it's fine...so I'm confident this isn't (by itself) the issue. The query I would like to use would be similar to: DateTime_2 = fme_name_1b and MMSI = record11 import arcpy year_1 = "2019" month_1 = "01" Q2_A = "201901" record11 = 316003484 fme_name_1b = ((year_1) + (month_1)) #fme_name_1b now = 201901 print ("") print ("fme_name_1b = " + str(fme_name_1b)) fc1 = r'F:\ArcGIS_Pro_Projects\AIS\AIS_Data\AIS_Data.gdb\TEST_AIS_Data' #expression1 = ("DateTime_2 = '" + str(Q2_A) + "' or DateTime_2 = '" + str(Q2_B) + "' or DateTime_2 = '" + str(Q2_C) + "'") # WORKS PERFECTLY #expression2 = ("DateTime_2 = '" + str(fme_name_1b) + "' and MMSI = '" + str(record11) + "'") # DOES NOT WORK #expression3 = ("DateTime_2 = '201907' And MMSI = 316012834") # WORKS PERFECTLY #expression4 = ("DateTime_2 = '" + str(Q2_D) + "'") # WORKS PERFECTLY expression5 = ("MMSI = 316003484") # WORKS PERFECTLY #expression6 = ("MMSI = '" + str(record11) + "'") # DOES NOT WORK print ("") print (expression5) print ("") print ("Selecting records in fc1 that correspond to MMSI found in Summary Table") SelectResult = arcpy.SelectLayerByAttribute_management((fc1), "NEW_SELECTION", (expression5))#Select records in feature class based on unique identifier (MMSI) CountTotal = arcpy.GetCount_management((fc1)) CountResult = arcpy.GetCount_management(SelectResult) print ("") print (str(CountResult) + " Of " + str(CountTotal) + " Records Selected")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Thank you

‎01-20-2020

Done.

‎01-20-2020

Dan: Thanks very much for this. I will most definitely experiment with this code...all of the suggestions here actually. I still have a bit to learn about arcpy I see :-). I will spend some time learning more about braces "{ }" too...I wasn't aware of this method but I can certainly see how it would be a powerful tool. It will probably take me a few months, but by the end of 2020 I expect to be a LOT more fluent in Python for ArcGIS. Thanks again.

‎01-20-2020

Randy: I liked this solution as it's so very close to what I already had. I can see I need to definitely work on my double and single quotes...I always seem to have difficulty with those :-). Thanks again

‎01-20-2020

Joshua: It just so happens I was trying (unsuccessfully) to do a query very similar to this but of course much less complex LOL. I just ran into issues when trying to use a value in a variable with the "LIKE" operator. I'm definitely going to experiment with this one as this seems to be something that would work in my case. I need to be able to select multiple records based on a DateTime extracted from the original data using FME. The DateTime data looks something like, "20190401091323", with the last 8 characters pertaining to hours, minutes and seconds etc. However, I only needed to select by year and month (the first 6 characters) but to be able to single out those records after each conversion, and using "LIKE" with a variable, apparently, is not supported. So the only way I managed to do this (with my limited Python knowledge) was using a new field that I had already created for that feature class (DateTime_2) that was only YearMonth, which would be NULL after each conversion (as the previous month's records would have had that already added)...so all records = NULL sufficed to calculate that field to YearMonth. But I also don't like querying that way either. Your method here seems to be a much better way to approach this...so I will definitely revisit my code and rewrite it. Thanks.

‎01-20-2020

Thank you all for such awesome (and timely) advice. I got my script to work, so I'm glad I posted the question. I have only just recently returned to Python scripting so I'm in the process of relearning the syntax. Obviously I need to learn a few things...even simple things, like the difference between single and double quotes, the combinations of these, and when to use them. A couple of these responses have also opened up other options and approaches to solving the same problem...something of which I really appreciate. I'm going to put all of these responses into my "How 2" folder for future reference for sure. I'm scheduled to take the first ESRI Python course in February, on the 20th, with plans to take the more advanced course in probably June if possible. I might as well really sink my teeth into this as the benefits are HUGE when it comes to geoprocessing for sure (and much more). I delved into arcpy in August-September as I needed to write a script that would automate a process from FME data conversion to Portal for ArcGIS publishing. I got it to work entirely in December, but then I ran into problems. The file geodatabase to which FME adds the converted AIS data has grown to be very cumbersome...it is now over 100-million points and I have only completed data from January to August (the plan is to convert data for each year). However, before it's uploaded to Portal I filter it using the MOD option to select 1 in 10 records...so the final file size is much more manageable (I'm thinking that I should perhaps get FME to do this before it adds to the GDB too). What I'm planning to do is every 4 months (using the Select script you helped me fix) select the previous 3 months of AIS data and then dump those records into a Q1, Q2 etc. file geodatabase, which will keep the main file geodatabase at a much more management size. With over 100-million points my script currently takes over 40 hours to run, but it's only 3-4 hours if it's 3-4 months worth of data (the script will just run after midnight on the first of each month). The final data in Portal is then used to create heat maps where our Operations Department can then analyze to see where ship traffic is mostly concentrated in and around our harbor. Thank you very much again

‎01-17-2020

I am using ArcGIS Pro 2.4.3, and just running this script through IDLE for ArcGIS Pro. I am having some difficulty getting a standard expression in Python to return what I need. I have created a simple query to try and run...if I can get this to work then I will insert this into my main program with added functions. When I run the script it only returns 16 records (i.e. records for only for variable Q1_A). I'm trying to get it to return 18 records (which would include Q1_B). import arcpy Q1_A = "CHARLES HAYS" Q1_B = "SMIT CLYDE" fc1 = (r"F:\ArcGIS_Pro_Projects\AIS\AIS_Data\AIS_Data.gdb\TEST_AIS_Data") #expression1 = ("Vessel_Name = 'CHARLES HAYS' Or Vessel_Name = 'SMIT CLYDE'") expression1 = ("Vessel_Name = '" + str(Q1_A) + "'" or "Vessel_Name = '" + str(Q1_B) + "'") print ("") print (expression1) SelectResult = arcpy.SelectLayerByAttribute_management((fc1), "NEW_SELECTION", (expression1)) CountTotal = arcpy.GetCount_management((fc1)) CountResult = arcpy.GetCount_management(SelectResult) print ("") print (str(CountResult) + " Of " + str(CountTotal) + " Records Selected") arcpy.SelectLayerByAttribute_management((fc1), "CLEAR_SELECTION")‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍ Hardcoding it (as per what is commented out) works exactly as I need it to work, and returns 18 records. If I use an "and" instead of an "or" it returns 2 records (only for Q1_B). Logic tells me that using an "or" it should select 18 records (i.e. for both Q1_A and Q1_B). The print statement (e.g. print (expression1)) also only prints the variable Q1_A. What am I doing wrong here? How do I get the variable "expression1" to read the entire string? I know I'm missing something, but for the life of me I just can't see it...and I have been spinning my wheels on this all afternoon. Suggestions? Thank you

‎01-16-2020

When I try to draw a relatively large data set (e.g. 36,000 points) in Portal for ArcGIS at a small scale (e.g. all of North America at about 1:40,000,000 scale), it's so slow it's just not usable. However, I can draw 10-million+ points in a 90,000 km² area (at about 1:1,700,000 scale) in Portal without any issues at all. Why is that? Why does scale seem to play such a big role in GIS drawing performance? I can also draw 200 points of locations all across North America without any performance issues at all...so quantity of points is (obviously) the main part of the performance issue. But why can I draw larger data sets at large scales much more efficiently than moderately smaller data sets at small scales? What's happening on the server that slows this performance? On my desktop (ArcGIS Pro - 2.4.3) the same data stored locally draws very quickly too. It's only when it's running on our server from our federated Server/Portal servers at small scales does large data sets seem to be an issue. Normally I would just cache the data and be done with it...but the user wants to be able to make small edits to the data in future discussions and meetings and of course editing cached data is a problem (I'm surmising that this is probably because one cannot cache a feature layer, only a map image layer?). What do other people do when they want to display large data sets at small scales using Portal for ArcGIS? Thanks

‎12-13-2019

Kory: Thanks for the feedback...most appreciated. I will take a look at your suggestions and video today. This data layer is eventually going to be published to our Portal (from ArcGIS Pro) so I wasn't sure how much this many layers would slow that server down. However, your suggestions might just offer a viable alternative so there's definitely an advantage to learn what you are suggesting for sure. Thanks again.

‎12-12-2019

Thanks Mark: I'm surmising that I have to do the Definition Query for each railroad? I might just go back to the requester and tell them that it cannot be done this way LOL. I mean, I can do it, but there are over 34,000 records in this feature class, and running a Definition Query 522 times would in essence mean that I'm copying that feature class 522 times...it might be a tad bit slow when it reaches Portal :-). Not to worry though...your suggestion still does help. If this is the only way it can be done then I might just show them how to query what they need in Portal instead :-).

‎12-12-2019

Problem solved: My coding was "almost" right LOL. I had neglected to add a variable to my SelecdtLayerByAttribute command, and then use that to display (print) how many records were selected and to then calculate the field. I think I just thought that once selected then (just like it happens in ArcGIS/ArcGIS Pro) it automatically knows what I would have selected. Now this works exactly how it should work. It's slow, but it's going to run after midnight anyway, so it really doesn't matter. Thanks everyone for their help though...everything helped steer me in the right direction.

‎12-12-2019

I was given a spreadsheet of railway data and the data is all broken down by each railway company. The request asks that they be able to turn/toggle on and off each railway. The problem is that there are 522 railway companies in this North American data set (many of which are probably not even active), a sample TOC of which is listed below. I have symbolized the data using the Unique Values option. The data is point data and the rail values are symbolized by Standard Carrier Alpha Code. Is there a way to configure ArcGIS Pro so that I would be able to physically turn on and off each point symbol in the TOC? Or is the only way to achieve this is to create 522 separate feature classes, add them to the TOC in APRX file, and then group them to achieve this? The data for North America looks like this: Currently, my TOC appears as per this segment: Thanks

‎11-25-2019

FC Basson: Thanks for the tip. I will explore this further and hopefully improve this current script (and future code too). I have a few things to learn still, but I believe I'm making progress too. There's writing code, and then there's writing code properly, so I'm always striving to improve. Thanks again.

‎11-25-2019

Luke: Thanks for the response. It's always great when the community can help out. I did narrow down the problem to that one specific Select command...but I still couldn't see exactly where the issue is. However, as per below, a resolution was found. You're right though...the code was good...except for one small oversight :-). The script does indeed work now (and does exactly what I wanted it to do)...and that marks the first time in years where I have successfully written a arcpy script in ArcGIS (ArcGIS Pro in my case). This one script will save me days of work every time I need to run it (the manual alternative is both tedious and time consuming. Thanks again

Online Status	Offline
Date Last Visited	‎04-30-2023 12:47 PM

My Ideas

Latest Contributions by Mark_Hotz

Re: Why am I having another "Select" issue using a variable?

Why am I having another "Select" issue using a variable?

Re: How Perform A Select Using "OR"?

Re: How Perform A Select Using "OR"?

Re: How Perform A Select Using "OR"?

Re: How Perform A Select Using "OR"?

Re: How Perform A Select Using "OR"?

How Perform A Select Using "OR"?

Why is Portal for ArcGIS Drawing Speed so Slow?

Re: Can One Toggle Layer Symbology On and Off?

Re: Can One Toggle Layer Symbology On and Off?

Re: I cannot seem to get SelectLayerAttribute to work using Python?

Can One Toggle Layer Symbology On and Off?

Re: I cannot seem to get SelectLayerAttribute to work using Python?

Re: I cannot seem to get SelectLayerAttribute to work using Python?

Can Geoprocessing be done using SQL Server?

Can ArcGIS Pro convert Ground coordinates to UTM?

Re: Is it possible to edit the Portal for ArcGIS S...

Re: ArcGIS Server Upgrade Question (10.8.1)

Re: ArcGIS Data Store Questions