Latest Contributions by LindsayRaabe_FPCWA

Preamble In our organisation, we've been using ArcGIS Online since the early days after it's initial release. This means we have many layers that have been around for quite some time. One thing we never really understood early on, and have found to be problematic in the last couple of years, is not having well considered and consistent spatial references for all our feature services.  Whilst the default spatial reference in ArcGIS Online WebMaps is WGS 1984 Web Mercator (auxilary sphere), feature services can be published using any coordinate system ArcGIS Online then projects them on-the-fly in webmaps. This usually is not an issue. We have found it though to be problematic when using the layers in ArcGIS Pro, and when there are a mix of spatial references that use both GDA94 and GDA2020. This mix was introducing data shifts that could not be dealt with through transformation selections. We have since decided the most robust way forward would be to republish everything in ArcGIS Online to use the same projection: WGS 1984 Web Mercator. This way, when loaded into a ArcGIS Pro project, we don't have to worry about incompatibility between GDA94 and 2020 being an issue any more.  NOTE: We still have many non- AGOL datasets that need to be reprojected as well, but thats a much simpler task.  Considerations As I'm sure many could appreciate, making significant changes to many feature services is no small feat. This meant scheduling a freeze period where users would not be able to capture or edit data without risking that data being lost. It also meant that all offline areas in Field Maps would need to be recreated (which is quite a lot in our org)! We estimated it would take our team 3 days to republish our feature services , check them then update impacted WebMaps & ArcGIS Pro templates. We had 102 feature services in total that needed their spatial reference updated. These then had another 120 associated hosted views that needed to be considered. We also wanted to preserve the Editor Tracking history in all our layers (something easily lost without proper consideration).  You may have seen recently issues around Editor Tracking fields and new default high-precision date fields in ArcGIS Online (Solved: Editor Tracking – High Precision Date Fields issue... - Esri Community). To mitigate these issues and speed up the process as much as possible, I built a model builder tool that would download each sub-layer/table in a feature service and recreate the Editor Tracking fields, preserving the creation/editor history and enabling editor tracking on the local (temp) data prior to publishing. This speeds up 's Solution in the above post. These tools are attached for download (one for feature services with and without tracking and one for tables with tracking).  NOTE: It looks for the default Creator/CreationDate/Editor/EditDate fields created in ArcGIS Online. If your fields vary from this, the model will need to be modified to recognise these.  Model to download feature service and maintain editor tracking     Process ArcGIS Pro Setup Open new ArcGIS Pro project and insert a new map Check the “Enable assignment of Unique ID’s” option in the Map Properties > General window Ensure map projection is set to WGS 1984 Web Mercator (auxiliary sphere) Consider using the setting Project > Options > Map and Scene > Spatial Reference > “Choose spatial reference” instead of “Use spatial reference of first operational layer” to prevent the map changing to a different spatial reference when adding data Layer Setup Add hosted feature service to ArcGIS Pro from Portal Run “Download FS with Editor Tracking” tool (W:\Mapping.atbx) NOTE: Select the input feature service by browsing through Portal – don’t select the layer in the map. Ensure you select the sub-layer so that the URL has the layer ID number at the end (e.g. /0) If the tool fails, it is due to the layer not having matching Editor Tracking fields. Run the version of the tool called “Download FS without Editor Tracking” or update the tool to match your editor tracking field names.  If a feature service has more than one sub-layer, run this process for each sublayer. Right click feature service layers you added to the map in Step 1 > Copy then right click the corresponding layer/sublayer output from Step 2 > Paste Properties. Repeat for each downloaded (temp) sublayer until all are are done.  Rename the layers to match the existing feature service sublayers.  Update the new layer properties to match the feature services ID (number) at the end of the service URL (data source) – ensure you do this last as “Paste Properties” will change this   Remove the layer added in Step 1 (existing feature service) leaving only your temporary data Confirm your map projection is still the desired spatial reference Updating Feature Service Go to ArcGIS Online and disable Sync on any Hosted Views (and the base if applicable) – all replicas need to be deregistered or the next step won’t be able to proceed Go to the Share ribbon > Web Layer > Overwrite Web Layer Select the original feature service and click OK In the Configuration section, ensure you check the “Preserve editor tracking info” (unless no editor tracking was found when downloading the feature service) Update the Time Zones (if required)   Analyse the layer for any errors and rectify as required Publish the feature service NOTE: If the republish fails, check that you didn't miss any Sync options on Views. If you can't get it to publish successfully, you may need to publish the layer as a V2 feature service and then change the data source of any associated Views to the new feature service. See "Learnings" below for more details.  Check feature service and all hosted views to ensure that they work (check the Data tab in ArcGIS Online). If you find any views where the Data tab shows “Data Error”, try republishing the layer again (this will likely fix the issue). Also ensure you check that you can select a feature in your feature service View attribute tables and zoom to the features. We found occasionally a view appeared to be working (showed records) but no features were visible in the map. See "Learnings" below for more details.  Reenable Sync on any Views Learnings When overwriting a feature service, you get a warning window that states you may lose all of it's associated data and popups. Data should be retained through the download process undertaken at the beginning. Popups and other styling (symbology, labels, etc) should also be retained through the Copy > Paste Properties steps undertaken earlier. This warning does NOT affect any views of the feature service. They will remain as they were before the layer was overwritten.    As mentioned in Step 8 above, sometimes we successfully overwrote a feature service, but a view may fail to reflect the updates, returning a Data Error on the Data tab of the items page on ArcGIS Online Item page.  Sometimes clicking Publish in ArcGIS Pro again would fix this. If you tried the Publish step again and you still can’t get all Views working, you may need to recreate the view manually and replace it in all WebMaps and ArcGIS Pro templates that it exists in. We found this was easiest done with the old one and new one side-by-side on screen so you can step through the defined view settings (Settings > Update View), Visualisation and Settings pages.   We found that some of our feature services simply refused to be overwritten. These ones we had to publish again from scratch (with "V2" appended to the name - see screenshots below). It's important to remember that you can't change the source of a hosted view to another feature service unless they have the same spatial reference as the original, and all the required fields are present (amongst other things - further reading here: Swap the source of a hosted feature layer view—ArcGIS Online Help | Documentation). Because we were going through this process in order to update spatial references, this meant publishing the V2 layer with the original spatial reference (e.g. GDA94 MGA Zone 50), changing the sources of all the views to the V2 feature service, and then we would go through the above republish (overwrite) process again on the V2 layer to update it's spatial reference (to WGS 1984 Web Mercator) with much more success.      Summary Despite a slow start as we ironed out the kinks in our process and ran at 2/3 team capacity for a while, we soon got on a roll and were churning through layers at a good pace. It certainly helps having someone involved who is intimately knowledgeable about the data being worked with as you can soon land in big mess if you misunderstand the relationships between hosted feature services, their views and the map/app products they exist in (not to mention the potential complications that related tables and other such things bring to the party - thankfully we only had 1 of those).  Overall, it took us 3 days to: Republish (or recreate if necessary) all 100 of the target feature services Check that they (and their views) were functional Check WebMaps and ArcGIS Pro templates for broken data sources and update them Ensure that all offline enabled WebMaps were still ready to work offline (and fix if required) Beyond the 3 day "freeze" period, a few small oversights were found and were fixed (thankfully nothing major) and Experiences were updated to reflect changed data sources (resulting in broken expressions).  ... View more

Hi All. I've been attempting to create an Arcade code that would be used in a Field Maps field calculation. The idea is that the feature being created would intersect with features from a series of layers. The layers to intersect with would all start with the same text, but finish with sequential Australian financial years (e.g. Seeding 2021-2022, Seeding 2022-2023, Seeding 2023-2024...). Once the layers have been filtered (excluding map layers that don't start with that text), the features would be intersected, and the feature with the highest value would be returned as a value for recording in the field.  The value at the end of the layer names is also recorded as an attribute in the layers under "SeedingSeason" field (e.g. 2022-2023).  Does anyone know how to achieve this? I was working with Copilot, which returned a few options, but none of them worked. I suspect it was the listing and filtering of layers in the first instance it was struggling with (there are other layers in the map we don't want).  ... View more

With crunch time coming and ArcMap set to become a thing of the past for many GIS users, even after migrating to ArcGIS Pro for the majority of works, there are still cases where ArcMap has been held on to as a way of accessing datasets that were stored in Personal geodatabases (.mdb). However, once ArcMap is gone, there will be no way to view this data using ArcGIS Pro. If you only have a few .mdb's, this isn't a big deal as you can manually create a new .gdb and migrate your datasets across using the tools already at your disposal in ArcMap.  But....if you have many .mdb's, this soon becomes unfeasible. We fell into this category where we had many archived and backup datasets stored as .mdb's that we may want to access in the future. With ArcMap planned to be uninstalled from all machines across our organisation later this year, we decided it was time to convert these to file geodatabases before that happened to guarantee future accessibility.   To achieve this, I used ArcMap's python window with 2 sets of 2 scripts as detailed below.  NOTE: When running the below scripts to update data on our network, they were very slow to perform almost every task. Moving the data (make a copy!) to the local C drive of my machine sped up the processing time immensely (Many many hours down to minutes!). It also reduced the number of failed copy tasks to almost 0.    Script 1: This script would be copy and pasted into ArcMaps python window. Update the input_folder parameter with the parent folder containing personal geodatabases. This will also loop through all subfolders and their contained .mdb's. The loop passes the .mdb pathway to the second script. Update the second_script_path parameter with the pathway to the second script.  Script 2: Called by script 1, this script is saved in a location of your choosing. It undertakes the creation of a matching .gdb and copies across any datasets, feature classes or tables found in the input .mdb.  The process was broken out into 2 scripts instead of being run in a single instance (script) because I found that after the first folder in the loop, all .mdb's found in subsequent folders would fail to have their data copied, returning many error messages. The second script is executed per .mdb found and this resets all the inputs, removing whatever issue caused it to fail when looped in the same instance.  Script 1     import os import subprocess from datetime import datetime def iterate_folders(input_folder, second_script_path): # Create a log file log_file = os.path.join(r"C:\temp", "MDB_to_GDB_log_{}.txt".format(datetime.now().strftime('%Y%m%d_%H%M%S'))) def log_message(message): print(message) with open(log_file, 'a') as log: log.write(message + '\n') # Iterate through all subfolders and files for root, dirs, files in os.walk(input_folder): for file in files: # Check if the file is a .mdb file if file.endswith(".mdb"): mdb_path = os.path.join(root, file) log_message("Processing MDB: " + mdb_path) # Call the second script to create GDB and copy items subprocess.call(["python", second_script_path, root, log_file]) # Example usage input_folder = r"C:\temp\FoldersWithMDBs" second_script_path = r"C:\Temp\Script2.py" # Specify the full path to the second script if not os.path.exists(input_folder): print("The path " + input_folder + " does not exist.") else: iterate_folders(input_folder, second_script_path)     Script 2     import os import sys import arcpy from datetime import datetime def create_gdb_and_copy_items(folder_path, log_file): def log_message(message): print(message) with open(log_file, 'a') as log: log.write(message + '\n') # Iterate through all files in the folder for file in os.listdir(folder_path): # Check if the file is a .mdb file if file.endswith(".mdb"): mdb_path = os.path.join(folder_path, file) gdb_name = os.path.splitext(file)[0] + ".gdb" gdb_path = os.path.join(folder_path, gdb_name) log_message("Processing MDB: " + mdb_path) # Check if the GDB already exists if not arcpy.Exists(gdb_path): try: # Create a new file geodatabase using arcpy arcpy.management.CreateFileGDB(folder_path, gdb_name) except Exception as e: log_message("Error creating GDB {}: {}".format(gdb_path, str(e))) continue else: log_message("GDB already exists: " + gdb_path) try: # List all datasets in the .mdb arcpy.env.workspace = mdb_path datasets = arcpy.ListDatasets("*", "All") log_message("Datasets: " + str(datasets)) except Exception as e: log_message("Error listing datasets in {}: {}".format(mdb_path, str(e))) datasets = [] try: # Copy each dataset to the new .gdb for dataset in datasets: if not arcpy.Exists(os.path.join(gdb_path, dataset)): arcpy.management.Copy(dataset, os.path.join(gdb_path, dataset)) log_message(str(dataset) + " copied") else: log_message(str(dataset) + " already exists in GDB") except Exception as e: log_message("Error copying dataset {}: {}".format(dataset, str(e))) continue try: # List all feature classes in the .mdb arcpy.env.workspace = mdb_path fcs = arcpy.ListFeatureClasses("*", "All") log_message("Feature classes: " + str(fcs)) except Exception as e: log_message("Error listing feature classes in {}: {}".format(mdb_path, str(e))) fcs = [] try: # Copy each feature class to the new .gdb for fc in fcs: if not arcpy.Exists(os.path.join(gdb_path, fc)): arcpy.management.Copy(fc, os.path.join(gdb_path, fc)) log_message(str(fc) + " copied") else: log_message(str(fc) + " already exists in GDB") except Exception as e: log_message("Error copying feature class {}: {}".format(fc, str(e))) continue try: # List all tables in the .mdb arcpy.env.workspace = mdb_path tables = arcpy.ListTables("*", "All") log_message("Tables: " + str(tables)) except Exception as e: log_message("Error listing tables in {}: {}".format(mdb_path, str(e))) tables = [] try: # Copy each table to the new .gdb, excluding those starting with 'dbo_' for table in tables: if not table.startswith("dbo_"): if not arcpy.Exists(os.path.join(gdb_path, table)): arcpy.management.Copy(table, os.path.join(gdb_path, table)) log_message(str(table) + " copied") else: log_message(str(table) + " already exists in GDB") else: log_message("Skipping table: " + str(table)) except Exception as e: log_message("Error copying table {}: {}".format(table, str(e))) continue log_message("Converted " + mdb_path + " to " + gdb_path) if __name__ == "__main__": if len(sys.argv) != 3: print("Usage: python create_gdb_and_copy_items.py <folder_path> <log_file>") else: folder_path = sys.argv[1] log_file = sys.argv[2] create_gdb_and_copy_items(folder_path, log_file)     Script 2 notes: You may have more than just datasets, feature classes and tables to migrate and thus may need to customise the script to suit your needs. Also, I've had to filter out tables starting with dbo_ as these were connections to long lost databases and caused the script to get stuck trying to access those databases.      The next 2 scripts were intended mainly as a double checking process. Once the first scripts had run, occasionally I would see error messages where a feature class or table failed to copy. This would be caught and logged but the script would continue on. Reviewing the log file helped to identify how many of these failures occurred. One or 2 and you may choose to fix manually.  Script 3: Very similar to Script 1 in that it loops through folders and personal geodatabases. The difference is here is that it would generate the path for .gdb's that should now exist from the located .mdb and would pass this to Script 4 as well. It is executed the same way (by pasting into ArcMaps python window and calls Script 4 using the referenced pathway) Script 4: This process lists all datasets/feature classes/tables in the input .mdb AND .gdb for comparison. If the lists match, nothing happens, but if they don't match, it attempts to copy the missing items across to complete the list.  NOTE: It has the same filter as Script to exclude "dbo_" tables.  Script 3     import os import subprocess from datetime import datetime def iterate_folders(input_folder, second_script_path): # Create a log file log_file = os.path.join(r"C:\temp", "MDB_to_GDB_log_{}.txt".format(datetime.now().strftime('%Y%m%d_%H%M%S'))) def log_message(message): print(message) with open(log_file, 'a') as log: log.write(message + '\n') # Iterate through all subfolders and files for root, dirs, files in os.walk(input_folder): for file in files: # Check if the file is a .mdb file if file.endswith(".mdb"): mdb_path = os.path.join(root, file) gdb_name = os.path.splitext(file)[0] + ".gdb" gdb_path = os.path.join(root, gdb_name) log_message("Processing MDB: " + mdb_path) # Call the second script to compare and copy items subprocess.call(["python", second_script_path, mdb_path, gdb_path, log_file]) # Example usage input_folder = r"C:\temp\FolderWithMDBs" second_script_path = r"C:\Temp\Script 4.py" # Specify the full path to the second script if not os.path.exists(input_folder): def log_message(message): print(message) with open(log_file, 'a') as log: log.write(message + '\n') log_message("The path " + input_folder + " does not exist.") else: iterate_folders(input_folder, second_script_path)     Script 4     import os import sys import arcpy def compare_and_copy_items(mdb_path, gdb_path, log_file): def log_message(message): print(message) with open(log_file, 'a') as log: log.write(message + '\n') try: # Ensure the GDB exists if not arcpy.Exists(gdb_path): log_message("GDB does not exist: " + gdb_path) return # List all datasets in the .mdb arcpy.env.workspace = mdb_path mdb_datasets = arcpy.ListDatasets("*", "All") or [] log_message("MDB Datasets: " + str(mdb_datasets)) mdb_fcs = arcpy.ListFeatureClasses("*", "All") or [] log_message("MDB Feature Classes: " + str(mdb_fcs)) mdb_tables = arcpy.ListTables("*", "All") or [] log_message("MDB Tables: " + str(mdb_tables)) # List all datasets in the .gdb arcpy.env.workspace = gdb_path gdb_datasets = arcpy.ListDatasets("*", "All") or [] log_message("GDB Datasets: " + str(gdb_datasets)) gdb_fcs = arcpy.ListFeatureClasses("*", "All") or [] log_message("GDB Feature Classes: " + str(gdb_fcs)) gdb_tables = arcpy.ListTables("*", "All") or [] log_message("GDB Tables: " + str(gdb_tables)) # Compare and copy missing datasets for dataset in mdb_datasets: if dataset not in gdb_datasets: try: arcpy.management.Copy(os.path.join(mdb_path, dataset), os.path.join(gdb_path, dataset)) log_message("Copied dataset: " + dataset) except Exception as e: log_message("Error copying dataset {}: {}".format(dataset, str(e))) # Compare and copy missing feature classes for fc in mdb_fcs: if fc not in gdb_fcs: try: arcpy.management.Copy(os.path.join(mdb_path, fc), os.path.join(gdb_path, fc)) log_message("Copied feature class: " + fc) except Exception as e: log_message("Error copying feature class {}: {}".format(fc, str(e))) # Compare and copy missing tables, excluding those starting with 'dbo_' for table in mdb_tables: if not table.startswith("dbo_") and table not in gdb_tables: try: arcpy.management.Copy(os.path.join(mdb_path, table), os.path.join(gdb_path, table)) log_message("Copied table: " + table) except Exception as e: log_message("Error copying table {}: {}".format(table, str(e))) else: log_message("Skipping table: " + table) except Exception as e: log_message("Error processing {}: {}".format(mdb_path, str(e))) if __name__ == "__main__": if len(sys.argv) != 4: print("Usage: python compare_and_copy_items.py <mdb_path> <gdb_path> <log_file>") else: mdb_path = sys.argv[1] gdb_path = sys.argv[2] log_file = sys.argv[3] compare_and_copy_items(mdb_path, gdb_path, log_file)       Conclusion I hope that this may be use to someone in their migration efforts. It took a fair bit of trial and error to make it work and consider all the unique situations that occur with differing data management strategies over time, so will likely need to be customised for your particular need. A solid starting point and methodology though will hopefully make your efforts a lot simpler! ... View more