topic Re: Updating 2 tables related by 3 ID fields in Python Questions

Updating 2 tables related by 3 ID fields

DavinShokes1 — Sat, 11 Dec 2021 22:07:13 GMT

I'm having a little trouble setting up this script. Here are the bullet points.

2 Tables have the same three fields. [SiteID], [EasyID], [FeatureID]. For simplicity, lets pretend these are the only fields (besides OID).
[FeatureID] is a unique integer for each feature created in Table 1.
[SiteID] is a non-unique integer and groups features by location.
[EasyID] is a string (typically a number) unique within the same [SiteID], but not unique in the field.
Table 1 - The master table. New features have all three ID fields populated.
Table 2 - New features always have [SiteID], sometimes [EasyID], and never a [FeatureID].
Table 2 - Features with [SiteID] and [EasyID], and no [FeatureID] may have a matching [SiteID] and [EasyID] in Table 1. If so, update [FeatureID] in Table 2.
Table 2 - Features missing [EasyID] need to be assigned the next number for that [SiteID] in either Table 1 or 2. If current EasyID's for a site are '1', '3', '4A', '6-7','S', and '55', new features would be '2', '4', '5', '6', '7', '8'....etc.
Table 2 - New features given a [FeatureID] when inserted into Table 1. The features are then updated in Table 2.
Table 1 - Finally, any features with a [SiteID] in Table 2, but the [FeatureID] and [EasyID] are not in Table 2, are inserted into Table 2
At the end of the script both tables should match.

I've looked through using model builder with no luck. With python dictionaries and search cursors i'm getting stuck trying to join against [SiteID] and [EasyID] at the same time. I am also don't know how to return the dictionaries with just integer EasyID's and loop through updating the next smallest integer.

Here is what I've got so far. Much of it stems from what I read from Richard Fairhurst's post Turbo Charging Data Manipulation with Python Cursors and Dictionaries

import arcpy

#Tables
T1 = r"C:\Python\Scratch.gdb\Table1"
T2 = r"C:\Python\Scratch.gdb\Table2"
fields = ["FeatureID", "SiteID", "EasyID"]

# Get FeatureID dictionaries for each Table
T1Dict = {r[0]:(r[0:]) for r in arcpy.da.SearchCursor(T1, fields)}
T2Dict = {r[0]:(r[0:]) for r in arcpy.da.SearchCursor(T2, fields)}

# Get SiteID+EasyID dictionaries for each Table
T1ConcatDict = {str(r[1]) + "," + str(r[2]):(r[0]) for r in arcpy.da.SearchCursor(T1, fields)}
T2ConcatDict = {str(r[1]) + "," + str(r[2]):(r[0]) for r in arcpy.da.SearchCursor(T2, fields)}

#First, If T2.FeatureID is Null but T2.EasyID and T2.SiteID are in T1, Update T2.FeatureID
with arcpy.da.UpdateCursor(T2, fields) as updateRows:  
    for updateRow in updateRows:  
        # store the Join value by combining 3 field values of the row being updated in a keyValue variable  
        keyValue = str(updateRow[1]) + "," + str(updateRow[2])  
        # verify that the keyValue is in the Dictionary  
        if keyValue in T1ConcatDict & updateRow[0] is None & updateRow[1] is not None:  
            # transfer the value stored under the keyValue from the dictionary to the updated field: FeatureID.  
            updateRow[0] = T1ConcatDict[keyValue][0]  
            updateRows.updateRow(updateRow)  

#Rebuild Dictionary if it is needed again
T2ConcatDict = {str(r[1]) + "," + str(r[2]):(r[0]) for r in arcpy.da.SearchCursor(T2, fields)}
T2Dict = {r[0]:(r[0:]) for r in arcpy.da.SearchCursor(T2, fields)}


'''
#Get Max(EasyID) within SiteID
NumberList = []
for value in T1Dict[2]:
    try:
        NumberList.append(int(value))
    except ValueError:
        continue
T1EasyNumberDict = [s[2] for s in T1Dict[2] if s.isdigit()]
T1MaxEasyDict = max(T1EasyNumberDict)
'''


#Second, If T2.FeatureID and T2.EasyID are Null, Update T2.EasyID with next smallest number (as string) in either T1 or T2 for the specific SiteID
with arcpy.da.UpdateCursor(T2, fields) as updateRows:  
    for updateRow in updateRows:  
        # store the Join value by combining 3 field values of the row being updated in a keyValue variable  
        keyValue = str(updateRow[1]) + "," + str(updateRow[2])  
        # verify that the keyValue is in the Dictionary  
        if keyValue in T1Dict & updateRow[0] is None & updateRow[1] is None:  
            # transfer the value stored under the keyValue from the dictionary to the updated field.  
            # Perhaps retrieve max int occurs here?
  updateRow[2] = max(T1Dict[keyValue][2], T2Dict[keyValue[2]) 
            updateRows.updateRow(updateRow)  


#Third, Insert into T1 if T2.SiteID is not null and T2.EasyID is not null


#Forth, Update T2.FeatureID with T1.FeatureID from previous insert where T2.SiteID=T1.SiteID and T2.EasyID=T1.EasyID


#Lastly, Insert any T1 features into T2 where T1.EasyID not in (Select EasyID from T2 where T2.SiteID = T1.SiteID) and T1.FeatureID  not in (Select FeatureID from T2)

Thank you for any advise.

Edit: Here is a sample of sites with 2 explanatory columns

T1.FeatureID	T2.FeatureID	T1.SiteID	T2.SiteID	T1.EasyID	T2.EasyID	Status	Result
358589	358589	136238	136238	1	1	Existing T1 and T2 Feature	No Change
358590	358590	136238	136238	2	2	Existing T1 and T2 Feature	No Change
358594	358594	136238	136238	4	4	Existing T1 and T2 Feature	No Change
652538	652538	136238	136238	5	5	Existing T1 and T2 Feature	No Change
486028	486028	136238	136238	8	8	Existing T1 and T2 Feature	No Change
486029	486029	136238	136238	9	9	Existing T1 and T2 Feature	No Change
525300	525300	136238	136238	34	34	Existing T1 and T2 Feature	No Change
574802	574802	136238	136238	998	998	Existing T1 and T2 Feature	No Change
670911		136238		300		New T1 Feature	(Step 6) Inserted Into T2
493840		136238		9996		New T1 Feature	(Step 6) Inserted Into T2
493839		136238		9997		New T1 Feature	(Step 6) Inserted Into T2
493831		136238		9999		New T1 Feature	(Step 6) Inserted Into T2
696019		136238		105-106		New T1 Feature	(Step 6) Inserted Into T2
696037		136238		9999N		New T1 Feature	(Step 6) Inserted Into T2
696014		136238		Area1		New T1 Feature	(Step 6) Inserted Into T2
670910		136238		N		New T1 Feature	(Step 6) Inserted Into T2
580636		136238		N30c		New T1 Feature	(Step 6) Inserted Into T2
360401		136401		AC		Exiting T1 Feature	(Skip) Not Inserted since no T2.SiteID match
360402		136401		SP		Exiting T1 Feature	(Skip) Not Inserted since no T2.SiteID match
360510		136427		Area 1		Exiting T1 Feature	(Skip) Not Inserted since no T2.SiteID match
362653		136635		15		Exiting T1 Feature	(Skip) Not Inserted since no T2.SiteID match
362943	362943	136698	136698	1	1	Existing T1 and T2 Feature	No Change
362944	362944	136698	136698	2	2	Existing T1 and T2 Feature	No Change
362945	362945	136698	136698	3	3	Existing T1 and T2 Feature	No Change
362946	362946	136698	136698	4	4	Existing T1 and T2 Feature	No Change
362947	362947	136698	136698	5	5	Existing T1 and T2 Feature	No Change
362950	362950	136698	136698	11C	11C	Existing T1 and T2 Feature	No Change
362948	362948	136698	136698	8		New T2 Feature, Exists in T1	(Step 5) Update T2.EasyID
362949	362949	136698	136698	9		New T2 Feature, Exists in T1	(Step 5) Update T2.EasyID
362951		136698	136698	15	15	New T2 Feature, Exists in T1	(Step 1) Update T2.FeatureID
362954		136698	136698	16	16	New T2 Feature, Exists in T1	(Step 1) Update T2.FeatureID
362955		136698	136698	17	17	New T2 Feature, Exists in T1	(Step 1) Update T2.FeatureID
362956		136698	136698	18	18	New T2 Feature, Exists in T1	(Step 1) Update T2.FeatureID
362957		136698	136698	19	19	New T2 Feature, Exists in T1	(Step 1) Update T2.FeatureID
			136698		20	New T2 Feature	(Step 3,4) Insterted into T1, Update T2.FeatureID
			136698		21	New T2 Feature	(Step 3,4) Insterted into T1, Update T2.FeatureID
			136698		22	New T2 Feature	(Step 3,4) Insterted into T1, Update T2.FeatureID
			136698		25	New T2 Feature	(Step 3,4) Insterted into T1, Update T2.FeatureID
			136698			New T2 Feature	(Step 2,3,4) Get next lowest integer for T2.EasyID -> 6
			136698			New T2 Feature	(Step 2,3,4) Get next lowest integer for T2.EasyID -> 7
			136698			New T2 Feature	(Step 2,3,4) Get next lowest integer for T2.EasyID -> 10
			136698			New T2 Feature	(Step 2,3,4) Get next lowest integer for T2.EasyID -> 11

Re: Updating 2 tables related by 3 ID fields

XanderBakker — Thu, 26 Feb 2015 12:14:15 GMT

I think this is one of these cases where it would be so valuable to have to sample data attached to the thread. Also you could mention Richard Fairhurst‌ (see Tagging people, places, and content within your post ) since you based the script on his post. This will send him a notification and he might be able to look at your code.

Anyway, please attach a sample of the data, to be able to debug the code.

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Thu, 26 Feb 2015 13:39:40 GMT

In what way are you getting stuck trying to join against [SiteID] and [EasyID] at the same time? Are you getting errors? The basic approach to a combined key will work as shown for the first 25 lines of code (I can't follow your overall logic beyond that).

Null values in the key most likely cause most of problems, so I would restructure the code order to deal with the second part of your script first to fill in Null values in the EasyID field before worrying about the FeatureID field at all. That involves processing a list of EasyIDs in the single key dictionary of just SideID key values first to verifying the unique value assumption for the non-Null EasyID values as well as filling in the Null EasyID values. Don't build dictionaries for T2 at all until they can be used. In any case, the cursor dictionary approach is your best option and can handle this whole set of processes, but each must happen in the correct order to avoid faulty assumptions about what set of fields contain unique values at each stage of the script.

Clearly you are dealing with a high complex interrelationship between these two tables and a large set of rules that I have yet to understand. I have no context how these records and values came into existence of what uses they will serve in the future. More crucially, You have given me no information about the interrelationship this script has with user actions. Every step you expect a user to do or not do creates a point of failure for your script and any of your rules and assumptions. If the user has to manually set off the script, you must always start by verifying they did everything you expected them to do and didn't do anything you didn't expect them to do related to your script assumptions.

Also, Xander is correct that mentioning my full name in a post puts a message in my inbox, which is the reason I saw this post when he did that.

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Thu, 26 Feb 2015 15:09:17 GMT

Good points Xander Bakker‌. I have included example info and fixed the mentioning. Thanks.

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Thu, 26 Feb 2015 15:43:35 GMT

What distinguishes the skipped records from the records that were step 6 inserted into the T2 table? I see no way to keep track of what is new in T1 since the last time the script was run to make that choice. All of the T2 insertions and the skipped records have no SiteID, so that is not a difference to make that choice. I don't agree that you have the steps in the correct order from what I can see. Your logic appears backwards, since normally I would deal with Nulls and new verses old records as my first steps in any comparison script, not towards the end. My scripts deal with new verses old by renaming existing data and deriving current data from another source, so that the comparison is easy to make. Possibly a variation of that approach would apply here, so that a last run version of the data is created so you can make sure you know what is really new and what you have processed before.

I would also add the ObjectID field for both tables to your field list and reorder the fields as:

["SideID", "EasyID", "FeatureID", "OID@"]

The ObjectID would be used in subroutines to impose order on Null values and to validate your assumptions of unique keys.

Anyway you never said if errors are occurring of if just unexpected values are being assigned. Unexpected values indicate a logic failure, while errors indicate a syntax or data validation failure. I am almost certain you will experience many logic errors developing and testing the script since there are so many dependencies at each stage that have to be considered, so develop only on test data and back up your data before trying it out on your live data.

My blog avoided going into anything this complex, because it hopes to make the core of the principles and the approach I was demonstrating easy to follow. You may want to look at this post to see an example of where I adapted the approach to deal with a much more complex many-to-many relationship between tables for further ideas about ways to vary the basic approach outlined in the Blog.

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Thu, 26 Feb 2015 16:21:48 GMT

This script will run at night. The T1 features that don't have a Site ID in T2 are basically ignored until the T2 user adds the first feature with that SiteID. The user may have manually matched the SiteID and EasyID before the script runs. Additionally, T2 users adding the first occurances of features in either table will skip adding EasyID and rely on unique ones being generated.

One other note is once a feature has all 3 ID'S it will never be changed by the users.

Perhaps if I pre filter T1 to T2 SiteID, the script would be simpler. I believe you have identified step 1 as redundant since it occurs later and max(EasyID) is gained across both tables.

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Fri, 27 Feb 2015 15:40:35 GMT

Richard,

I haven't gotten far enough to get any errors. I don't know how to build/filter the dictionary for T2.OID T2.SiteID string EasyID, then gather only integer EasyIDs, and return the 1st missing integer starting from 1. Without it I can't test updating Null EasyIDs in new T2 features.

Thank you for posting that link to stackexchange! It looks very similar the components of my scenario. I will update after testing the code.

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Sat, 11 Dec 2021 22:07:16 GMT

Your approach still mystifies me and I still don't understand your business rules. Your rules may make sense to you and may be correct for your business needs, but on the surface they at least partially conflict with my experience in synchronizing data and matching tables. The picture in your head of how everything should work is not transferring into mine yet.

I highly recommend that you reconsider the rule that says:

Table 2 - Features missing [EasyID] need to be assigned the next number for that [SiteID] in either Table 1 or 2. If current EasyID's for a site are '1', '3', '4A', '6-7','S', and '55', new features would be '2', '4', '5', '6', '7', '8'....etc.

The EasyID is anything but easy to understand or program as you have described it. Filling in these blanks this way seems arbitrary to me especially given that the natural sort of the strings is actually '1', '3', '4A', '55', '6-7', 'S'. Why fill in blanks at all? Over time that means any deleted records will have their SiteID + EasyID combination reused for an entirely unrelated record, and therefore that key is only unique within the snapshot in time before the script reuses it. In other words, you will never be able to use the SiteID + EasyID key if you ever have to compare two different data snapshots that were taken before and after the script ran. This rule may make sense to you, but in my experience this is a bad database practice. Unique keys (single or multi-field) are only valuable in my experience if they are unique to one record over all time or support actual data relationships and become a problem if they are ever reused for completely unrelated records. I personally don't want to help implement this rule, since its seems excessively complicated to me, and I believe from experience that a day will come when you will want to use that key to recover from a data corruption event and the code that implements this rule will make that recovery nearly impossible. You also will greatly increase the likelihood of creating data corruption if you accidentally link together two snapshots that reassigned the same keys to different records.

I have several other questions about this EasyID field. How many characters are allowed in this field? Why does it contain letters and what is the significance of those letters? Why are there dashes to combine two numbers? Since this field is a string field, how do your users handle the fact that it will never sort numerically in any table, since you don't include leading spaces or strings to right-justify them? What type of business are you working for where this business process was developed to track any of this data in either table?

So what little I do understand (or think I understand) I will try to present some code that should fit your needs. This code is more or less what I would start with. Key fields always should come first in the field list and value fields always should follow. I would incorporate the OID field into the code processes and dictionaries as a fail safe unique key for linking back to the original table where ever the user defined keys turn out to be duplicated and not unique.

The code below handles both a 1:1 and 1:M relationship possibility, so even if the key value is not unique you will be able to trap that and fix it.

import arcpy
import sys

#Tables  
T1 = r"C:\Python\Scratch.gdb\Table1"  
T2 = r"C:\Python\Scratch.gdb\Table2"  
fields = ["SiteID", "EasyID", "FeatureID", "OID@"]  

# Intialize T1 as a dictionary
T1Dict = {}
# Initialize a list to hold any concatenated key duplicates found
T1KeyDups = []
# Open a search cursor and iterate rows
with arcpy.da.SearchCursor(T1, fields) as searchRows:  
  for searchRow in searchRows:
    # Build a composite key value from 2 fields
    keyValue = '{};{}'.format(searchRow[0], searchRow[1])
    if not keyValue in T1Dict:
      # Key not in dictionary.  Add Key pointing to a list of a list of field values
      T1Dict[keyValue] = [list(searchRow[2:])]  
    else:  
      # Key in dictionary is not unique.
      T1KeyDups.append(keyValue)
      # Append a list of field values to the list the Key points to
      T1Dict[keyValue].append(list(searchRow[2:]))
del searchRows, searchRow

# Sample of how to access the keys, record count, and record values of the dictionary
for keyValue in T1Dict.keys():
  for i in range(0, len(T1Dict[keyValue])):
    print "The SiteID;EasyID key is {} with {} record(s).  Record {} has FeatureID {} and ObjectID {}.".format(keyValue, len(T1Dict[keyValue]), i+1, T1Dict[keyValue][0], T1Dict[keyValue][1])

if len(T1KeyDups) > 0:
  # Duplicate keys exist in T1
  # Give a warning and either exit the script or else do a fix of T1 before proceeding
  print("Duplicate keys found!  They are:")
  for keyValue in T1KeyDups:  
    for i in range(0, len(T1Dict[keyValue])):  
      print "The SiteID;EasyID key is {} with {} record(s).  Record {} has FeatureID {} and ObjectID {}.".format(keyValue, len(T1Dict[keyValue]), i+1, T1Dict[keyValue][0], T1Dict[keyValue][1])
  # Either exit or fix T1 here
  sys.exit(-1)

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Sat, 11 Dec 2021 22:07:19 GMT

if you are determined to implement the rule as you have describe it here is some code that should work to get a siteID dictionary for T2 that should find the sorted list of integers that are not yet used for each SiteID and assign the next unused number to the Null EasyID records associated with each SiteID. Also, do not bother filtering cursors if you intend to put it in a dictionary. It is faster to put everything into the dictionary and then do all of the logic tests, type validations, and list tracking in code. You might filter for the Null EasyID values prior running the updateCursor, but even if you have 1 million records to process they will take only about 5 minutes for the update cursor to run through all of them (and the SQL that filters for Null values might take longer, since Null values queries run pretty slow, especially if EasyID is not indexed).

import arcpy  
import sys  
  
#Tables    
T1 = r"C:\Python\Scratch.gdb\Table1"    
T2 = r"C:\Python\Scratch.gdb\Table2"    
fields = ["SiteID", "EasyID", "FeatureID", "OID@"]    

# Get the list of easyIDs associated with each SiteID in a dictionary for T1
T1SiteIDDict = {}
with arcpy.da.SearchCursor(T1, fields) as searchRows:    
  for searchRow in searchRows:  
    keyValue = searchRow[0]
    if not keyValue in T1Dict:  
      # Key not in dictionary.  Add Key pointing to a list of a list of field values  
      T1SiteIDDict[keyValue] = [searchRow[1]]    
    else:    
      # Append a list of field values to the list the Key points to  
      T1SiteIDDict[keyValue].append(searchRow[1])  
del searchRows, searchRow

# Get the list of easyIDs associated with each SiteID in a dictionary for T2
T2SiteIDDict = {}
with arcpy.da.SearchCursor(T1, fields) as searchRows:    
  for searchRow in searchRows:  
    keyValue = searchRow[0]
    if not keyValue in T1Dict:  
      # Key not in dictionary.  Add Key pointing to a list of a list of field values  
      T2SiteIDDict[keyValue] = [searchRow[1]]    
    else:    
      # Append a list of field values to the list the Key points to  
      T2SiteIDDict[keyValue].append(searchRow[1])  
del searchRows, searchRow

SideIDDict = {}
for keyValue in T2SiteIDDict.keys():
  intList = []
  for easyID in T2SiteIDDict[keyValue]:
    if easyID.isnumeric():
      # easyID is a number
      if float(easyID) == int(easyID):
        # easyID is an integer, so add it to the list
        intList.append(int(easyID))
  if keyValue in T1SiteIDDict:
    for easyID in T1SiteIDDict[keyValue]:
      if easyID.isnumeric():
        # easyID is a number
        if float(easyID) == int(easyID):
          # easyID is an integer, so add it to the list
          intList.append(int(easyID))
  # remove already used numbers out of the numbers from 1 to 9999
  # and get a sorted list stored for each SiteID
  SiteIDDict[keyValue] = sorted(set(range(1, 10000)) - set(intList))


with arcpy.da.UpdateCursor(T2, fields) as updateRows:    
  for updateRow in updateRows:  
    if updateRow[1] == None:
      templist = SiteIDDict[updateRow[0]]
      updateRow[1] = templist[0]
      # I believe updates of the list affect the list in the dictionary
      templist.remove(updateRow[1])
    updateRows.updateRow(updateRow)

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Sat, 28 Feb 2015 00:02:18 GMT

Wow thank you for these excellent resources! I will try to digest this over the weekend, but this looks very promising. Sorry that I didn't provide all of the aspects of this dilemma. I tried to simplify it enough that python masters like yourself wouldn't have to read a novel. As you ascertained, this is complex.

The business table (T1) sits on a 2005 SQL server, unsupported since 10.2. EasyID is a 7 char string used for labeling at a site and it is the syncing bane of my existence. Users are allowed to assign "C5-C8" so that FeatureID represents numerous real world things. C5, C6, C7, and C8 could also exist in T1 individually. It is a data quality mess. To combat this the groupings will not make it to T2. Instead C5, C6, C7, and/or C8 will be digitized by the T2 user, if so desired.

Luckily, users are unable to delete rows and can't view or change the FeatureID or SiteID. The other table (T2) and a copy of the Sites table are layers in a hosted service for collecting new site features. I update the Sites with a scheduled script and thanks to Collector for ArcGIS 10.3 honoring relationships, all features maintain their SiteID.

In short, users are limited to picking an EasyID during feature creation in either table and they could possibly update EasyID on T2. On the off chance they do change EasyID on a feature existing in T1 and T2 there will be a script that joins by FeatureID and updates the one with an earlier edit date.

I could go on and on, but don't want to spoil your weekend. Thanks again Richard Fairhurst‌!

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Sat, 11 Dec 2021 22:07:23 GMT

I've made some headway. My script is successfully insertings new rows into sql server with pyodbc. However, I've got an encoding issue when updating the hosted layer. The sql table varchar columns and I assume that hosted layers are nvarchar. The call to update the hosted layer succeeds but the text column doesn't actually update. Integer fields do.

Here is my code I've kept the actual column names so the SQL Server table fields are:

SiteID-->CustomerID, FeatureID-->PropertyID, and EasyID--> Location

The hosted layer fields are:

SiteID-->CustomerID, FeatureID-->PropertyID_WO, and EasyID--> TAG

I've match the sequence to prevent name issues. I'm don't know how to go further without figuring out how to update TAG with Location.

# -*- coding: utf-8 -*-
# ---------------------------------------------------------------------------
# 


import sys, os, datetime, arcpy, pyodbc, time, uuid


def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60.
    return "{}:{:>02}:{:>05.2f}".format(h, m, s)
# End hms_string


def timedif(end_datetime, start_datetime):
    seconds_elapsed = (end_datetime - start_datetime).total_seconds()
    return hms_string(seconds_elapsed)
# End timedif
start_time = datetime.datetime.now()
print "# Script start: {}".format(start_time)


from arcpy import env, da
from arcrest.security import AGOLTokenSecurityHandler
from arcrest.agol import FeatureService
from arcrest.agol import FeatureLayer
from arcrest.common.filters import LayerDefinitionFilter
from datetime import timedelta


print "# Imports loaded."


# Set Geoprocessing environments
Scratch_gdb = r'C:\Python\scratch.gdb'
arcpy.env.scratchWorkspace = Scratch_gdb
arcpy.env.workspace = Scratch_gdb




Local_Table = r'C:\Users\dkshokes\AppData\Roaming\ESRI\Desktop10.3\ArcCatalog\RTC03.sde\Rainbow.dbo.Property'
Output_Table_1 = "T1"
Expression_1 = r'"CustomerID" = 166602 AND "Location" is not Null'
Online_Trees = r'http://services.arcgis.com/###/arcgis/rest/services/###/FeatureServer/0'
username = "####"
password = "####"
proxy_port = None
proxy_url = None
agolSH = AGOLTokenSecurityHandler(username=username, password=password)
Output_Table_2 = "T2"
Expression_2 = '"CustomerID" = 166602'
fields_1 = ['CustomerID', 'Location', 'PropertyID', 'OID@', 'Description', 'CBH', 'PropertyQuantity', 'PropertyItemID', 'DateEntered', 'LastChangeDate', 'Special_Note']
fields_2 = ['CustomerID', 'TAG', 'PropertyID_WO', 'OID@', 'COMMON', 'DBH', 'QTY', 'PropertyItemID', 'CreationDate', 'EditDate', 'Notes', 'Creator', 'Editor', 'PruneYear', 'ProposalStatus', 'SoilYear']
print "\n# Online variables loaded.  "


# Local variables:
T1 = Scratch_gdb + "\\" + Output_Table_1
print "# T1 Table: {}".format(T1)
print "# T1 Fields: {}".format(fields_1)
T2 = Scratch_gdb + "\\" + Output_Table_2
print "# T2 Table: {}".format(T2)
print "# T2 Fields: {}".format(fields_2)
FieldMap_1 = r"PropertyID \"PropertyID\" true true false 50 Long 0 10 ,First,#," + Local_Table + ",PropertyID,-1,-1;CustomerID \"CustomerID\" true true false 4 Long 0 10 ,First,#," + Local_Table + ",CustomerID,-1,-1;Location \"Location\" true true false 7 Text 0 0 ,First,#," + Local_Table + ",Location,-1,-1;PropertyItemID \"PropertyItemID\" true true false 4 Long 0 10 ,First,#," + Local_Table + ",PropertyItemID,-1,-1;Description \"Description\" true true false 255 Text 0 0 ,First,#," + Local_Table + ",Description,-1,-1;PropertyQuantity \"PropertyQuantity\" true true false 4 Long 0 10 ,First,#," + Local_Table + ",PropertyQuantity,-1,-1;CBH \"CBH\" true true false 8 Double 1 18 ,First,#," + Local_Table + ",CBH,-1,-1;Special_Note \"Special_Note\" true true false 255 Text 0 0 ,First,#," + Local_Table + ",Special_Note,-1,-1;Status \"Status\" true true false 10 Text 0 0 ,First,#," + Local_Table + ",Status,-1,-1;DateEntered \"DateEntered\" true true false 36 Date 0 0 ,First,#," + Local_Table + ",DateEntered,-1,-1;PropertyGroup \"PropertyGroup\" true true false 2 Short 0 1 ,First,#," + Local_Table + ",PropertyGroup,-1,-1;Removed \"Removed\" true true false 2 Short 0 1 ,First,#," + Local_Table + ",Removed,-1,-1;LastChangeDate \"LastChangeDate\" true true false 36 Date 0 0 ,First,#," + Local_Table + ",LastChangeDate,-1,-1"


if arcpy.Exists(T1):
    arcpy.Delete_management(T1)
    print "Previous T1 Found so it was deleted."
if arcpy.Exists(T2):
    arcpy.Delete_management(T2)
    print "Previous T2 Found so it was deleted."


#print "\n# Local variables loaded...Making temporary business table."
arcpy.TableToTable_conversion(Local_Table, Scratch_gdb, Output_Table_1, Expression_1, FieldMap_1, "")
#print "# Successfully made temporary business table."


#print "\n# Querying Feature Layer and Making Temporary Feature Class."
fl = FeatureLayer(Online_Trees, securityHandler=agolSH, initialize=False,proxy_url=proxy_url, proxy_port=proxy_port)
print fl.query(where=Expression_2, out_fields='*', returnGeometry=True, returnIDsOnly=False, returnCountOnly=False, returnFeatureClass=True, out_fc=Output_Table_2)
print "\n# Ready to compare." + "  Time elapsed: {}".format(timedif(datetime.datetime.now(), start_time))


print "\n# Intialize T1 as a concatenated dictionary"
T1Dict = {}
# Initialize a list to hold any concatenated key duplicates found
T1KeyDups = []
# Open a search cursor and iterate rows
with arcpy.da.SearchCursor(T1, fields_1) as searchRows:
    for searchRow in searchRows:
        # Build a composite key value from 2 fields: CustomerID;TAG
        keyValue = '{};{}'.format(searchRow[0], searchRow[1]).decode('utf-8')
        if not keyValue in T1Dict and searchRow[1] is not None:
            # Key not in dictionary.  Add Key pointing to a list of a list of field values
            T1Dict[keyValue] = [list(searchRow[0:])]
            #print "T1Dict keyValue: {}".format(keyValue)
        elif searchRow[1] is not None:
            # Key in dictionary.  Append a list of field values to the list the Key points to
            T1KeyDups.append(keyValue)
            T1Dict[keyValue].append(list(searchRow[0:]))
        #print T1Dict[keyValue][0:]
del searchRows, searchRow


##print "\n# Sample of how to access the keys, record count, and record values of the dictionary"
##for keyValue in T1Dict.keys():
##    for i in range(0, len(T1Dict[keyValue])):
##        print "T1Dict Key CustomerID;TAG is {} Record:{} of {} PropertyID_WO:{} ObjectID:{}.".format(keyValue, i+1, len(T1Dict[keyValue]), T1Dict[keyValue][0], T1Dict[keyValue][1])


if len(T1KeyDups) > 0:
    print "T1KeyDups keys found! {}".format(T1KeyDups)
    # fix T1 here
else:
    print ("No Duplicates!")
print "\n# T1Dict Complete:" + "  Time elapsed: {}".format(timedif(datetime.datetime.now(), start_time))


print "\n# Get the list of TAGs associated with each CustomerID in a dictionary for T1"
T1CustomerIDDict = {}
with arcpy.da.SearchCursor(T1, fields_1) as searchRows:
    for searchRow in searchRows:
        keyValue = searchRow[0]
        if not keyValue in T1CustomerIDDict:
            # Key not in dictionary.  Add Key pointing to a list of a list of field values
            T1CustomerIDDict[keyValue] = [searchRow[0:]]            
        else:
            # Append a list of field values to the list the Key points to
            T1CustomerIDDict[keyValue].append(searchRow[0:])
        #print T1CustomerIDDict[keyValue]
del searchRows, searchRow
print "\n# T1CustomerIDDict Complete"


print "\n# Get the list of TAGs associated with each CustomerID in a dictionary for T2"
T2CustomerIDDict = {}
with arcpy.da.SearchCursor(T2, fields_2) as searchRows:
    for searchRow in searchRows:
        keyValue = searchRow[0]
        if not keyValue in T2CustomerIDDict:
            T2CustomerIDDict[keyValue] = [searchRow[0:]]
        else:
            T2CustomerIDDict[keyValue].append(searchRow[0:])
del searchRows, searchRow
print "\n# T2CustomerIDDict Complete"


print "\n# Creating CustomerIDDict"
CustomerIDDict = {}
for keyValue in T2CustomerIDDict.keys():
    intList = []
    for TAG in T2CustomerIDDict[keyValue]:
        try:
            if TAG[1]:
                TAG = int(TAG[1])
                intList.append(TAG)
        except:
            print "Error with T2.TAG"
    if keyValue in T1CustomerIDDict:
        for TAG in T1CustomerIDDict[keyValue]:
            try:
                if TAG[1]:
                    TAG = int(TAG[1])
                    intList.append(TAG)
            except:
                print "Error with T1.TAG"          
    #print "\nnumeric T2.TAGs: {}".format(set(intList))
    #print "\nnumeric T1.TAGs: {}".format(set(intList))
    # remove already used numbers out of the numbers from 1 to 9999
    # and get a sorted list for use with an update cursor.
    CustomerIDDict[keyValue] = sorted(set(range(1, 200)) - set(intList))
print "\n# CustomerIDDict Complete"


print "\n# Inserting and Updating" + "  Time elapsed: {}".format(timedif(datetime.datetime.now(), start_time))
### This cursor sucessfully inserts into sql server a new row with an unused TAG and attributes from the hosted feature.
### It then updates the hosted layer fields propertyID_WO and TAG. 
##  propertyID_WO succeeds but TAG is blank
#
print "#  Working with pyodbc sql stuff"
cnxn_SQL = pyodbc.connect(Trusted_Connection='yes', Driver = '{SQL Server Native Client 10.0}', Server = '###', database = '###')
cursor_SQL = cnxn_SQL.cursor() 
with arcpy.da.UpdateCursor(T2, fields_2) as updateRows:
     for updateRow in updateRows:            
        # if T2.Row has TAG and CustomerID but no PropertyID compare and look for match in T1
        if (updateRow[2] == None) and not (updateRow[1] == None):
            print "1st if TAG {} PropertID {}".format(updateRow[1],updateRow[2])
            for valu in T1CustomerIDDict[updateRow[0]]:
                if valu[1] == updateRow[1]:
                    try:
                        updateRow[2] = valu[2]
                        print "# Updating PropertyID_WO for TAG:{} hosted layer with {}".format(updateRow[1],updateRow[2])
                        fl.calculate(where="OBJECTID={} AND CustomerID={} AND TAG={}".format(updateRow[3], updateRow[0], updateRow[1]), calcExpression={"field" : "PropertyID_WO", "value" : "{}".format(updateRow[2])})
                        print "# Success"
                    except:
                        print "# Failed Hosted Update of PropertyID"
        if (updateRow[1] == None) and (updateRow[2] == None):
            print "3rd if TAG {} PropertID {}".format(updateRow[1],updateRow[2])
            newPropID = None
            print CustomerIDDict[updateRow[0]][0]
            # Set templist to show available integer TAGs for 1 CustomerID at a time.
            templist = CustomerIDDict[updateRow[0]]
            # Set Null tag to first available integer
            print updateRow[1:3]
            updateRow[1] = templist[0]
            # Insert to local sql server table, update online from sql results, then Remove used tag.
            print updateRow[1:3]
            try:
                # Get ItemID required by insert trigger
                propitemid = cursor_SQL.execute("SELECT [PropertyItemID] from [TREESPECIESRELATIONSHIPTABLE] Where [COMMON__GIS_name_] = ?",updateRow[4]).fetchone()[0]
                # Insert New trees into T1
                cursor_SQL.execute("IF NOT EXISTS (SELECT 1 FROM Property WHERE CustomerID = ? and Location = ?) INSERT INTO Property(CustomerID, Location, Description, CBH, PropertyQuantity, PropertyItemID, DateEntered, LastChangeDate, Special_Note) values (?,?,?,?,?,?,?,?,?)", updateRow[0], updateRow[0], updateRow[0],updateRow[1],updateRow[4],updateRow[5],updateRow[6],propitemid,updateRow[8],updateRow[9],updateRow[10])
                cnxn_SQL.commit()
                # Grab to primary key for updating hosted layer
                newPropID = cursor_SQL.execute("Select PropertyID From Property where CustomerID=? AND Location=?", updateRow[0], updateRow[1]).fetchone()[0]
                print "Finished Insert for New PropertyID {}".format(newPropID)
                cnxn_SQL.commit()
                updateRow[2] = newPropID
                templist.remove(updateRow[1])
                print updateRow[1:3]
            except:
                print "# Failed SQL Server Insert"
            try:
                print "# Updating TAG and PropertyID_WO for hosted layer"
                print fl.calculate(where="OBJECTID={} AND CustomerID={}".format(updateRow[3], updateRow[0]), calcExpression={"field" : "TAG", "value" : "{}".format(updateRow[1]),"field" : "PropertyID_WO", "value" : "{}".format(updateRow[2])})
            except:
                print "# Failed Hosted Update of TAG and PropertyID"
        updateRows.updateRow(updateRow)
del updateRows, updateRow
cnxn_SQL.close()         
#
##
###
###
print "Finished!" + "  Time elapsed: {}".format(timedif(datetime.datetime.now(), start_time))

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Tue, 03 Mar 2015 17:18:57 GMT

I am not clear which part of the code is causing the problem and which field name is for an integer and which is for a string. Generally to use Python to write to a string field you have to enclose non-strings in the str() method. For converting strings to integers you must always test that the string is actually an integer and use Null (or some other default value for blank numbers) when the strings is not numeric. For example, " " is not an integer and will fail if you try to convert it to an integer. Potentially Null will also cause a problem.

Anyway, any time type conversion is involved in a script it is easy to produce errors and unexpected results until you include enough logic to handle both the values that can convert and the values that can't convert at each conversion step. To confirm for yourself what is failing you should include a print statement in your try except block to show you the actual value read from the source field that you are trying to write to the field that is rejecting the value or producing bad results.

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Tue, 03 Mar 2015 18:09:47 GMT

The last try at line 230 is where it is succeeding the REST call, but the layer's field values stay null. A few lines prior, updateRow[1] gets an integer from the dictionary CustomerIDDict.

# Set Null tag to first available integer

updateRow[1] = templist[0]

Then updateRow[1] gets put into...

print fl.calculate(where="OBJECTID={} AND CustomerID={}".format(updateRow[3], updateRow[0]), calcExpression={"field" : "TAG", "value" : "{}".format(updateRow[1]),"field" : "PropertyID_WO", "value" : "{}".format(updateRow[2])})

I'm guessing it turns into: calcExpression={"field" : "TAG", "value" : "4",....},but I'm not certain.

Also, at line 97 added a decode and a typeerror went away. I can't tell if this decode causes issues in the dictionary later on though.

# Build a composite key value from 2 fields: CustomerID;TAG

keyValue = '{};{}'.format(searchRow[0], searchRow[1]).decode('utf-8')

Re: Updating 2 tables related by 3 ID fields

RichardFairhurst — Tue, 03 Mar 2015 19:32:38 GMT

Actually, line 230 is a print statement. Are you sure it is executing? You should break that whole expression down to multiple prints statements so you are certain exactly what arguments are being passed to the query and calculation. For example, you could try:

print "OBJECTID={} AND CustomerID={}".format(updateRow[3], updateRow[0])

print "{}".format(updateRow[1])

print "{}".format(updateRow[2])

Then, if all of those inputs makes sense, execute the fl.calculate outside of a print operation.

There are some details to pay attention to. If CustomerID is numeric and the value makes sense, then the SQL should be fine. But if CustomerID is a string value then the SQL should fail, since the value is not within quotes. For a string CustomerID you would need: "OBJECTID={} AND CustomerID='{}'".format(updateRow[3], updateRow[0])

Re: Updating 2 tables related by 3 ID fields

DavinShokes1 — Tue, 03 Mar 2015 22:36:07 GMT

I figured it out. I was trying to update 2 different fields at once and field calculator understandably only takes one. Its crazy how much time I looked up information on unicode stuff only to find it was a misuse of the parameter.

Can't thank you enough Richard. I should be able to fumble through the rest of the conditionals.