topic Re: Append data not appending attributes in Python Questions

Append data not appending attributes

MartinSirkovsky1 — Fri, 23 Mar 2018 16:21:52 GMT

I create a script to append data from feature class in FGDB to SDE (MS Sql DBMS). I use "no_test" on append.

Script works fine, data get appended, but no attributes are transfered, just geometry. When I run the command manually, or if I set target to be another FGDB, it works.

Does anybody experienced this before?

Re: Append data not appending attributes

XanderBakker — Fri, 23 Mar 2018 16:25:14 GMT

After you manually run the tool the way it works copy the Python snippet from the Results window and compare the code with what you are using in your script. I'm sure it will contain information on the field mapping.

Re: Append data not appending attributes

JustinPierre — Fri, 23 Mar 2018 17:23:08 GMT

It's probably related to the field mapping from the file geodatabase to the SDE database.

Field mapping and Python scripting | ArcGIS Blog

As Xander suggests, if you copy the Python snippet from ArcMap it will include the field mapping you need

arcpy.Append_management(

inputs="some featureclass'",

target="some sde",

schema_type="NO_TEST",

field_mapping='a very long string of mapped attributes',

subtype=""

)

Re: Append data not appending attributes

MartinSirkovsky1 — Fri, 23 Mar 2018 19:14:45 GMT

I don't think there is a problem in my field mapping. This are the reasons why:

I have copied it from python snippet
When I run it against FGDB it works
When I run my append code in Python's console in ArcMAP it works (target is SDE)

I am wondering if it might be some issue with MS SQL. It is a first time I have experienced such a strange behaviour, and I did append many many times via scripting, however never against MS SQL.

Re: Append data not appending attributes

JoshuaBixby — Sun, 25 Mar 2018 20:57:45 GMT

How are you executing your code when not in the interactive Python window in ArcMap?

Re: Append data not appending attributes

MartinSirkovsky1 — Sun, 25 Mar 2018 21:27:40 GMT

I am using Sublime text 3 to execute my code. But I think I also tried directly from command prompt. I did this operation 1000 times before. Please notice, that all the features are appended to target FC, just all attributes have value NULL.

Re: Append data not appending attributes

MatthewDobson — Sun, 25 Mar 2018 23:34:09 GMT

Martin.

What version of ArcGIS are you using, and are there map services consuming the featureclasses that you are attempting to append to??

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 09:14:10 GMT

Hey Matthew.

I am using ArcGIS 10.5.1, and there are currently no Feature Classes consuming the services.

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 09:44:58 GMT

So I did some testing. Here is something interesting.

I have set up the input feature class in a way that I can append it with "TEST" value in "schema_type" parameter.

Running script from ArcMAP both with NO_TEST and TEST as schema type gives correct result
Running the script as a python script with NO_TEST as schema type results in FC being appended but no attributes are transfered
Running the script as a python script with NO_TEST as schema type results in FC being appended together with attributes

So it seems that there some kind of bug, when running append command from python script with "NO_TEST" as schema type against feature class in MS SQL SDE.

Anyone have any idea how to solve this issue?

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 09:46:55 GMT

CORRECTION TO ABOVE POST:

Running script from ArcMAP both with NO_TEST and TEST as schema type gives correct result
Running the script as a python script with NO_TEST as schema type results in FC being appended but no attributes are transfered
Running the script as a python script with TEST as schema type results in FC being appended together with attributes

Re: Append data not appending attributes

XanderBakker — Mon, 26 Mar 2018 11:22:15 GMT

Can you copy and paste the Python snippet of the code that works in ArcMap?

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 12:03:19 GMT

I tried that. It doesn't work. And when I run the code it returns no error.

If you think about it, why would it work if the target FC is FGDB, and not work when the target FC is SDE? And why would it work if the schema type is TEST and not work when the schema type is NO TEST?

It seems to be that there is some issue with DBMS.

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 13:44:47 GMT

If you wanted to see code and a subset of data I want to import, you can download it from here:
Dropbox - sample_code_and_FGDB.zip

The code is also here:

# -*- coding: utf-8 -*-
#!/usr/bin/python

"""
Script tries to append data to Kloakk SDE (FORSYNING)
The script is testing append command, which appears not to be working
"""
print "Starting script...."
import arcpy
inputFC = "C:/temp/kloakk.gdb/kloakk_lutir_180326"
sde = "L:/Forriting/sde connections/gisedit/KLOAK@FORSYNING@tksqlgis.sde/"
targetFC = "kloakk_lutir_fyribils"
arcpy.env.workspace = sde
fieldMapping = 'mslink "mslink" true true false 4 Long 0 10 ,First,#,kloakk_lutir_180326,mslink,-1,-1;name "navn" true true false 150 Text 0 0 ,First,#,kloakk_lutir_180326,name,-1,-1;city "stað" true true false 50 Text 0 0 ,First,#,kloakk_lutir_180326,city,-1,-1;zip "postnr." true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,zip,-1,-1;owner "eigari" true true false 2 Text 0 0 ,First,#,kloakk_lutir_180326,owner,-1,-1;liquid "veska" true true false 2 Text 0 0 ,First,#,kloakk_lutir_180326,liquid,-1,-1;feature_type "slag" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,feature_type,-1,-1;material "tilfar" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,material,-1,-1;net_type "netverkslag" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,net_type,-1,-1;diameter "tvørmát" true true false 4 Float 3 6 ,First,#,kloakk_lutir_180326,diameter,-1,-1;elevation_cover "deksulkota" true true false 4 Float 2 6 ,First,#,kloakk_lutir_180326,elevation_cover,-1,-1;elevation_bottom "botnkota" true true false 4 Float 2 6 ,First,#,kloakk_lutir_180326,elevation_bottom,-1,-1;location "staðseting" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,location,-1,-1;comment "viðmerking" true true false 200 Text 0 0 ,First,#,kloakk_lutir_180326,comment,-1,-1;construction_year "innlagt ár" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,construction_year,-1,-1;status "støða" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,status,-1,-1;street_name "gøta" true true false 30 Text 0 0 ,First,#,kloakk_lutir_180326,street_name,-1,-1;house_nr "húsanr." true true false 10 Text 0 0 ,First,#,kloakk_lutir_180326,house_nr,-1,-1;collection_type "mátiháttur" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,collection_type,-1,-1;collection_date "skráset dagf." true true false 8 Date 0 0 ,First,#,kloakk_lutir_180326,collection_date,-1,-1;collected_by "skráset hevur" true true false 10 Text 0 0 ,First,#,kloakk_lutir_180326,collected_by,-1,-1;imported_date "innlisið frá microstation" true true false 8 Date 0 0 ,First,#,kloakk_lutir_180326,imported_date,-1,-1'

arcpy.Append_management(
inputs=inputFC,
target=targetFC,
schema_type="NO_TEST",
field_mapping = fieldMapping
)

# PYTHON SNIPPET EXPORTED FROM ARCMAP
# # Replace a layer/table view name with a path to a dataset (which can be a layer file) or create the layer/table view within the script
# # The following inputs are layers or table views: "FORSYNING.KLOAK.kloakk_lutir_fyribils"
# arcpy.Append_management(inputs="kloakk_lutir_180326", target="FORSYNING.KLOAK.kloakk_lutir_fyribils", schema_type="NO_TEST", field_mapping='mslink "mslink" true true false 4 Long 0 10 ,First,#,kloakk_lutir_180326,mslink,-1,-1;name "navn" true true false 150 Text 0 0 ,First,#,kloakk_lutir_180326,name,-1,-1;city "stað" true true false 50 Text 0 0 ,First,#,kloakk_lutir_180326,city,-1,-1;zip "postnr." true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,zip,-1,-1;owner "eigari" true true false 2 Text 0 0 ,First,#,kloakk_lutir_180326,owner,-1,-1;liquid "veska" true true false 2 Text 0 0 ,First,#,kloakk_lutir_180326,liquid,-1,-1;feature_type "slag" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,feature_type,-1,-1;material "tilfar" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,material,-1,-1;net_type "netverkslag" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,net_type,-1,-1;diameter "tvørmát" true true false 4 Float 3 6 ,First,#,kloakk_lutir_180326,diameter,-1,-1;elevation_cover "deksulkota" true true false 4 Float 2 6 ,First,#,kloakk_lutir_180326,elevation_cover,-1,-1;elevation_bottom "botnkota" true true false 4 Float 2 6 ,First,#,kloakk_lutir_180326,elevation_bottom,-1,-1;location "staðseting" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,location,-1,-1;comment "viðmerking" true true false 200 Text 0 0 ,First,#,kloakk_lutir_180326,comment,-1,-1;construction_year "innlagt ár" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,construction_year,-1,-1;status "støða" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,status,-1,-1;street_name "gøta" true true false 30 Text 0 0 ,First,#,kloakk_lutir_180326,street_name,-1,-1;house_nr "húsanr." true true false 10 Text 0 0 ,First,#,kloakk_lutir_180326,house_nr,-1,-1;collection_type "mátiháttur" true true false 2 Short 0 5 ,First,#,kloakk_lutir_180326,collection_type,-1,-1;collection_date "skráset dagf." true true false 8 Date 0 0 ,First,#,kloakk_lutir_180326,collection_date,-1,-1;collected_by "skráset hevur" true true false 10 Text 0 0 ,First,#,kloakk_lutir_180326,collected_by,-1,-1;imported_date "innlisið frá microstation" true true false 8 Date 0 0 ,First,#,kloakk_lutir_180326,imported_date,-1,-1', subtype="")

Re: Append data not appending attributes

JoshuaBixby — Mon, 26 Mar 2018 14:06:34 GMT

One difference I see between the ArcMap code snippet and your code is that in ArcMap you are passing a layer or table view name whereas you are passing a feature class outside of ArcMap. Although it should not make a difference, have you tried (outside of ArcMap) making a feature layer first and then passing that to Append instead of the feature class directly?

Re: Append data not appending attributes

MartinSirkovsky1 — Mon, 26 Mar 2018 16:17:02 GMT

Hey Joshua.

I have tried that as well. No difference.

Re: Append data not appending attributes

MartinSirkovsky1 — Tue, 27 Mar 2018 09:50:05 GMT

I got it to work. It is a workaround, and I wish it wouldn't be necessary.

The attributes get appended if I defined the field mappings using a field mapping object, instead of the field mapping definition string which can be found in python snippet.

Re: Append data not appending attributes

CharlesFried — Tue, 26 Mar 2019 20:15:46 GMT

APPEND tool really sucks!

What is a "field mapping object" and how did you create that?

Thanks

Re: Append data not appending attributes

MartinSirkovsky1 — Sun, 12 Dec 2021 16:48:09 GMT

You can read about field map object here:

FieldMappings—ArcPy classes | ArcGIS Desktop

What I have done, I have create a small function where I send the 2 paramters. The first is path to Input feature class, and second is array of arrays of [field name in input FC, corresponding field in target FC]. If the names are the same, I just pass the one item in the subarray.

The functions returns field mapping object, which I then use in append function.

Here is a sample from the code, hope it helps.

# Function which created a field mapping object

def createFieldMap(inputFC, fieldMaps):
  """
  Creates a field map object use to define field mapping for tools which use fieldmaps

  Args:
  inputFC(string): the input feature class
  fieldMaps(array): array of input / output fields.
  First item is input field name, second is output field name.
  If output field name is not defined it is set to same as input field
  Returns:
  (object): Field mapping object which can be used to define field mappings
  """

  # If field out is not defined set it to the same name as field in
  fms = arcpy.FieldMappings()
  print ("\nCreating field map")

  for fieldMap in fieldMaps:

  # print ("\nProcessing field {0}".format(fieldMap[0]))

  fieldIn = fieldMap[0]
  fieldOut = fieldMap[0] if len(fieldMap) == 1 else fieldMap[1]

  # print ("Field in {0}".format(fieldIn))
  # print ("Field out {0}".format(fieldOut))


  # Define field map
  fm = arcpy.FieldMap()
  fm.addInputField(inputFC, fieldIn)

  # Set ouput field properties
  fmOutputField = fm.outputField
  fmOutputField.name = fieldOut
  fm.outputField = fmOutputField

  fms.addFieldMap(fm)

  return fms

fms = createFieldMap(
  pathToinputFC, 
  [
    ['feature_id', 'id'], # values from input FC field "feature_id" will be appended to field in target FC with name "id"
    ['owner'], # the name of the field in input and target FC is save, "owner"
    ['liquid'] # the name of the field in input and target FC is save, "liquid"
  ],
)


# RUN Append command
arcpy.Append_management(
  inputs=pathToinputFC,
  target=targetFC,
  schema_type="NO_TEST",
  field_mapping = fms
)

Re: Append data not appending attributes

MR14 — Fri, 19 Apr 2024 13:07:12 GMT

Thanks Martin! Using a field mapping object was the only working solution for appending to an sde feature class