Anyone had an issue with CSV Overwrites in Portal due to Col A Ordering?

281
0
06-10-2020 01:14 PM
JohnSteed1
New Contributor III

I submitted a bug to Esri a while ago (several months) and am still waiting on resolution to this issue.  We have been passed around to several Technical Support personnel.  I thought others here might have also encountered this issue in Portal and might have insight, or might find it helpful.

Problem:

Overwriting a csv in Portal fails with an unknown error that also throws a error in the server log stating "The dataset has an invalid definition... Unknown field type in schema.ini. [...]"  We have traced this down to an issue that arises when the following occurs:

  1. Column A is stored as an integer.
  2. The data order is changed at the top of the csv between overwrites (ie - rows 1-50etc are not the same as the new rows 1-50etc).  

Main point: It appears that the issue comes down to the way Column A is sorted, compared to the initial source file.
  1. Focused on Column A given the error that gets spit out in the logs: "The dataset has an invalid definition... Unknown field type in schema.ini. Col1=Tracking Number Long] Failed to execute (Publish Portal Service)". The error shows you where to look.
  2. Comparing both CSVs (overwritten and new), we noticed that the Tracking Numbers in the "Overwrite" CSV were not in the same order as the "Initial" CSV.
  3. Because Column A are integers, we believe Portal may handle them like ObjectIDs. So a discrepancy between an initial CSV to an updating one with the ObjectID fields in different orders would cause an schema error when Portal reads it
  4. This seems to only matter when Column A is an integer and not a string. We tested with changing Column A from integer to string and it worked.
  5. Sorting the "Overwrite" CSV to mirroring the "Initial" CSV Tracking Numbers back to the top allows the overwrite to go through.  Hence it is NOT the data itself causing the overwrite issue, but the order of the data that is the problem.
Additional notes:
  • We don't exactly know what the threshold is where the columns can be misordered.

Summary: We reported this as bug, specifically for using an integer in Column A, as this is not specific to our Portal because we tested the same workflow on THREE other Portals. Also not sure if this issue is specific to CSVs or if Excels also run into this issue.  All Portals we tested this on were 10.7.1.
Question: Have others experienced this before?
UPDATE as of 27 July 2020
After several months of testing, Esri elevated this to a bug.  FYI
0 Kudos
0 Replies