Apply Asset Package: float changes to double in enterprise geodatabase

321
11
Jump to solution
2 weeks ago
Andy_Morgan
New Contributor III

I'm using ArcGIS Pro 3.0.3 (UN schema version 3, untools version 3.0.0). Apply Asset Package changes my Float data type fields to Double data type in the output enterprise geodatabase.

Is this a known bug? Is there any way to prevent this conversion? Was it fixed in 3.1 or later? I don't understand why this happens if Float is a valid data type in EGDBs, too. Why can't it be retained? If asset packages cannot handle Float then that means I change my entire workflow and create all of them directly in the EGDB...which makes for a very inconvenient deployment preparation. Not only that, if you were to export the EGDB to asset package and bring it back in, you'd have to re-create all your fields again? ...or just accept Double 38 precision, 8 scale, instead of Float which is not ideal if you want to limit numbers, have smaller decimal places and save a little data storage.  

0 Kudos
2 Solutions

Accepted Solutions
PaulLeBlanc1
Esri Regular Contributor

The code above was to show that despite telling the system to create a FLOAT, it created a DOUBLE (implicit upcast). There is a very specific edge case where apply AP does explicitly upcast a FLOAT to a DOUBLE (i.e., it creates the field as a DOUBLE to begin with). You get a runtime warning informing you about this.

If you create a FLOAT field and don't specific scale/precision, then you get a DOUBLE. Tested with the same software as above, so you'll see this behavior even after you upgrade your client.

PaulLeBlanc1_0-1715370213132.png

 

Edit:

Apply AP has been creating FLOAT fields with the following payload since ~Pro 2.1. Scale/Precision are both 0 because that is how they are stored in FGDB.

<AddField>
   <field_name>f_05_03</field_name>
   <field_type>FLOAT</field_type>
   <field_precision>0</field_precision>
   <field_scale>0</field_scale>
   <field_length>4</field_length>
   <field_alias>f_05_03</field_alias>
   <field_is_nullable>True</field_is_nullable>
   <field_is_required>False</field_is_required>
</AddField>

 

View solution in original post

0 Kudos
PaulLeBlanc1
Esri Regular Contributor

untools 3.1.1.1 is uploaded here and can be used like so. This is a RC build, so caveat emptor, not supported, etc. Please reach out to pleblanc at esri dot com (or direct message me here) with any issues/questions.

 

import untools
import arcpy

# Override precision and scale for all float fields.
untools.common.settings.SINGLE_PRECISION_SCALE = (6, 1)
untools.common.settings.DOUBLE_PRECISION_SCALE = None  # The default.

arcpy.pt.AssetPackageToUtilityNetwork(...)

 

To install, open a python command prompt and run conda install solutionsdev/label/pro3.1::untools

View solution in original post

11 Replies
PaulLeBlanc1
Esri Regular Contributor

File geodatabases don't support scale or precision (https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/add-field.htm) so round-tripping is problematic unless we stored the desired scale/precision in a side table and read from that during field creation.

What DBMS are you targeting? untools is not explicitly upcasting  the field, so I'd need to dig further to understand where that is happening.

 

0 Kudos
Andy_Morgan
New Contributor III

Paul, I really do appreciate the quick responses on this board. I'm using SQL Server:

Microsoft SQL Server 2019 (RTM) - 15.0.2000.5 (X64)

While this is a development server hosted on a virtual Windows server, the SQL database is nearly identical to our production server where I'll eventually deploy our UN within 6 months. Here's the production server version:

Microsoft SQL Server 2019 (RTM-CU19) (KB5023049)

I'm trying to think if there is anything else special about what I'm doing. I don't want to waste anyone's time. Let me know if you think of anything else that might be different from the normal scenario. Even though FGDB don't support scale/precision, it still surprised me that it was casted differently for this particular process/tool. I will eventually upgrade Pro to 3.1 and could re-try this process, but I figured others out there with 3.1+ already know if it does or doesn't work.

0 Kudos
PaulLeBlanc1
Esri Regular Contributor

I'm testing with Pro 3.3 and SQL Server 16.0.4120.1 (what I have handy).

import arcpy

fields = []
for p in range(1, 10):
    for s in range(1, p):
        fields.append(
            dict(
                field_name=f"f_{p:02}_{s:02}",
                field_type="FLOAT",
                field_alias=f"Precision: {p}, Scale: {s}",
                field_precision=p,
                field_scale=s,
            )
        )

for f in fields:
    arcpy.AddField_management(
        r'<my table>',
        **f
    )
    print(arcpy.GetMessages())

 The fields are created as numeric(p, s), but ArcGIS treats any precision >6 as a DOUBLE:

PaulLeBlanc1_0-1715350691605.pngPaulLeBlanc1_1-1715350722582.png

 

0 Kudos
Andy_Morgan
New Contributor III

Same result here with my version of SQL Server:  changes over to Double at the "f_07_01" field (precision > 6).

This clearly illustrates what happens in general when manually creating new fields. I was mainly wondering if anyone out there can confirm "Apply Asset Package" still converts from Float to Double in later versions of Pro / UN schema (3.1+). If I need to upgrade one of our test computers and run through the whole process on it then I will, but there must be many users already on 3.1 right now who have tried this. I'm not totally clear if you're implying the answer is yes (3.1 and higher also convert Float to Double). If that's the case, then I'm confused on how Apply Asset Package is not "explicitly upcasting the field" and yet the field is in fact different in the target EGDB.  

 

Float_to_Double.jpg

 

0 Kudos
PaulLeBlanc1
Esri Regular Contributor

The code above was to show that despite telling the system to create a FLOAT, it created a DOUBLE (implicit upcast). There is a very specific edge case where apply AP does explicitly upcast a FLOAT to a DOUBLE (i.e., it creates the field as a DOUBLE to begin with). You get a runtime warning informing you about this.

If you create a FLOAT field and don't specific scale/precision, then you get a DOUBLE. Tested with the same software as above, so you'll see this behavior even after you upgrade your client.

PaulLeBlanc1_0-1715370213132.png

 

Edit:

Apply AP has been creating FLOAT fields with the following payload since ~Pro 2.1. Scale/Precision are both 0 because that is how they are stored in FGDB.

<AddField>
   <field_name>f_05_03</field_name>
   <field_type>FLOAT</field_type>
   <field_precision>0</field_precision>
   <field_scale>0</field_scale>
   <field_length>4</field_length>
   <field_alias>f_05_03</field_alias>
   <field_is_nullable>True</field_is_nullable>
   <field_is_required>False</field_is_required>
</AddField>

 

0 Kudos
Andy_Morgan
New Contributor III

OK, last response. I'll accept this as an answer, but my conclusion is that everyone who builds a Utility Network on SQL Server (and some other DBMS types?) has two options for using Float fields:

  1. Accept the fact that your non-integer numeric fields are going to be Double with the default precision/scale (38/8)
  2. If you really want Float data type fields and you want all your fields added in a specific order then you must omit them from the AP. After applying the AP, you could run a script to add all your fields with the numeric data types having an exact precision/scale defined during that stage. If, however, you were to export your EGDB UN to an AP to start fresh with an AP having the latest properties, you might not be able to use this AP for pushing updates.   

 In #2 I assume this situation will cause Apply Asset Package to fail:

"Elevation" field data type in Asset Package "Elevation" field data type in EGDB UN Device FC 
FloatDouble

 

Who really wants to go with option #2, though. It somewhat defeats the purpose of using an AP to me.

0 Kudos
PaulLeBlanc1
Esri Regular Contributor

Few questions:

  1. Do you want all your floats in all tables to have the same scale/precision?
  2. Do you want all your doubles in all tables to have the same scale/precision?

If yes, then supporting this would be quite straightforward and you could have a RC version of untools by Monday morning that would solve this. I've been toying around with this approach and it works nicely.

If no, and you want certain fields to be numeric(x,y) and others to be numeric(a,b) then it would take  longer (due to priorities and such) but would provide a nice level of granularity.

0 Kudos
Andy_Morgan
New Contributor III

I'd gladly take a generic version with all Floats having P: 6, S: 1 (example: 42.5) and all Doubles having the current default of P: 38, S: 8.  

But we (organizational level) may be upgrading our version of Pro to something higher over the next 6-12 months as Enterprise gets upgraded and UN comes into the picture. Would it make more sense to target untools for 3.1 for the RC? I imagine our whole organization will be on Pro 3.1 or higher before the end of the year. I may jump ahead from 3.0.3 to 3.1 for UN prep.

Thanks. I realized my thinking was a bit narrow in terms of precision/scale for a Float. There is a limit for those values, but not everyone necessarily wants scale of 1 for their Floats. I'm not sure how you'd decide what's best for the average need (a precision of  5 or 6, maybe, with a scale of 1 or 2 at most?). Maybe you're saying you define a universal P/S for those two types. It would definitely be an improvement. I'm trying to represent others out there, too, hopefully.   

0 Kudos
PaulLeBlanc1
Esri Regular Contributor

It wouldn't be a default; you would have to opt in because we don't want to break backwards compatibility. It's been this way for 6 (!) years and you are the first I'm aware of that is raising this. 😄

Still ironing it out, but it would be something like this. If you left off that line, or set it = None, it would be the current behavior.

 

import untools
import arcpy

# Override precision and scale for all float fields.
untools.common.settings.SINGLE_PRECISION_SCALE = (6, 1)

arcpy.pt.AssetPackageToUtilityNetwork(...)

 

 

Yes, we would likely do this on Pro 3.1/3.3 only because those are Network Management Release versions.