changes to spatial data frame?

StacyLynnBlanco — Mon, 03 Oct 2022 16:17:35 GMT

I often use pd.DataFrame.spatial.from_layer(***) in my scripts to convert the layer to a pandas dataframe for running reports or otherwise manipulating the data. My scripts have worked fine for the past year or more, but I'm getting an error, now.

Last week, I updated ArcPro to version 3.0.2, and it currently says my ArcGIS Pro version is up to date. I did discover that the Python versions have changed, and since I often run my scripts from Anaconda, I re-cloned the python environment so I'm using the same versions of the arcpy libraries.

However, I'm noticing some funny behaviors with the DataFrame.spatial.from_layer(***) command. I am getting an error due to having nulls in the layers. For one layer, I was able to fix this by setting all nulls to 0 or 1, since these are realistic default codes. However, in other layers, this is not possible. The data represents information being collected in the field, and assuming a particular value when nothing is entered is a bad data management practice. These fields allow nulls and that is required.

Up until last week, the conversion to a dataframe worked fine with nulls. However, I am now getting the error that it is trying to convert null values to an integer, even if I don't have any integer-type classes in the layer. I'm not sure what changed or how to fix this. Please help as I need to run these scripts ASAP (regulatory reporting for the end of the quarter). I am desperate for any help you can provide!

Here is the full error:

---------------------------------------------------------------------------
IntCastingNaNError                        Traceback (most recent call last)
C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\arcgis\features\geo\_accessor.py in from_layer(layer)
   2687 
-> 2688             return from_layer(layer=layer)
   2689         except ImportError:

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\arcgis\features\geo\_io\serviceops.py in from_layer(layer, query)
    186         raise ValueError("Invalid inputs: must be FeatureLayer or Table")
--> 187     sdf = layer.query(where=query, as_df=True)
    188     sdf.spatial._meta.source = layer

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\arcgis\features\layer.py in query(self, where, out_fields, time_filter, return_count_only, return_ids_only, return_distinct_values, group_by_fields_for_statistics, statistic_filter, result_offset, result_record_count, object_ids, gdb_version, order_by_fields, out_statistics, return_all_records, historic_moment, sql_format, return_exceeded_limit_features, as_df, having, **kwargs)
   3793 
-> 3794                     df = self._query_df(url, params)
   3795                     count += len(df)

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\arcgis\features\layer.py in _query_df(self, url, params, **kwargs)
   3307         if dtypes:
-> 3308             df = df.astype(dtypes)
   3309 

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors)
   5798                     results.append(
-> 5799                         col.astype(dtype=dtype[col_name], copy=copy, errors=errors)
   5800                     )

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors)
   5814             # else, only a single dtype is given
-> 5815             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   5816             return self._constructor(new_data).__finalize__(self, method="astype")

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\internals\managers.py in astype(self, dtype, copy, errors)
    417     def astype(self: T, dtype, copy: bool = False, errors: str = "raise") -> T:
--> 418         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    419 

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\internals\managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
    326                 else:
--> 327                     applied = getattr(b, f)(**kwargs)
    328             except (TypeError, NotImplementedError):

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\internals\blocks.py in astype(self, dtype, copy, errors)
    590 
--> 591         new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
    592 

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\dtypes\cast.py in astype_array_safe(values, dtype, copy, errors)
   1308     try:
-> 1309         new_values = astype_array(values, dtype, copy=copy)
   1310     except (ValueError, TypeError):

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\dtypes\cast.py in astype_array(values, dtype, copy)
   1256     else:
-> 1257         values = astype_nansafe(values, dtype, copy=copy)
   1258 

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\dtypes\cast.py in astype_nansafe(arr, dtype, copy, skipna)
   1167     elif np.issubdtype(arr.dtype, np.floating) and np.issubdtype(dtype, np.integer):
-> 1168         return astype_float_to_int_nansafe(arr, dtype, copy)
   1169 

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\pandas\core\dtypes\cast.py in astype_float_to_int_nansafe(values, dtype, copy)
   1212     if not np.isfinite(values).all():
-> 1213         raise IntCastingNaNError(   1214             "Cannot convert non-finite values (NA or inf) to integer"

IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-15-1882b171487b> in <module>
      8 
      9 assets_df = pd.DataFrame.spatial.from_layer(nf1)
---> 10 poles_df = pd.DataFrame.spatial.from_layer(nt1)
     11 #circ_df = pd.DataFrame.spatial.from_layer(nt2)

C:\ProgramData\Anaconda3\envs\arcgispy3\lib\site-packages\arcgis\features\geo\_accessor.py in from_layer(layer)
   2695             )
   2696         except Exception as e:
-> 2697             raise Exception("Could not load the dataset: %s" % str(e))
   2698 
   2699     # ----------------------------------------------------------------------

Exception: Could not load the dataset: Cannot convert non-finite values (NA or inf) to integer

And here is the analysis of my fields:

I do have some fields that are type Long, but they also do allow nulls.

Any suggestions?

Re: changes to spatial data frame?

CMV_Erik — Mon, 03 Oct 2022 16:49:10 GMT

This post sounds similar and might include a workaround (apparently layers have a .sdf method?).

IntCastingNaNError: Cannot convert non-finite valu... - Esri Community

Re: changes to spatial data frame?

StacyLynnBlanco — Mon, 03 Oct 2022 17:12:34 GMT

That work around does work...I was not aware of the .sdf method for layers. It also worked for tables, so I can at least go on for now. It would be nice to get the old functions working again for nulls since this is just a workaround, but thank you so much for pointing this out!

Re: changes to spatial data frame?

CMV_Erik — Mon, 03 Oct 2022 17:35:50 GMT

Thank you for letting me know it worked - I expect I will need to review this when I'm ready to upgrade!

topic Re: changes to spatial data frame? in ArcGIS API for Python Questions

changes to spatial data frame?

Re: changes to spatial data frame?

Re: changes to spatial data frame?

Re: changes to spatial data frame?