IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

Question

arcgis module 2.0.1

Have a hosted layer in ArcGIS Online. It contains Survey123 results. I wanted to load the results as a spatially enabled dataframe using the ArcGIS API for Python, in a Notebook currently but will move to a script eventually. There are fields of type Integer in the hosted layer. Some or all of these fields contain null/empty values. I get the item using gis.content.get(), providing the item id, without issue. I then retrieve the layer, by index position 0 as there is only 1. After retrieving the layer I call (below), all per the instructions at https://developers.arcgis.com/python/guide/part2-data-io-reading-data/ .This has worked in the past on other projects with no issues.

pd.DataFrame.spatial.from_layer(my_layer)

The issue is this time I get an exception and cannot create the dataframe.

IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

It does not appear that you can provide dtypes, or fillna, to head this issue off before it occurs. The from_layer() does not appear to accept kwargs and takes the schema of the hosted layer. After much searching for a way around I tried an open query of the layer, without any filtering etc. Just simply called

sdf = layer.query().sdf

And this works, without raising an exception.

BUT, if you pass as_df=True to query() you will get the same IntCastingNaNError I mentioned earlier.

query(as_df=True)

This seems like inconsistent behavior. Is this a bug/mishandling or expected and why? It seems like these methods should perform similarly/consistently.

Thanks

CarlVricella · Answer

That is not the direct reason...as the error states it's an underlying exception raised by pandas due to non finite values being in one of the columns/fields; I get the same error with fields that have no coded domains. My guess is that in older versions of the API they were catching this error or doing a conditional check before fixing and returning from the query method but for whatever reason the workflow has been axed in the newest version. Could also be to a newer pandas version making a change. Either way, I'm not going looking through all that code to figure it out haha.

To check for the field you could iterate through every field in the layer and try a query with as_df =True for each of them...whatever fields are causing the error will fail.

@ConradSchaefer__DOIT_ good pick up on the .sdf property still working...it's not ideal but it is a workable fix until this is addressed (for me atleast). I haven't tested, but another solution would be to do what I am suggesting above to find the fields and then doing an update to update any NA or INF values (or values that will convert to those types in pandas rather) to some default value. Hopefully ESRI will address this soon.

Clubdebambos · Answer

Adding to this thread as I am experiencing the same issue using the Geoaccessor.

from arcgis import GIS
from arcgis.features import GeoAccessor

## connect to agol
agol = GIS("home")

## get feature service item
item = agol.content.get("***ITEM_ID***")

## access feature layer
lyr = item.layers[0]

## create a spatial data frame from the lyr
sdf = GeoAccessor.from_layer(lyr)

#############################################################################
## ERROR

"Cannot convert non-finite values (NA or inf) to integer"
pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer
Exception: Could not load the dataset: Cannot convert non-finite values (NA or inf) to integer

As @ConradSchaefer__DOIT_ states, there doesn't seem to be a way to fillna at time of or before conversion. There is also no hint as to the field that is the culprit. Its apparent its an integer field, but which one?

UPDATE: The error occurs when using coded domains with integers, the attribute cannot be -empty- as shown in the dropdown list. There must be a value!