topic Re: pd.DataFrame.spatial.from_table() is loading column labels but not data? in ArcGIS API for Python Questions

pd.DataFrame.spatial.from_table() is loading column labels but not data?

SamuelStroebel — Tue, 10 Nov 2020 15:19:49 GMT

as the title says, trying to access a local file in a GDB, a dataframe is created with the correct number of columns but no data?

Code:

import pandas as pd
from arcgis.features import GeoAccessor, GeoSeriesAccessor
from arcgis import GIS
import arcpy

df = pd.DataFrame.spatial.from_table(r'some_file_path_to_a_table_in_a_gdb')

df.shape

output: 0 x number of cols in input table

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

JoshuaBixby — Tue, 10 Nov 2020 16:42:59 GMT

It always helps to show your code.

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

SamuelStroebel — Tue, 10 Nov 2020 16:49:55 GMT

thanks, updated

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

JoshuaBixby — Tue, 10 Nov 2020 16:52:04 GMT

What version of the ArcGIS API for Python are you using?

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

SamuelStroebel — Tue, 10 Nov 2020 16:56:48 GMT

1.8.2, in a cloned environment

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

JoshuaBixby — Tue, 10 Nov 2020 17:02:51 GMT

I see what you are running into, clearly a defect. To work around it, specify the fields you want in a field list, and it will work. It appears the wildcard for all fields isnt' working.

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

SamuelStroebel — Tue, 10 Nov 2020 17:09:07 GMT

Your work around worked perfectly, thanks!

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

JoshuaBixby — Tue, 10 Nov 2020 17:23:20 GMT

Please mark my response as Correct to close out the thread.

Also, I opened in issue on GitHub: GeoAccessor.from_table Returns Empty Table When Using All Fields Wildcard · Issue #829 · Esri/arcgis-python-api · GitHub

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

Hudi — Mon, 08 Feb 2021 20:50:28 GMT

I'm not entirely sure the solution response is the entire picture. It appears to me to be a failure of spatial.from_table to deal with null values.

The default behavior looks like it will cut down the rows it actually reads into the dataframe such that it can have at least one complete row of non-null data that includes values from all specified fields. So as you add more fields to the list the chances of an entire row containing non-null values diminishes.

Therefore specifying fields probably means also excluding many fields you don't care about. That explains why specifying fields works. If you used arcpy.ListFields() and built a list of all field names it should behave the same as default where no fields are entered, which is the same as fields='*'.

So to handle this behavior you can add something like null_value='-9999' and then in the dataframe switch that value back to NaN if that is desired.

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

gallgher55 — Thu, 24 Jun 2021 14:21:21 GMT

I've confirmed this is the actual issue. I supplied my fields via a list from arcpy.ListFields() to no avail. Using the null_value is what actually made it work for me.

Re: pd.DataFrame.spatial.from_table() is loading column labels but not data?

EricAnderson_DHS — Tue, 06 Dec 2022 16:58:52 GMT

skip_nulls=False also worked for me

df = pd.DataFrame.spatial.from_table(filename=os.path.join(sde,"Table"),skip_nulls=False)