Can't open spatial dataframe in Spyder 4.2.1 Variable Explorer - not picklable

5003
6
Jump to solution
02-24-2021 01:26 PM
Zeke
by
Regular Contributor III

I upgraded to Spyder 4.2.1 from 3.6, and now when I run my script that creates a spatially enabled dataframe, the dataframe 'sdf' won't open in Variable Explorer. Instead it gives a message stating 'The variable is not picklable'. The console pane shows a long traceback ending in '
_pickle.PicklingError: Can't pickle 'geoprocessing server result object' object: <geoprocessing server result object object at 0x000001F686A82A80>'.

This was working before the upgrade, and I'm thinking after it for a while as well (might be wrong there). running things like sdf.head() works fine. Code below.

Running Python 3.7.9 64-bit | Qt 5.9.6 | PyQt 5.9.2 | Windows 10 | Spyder 4.2.1

 

 

 

import time as t
import os
import sys
import pandas as pd
import arcpy
from arcpy import env
from arcgis.features import GeoAccessor, GeoSeriesAccessor

gdb = r"C:\Users\Owner\Desktop\Tract_Aggregation\Tract_aggregation.gdb"
fds = 'Detroit_tracts'
# fc = 'All_Detroit_Tracts_TL_Data_lt_4200'
fc = 'Tract_Agg_1_021621_New'
census_tracts = os.path.join(gdb, fds, fc)
mem_tracts = arcpy.management.MakeFeatureLayer(census_tracts, r"memory\tracts")
env.overwriteOutput = True
env.workspace = os.path.join(gdb, fds)
cutoff = 4200
used_gids = []


def create_sdf():
    """
    Returns
    -------
    df : Pandas Dataframe
        Census tracts.
    """
    print('\tCreating Spatial DataFrame...')
    drop_flds = ['OBJECTID',
                 'objectid_1',
                 'object_id']
    try:
        df = pd.DataFrame.spatial.from_featureclass(mem_tracts)
        df.dropna(subset=['geoid'], inplace=True)
        df.drop(drop_flds, axis=1, inplace=True)
        df['under125pe'] = df['under125pe'] * 1.0
        df['under200pe'] = df['under200pe'] * 1.0
        df['under125pe'].fillna(0.0)
        df['under200pe'].fillna(0.0)
        df.set_index('geoid', inplace=True, drop=False)
        df.rename_axis('gid', inplace=True)
        df.sort_values(by='total_hh', axis=0,
                       ascending=False, inplace=True)
        print('\tSpatial DataFrame created\n')
        return df
    except Exception as e:
        print(sys.exc_value)

 

 

 

 

0 Kudos
1 Solution

Accepted Solutions
DanPatterson
MVP Esteemed Contributor

was the arcgis module updated during the process?  Lots of Issues on their github site with the transition from 1.8.2 to 1.8.3 to 1.8.4


... sort of retired...

View solution in original post

6 Replies
DanPatterson
MVP Esteemed Contributor

What version of arcgis?

There were recent issues with pandas dataframes

Spatially Enabled Data Frame not working with pandas datetime · Issue #874 · Esri/arcgis-python-api ...

and 

 

conda upgrade --no-pin arcgis

 

this was the solution 

Not a spyder issue that I could find on their Issues link

Issues · spyder-ide/spyder (github.com)


... sort of retired...
Zeke
by
Regular Contributor III

Thanks @DanPatterson. Using ArcGIS Pro 2.7.1, cloned Python environment from ArcGIS default to which I've added Spyder.

Default Spyder install via this method is Spyder 3.6. This runs but does nothing when I try to view the spatial dataframe in Variable Explorer. Upgrading Spyder to 4.2.1 also runs, but gives error message the pickling error message in original post. Using the upgrade arcgis method you posted tells me that it's already installed. This is really baffling me. Thanks!

0 Kudos
DanPatterson
MVP Esteemed Contributor

I suspect you won't be able to since it isn't a true Pandas df.  I doubt the spyder crew will have much interest in providing support unless there is a widespread need.  Numpy and Pandas work fine


... sort of retired...
0 Kudos
Zeke
by
Regular Contributor III

Thanks @DanPatterson. Maybe so, but prior to a few days ago it would display it. Oh well.

0 Kudos
DanPatterson
MVP Esteemed Contributor

was the arcgis module updated during the process?  Lots of Issues on their github site with the transition from 1.8.2 to 1.8.3 to 1.8.4


... sort of retired...
Zeke
by
Regular Contributor III

That may be it. I did get the latest (afaik) arcgis module.

0 Kudos