Spatial dataframe does not plot to a mapwidget due to spatial reference?

Question

I'm doing some data processing in Jupyter Notebook with arcpy geoprocessing tools and storing the result into file geodatabase as a feature class. To check the results inside the notebook I imported the feature class to spatial dataframe (sdf) with from_featureclass() method. Everything seems to be ok and sdf has a SHAPE column and the first row contents of the SHAPE column looks good when printed out:

print(sdf.iloc[0, sdf.columns.get_loc('SHAPE')])

and the SHAPE field referenced by itself

sdf.iloc[0, sdf.columns.get_loc('SHAPE')]

results in a polygon being drawn to the cell's output.

However, if i try to plot it to a map widget with:

mw = gis.map("Center of the Map")
sdf.spatial.plot(map_widget=mw, 
                 col='columnName')
mw

it results in a map widget drawn with only the default base map.

On the other hand, if I export it to a feature layer in AGOL with:

lyr = sdf.spatial.to_featurelayer('export_test', folder='exportDemo')

It draws correctly on the Map Viewer, but there are still some issues regarding the calculated values (it only allows unique values symbology and not "Counts and Amounts"), but that I think is just a data type issue.

The usual suspect in this type of problem is often an issue with the spatial references. My SDF has a projected national grid spatial reference, and the widget basemap probably is Web Mercator. There might be some projection issue even if no error is displayed. But the reference in the SDF should be okay, because it plots correctly in AGOL.

Do you have any ideas or recommendations on how to tackle this? Should I try to re-project my data to Web Mercator before plotting, or can/should I define my national grid as the default spatial reference?

Edit: After doing the stuff above, I exported the SDF to file geodatabase with:

sdf.spatial.to_featureclass("path to file geodatabase feature class")

That feature class also plots correctly in ArcGIS Pro. Interestingly, export changed the "columnName" style column names to the "column_name" style. I tried to re-import that feature class back to the notebook and plot it in the map widget, but I got the same result: it did not plot.

Edit: tidying up extra linebreaks.

TimMatisziw · Answer

I've noticed some changes in the behavior of some of the spatial methods (e.g., from_featureclass) recently. For me, it seems that it is due to the presence of fields with null or zero values. In particular, if there are latitude/longitude fields that are null or contain zeros. In my case, removing all empty/null fields fixed it. I'm guessing that the map widget first looks for fields named latitude/longitude to use that info for positioning the symbology.

PetriSmolander1 · Answer

Interestingly enough, when you open the post for editing after posting the empty lines before and after a code block increases by one.

PetriSmolander1 · Answer

The problem is that I would struggle to find a base map with my national grid from AGOL, at least the free ones. Since my original post, I have also tried manually reproject the data to Web Mercator with:

# sdf contains the original Spatial Data Frame
reproj = sdf.copy()
reproj.spatial.project(3857)
# or alternately
# reproj.spatial.project(102100)

reproj.spatial.plot(map_widget=mw)

If I understood correctly, the WKID 3857 should be correct for AGOL's world base maps.

When looking at the converted coordinates in the SHAPE column, they have changed, and spatial reference has also changed and is correct. My original spatial reference is more or less an ETRS-TM coordinate system zone, expanded to cover the entire country. So, the transformation isn't really specific to this case. Also, if I find the extent of the reprojected data frame and zoom the map widget to that, it zooms into the correct location and extent of my data with:

lyrextent = reproj.spatial.full_extent

mw.extent = {'spatialReference': {'latestWkid': 3857, 'wkid': 102100},
 'xmin': lyrextent[0],
 'ymin': lyrextent[1],
 'xmax': lyrextent[2],
 'ymax': lyrextent[3]}

mw

So, I'm thinking that there is something else failing.

Unfortunately, my data is somewhat restricted, so, I can't attach the original data for review.

EDIT: tidying up extra linebreaks.