This may not be wholly an ArcGIS problem but I always seem to get the best answers from this group! I'm working with the new ESRI Tracker feature service and trying to summarize the json output a bit to generate reports. The specific task is to maintain a running total of seconds between successive point features using the "location_timestamp" column. I won't go into the query portion where we are acquiring the json from the feature service (it's all basic/simple stuff just querying the REST interface with urllib2.Request). I'll put 2 versions (short & long version has all details, just read everything after the *****) The short version -- grouping to sum an "elapsedSeconds" column:grouped = df . groupby ( [ 'location_day' , 'PEP_land_name' , 'PEP_land_rate' , 'created_user' ] ) [ 'elapsedSeconds' ] . sum ( ) . reset_index ( ) Revert this grouped dataframe back into json:dfjson = grouped . to_json ( orient = 'records' ) This is where I end up with an "overflowerror maximum recursion level reached".************ The long version with more detail about what I'm doing. To start, I'm just querying the REST of a hosted feature service (the ESRI tracker service):tracksReq = urllib2 . Request ( urlTrackerMain + '/query' , tracksParams ) tracksResponse = urllib2 . urlopen ( tracksReq ) tracksResult = json . load ( tracksResponse ) And whammo, I have the json output of the tracker service. From here I add some columns to the json for various things: if tracksResult is not None : output = [ ] for jj in tracksResult [ 'features' ] : int_num = int ( str ( json . dumps ( jj [ 'attributes' ] [ 'location_timestamp' ] ) ) ) utc = time . strftime ( '%Y-%m-%d %H:%M:%S' , time . gmtime ( int_num / 1000.0 ) ) convTimeStamp = time . strftime ( '%Y-%m-%d %H:%M:%S' , time . gmtime ( int_num / 1000.0 ) ) convDayStamp = time . strftime ( '%Y-%m-%d' , time . gmtime ( int_num / 1000.0 ) ) jj [ 'attributes' ] [ 'location_timestamp_local' ] = convTimeStamp jj [ 'attributes' ] [ 'location_day' ] = convDayStamp output . append ( jj [ 'attributes' ] ) So now I have a new output with some extra columns (converted those epoch datetime values to something meaningful). Now from here I am ready to do some grouping/summarizing using Pandas:df = pd . DataFrame . from_dict ( output , orient = 'columns' ) df [ 'location_timestamp_local' ] = pd . to_datetime ( df [ 'location_timestamp_local' ] ) df = df . sort ( 'location_timestamp_local' ) Here's where I am creating that running total column in the dataframe: #determine duration in seconds between the previous row's datetime value. Do this for each day. df [ 'elapsedSeconds' ] = df . sort ( 'location_timestamp_local' ) . groupby ( [ 'location_day' , 'PEP_land_name' ] ) [ 'location_timestamp_local' ] . diff ( ) / 1000 All is good and well. Now for some grouping to sum that "elapsedSeconds" value.grouped = df . groupby ( [ 'location_day' , 'PEP_land_name' , 'PEP_land_rate' , 'created_user' ] ) [ 'elapsedSeconds' ] . sum ( ) . reset_index ( ) And finally revert this grouped dataframe back into json:dfjson = grouped . to_json ( orient = 'records' ) This is where I end up with an "overflowerror maximum recursion level reached". It doesn't seem to error when I run in pyScripter v2.6 x86 but it fails when I hook up this .py script to a Geoprocessing tool source. This is likely the issue (32-bit vs. 64?), I'm unsure and really just looking for workarounds if known.Anyway -- thanks for looking and should be a fun one to figure out!ArcGIS 10.4 Pandas 0.16.1

Sort of a hack workaround I guess (no problem taking that criticism) but if I just convert the grouped result to a csv then back to a json output via each line as json.dumps(), I get the result I'm after. Just added this simple def() to create the csv into the scratchFolder then return the json: def generateOutput ( df ) : outputT = 'SessionFile_{}.{}' . format ( str ( uuid . uuid1 ( ) ) , "csv" ) Output_File = os . path . join ( arcpy . env . scratchFolder , outputT ) df . to_csv ( Output_File , index = False ) fieldnames = ( 'location_day' , 'PEP_land_name' , 'PEP_land_rate' , 'created_user' , 'elapsedSeconds' ) csvfile = open ( Output_File , 'r' ) reader = csv . DictReader ( csvfile ) outJson = json . dumps ( [ row for row in reader ] ) csvfile . close ( ) arcpy . Delete_management ( Output_File ) return outJson grouped = df . groupby ( flds ) [ 'elapsedSeconds' ] . sum ( ) . reset_index ( ) grouped [ 'elapsedSeconds' ] = ( ( grouped [ 'elapsedSeconds' ] / np . timedelta64 ( 1 , 's' ) ) * 1000 ) . astype ( str ) dfjson = generateOutput ( grouped ) arcpy . AddMessage ( dfjson )

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Pandas: overflowerror maximum recursion level reached

JamesCrandall

This may not be wholly an ArcGIS problem but I always seem to get the best answers from this group!

I'm working with the new ESRI Tracker feature service and trying to summarize the json output a bit to generate reports. The specific task is to maintain a running total of seconds between successive point features using the "location_timestamp" column. I won't go into the query portion where we are acquiring the json from the feature service (it's all basic/simple stuff just querying the REST interface with urllib2.Request).

I'll put 2 versions (short & long version has all details, just read everything after the *****)

The short version -- grouping to sum an "elapsedSeconds" column:

grouped <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>groupby<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_name'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_rate'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'created_user'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>sum<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reset_index<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

Revert this grouped dataframe back into json:

dfjson <SPAN class="operator token">=</SPAN> grouped<SPAN class="punctuation token">.</SPAN>to_json<SPAN class="punctuation token">(</SPAN>orient<SPAN class="operator token">=</SPAN><SPAN class="string token">'records'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

This is where I end up with an "overflowerror maximum recursion level reached".

************

The long version with more detail about what I'm doing. To start, I'm just querying the REST of a hosted feature service (the ESRI tracker service):

tracksReq <SPAN class="operator token">=</SPAN> urllib2<SPAN class="punctuation token">.</SPAN>Request<SPAN class="punctuation token">(</SPAN>urlTrackerMain <SPAN class="operator token">+</SPAN> <SPAN class="string token">'/query'</SPAN><SPAN class="punctuation token">,</SPAN> tracksParams<SPAN class="punctuation token">)</SPAN>
tracksResponse <SPAN class="operator token">=</SPAN> urllib2<SPAN class="punctuation token">.</SPAN>urlopen<SPAN class="punctuation token">(</SPAN>tracksReq<SPAN class="punctuation token">)</SPAN>
tracksResult <SPAN class="operator token">=</SPAN> json<SPAN class="punctuation token">.</SPAN>load<SPAN class="punctuation token">(</SPAN>tracksResponse<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

And whammo, I have the json output of the tracker service. From here I add some columns to the json for various things:

<SPAN class="keyword token">if</SPAN> tracksResult <SPAN class="keyword token">is</SPAN> <SPAN class="operator token">not</SPAN> None<SPAN class="punctuation token">:</SPAN>
     output <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="keyword token">for</SPAN> jj <SPAN class="keyword token">in</SPAN> tracksResult<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
	int_num <SPAN class="operator token">=</SPAN> int<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">(</SPAN>json<SPAN class="punctuation token">.</SPAN>dumps<SPAN class="punctuation token">(</SPAN>jj<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'attributes'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_timestamp'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
	utc <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%Y-%m-%d %H:%M:%S'</SPAN><SPAN class="punctuation token">,</SPAN> time<SPAN class="punctuation token">.</SPAN>gmtime<SPAN class="punctuation token">(</SPAN>int_num<SPAN class="operator token">/</SPAN><SPAN class="number token">1000.0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
	convTimeStamp <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%Y-%m-%d %H:%M:%S'</SPAN><SPAN class="punctuation token">,</SPAN> time<SPAN class="punctuation token">.</SPAN>gmtime<SPAN class="punctuation token">(</SPAN>int_num<SPAN class="operator token">/</SPAN><SPAN class="number token">1000.0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
	convDayStamp <SPAN class="operator token">=</SPAN> time<SPAN class="punctuation token">.</SPAN>strftime<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'%Y-%m-%d'</SPAN><SPAN class="punctuation token">,</SPAN> time<SPAN class="punctuation token">.</SPAN>gmtime<SPAN class="punctuation token">(</SPAN>int_num<SPAN class="operator token">/</SPAN><SPAN class="number token">1000.0</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
	jj<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'attributes'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> convTimeStamp
	jj<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'attributes'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> convDayStamp

	output<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>jj<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'attributes'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

So now I have a new output with some extra columns (converted those epoch datetime values to something meaningful). Now from here I am ready to do some grouping/summarizing using Pandas:

df <SPAN class="operator token">=</SPAN> pd<SPAN class="punctuation token">.</SPAN>DataFrame<SPAN class="punctuation token">.</SPAN>from_dict<SPAN class="punctuation token">(</SPAN>output<SPAN class="punctuation token">,</SPAN> orient<SPAN class="operator token">=</SPAN><SPAN class="string token">'columns'</SPAN><SPAN class="punctuation token">)</SPAN>
df<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="operator token">=</SPAN> pd<SPAN class="punctuation token">.</SPAN>to_datetime<SPAN class="punctuation token">(</SPAN>df<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
df <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>sort<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Here's where I am creating that running total column in the dataframe:

<SPAN class="comment token">#determine duration in seconds between the previous row's datetime value. Do this for each day.</SPAN>
df<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>sort<SPAN class="punctuation token">(</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>groupby<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_name'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_timestamp_local'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>diff<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="operator token">/</SPAN><SPAN class="number token">1000</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

All is good and well. Now for some grouping to sum that "elapsedSeconds" value.

grouped <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>groupby<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_name'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_rate'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'created_user'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>sum<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reset_index<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

And finally revert this grouped dataframe back into json:

dfjson <SPAN class="operator token">=</SPAN> grouped<SPAN class="punctuation token">.</SPAN>to_json<SPAN class="punctuation token">(</SPAN>orient<SPAN class="operator token">=</SPAN><SPAN class="string token">'records'</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

This is where I end up with an "overflowerror maximum recursion level reached". It doesn't seem to error when I run in pyScripter v2.6 x86 but it fails when I hook up this .py script to a Geoprocessing tool source. This is likely the issue (32-bit vs. 64?), I'm unsure and really just looking for workarounds if known.

Anyway -- thanks for looking and should be a fun one to figure out!

ArcGIS 10.4
Pandas 0.16.1

Find more posts tagged with

Accepted answers

JamesCrandall

Sort of a hack workaround I guess (no problem taking that criticism) but if I just convert the grouped result to a csv then back to a json output via each line as json.dumps(), I get the result I'm after.

Just added this simple def() to create the csv into the scratchFolder then return the json:

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">generateOutput</SPAN><SPAN class="punctuation token">(</SPAN>df<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>

    outputT <SPAN class="operator token">=</SPAN> <SPAN class="string token">'SessionFile_{}.{}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">(</SPAN>uuid<SPAN class="punctuation token">.</SPAN>uuid1<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"csv"</SPAN><SPAN class="punctuation token">)</SPAN>
    Output_File <SPAN class="operator token">=</SPAN> os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>join<SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>scratchFolder<SPAN class="punctuation token">,</SPAN> outputT<SPAN class="punctuation token">)</SPAN>

    df<SPAN class="punctuation token">.</SPAN>to_csv<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">,</SPAN> index<SPAN class="operator token">=</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">)</SPAN>
    fieldnames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_name'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_rate'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'created_user'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">)</SPAN>
    csvfile <SPAN class="operator token">=</SPAN> open<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'r'</SPAN><SPAN class="punctuation token">)</SPAN>
    reader <SPAN class="operator token">=</SPAN> csv<SPAN class="punctuation token">.</SPAN>DictReader<SPAN class="punctuation token">(</SPAN> csvfile<SPAN class="punctuation token">)</SPAN>
    outJson <SPAN class="operator token">=</SPAN> json<SPAN class="punctuation token">.</SPAN>dumps<SPAN class="punctuation token">(</SPAN> <SPAN class="punctuation token">[</SPAN> row <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> reader <SPAN class="punctuation token">]</SPAN> <SPAN class="punctuation token">)</SPAN>

    csvfile<SPAN class="punctuation token">.</SPAN>close<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
    arcpy<SPAN class="punctuation token">.</SPAN>Delete_management<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">)</SPAN>

    <SPAN class="keyword token">return</SPAN> outJson


grouped <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>groupby<SPAN class="punctuation token">(</SPAN>flds<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>sum<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reset_index<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
grouped<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">(</SPAN>grouped<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">/</SPAN> np<SPAN class="punctuation token">.</SPAN>timedelta64<SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'s'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">*</SPAN><SPAN class="number token">1000</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>astype<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">)</SPAN>
dfjson <SPAN class="operator token">=</SPAN> generateOutput<SPAN class="punctuation token">(</SPAN>grouped<SPAN class="punctuation token">)</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN>dfjson<SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

All comments

JamesCrandall

Just added this simple def() to create the csv into the scratchFolder then return the json:

<SPAN class="keyword token">def</SPAN> <SPAN class="token function">generateOutput</SPAN><SPAN class="punctuation token">(</SPAN>df<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>

    outputT <SPAN class="operator token">=</SPAN> <SPAN class="string token">'SessionFile_{}.{}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">(</SPAN>uuid<SPAN class="punctuation token">.</SPAN>uuid1<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"csv"</SPAN><SPAN class="punctuation token">)</SPAN>
    Output_File <SPAN class="operator token">=</SPAN> os<SPAN class="punctuation token">.</SPAN>path<SPAN class="punctuation token">.</SPAN>join<SPAN class="punctuation token">(</SPAN>arcpy<SPAN class="punctuation token">.</SPAN>env<SPAN class="punctuation token">.</SPAN>scratchFolder<SPAN class="punctuation token">,</SPAN> outputT<SPAN class="punctuation token">)</SPAN>

    df<SPAN class="punctuation token">.</SPAN>to_csv<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">,</SPAN> index<SPAN class="operator token">=</SPAN><SPAN class="token boolean">False</SPAN><SPAN class="punctuation token">)</SPAN>
    fieldnames <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="string token">'location_day'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_name'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'PEP_land_rate'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'created_user'</SPAN><SPAN class="punctuation token">,</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">)</SPAN>
    csvfile <SPAN class="operator token">=</SPAN> open<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'r'</SPAN><SPAN class="punctuation token">)</SPAN>
    reader <SPAN class="operator token">=</SPAN> csv<SPAN class="punctuation token">.</SPAN>DictReader<SPAN class="punctuation token">(</SPAN> csvfile<SPAN class="punctuation token">)</SPAN>
    outJson <SPAN class="operator token">=</SPAN> json<SPAN class="punctuation token">.</SPAN>dumps<SPAN class="punctuation token">(</SPAN> <SPAN class="punctuation token">[</SPAN> row <SPAN class="keyword token">for</SPAN> row <SPAN class="keyword token">in</SPAN> reader <SPAN class="punctuation token">]</SPAN> <SPAN class="punctuation token">)</SPAN>

    csvfile<SPAN class="punctuation token">.</SPAN>close<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
    arcpy<SPAN class="punctuation token">.</SPAN>Delete_management<SPAN class="punctuation token">(</SPAN>Output_File<SPAN class="punctuation token">)</SPAN>

    <SPAN class="keyword token">return</SPAN> outJson


grouped <SPAN class="operator token">=</SPAN> df<SPAN class="punctuation token">.</SPAN>groupby<SPAN class="punctuation token">(</SPAN>flds<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>sum<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>reset_index<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
grouped<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">(</SPAN>grouped<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'elapsedSeconds'</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">/</SPAN> np<SPAN class="punctuation token">.</SPAN>timedelta64<SPAN class="punctuation token">(</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'s'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">*</SPAN><SPAN class="number token">1000</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>astype<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">)</SPAN>
dfjson <SPAN class="operator token">=</SPAN> generateOutput<SPAN class="punctuation token">(</SPAN>grouped<SPAN class="punctuation token">)</SPAN>
arcpy<SPAN class="punctuation token">.</SPAN>AddMessage<SPAN class="punctuation token">(</SPAN>dfjson<SPAN class="punctuation token">)</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JamesCrandall

Thanks Dan. Probably something to do with it. I just dev off the resources I'm provided but will have to check with sys engineering to determine what's what I guess.

DanPatterson_Retired

Is the issue...

python 3.6.9, pandas 0.25.1 (ArcGIS Pro) vs python 2.7.x, pandas 0.16.1 (ArcMap 10.4)

Or is this only a python 2.7.x 32 bit vs 64 bit issue?

Because there are 5 years of pandas changes, not to mention numpy and python changes if it is the former

Release Notes — pandas 0.25.3 documentation

JamesCrandall

EDIT (may help to answer):

Just trying a different tact I exported the grouped result to a .csv file to see what the contents were. It looks like that "elapsedSeconds" column is in a timedelta?

I'm not sure but perhaps this is the cause of my error, but I'm still unsure what to do about it:

PEP_land_name,PEP_land_rate,elapsedSeconds
Campus 1,RP,0 days 00:00:00.022000000
Campus 2,PRE,0 days 00:00:00.009000000
Campus B2 Entrance East 1,PRE,0 days 00:00:00.013000000
Campus Parking Lot NE 1,RP,0 days 00:00:00.040000000
Campus Parking Lot NE 2,PRE,0 days 00:00:00.039000000

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.