POST
|
I have reported this problem to ESRI Japan and sent data(tmp_table). They confirmed that they reproduced this phenmenon in their environment.
... View more
01-15-2018
11:41 PM
|
0
|
0
|
609
|
POST
|
Thanks Dan and Xander Sorry for my impolite > Dan Xander's code worked well and the code below also worked ... arcpy.CalculateField_management(in_table="C:/Work01/scratch_00.gdb/tmp_table",
field="test",
expression="[COUNT]",
expression_type="VB") But, only this code doesn't work. Of course, it works well for other tables. (I done it manually for the one that worked(Only changed table name) and copied the python snippet from the results window.) arcpy.CalculateField_management(in_table="C:/Work01/scratch_00.gdb/tmp_table",
field="test",
expression="!COUNT!",
expression_type="PYTHON_9.3") tmp_table: OID Value COUNT SUM test -1 6 5E+08 1.61E+09 For example, this code works well (copied the python snippet from the results window). arcpy.CalculateField_management(in_table="C:/Work01/scratch_00.gdb/tmp_table2",
field="test",
expression="!COUNT!",
expression_type="PYTHON_9.3") tmp_table2: OID Value COUNT SUM test -1 6 5.1E+08 1.61E+09 Thanks again Dan and Xander
... View more
01-15-2018
07:14 PM
|
3
|
0
|
609
|
POST
|
Oh...sorry I use "C:/Work01/scratch_00.gdb/tmp_table" and try again. The result is .... ERROR 000539: SyntaxError: unexpected EOF while parsing (<expression>, line 1) same error... I think this may not be the Unicode issue....
... View more
01-15-2018
06:07 PM
|
0
|
2
|
609
|
POST
|
Thanks Dan I tried again according to your advice. 1 arcpy.CalculateField_management(in_table=u"C:\Work01\scratch_00.gdb\tmp_table", field="test", expression="!COUNT!", expression_type="PYTHON_9.3", code_block="") I got parameter error.This is due to prefix of path(ur). Runtime error Traceback (most recent call last): File "<string>", line 1, in <module> File "c:\program files (x86)\arcgis\desktop10.5\arcpy\arcpy\management.py", line 3663, in CalculateField raise e ExecuteError: 実行できません。パラメーターが不正です。 ERROR 000732: 入力テーブル: データセット C:\Work01\scratch_00.gdb mp_table が存在しないか、サポートされていません (CalculateField) を実行できませんでした。 2 arcpy.CalculateField_management(in_table=ur"C:\Work01\scratch_00.gdb\tmp_table", field="test", expression="!COUNT!", expression_type="PYTHON_9.3", code_block="") I got the same error. Runtime error Traceback (most recent call last): File "<string>", line 1, in <module> File "c:\program files (x86)\arcgis\desktop10.5\arcpy\arcpy\management.py", line 3663, in CalculateField raise e ExecuteError: ERROR 000539: SyntaxError: unexpected EOF while parsing (<expression>, line 1) (CalculateField) を実行できませんでした。 I think this may not be the Unicode issue....
... View more
01-15-2018
04:53 PM
|
0
|
4
|
2126
|
POST
|
Sorry for my poor info. table info tmp_table:table class tmp_table:C:\Work01\scratch_00.gdb\tmp_table the definition of both fields COUNT:Float test:Float Result window original メッセージ 実行: CalculateField tmp_table test !COUNT! PYTHON_9.3 # 開始時間: Mon Jan 15 15:01:42 2018 ERROR 000539: SyntaxError: unexpected EOF while parsing (<expression>, line 1) (CalculateField) を実行できませんでした。 失敗 (Mon Jan 15 15:01:42 2018) (経過時間: 0.01 秒) translation message execute: CalculateField tmp_table test !COUNT! PYTHON_9.3 # start: Mon Jan 15 15:01:42 2018 ERROR 000539: SyntaxError: unexpected EOF while parsing (<expression>, line 1) Failed to execute(CalculateField) Failure (Mon Jan 15 15:01:42 2018) (elapsed time: 0.01 sec) I checked two scripts below,but I got the same error. arcpy.env.workspace = ur"C:\Work01\scratch_00.gdb"
arcpy.CalculateField_management(in_table="tmp_table", field="test", expression="!COUNT!", expression_type="PYTHON_9.3", code_block="")
arcpy.CalculateField_management(in_table="C:\\Work01\\scratch_00.gdb\\tmp_table", field="test", expression="!COUNT!", expression_type="PYTHON_9.3", code_block="")
... View more
01-15-2018
04:29 PM
|
0
|
6
|
2126
|
POST
|
Hi Jayanta, Here is my script. In addition, I got the same error when using ArcMap toolbox. # Replace a layer/table view name with a path to a dataset (which can be a layer file) or create the layer/table view within the script
# The following inputs are layers or table views: "tmp_table"
arcpy.CalculateField_management(in_table="tmp_table", field="test", expression="!COUNT!", expression_type="PYTHON_9.3", code_block="")
... View more
01-15-2018
12:08 AM
|
0
|
1
|
2126
|
POST
|
Hi! I tried copy values from COUNT attribute to test attribute. (expression ="!COUNT!") I used CalculateField_management, but CalculateField_management generates ERROR 000539 depending on values. TEST1:I tried CalculateField using this table. OID Value COUNT SUM test -1 6 5E+08 1.61E+09 I got the error below. ExecuteError: ERROR 000539: <type 'exceptions.SyntaxError'>: unexpected EOF while parsing (<expression>, line 1) Failed to execute (CalculateField). TEST2:I tried CalculateField using this table whose value in COUNT attribute changed . OID Value COUNT SUM test -1 6 5.1E+08 1.61E+09 This time, CalculateField worked well. Why value"5.1E+08" worked well and did not work value"5E+08" ? How should I avoid this problem? my arcGIS version is 10.5 Thanks in advance.
... View more
01-14-2018
10:26 PM
|
0
|
15
|
3653
|
POST
|
Thank you for your information. According to your advice, I added the code below. arcpy.SetLogHistory(False) SetLogHistory: http://desktop.arcgis.com/ja/arcmap/10.3/analyze/arcpy-functions/setloghistory.htm Then, you are right.Log setting caused mainly the issue.The result is first time:about10sec 51st time:about 25sec(improved!) But, the itrate process still slightly slow down... It can not be helped,can it?
... View more
12-10-2017
11:23 PM
|
0
|
1
|
1925
|
POST
|
Additional information I changed python version from 3.X to 2.7(changed Arc from pro to Map). This slow down has not happened! Is this the matter of ArcGIS pro 2.0???
... View more
12-08-2017
01:07 AM
|
0
|
4
|
1925
|
POST
|
I try to the process below. spatial search tile raster index polygon with feature rasterize feature combine rasterized feature and MODIS raster and export combine raster attribute table (loop for tiles) table operation with pandas and arcpy This process works well.However, process gradually slow down. I try this process for 51 times and check time. All input features are same. The result is below.Each time_1,time_2 time_3 is described below. time_1:finish process 2(rasterize feature) time_2:finish process 3(combine and export table) time_3:finish process 4(table operation) elapsed_time:finish 1 feature process(save output in gdb file) first time INFO:root:land_00244 1/51 START INFO:root:time_1:2.9405651092529297[sec] INFO:root:time_2:3.73823618888855[sec] INFO:root:time_2:4.30132269859314[sec] INFO:root:time_2:4.817508697509766[sec] INFO:root:time_2:5.239825248718262[sec] INFO:root:time_3:9.50986933708191[sec] INFO:root:elapsed_time:10.557835817337036[sec] 51 time INFO:root:land_00244_49 51/51 START INFO:root:time_1:14.089014768600464[sec] INFO:root:time_2:19.874730348587036[sec] INFO:root:time_2:25.050605058670044[sec] INFO:root:time_2:30.117022275924683[sec] INFO:root:time_2:35.16779851913452[sec] INFO:root:time_3:51.1801700592041[sec] INFO:root:elapsed_time:53.90104866027832[sec] Even though same feature, process time increased 5 times! All processes slow down.Why? My environment is below. OS:windows 10 pro ArcGIS:10.5.1 ArcGIS pro:2.0 But, in windows 7 and ArcGIS(10.4.1 and pro 1.4), this slow down didn't occur. Thanks in advance! from __future__ import print_function, unicode_literals, absolute_import
try:
import urllib2 # Python 2
except ImportError:
import urllib.request as urllib2 # Python 3
# Import arcpy module
import pandas as pd
import arcpy
import sys
import os
import socket
import gc
import time
import numpy as np
import logging
# Set overwrite option
arcpy.env.overwriteOutput = True
# datapath
DATA_DIR = r"C:\Data"
WORK_DIR = r"C:\Work01"
MODIS = DATA_DIR + "\\" + "MODIS_2001.gdb"
INDEX_POLY = DATA_DIR + "\\" + "MODIS_index.gdb"
def proc_zonalStat(feature_lyr, feature_path, list_index):
arcpy.env.workspace = "in_memory"
arcpy.SelectLayerByLocation_management(list_index[0], "INTERSECT", feature_lyr)
raslist_modis = []
cursor = arcpy.da.SearchCursor(list_index[0], ["F_NAME"])
for row in cursor:
raslist_modis.append(MODIS + "\\" + row[0])
del cursor
ras_feature = "in_memory"+ "\\ras_feature"
arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326)
arcpy.PolygonToRaster_conversion(feature_lyr, "OBJECTID", ras_feature, cellsize=0.0041666667)
time_1 = time.time() - start_time
logging.info("time_1:{0}".format(time_1) + "[sec]")
count = 0
aggreFlg = True
for inValueRaster in raslist_modis:
try:
tableName = "MODIS_tbl" + str(count).zfill(3)
outCombine = arcpy.sa.Combine([ras_feature, inValueRaster])
arcpy.TableToTable_conversion(outCombine, scratchGdb, tableName)
arcpy.Delete_management(outCombine)
count += 1
time_2 = time.time() - start_time
logging.info("time_2:{0}".format(time_2) + "[sec]")
fieldName = "MODIS_254"
expression = "0"
arcpy.CalculateField_management(feature_lyr, fieldName, expression, "PYTHON_9.3")
arcpy.Delete_management(ras_feature)
tables = arcpy.ListTables("MODIS_tbl*")
list_df_MODIS = []
fields = ['ras_feature', 'MODIS_landCover_', 'COUNT']
for table in tables:
# numpy array -> pandas dataframe
arr = arcpy.da.TableToNumPyArray(table, fields)
df = pd.DataFrame(arr, columns=fields)
list_df_MODIS.append(df)
del arr
gc.collect()
del tables
gc.collect()
df_all = pd.concat(list_df_MODIS, ignore_index=True)
add = pd.DataFrame([[0,0,0],
[0,1,0],
[0,2,0],
[0,3,0],
[0,4,0],
[0,5,0],
[0,6,0],
[0,7,0],
[0,8,0],
[0,9,0],
[0,10,0],
[0,11,0],
[0,12,0],
[0,13,0],
[0,14,0],
[0,15,0],
[0,16,0],
[0,254,0]],
columns=['ras_feature', 'MODIS_landCover_', 'COUNT'])
df_all = df_all.append(add)
del add
grouped = df_all.groupby([df_all['ras_feature'], df_all['MODIS_landCover_']], as_index=False).sum()
arr_MODIS = grouped.pivot_table(index='ras_feature', columns=['MODIS_landCover_'],values=['MODIS_landCover_','COUNT'],fill_value=0)
arr_MODIS = arr_MODIS.reset_index().values
arr_MODIS = arr_MODIS.astype(np.float32)
struct_array = np.core.records.fromarrays(arr_MODIS.transpose(),
np.dtype([('Value', np.uint64),
('MODIS_0', np.float32),
('MODIS_1', np.float32),
('MODIS_2', np.float32),
('MODIS_3', np.float32),
('MODIS_4', np.float32),
('MODIS_5', np.float32),
('MODIS_6', np.float32),
('MODIS_7', np.float32),
('MODIS_8', np.float32),
('MODIS_9', np.float32),
('MODIS_10', np.float32),
('MODIS_11', np.float32),
('MODIS_12', np.float32),
('MODIS_13', np.float32),
('MODIS_14', np.float32),
('MODIS_15', np.float32),
('MODIS_16', np.float32),
('MODIS_254', np.float32)]))
tmp_table = tmp_proc_Gdb + "\\tmp_table"
arcpy.da.NumPyArrayToTable(struct_array, tmp_table)
del struct_array, grouped, arr_MODIS, df_all, list_df_MODIS
gc.collect()
tmp_table_view = tmp_proc_Gdb + "\\tmp_table_view"
arcpy.MakeTableView_management(tmp_table, tmp_table_view, "NOT Value = 0")
arcpy.AddJoin_management(feature_lyr, "OBJECTID", tmp_table_view, "Value")
feature_name = os.path.basename(feature_path)
for i in range(0,17):
fieldName = feature_name + ".MODIS_" + str(i)
expression = "!tmp_table.MODIS_" + str(i) + "!"
arcpy.CalculateField_management(feature_lyr, fieldName, expression, "PYTHON_9.3")
fieldName = feature_name + ".MODIS_254"
expression = "!tmp_table.MODIS_254!"
arcpy.CalculateField_management(feature_lyr, fieldName, expression, "PYTHON_9.3")
arcpy.RemoveJoin_management(feature_lyr)
arcpy.Delete_management(tmp_table_view)
arcpy.Delete_management(tmp_table)
time_3 = time.time() - start_time
logging.info("time_3:{0}".format(time_3) + "[sec]")
# Set scratchWorkspace
arcpy.env.scratchWorkspace = "in_memory"
# Set local_outGDB
THREAD = 0
hostName = socket.gethostname()
local_outGdb_name = 'tmp_' + hostName + '_' + str(THREAD).zfill(2) + '.gdb'
scratchGdb = "in_memory"
local_outGdb = WORK_DIR + '\\' + local_outGdb_name
tmp_proc_Gdb = "in_memory"
logfile_name = WORK_DIR + "\\log_test.txt"
logging.basicConfig(filename=logfile_name,level=logging.DEBUG)
logging.info("************** MAIN PROCESS START **************")
arcpy.env.workspace = WORK_DIR
for gdb in arcpy.ListFiles("test.gdb"):
if not arcpy.Exists(WORK_DIR + '\\' + local_outGdb_name):
arcpy.CreateFileGDB_management(WORK_DIR, local_outGdb_name)
if arcpy.Exists(WORK_DIR + '\\Res\\' + gdb):
arcpy.env.workspace = WORK_DIR + '\\Res\\' + gdb
features_end = arcpy.ListFeatureClasses()
arcpy.env.workspace = WORK_DIR + '\\' + gdb
features_tmp = arcpy.ListFeatureClasses()
features = [ feature for feature in features_tmp if feature not in features_end ]
else:
arcpy.env.workspace = WORK_DIR + '\\' + gdb
features = arcpy.ListFeatureClasses()
totalNum = len(features)
counter = 0
index_MODIS_lyr = "in_memory" + "\\" + "idx_MODIS_lyr"
list_index = [index_MODIS_lyr]
arcpy.MakeFeatureLayer_management(INDEX_POLY + "\\MODIS_landCover_2001", index_MODIS_lyr)
for feature in features:
try:
start_time = time.time()
arcpy.env.workspace = WORK_DIR + '\\' + gdb
arcpy.env.parallelProcessingFactor = "100%"
counter += 1
logging.info(feature+" "+str(counter) + '/' + str(totalNum)+" START")
tmp_feature = tmp_proc_Gdb + "\\tmp_feature"
arcpy.CopyFeatures_management(feature, tmp_feature)
feature_lyr = tmp_proc_Gdb + "\\feature_lyr"
arcpy.MakeFeatureLayer_management(tmp_feature, feature_lyr)
rows = arcpy.GetCount_management(feature_lyr)
countFeature = int(rows.getOutput(0))
del rows
arcpy.AddField_management(feature_lyr, "MODIS_0" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_1" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_2" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_3" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_4" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_5" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_6" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_7" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_8" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_9" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_10" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_11" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_12" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_13" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_14" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_15" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_16" , "FLOAT")
arcpy.AddField_management(feature_lyr, "MODIS_254" , "FLOAT")
proc_zonalStat(feature_lyr, tmp_feature, list_index)
out_feature = local_outGdb + "\\" + feature
arcpy.CopyFeatures_management(tmp_feature, out_feature)
arcpy.Delete_management(feature_lyr)
arcpy.Delete_management(tmp_feature)
elapsed_time = time.time() - start_time
logging.info("elapsed_time:{0}".format(elapsed_time) + "[sec]")
except arcpy.ExecuteError:
logging.info(arcpy.GetMessages(2))
arcpy.Delete_management("in_memory")
arcpy.MakeFeatureLayer_management(INDEX_POLY + "\\MODIS_landCover_2001", index_MODIS_lyr)
outDir = WORK_DIR + "\\Res"
outGdbPath = outDir + "\\" + gdb
if not arcpy.Exists(outGdbPath):
arcpy.CreateFileGDB_management(outDir, gdb)
arcpy.env.workspace = local_outGdb
features = arcpy.ListFeatureClasses()
arcpy.FeatureClassToGeodatabase_conversion(features, outGdbPath)
arcpy.Delete_management(local_outGdb)
logging.info("************** MAIN PROCESS E N D **************")
... View more
12-07-2017
10:13 PM
|
0
|
6
|
2981
|
POST
|
Thanks Though adding Delete_management , the problem still occurred. I tried the script below. arcpy.env.overwriteOutput = True arcpy.env.workspace = r"C:\Data\check.gdb" features = arcpy.ListFeatureClasses() feature = features[0] feature_lyr = "in_memory\\featureLyr" arcpy.MakeFeatureLayer_management(feature, feature_lyr) ras_feature = "in_memory\\RasFeature" arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326) arcpy.PolygonToRaster_conversion(feature_lyr, "OBJECTID", ras_feature, cellsize=0.00025) When arcpy.PolygonToRaster_conversion, the script crashed. Because saving to the disk work, I try the modified script. arcpy.env.overwriteOutput = True arcpy.env.workspace = r"C:\Data\check.gdb" features = arcpy.ListFeatureClasses() feature = features[0] feature_lyr = "in_memory\\featureLyr" arcpy.MakeFeatureLayer_management(feature, feature_lyr) tmp = r"C:\Data\check.gdb\tmpRas" ras_feature = "in_memory\\RasFeature" arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326) arcpy.PolygonToRaster_conversion(feature_lyr, "OBJECTID", tmp, cellsize=0.00025) arcpy.BuildRasterAttributeTable_management(tmp, "Overwrite") arcpy.CopyRaster_management(tmp, ras_feature) In this script, crash happened when arcpy.CopyRaster_management.... As you said, it's hard to pin down...
... View more
11-08-2017
05:21 PM
|
0
|
1
|
658
|
POST
|
>Also, report your cell size, extent, projection and snap raster used I add these information. extent and snap raster are not used. cell size and projection are below. arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(4326) arcpy.PolygonToRaster_conversion(feature_lyr, "OBJECTID", ras_feature, cellsize=0.00025)
... View more
11-07-2017
10:59 PM
|
0
|
0
|
658
|
POST
|
Sorry for late reply >Dan did it fail when saved to disk instead of in_memory? did you monitor system resources during a run how about trying the same workflow... but use a smaller file I answer your questions. 1:No,it didn't fail.Saving to disk worked well. 2:I monitored memory consumption.When start PolygonToRaster, it increased about 10GB,then crash. it doesn't seem to be memory capacity problem. 3:trying some smaller polygon,it didn't fail(crash)... it's the matter of polygon(complex shape, a lot of vertices, lots of features in polygon)? >Jayanta Thanks for your information! I set scratch workspace to "in_memory". I'm sorry that this is not the problem of model builder...
... View more
11-07-2017
09:46 PM
|
0
|
1
|
658
|
POST
|
Hi, I try to zonal statistics as Table in script. my workflow is Copy feature to "in_memory" Make layer in "in_memory" PolygonToRaster zonal statistics as Table In this workflow, output of "PolygonToRaster" is temporal (finally delete). So, I tried to save it at "in_memory" workspace, but not worked(crashed). output raster size is about 5GB(save to gdb on HDD and check size), My memory is 32GB.Memory seems to be enough. What should I do? I use ArcGIS pro 1.4.1 Thanks in advance!
... View more
10-31-2017
08:11 PM
|
0
|
7
|
933
|
POST
|
Hi,Shaun, version of ArcGIS Pro:1.4.1 version of Windows:windows 7 professional SP1 (64bit) I ran your code, and it porduced the same error! import locale, time locale.setlocale(locale.LC_ALL, 'ja') 'ja' time.strftime('%a') Runtime error Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: embedded null byte I wish this was helpful for you.
... View more
10-24-2017
07:39 PM
|
0
|
1
|
2492
|
Title | Kudos | Posted |
---|---|---|
3 | 01-15-2018 07:14 PM |
Online Status |
Offline
|
Date Last Visited |
11-11-2020
02:24 AM
|