topic Re: Error prepare data in ArcGIS API for Python Questions

Error prepare data

Big_DataManado — Mon, 02 Dec 2019 07:13:58 GMT

I'm trying to prepare training data with this detail :

ArcGIS Pro 2.4.2

inRaster = "E:\Work\ArcDL\Base\MANADO_FIX.tif" (raster 8bit unsigned, thematic, with nodata=0)
out_folder = "E:\Work\ArcDL\gtg6"
in_training = "E:\Work\ArcDL\Experiment_BuildFoot_2\MyProject\Shape\House.shp" (shp that has Classvalue, Classname, RED, GREEN, BLUE field)
image_chip_format = "TIFF"
tile_size_x = "256"
tile_size_y = "256"
stride_x="128"
stride_y="128"
output_nofeature_tiles="ONLY_TILES_WITH_FEATURES"
metadata_format="Labeled_Tiles"
start_index = 0
classvalue_field = "Classvalue"
buffer_radius = 1
in_mask_polygons = "E:\Work\ArcDL\Experiment_BuildFoot_2\MyProject\Shape\Mask.shp"
rotation_angle = 0

I run my prepare data syntax just like this :
data = prepare_data('gtg6')

But I always ended up in this kind of error for labeled_tiles metadata format, there's no further explanation and I cant use the data for training since data.show_batch() will result in OSError: -2

Any solution?

Thanks in advance.

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 09:33:17 GMT

Did you set a reasonable cell size in the environment variables?

Also can you post a screenshot of your exported tiles here so that we can have a look at them ?

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 09:44:46 GMT

Thank you.., I set my cell size to 0.3, and this is my screenshot at my exported files :

Because I did this based on tutorial Feature Categorization using Satellite Imagery and Deep Learning | ArcGIS for Developers then I try to implement that tutorial using the same data based on that tutorial (with cell size 0.1) and this is my screen shot at my exported tiles based on the tutorial :

but it still have the same error and in jupyter notebook it says Cannot read TIFF header :

anything wrong?
thanks in advance

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 09:52:55 GMT

You can try exporting your data in any other format like PNG and check if that works.

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 09:55:22 GMT

I did have... and no, it not working (I also test JPEG) :

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 09:57:56 GMT

I can see .tiff as well as .png files in this folder, can you try exporting your data to a new folder ?

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 10:00:27 GMT

I use different folder name (you can check at my previous I use "gtg6" for tiff and "gtg12" for png) but however the tiff always follow along... is there something wrong?

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 10:03:57 GMT

try exporting it in a new empty folder and let me know how it goes.

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 10:15:21 GMT

Here's the result :

I always exported to new folder since ArcgisPro won't allow if the output folder already exist.

Here's my environment :

and my ArcGIS Pro version

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 10:21:27 GMT

Let me try to reproduce this issue.

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 10:23:51 GMT

Thank you, I've been struggling for weeks in these prepare data thing... I don't have problem with pascal VOC or Classified Tiles metadata format from before.

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 10:39:48 GMT

Can you try install pillow again

pip uninstall pillow

and then install it again using

pip install pillow

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 11:17:09 GMT

To verify your pillow installation try to load an exported image chip directly using the following code

from PIL import Image
Image.open(image_path)

it should give a similar output as below

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 11:34:18 GMT

Now it seems to make sense, it resulted in an error :

so what should I do ? in another side I need this conda install -c pytorch -c fastai fastai=1.0.39 pytorch=1.0.0 torchvision command to prepare_data()

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 11:40:47 GMT

One more thing, it can read normal image png, but it won't work from image created from export training data

Re: Error prepare data

Anonymous User — Wed, 04 Dec 2019 11:54:36 GMT

Can you uninstall pillow

pip uninstall pillow

and then install it again using

pip install pillow

If still you are not able to read the image please attach the image here.

Re: Error prepare data

Big_DataManado — Wed, 04 Dec 2019 12:05:14 GMT

Okay, now I can read the Image from export training data but it leaves some error and I'm afraid it will cause something wrong later :

image successfully read :

should I run :

conda install -c fastai nvidia-ml-py3‍

since I'm using GPU for this

Re: Error prepare data

Anonymous User — Fri, 06 Dec 2019 06:53:34 GMT

Big Data Manado‌ we can ignore this warning for now.

Re: Error prepare data

Big_DataManado — Sat, 07 Dec 2019 00:08:05 GMT

Ok, Thank you for your help.

Re: Error prepare data

NicholasPiper — Sun, 26 Jan 2020 19:44:11 GMT

Sandeep,

I ran pip uninstall pillow and pip install pillow. I can now import an image with pillow. However, this appears to have broken my fastai:

Is there a way to fix pillow with out breaking fastai? Thank you for your time!

-Nick