topic Re: “Shift” tool has performance issue, in ArcGIS Enterprise Questions

“Shift” tool has performance issue,

JamalNUMAN — Sat, 20 Jul 2019 07:37:14 GMT

“Shift” tool has performance issue,

I observed that the “Shift” tool has performance issue. Is it normal that a raster of 113 MB takes around 4 minutes in order to be shifted (-50000, -500000)? My machine is with 4 Cores X 64GB RAM

Is there any recommendation to increase the performance of this tool?

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sat, 20 Jul 2019 07:59:03 GMT

Shift—Data Management toolbox | ArcGIS Desktop probably the same in arcmap

but anyway....

you didn't specify a file extension, so it isn't the shift that is taking a long time, but it is the conversion to an esri grid in the ZZ folder.

If you want to shift specify and output type by giving the raster an extension like *.tif... (ecw isn't on ArcGIS Pro's list of output types)

Re: “Shift” tool has performance issue,

JamalNUMAN — Sat, 20 Jul 2019 10:14:03 GMT

Many thanks for the input Dan.

I tried it with jpg extension. it took around 2 min and 15 sec.

I observed that as the ecw file (113 MB) is shifted with jpg extension, then ovr file is created with a size of 150 MB. This means that the 113 ecw file is ending up with 135 MB for jpg in addition to 150 MB for the ovr. This means that the size is tripled just to shift the raster!

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sat, 20 Jul 2019 11:11:11 GMT

try another format... esri usually recommends *.tif

Re: “Shift” tool has performance issue,

JamalNUMAN — Sat, 20 Jul 2019 11:40:47 GMT

Now the size of the tif and ovr are much bigger: 1.5 GB for the tif and 350 MB for the ovr

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sat, 20 Jul 2019 15:04:43 GMT

did you try convert first, then shift? when you convert, you can specify the compression etc

Re: “Shift” tool has performance issue,

JamalNUMAN — Sat, 20 Jul 2019 15:55:02 GMT

I have converted ecw to tif and then the tif is shifted:

-The size of tif file is much bigger than ecw

-When shifting tif, the size stays the same (as expected)

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sat, 20 Jul 2019 16:12:47 GMT

In the raster to other format tool... did you go to the environments tab and check the compression being used?

Compression (Environment setting)—Geoprocessing | ArcGIS Desktop

You can experiment with compression types and bit settings. You haven't indicated what bit type the input data are but I can assume 8-bit unsigned from this

Raster file formats—ArcGIS Pro | ArcGIS Desktop

But since ecw is proprietary, then you will have to go to something else. tif will be larger, *.png is smaller and lossless (I used it for all imagery/graphics regardless whether for gis or not).

So if raster size is the issue, you are fighting a lost cause... SID and ecw are the squished choices, tif, png are widely used and jpeg has had its day

Re: “Shift” tool has performance issue,

JamalNUMAN — Sun, 21 Jul 2019 21:10:25 GMT

I’ll be trying to go for the png option as it is better than other formats in terms of size. However, the performance stays low as one raster needs around 4 minutes to be converted and shifted.

This means that the 1,600 rasters need around 4.5 days to be converted and shifted

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sun, 21 Jul 2019 22:04:22 GMT

Was it a projection issue Jamal?

If it was, I am wondering if a batch reprojection wouldn't be better.

Also, the png format produces a world file. it might be faster to just go and edit it in a text editor (or via a script) to just adjust (aka, shift) the coordinates there. If the coordinates are embedded in the file then you can't do this, but if there is a world file, it can be done.

why don't you zip and post all the bits of the *.png file except the png… it the *.aux, *.pnw, *.ovr etc etc

Re: “Shift” tool has performance issue,

JamalNUMAN — Tue, 23 Jul 2019 16:49:05 GMT

Precisely Dan.

The issue here is to convert from “Israel_TM_Grid” to “Palestine_1923_Palestine_Grid”.

Unfortunately, none of the built-in algorithms available in ArcGIS can perform the job properly. We found out that the best way to do it is to move the rasters with -50000 in X and -500000 in Y. This way, the “Israel_TM_Grid” is converted precisely to “Palestine_1923_Palestine_Grid”.

Correct. Each raster has its aux:

<SRS>PROJCS["Israel_TM_Grid",GEOGCS["GCS_Israel",DATUM["Israel",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",219529.584],PARAMETER["False_Northing",626907.39],PARAMETER["Central_Meridian",35.20451694444445],PARAMETER["Scale_Factor",1.0000067],PARAMETER["Latitude_Of_Origin",31.73439361111111],UNIT["Meter",1.0]]</SRS>

</PAMDataset>

What might be the best practice to move the raster with the value (-50000, -500000) other than applying the shift tool?

However, we are forced to convert them from ecw to jpg due to the fact that the ArcGIS Server doesn’t support ecw when it comes to publishing.

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Tue, 23 Jul 2019 20:22:47 GMT

It is the png's world file *.pnw that I would be interested in seeing. In theory, subtracting those values prior to shifting that would save you the shifting slowness

Re: “Shift” tool has performance issue,

JamalNUMAN — Tue, 23 Jul 2019 20:46:06 GMT

The pgw and aux are attached

Thank you

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Sat, 11 Dec 2021 17:54:47 GMT

As a test, and if I read you right, I just subtracted 50,000 from X and 500,000 from y in your arl-3.pgw yielding.

Rename arl-3.pgw to arl-3.old

copy and paste the below into a text editor (ie Notepad) and name it arl-3.pgw. and save it in the same folder.

Add the image

Does it line up properly?

0.5000000000
0.0000000000
0.0000000000
-0.5000000000
169980.0000000000
170020.0000000000‍‍‍‍‍‍

Re: “Shift” tool has performance issue,

JamalNUMAN — Wed, 24 Jul 2019 21:03:38 GMT

Replacing the values of the pgw with the ones you have provides does stick the raster in the right place.

Does this mean that need to subtract the (-50000, -500000) from each pgw files in order to stick the 1600 raster in their correct position?

Re: “Shift” tool has performance issue,

DanPatterson_Retired — Wed, 24 Jul 2019 21:13:35 GMT

yes... time for some scripting to

read the *.pnw, rename them *.pnw_old,
get the appropriate lines containing the X, Y values,
do the math and
save the file out to a new copy so as to retain the old.

You wouldn't want to do this manually... even though it would be quicker than shifting.

Re: “Shift” tool has performance issue,

BojanŠavrič — Fri, 26 Jul 2019 15:20:24 GMT

Hello Jamal NUMAN! I just want to verify if I understand your comment correctly. Were you not able to line-up your data using any of the transformation paths below in map properties or by projecting it with project tool and available transformation paths below? Thanks!

Re: “Shift” tool has performance issue,

JamalNUMAN — Sat, 27 Jul 2019 06:23:34 GMT

Correct. None of the transformations paths make the data line-up precisely.

Despite the fact that the “Israel_TM_Grid” coordinate system is based on “GRS_1980” Spheroid and “Transverse_Mercator” projection while the “Palestine_1923_Palestine_Grid” is built from“Clarke_1880_Benoit” and Spheroid “Cassini” projection:, they can be converted from one another by subtracting\adding (50000, 500000) to their x and y respectively.

Re: “Shift” tool has performance issue,

ArwaShaqour — Mon, 19 Aug 2019 08:05:08 GMT

Hi All,

I managed to develop a python script that performs the required action as shown in the screenshots below. However, I’m struggling at the moment in how to create a tbx tool that can be used to run the script. The script is attached.

Ahmad SALEH‌ , Jamal NUMAN‌

Best,

Arwa

Re: “Shift” tool has performance issue,

AhmadSALEH1 — Mon, 19 Aug 2019 14:47:42 GMT

Arwa,

There are two approaches to get this job done, the most simple and easy one is by creating a script tool .

But first, you have to edit your script to access the parameters via GetParameterAsText() "see the attached file".

then go to arcmap > create a new toolbox

right click on the toolbox>add>script and follow the snapshots below

let me know if you have any questions,

Thanks,