topic Failure to apply a custom geographic transformation in ArcGIS Pro Questions

Failure to apply a custom geographic transformation

JoeLandersarup — Sat, 13 Mar 2021 23:25:41 GMT

Hello,

I have create a custom geographic transformation for my project. I want to use this transform data on the fly in my map in ArcGIS Pro, so I can view multiple dataset in different co-ordinate systems together. Some of these cannot be reproejcted, such as the WMS layer from the British Geological Society (BGS) Borehole Database.

I have these layers:

BGS Borehole WMS layer - British National Grid (In the form of a projected coord system OSGB_1936)
historical borehole data for project - project specific projected coordinate system (Created by a company called SnakeGrid. The project specific projected coord system is called TPEN11-TM, and has an accompanying geographic coord system called TPEN11-IRF.

I have created a custom transformation using the ArcGIS tool (and using the TPEN11-TM.prj, TPEN11-IRF and NTv2 .gsb files provided by Snakegrid for this project). This transformation is for converting between geographic system GCS_ETRS89 to the geographic system TPEN11_IRF. The ArcGIS Pro built-in transformers can then convert the data from ETRS89 tto OSGB 1936 (using OTSN02 NTv2 .gsb).

ArcGIS recognises the custom transformation I have created, see attached screenshot (select_transformation.png). The name of the custom transformation is 'TN15-ETRS89-to-TPEN11-IRF'. ArGIS Pro the adds '+OSGB 1936 to ETRS_1989_1' to convert it the rest of the way. However the problem is that once I click 'ok' to close the box it will not apply this transformation on my map. My data on the map doesn't transform and is not displaying together. When I re-open the map transformations list, the list is still at 'none'. See attached screenshot 'none_selected'. This is very frustrating.

To cut a long story short, Has anyone else come across this problem of ArcGIS Pro not applying a custom transformation that is clearly being recognised in the list by ArcGIS Pro and is selectable? Any ideas why it won't apply?

On a semi-related note, the custom transformation works perfectly to project data. I have been successful in using the 'project' tool to use my custom transformation to convert data from British National Grid into my project specific co-ordinate system (see image 'project_transformer.png' So there is clearly nothing wrong with the accuracy of the custom transformation, it's working perfectly using the 'project' tool, it just won't be applied by ArcGIS Pro for on-the-fly transformation (which is what I badly need).

I'm using ArcGIS Pro 2.7

Re: Failure to apply a custom geographic transformation

DanPatterson — Sun, 14 Mar 2021 02:23:53 GMT

@HeatherSmith posted a blog recently

Projection on the fly and geographic transformations (esri.com)

I am not sure whether that adding it to your "Favorites" will provide the capability to find it for coding or as defaults.

She will get a message about this post. Perhaps she can comment further

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Sun, 14 Mar 2021 14:52:01 GMT

Thanks Dan, I came across the blog by @HeatherSmith before making the post (it's an excellent post btw), however that isn't the problem I'm experiencing.

The problem is to do with ArcGIS Pro not applying the transformation when I select it. It always reverts back to 'none', even when I select the custom transformation in the list.

I can provide my dataset, if people want to test it.

Re: Failure to apply a custom geographic transformation

DanPatterson — Sun, 14 Mar 2021 15:13:11 GMT

I just checked bug reports to see if anything was remotely connected

I think this is the one

BUG-000096165: ArcGIS Pro does not honor on-the-fly custom transfor.. (esri.com)

others for interest

BUG-000135135: In ArcGIS Pro, an applied custom transformation meth.. (esri.com)

BUG-000125948: Arcpy.ListTransformations() is inconsistent when wor.. (esri.com)

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Sun, 14 Mar 2021 17:08:16 GMT

Thanks Dan for the list of bug reports (I didn't know about them as a resource). They describe the problem exactly. 😞

Re: Failure to apply a custom geographic transformation

DanPatterson — Sun, 14 Mar 2021 21:33:37 GMT

Perhaps you could close... sad answer that it was

Re: Failure to apply a custom geographic transformation

Tim_McGinnes — Sun, 14 Mar 2021 23:25:50 GMT

Maybe try manually putting the transformation in? Save the map to a .mapx file. Open in text editor and look for the datumTransforms section. Should look something like the below when you have one. Modify as required (the ____ parts), save, and Import Map into a new project. Does the ETRS89 to OSGB 1936 transform get added from the properties? Maybe put it in by itself before you export as a good starting point.

"datumTransforms" : [ { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : ____, "latestWkid" : ____, "transformForward" : false, "name" : "____" }, { "wkid" : ____, "latestWkid" : ____, "transformForward" : true, "name" : "____" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : ____, "latestWkid" : ____, "transformForward" : false, "name" : "____" }, { "wkid" : ____, "latestWkid" : ____, "transformForward" : true, "name" : "____" } ] } } ],

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Mon, 15 Mar 2021 00:17:31 GMT

Thanks for the suggestion Tim.

I saved as mapx, and opened in text editor. See below block.

There is alot of transformations in the code. I'm not sure why. However when I look at the list of transformations within the map properties, there is a very long list under the 'Additional transformations' stating that 'These transformations are currently not in use on the map'. See my attached screenshot. The weird thing is, the first one on that list, is exactly the one I'm trying to use apply to the map!!!

As for the list of transformations in the mapx file, the 'wkt' for the custom transformation is a very long piece of text. But also there isn't any 'wkid' number like there is for the other built-in transformations.

"datumTransforms" : [ { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5339, "latestWkid" : 5339, "transformForward" : false, "name" : "OSGB_1936_To_WGS_1984_7" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5339, "latestWkid" : 5339, "transformForward" : false, "name" : "OSGB_1936_To_WGS_1984_7" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : false, "name" : "TN15-ETRS89-to-TPEN11-IRF" }, { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : false, "name" : "OSGB_1936_To_ETRS_1989_1" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : false, "name" : "TN15-ETRS89-to-TPEN11-IRF" }, { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : false, "name" : "OSGB_1936_To_ETRS_1989_1" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : false, "name" : "TN15-ETRS89-to-TPEN11-IRF" }, { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : false, "name" : "OSGB_1936_To_ETRS_1989_1" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : false, "name" : "TN15-ETRS89-to-TPEN11-IRF" }, { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : false, "name" : "OSGB_1936_To_ETRS_1989_1" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : true, "name" : "OSGB_1936_To_ETRS_1989_1" }, { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5338, "latestWkid" : 5338, "transformForward" : true, "name" : "OSGB_1936_To_ETRS_1989_1" }, { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" }, { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" }, { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkt" : "GEOGTRAN[\"TN15-ETRS89-to-TPEN11-IRF\",GEOGCS[\"GCS_ETRS_1989\",DATUM[\"D_ETRS_1989\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],GEOGCS[\"GCS_TPEN11-IRF\",DATUM[\"D_TPEN11-IRF\",SPHEROID[\"GRS_1980\",6378137.0,298.257222101]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],METHOD[\"NTv2\"],PARAMETER[\"Dataset_uk/TN15-ETRS89-to-TPEN11-IRF\",0.0]]", "transformForward" : true, "name" : "TN15-ETRS89-to-TPEN11-IRF" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" } ] } }, { "type" : "CIMDatumTransform", "forward" : true, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5339, "latestWkid" : 5339, "transformForward" : true, "name" : "OSGB_1936_To_WGS_1984_7" }, { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" } ] } }, { "type" : "CIMDatumTransform", "forward" : false, "geoTransformation" : { "geoTransforms" : [ { "wkid" : 5339, "latestWkid" : 5339, "transformForward" : true, "name" : "OSGB_1936_To_WGS_1984_7" }, { "wkid" : 1149, "latestWkid" : 1149, "transformForward" : false, "name" : "ETRS_1989_To_WGS_1984" } ] } } ],

Re: Failure to apply a custom geographic transformation

BojanŠavrič — Mon, 15 Mar 2021 00:25:58 GMT

Hi @JoeLandersarup,

With the new Transformation UI in Pro 2.7, we resolved some issues regarding custom transformations described here. Can you please share with me your dataset, TPEN11-TM.prj, TPEN11-IRF and NTv2 .gsb files, so we can test it? I sent you my contact via direct message. Thanks!

Bojan

Re: Failure to apply a custom geographic transformation

BojanŠavrič — Mon, 15 Mar 2021 00:31:01 GMT

BUG-000096165 and BUG-000135135 (based on their repro steps) appear to be resolved with the latest release, and both issues should be soon marked appropriately. Thanks!

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Mon, 15 Mar 2021 18:34:22 GMT

Thanks very much @BojanŠavrič, I didn't receive a direct message, but I sent you a direct message with the dataset and my contact details.

Joe

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Mon, 15 Mar 2021 18:34:01 GMT

Thanks @Tim_McGinnes

I tried your suggestion and the .mapx had really long list of 'datumTransforms'. I didn't really know what I was looking at so decided to leave it. As Bojan has kindly offered to look at the dataset, I'll see if ESRI can solve my issue.

Joe

Re: Failure to apply a custom geographic transformation

JoeLandersarup — Wed, 17 Mar 2021 16:20:16 GMT

Update:
@BojanŠavrič identified the issue and provided a very simple temp fix as follows:

The .prj files provided had an unfortunate format error where the spheroid was set to: SHPEROID["GRD 1980"... when it should be SPHEROID["GRS_1980"... the space " " needed to be replaced with a "_".

This had a knockon effect on the custom transformation I created which also had the same space " " error for the SPHEROID.

Once I fixed this error in all the prj files and the gtf file, and cleared my cache for the files, the transformation started working and all my data aligned.

This is what was causing my bug in ArcGIS Pro (and also ArcMap 10.8.1). This is such a relief for our project!

I'm submitting a ticket for ESRI to track this type of bug (and maybe build in mitigation measures to prevent this in future?)

Thanks all for help.