topic Re: importing survey data in Coordinate Reference Systems Questions

importing survey data

francescarusso — Fri, 18 Nov 2016 15:37:50 GMT

Hi all,

I have received an hardrive from a geophysical survey with the following data:

MBES- DTM : Density xyz xyz file

DTM xyz xyz file

Slope xyg xyz file

MAG : text file

SSS: Mosaic: .tif and .twf file

Raw data: xtf data

SBP: .sgy files

I do not know how to import them into ArcGIS 10.2 Desktop and I do not have 3D Spatial Analyst tool. (But I have QGIS which sometimes makes the job).

Can anyone help me?

Thanks a lot

Re: importing survey data

NeilAyres — Fri, 18 Nov 2016 16:12:21 GMT

The xyz files will probably be text. So open them in something like Notepad++ and have a look.

tif is an image so that should be no problem.

No idea what a *.sgy file might be, but again, have a look at it in a text editor just to see what the contents are.

And, of most importance, you will need to know, up front, what the coord sys is for all this stuff.

Re: importing survey data

francescarusso — Mon, 21 Nov 2016 09:28:58 GMT

Morning Neil,

thanks for your answer. I had a look at the data in a notepad as suggested.

Please see below the contents:

this is the kind of data into the DTM folder :

I was expecting a column with something as depth and I cannot see it.

This is the content of the SLOPE folder:

With this one I have another "issue". When I paste into the excel with the "Use Text Import Wizard" to keep the separation of the columns, I got a message "Text extends beyond worksheet. Some text will be lost".

I really do not know what to do as it doesn't paste all the data.

I hope you can help.

Thanks

Re: importing survey data

NeilAyres — Mon, 21 Nov 2016 09:51:23 GMT

That first table looks a bit odd. Where's the separator? Looks like they are just merging into each other.

Possibly the first row should look like :

625079.50  4835170.50  2.41‍

But is that Z (3rd column) a realistic number for this area?

The second one seems to at least have a tab or space separating the columns.

Note that, for this snippet, the X is exactly spaced 1m(?) apart. Check the Y's as well.

Perhaps, the linefeeds / newlines are not being interpreted correctly when you import, and it is all coming across as one very long line, that's why you run out of columns.

Re: importing survey data

NeilAyres — Mon, 21 Nov 2016 09:55:35 GMT

Do you know MS Access. The text importer there is quite flexible. For example, that first table could be imported using a fixed column positions.

Re: importing survey data

francescarusso — Tue, 22 Nov 2016 16:29:16 GMT

Dear Neil,

thank you very much for your suggestions.

I have found out that other DTM xyz data are not merged as the example I sent to you. So probably it has been an issue for just one acquisition.(I will check with the Survey company).

As I have mentioned before I do not have the 3d Spatial Analyst tool. As already said when I paste all the data into excel I run out the rows because the data rows are more than what excel can handle. So I have tried to split all the data in different sheets for the same excel document (which requires lots of patient). When I try to save the document as Excel 97-2003 which is the excel version that GIS is able to open I get another message of not compatibility. So again I cannot import my excel into GIS. I do not have MS Access too. So do you have any suggestion to copy these data into GIS? (maybe with another version of excel?)

Thanks a lot for all your help.

Re: importing survey data

NeilAyres — Tue, 22 Nov 2016 16:55:04 GMT

Probably the quickest is to use some python magic.

If you can post some example files here, then we can import them to points.

But, if you haven't got SA, then turning them back into images will be a problem.

Do you know what the coordinate system is yet?

Re: importing survey data

francescarusso — Thu, 24 Nov 2016 10:04:35 GMT

Dear Neil,

thanks for coming back to me. I am in the process of asking to the survey company to deliver the data in a different format which is readable in GIS.

Before that, could you please confirm me that the GIS cannot read other excel versions rather than Excel 97-2003?

Is it true that even if I save the document into .csv I won't solve the problem of running out the excel's rows?

If I would have the 3d and spatial analyst tool. would be any difference? Will I be able to read the .xyz directly from GIS without importing in excel?

Thanks once more for your support.

Re: importing survey data

NeilAyres — Thu, 24 Nov 2016 11:44:32 GMT

Regarding the column thing.

I just suggested that, because of the spacing / tabs or whatever, the excel importer was not seeing the data correctly and was trying to import the whole file as a series of column (ie not row by row), that's why you ran out of columns.

Ask them to send you it in standard csv (comma separated) style. Then, hopefully excel would be able to interpret it correctly.

And, by the way, you should be able to import csv directly into Arc without going through excel.

When you get it into a point feature class, you will need spatial analyst to "rasterise" it.

And, because these points seem to be on a very regular grid, it looks to me as if this is a "text dump" of a raster anyway. Each row represents the cell centre coordinates and value.

Is this coming from Geosoft? If so, get them to send just the original geosoft "*.grd" files. This is Geosoft's own propriety raster format. You can download a plugin from geosoft for ArcGIS, which will enable you to read the raster, then save to an Arc format like file geodb or tiff.

And, don't forget about the coordinate system for this data.

Re: importing survey data

francescarusso — Thu, 24 Nov 2016 12:10:00 GMT

Hi Neil,

thanks for your prompt reply.

I tried to paste into excel the data that in the .xyz file were looking separated rather than the one merged. But I still got the issue of running out rows (to me looked like the importing was alright) is just that they are millions of data and as far as I know excel run out rows at some point. That is why I tried to split the data in multiple sheets (but this is crazy enough) and afterwards I have the problem of saving in a compatibility mode which anyway doesn't allow me to do the job.

I will try to figure out if they can send the .csv files and hopefully I will be able to open directly from the ArcGIS.

My idea is to skip the problem of not having the 3d tool:

- importing the xyz into ArcGIS

- Create a feature

- Interpolate it via QGIS

I have used QGIs for interpolation once (but I managed to achieve that because I already had a points shapefile rather than a xyz file).

I do not know about Geosoft. I will try to find out that one as well.

Thanks a lot and have a nice day.

Re: importing survey data

francescarusso — Wed, 07 Dec 2016 13:51:03 GMT

Hi Neil,

These "issue" for me has not been solved yet. We asked to the survey company to deliver the data in GIS format to let me able to use them.

In the meanwhile I have a couple of new things to ask.

Is the 3D Analyst licence the same of Spatial Analyst licence? We are in the process of taking some decision about purchasing a new licence but I am not sure what I really need now. I have another task for this project which is to compare the Bathy contour from two different survey in different years.

Let's say Bathy A contour - Bathy B contour.

I am not sure whether it something I can achieve with 3d analyst or Spatial Analyst (or whether once I buy the 3d analyst I will have automatically the Spatial analyst tools activated as well)

Thanks a lot

Francesca

Re: importing survey data

NeilAyres — Wed, 07 Dec 2016 14:26:56 GMT

No 3D analyst is not the same as Spatial Analyst. They are separate extensions (but some of use have wondered for years why they are not bundled together.

SA does most of the raster and raster surface stuff. 3DA tins and you can do quite a lot with 3d geometries (Multipatches).

If you are dealing with comparisons of surfaces (like bathymetry), you will probably want to do that with raster surfaces and use SA. But could also do this with 3DA using tins.

Confusing, huh...

Re: importing survey data

francescarusso — Wed, 07 Dec 2016 14:35:49 GMT

Hi Neil,

Thanks a lot for the prompt reply and for help. I do not think my company will be happy to buy2 licences at the same time (considering we use GIS few times).

So I might suggest to buy the 3d analyst. So do you confirm it is possible to do it with 3da using tins? ( I might need some help afterwards)

Thanks

Re: importing survey data

NeilAyres — Wed, 07 Dec 2016 14:47:39 GMT

Not sure, I would probably go with spatial analyst, but that's just me.

spatial-analyst-toolbox

3d-analyst-toolbox

Re: importing survey data

francescarusso — Wed, 07 Dec 2016 14:51:47 GMT

Thanks.

Do you know if is there any chance to do it with QGIS?

Re: importing survey data

NeilAyres — Wed, 07 Dec 2016 14:56:03 GMT

You can do a lot of stuff with QGIS, but this is the esri forum!

Re: importing survey data

francescarusso — Wed, 07 Dec 2016 16:12:10 GMT

Totally right. sorry about that. and thanks for your suggestions.

Have a nice day

Re: importing survey data

francescarusso — Fri, 23 Dec 2016 14:30:36 GMT

Hi Neil,

I have one more question about survey data.

I got other offshore survey data. the survey has been divided into Blocks.

For the same block let s say block 1 I have 3 files:

- LAT_XYG_Block1.xyz

- LAT_XYS_Block1.xyz

-LAT_XYZ_Block1.xyz

X and Y coordinates are the same, but from your experience, do you have any idea of what would be the main differences between the files?

As per our previous correspondences I am not able to import the data into GIS at the moment so I have no clue of what this data might be. Any suggestion will be much appreciated.

Thanks a lot in advance and I wish you a Merry Christmas!

Cheers,

Re: importing survey data

NeilAyres — Mon, 09 Jan 2017 08:49:55 GMT

Sorry for the long delay, holidays....

Can you post a snapshot of the first few lines in each file? What do the numbers look like?

Re: importing survey data

francescarusso — Fri, 13 Jan 2017 15:36:07 GMT

Hi Neil,

thank you for coming back to me. I wish I could take a screenshot of the data, but it seems that my computer cannot open them anymore with the Notepad (probably because they are huge and my computer is almost dying). Luckily it seems I do not have to deal with these raw data anymore so for the moment I have just given up.

Thank you anyway for your help

Have a nice weekend