topic Re: Error on shapefile generation... What's wrong ? in Developers Questions

Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Mon, 04 Apr 2016 08:16:26 GMT

Hi GeoNet people 🙂

I developped an application with a class able to read data from an Oracle database with MapXtreme (MapInfo), and convert them in Shapefile. This class follows the ESRI specifications from here : https://www.esri.com/library/whitepapers/pdfs/shapefile.pdf

It looks like an issue in my creation algorithm, but I really don't see what's wrong, it seems to follow well the ESRI specifications. I'm not a "GIS expert", so maybe some of you could help me to discover what's wrong.

Or if you have any GIS tool that can tell me what's not correct in my shapefile, it could really help me.

I already asked this question on QGIS forums and mailing list, where the generated shapefile cannot be well displayed, but I didn't got many help. I hope I'll have more luck here.

You can find in attachment the source code of my class in VB.Net and some generated shapefiles.

Many thanks !

Regards,

Re: Error on shapefile generation... What's wrong ?

VinceAngelo — Mon, 04 Apr 2016 15:09:49 GMT

You might get more responses if you provide a relevant code sample in the body of the question, along with a description of the error you are receiving.

- V

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Tue, 05 Apr 2016 07:58:56 GMT

Hello Vangelo,

Thank you for your answer.

I don't get any error message in my code, the issue comes from the

softwares used to open my generated shapefile.

For example, opening it in QGIS lead simply to not display it on the

map, completly or partially depending of file.

A QGIS user gave me a Python script showing that the spatial object is

simply "None", but that doesn't tell me what is wrong (maybe a EOF is off).

Putting all my code in my post isn't useful, because my code runs well,

this is the result that is wrong, but I don't see what.

That's why I'm asking some help to see if someone can check my code and

see if it follows well ESRI specifications, or if it exists a tool that

can analyze a shapefile to see if it follows ESRI specifications, or

anything that can tell me what I'm doing wrong.

Many Thanks.

Regards,

Le 04/04/2016 17:10, Vince Angelo a écrit :

GeoNet <https://community.esri.com/?et=watches.email.thread>

Error on shapefile generation... What's wrong ?
reply from Vince Angelo
<https://community.esri.com/people/vangelo-esristaff?et=watches.email.thread>
in /Developers/ - View the full discussion
<https://community.esri.com/message/599452?et=watches.email.thread#comment-599452>

Re: Error on shapefile generation... What's wrong ?

VinceAngelo — Tue, 05 Apr 2016 13:36:00 GMT

You need to debug your code, since it does not produce the correct output. We'd be willing to help, but you need to do the majority of the work. The other option would be to pay someone to review your code.

I've written a number of shapefile readers and writers in both Java and 'C', and the core code was under 100 lines. Once you've verified that the header isn't mangled, the code you'd need to post would be under 40 lines. If the problem is with dBase, then it might be as large as 60 lines (most of it in a large case statement by datatype).

Expecting folks to download and review your code without context may be the reason you didn't get help from the QGIS list. I know a question like this would be closed immediately in GIS StackExchange for failure to include code (an MCVE) in the post.

What we'd need to help you is for you to verify the dBase (.dbf) component, then review the 100 byte header contents, then review the individual shape contents. If you write exactly one shape then post the contents of bytes 101 through EOF as a hex dump (`od -t x1` is a wonderful tool), along with the code that generated that .shp record, we'd be able to tell you what was wrong.

- V

Re: Error on shapefile generation... What's wrong ?

ÁkosHalmai — Tue, 05 Apr 2016 14:50:19 GMT

It’s a quite interesting problem. If I add your shapefiles to ESRI ArcMap there are no visible glitches. There is 8 features in in the shapefile named “4480___327550” and 1 feature in the other one. Some of them are overlapping and duplicate, but rendering perfectly. I also tried to add it to the open source MapWindow GIS with the same result: no problem at all. After that I tried to import into GRASS GIS and it turned out that the GRASS (OGR) can’t import anything from “4480___S12_060_4”. The message is: “Error in fread() reading object of size 916 at offset 100 from .shp file”. I made a comparison in HEX viewer and there is no difference between the shapefiles, but the last byte of the shx file differs. If I replace the shx it works everywhere! So the problem is somewhere the value of “Content Length” in the index record.

Please, throw a comment if working.

Ákos Halmai

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Wed, 06 Apr 2016 07:35:01 GMT

Hi halmaia,

Thank you very much for your answer.

I don't know MapWindow GIS and GRASS GIS, I'll do a test as soon as I

can and do more investigation in the index file generation.

Many thanks !

Le 05/04/2016 16:50, Ákos Halmai a écrit :

GeoNet <https://community.esri.com/?et=watches.email.thread>

Error on shapefile generation... What's wrong ?
reply from Ákos Halmai
<https://community.esri.com/people/halmaia?et=watches.email.thread> in
/Developers/ - View the full discussion
<https://community.esri.com/message/599732?et=watches.email.thread#comment-599732>

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Wed, 06 Apr 2016 12:21:13 GMT

Hi Ákos Halmai,

I did your tests with MapWindow GIS and GRASS GIS and got same results.

In GRASS, I have many times error "Error in fread() reading object of

size 916 at offset 100 from .shp file", and warning "1 feature without

geometry skipped".

I'm just discovering GRASS, so which shx files are you comparing ? The

original I created and ... ? When I go to "grassdata" folder, I see a

"vector" folder with my export, but it doesn't look like a shapefile.

I did a test with the index file generation : writting content length in

little Endian instead of big Endian as said in specifications.

In that way, I have no more error messages, but always the warning "1

feature without geometry skipped".

Just wanted to check...

However, the index file generation is very simple : it has same header

than main file, and records of offset and content length.

Here is the code to generate the index file :

Snippet

Me.WriteBigEndian(9994, internalIndexStream)'File code (cf. ESRI Shapefile Technical Description)

Me.WriteBigEndian(0, internalIndexStream)'Unused (cf. ESRI Shapefile Technical Description)

Me.WriteBigEndian(indexFileLength, internalIndexStream)'File Length (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(1000, internalIndexStream)'Version (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(5, internalIndexStream)'Shape Type : 5<=>Polygone (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(boundingBoxXMin, internalIndexStream)'Xmin (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(boundingBoxYMin, internalIndexStream)'Ymin (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(boundingBoxXMax, internalIndexStream)'Xmax (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(boundingBoxYMax, internalIndexStream)'Ymax (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(0.0, internalIndexStream)'Zmin (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(0.0, internalIndexStream)'Zmax (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(0.0, internalIndexStream)'Mmin (cf. ESRI Shapefile Technical Description)

Me.WriteLittleEndian(0.0, internalIndexStream)'Mmax (cf. ESRI Shapefile Technical Description)

currentOffset = 50

For i = 0To containerOfLengthsForIndexFile.Count - 1

Me.WriteBigEndian(currentOffset, internalIndexStream)

Me.WriteBigEndian(containerOfLengthsForIndexFile(i), internalIndexStream)

currentOffset += containerOfLengthsForIndexFile(i)

Me.internalIndexStream.Close()

According to your tests, the last element of

"containerOfLengthsForIndexFile" should be invalid.

The counter is calculated each time writting a geometry (in

WriteGeometryToFile function, at record header) :

Snippet

Snippet Me.counterOfCurrentRecord = counterOfCurrentRecord + 1

Me.WriteBigEndian(counterOfCurrentRecord, internalMainTemporaryStream)

counterOfContentLength = 22'22 words of 16 bits

counterOfContentLength += (counterOfRings * 2)'Parts

counterOfContentLength += (counterOfPoints * (2 * 4))'Points

Me.WriteBigEndian(counterOfContentLength, internalMainTemporaryStream)

Me.containerOfLengthsForIndexFile.Add(counterOfContentLength + 4)

I don't understand why there is a "+4" at the last line.

This is code from a former employee, and I don't see anything in

specifications about that.

But by removing this "+4", it looks like well working now in any GIS

software !

I did a new export in that way. Can you just try to open it in other GIS

like ArcMap (I don't have it) to see if it opens well ?

It would be very nice.

Many thanks !

Le 05/04/2016 16:50, Ákos Halmai a écrit :

GeoNet <https://community.esri.com/?et=watches.email.thread>

Error on shapefile generation... What's wrong ?
reply from Ákos Halmai
<https://community.esri.com/people/halmaia?et=watches.email.thread> in
/Developers/ - View the full discussion
<https://community.esri.com/message/599732?et=watches.email.thread#comment-599732>

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Thu, 07 Apr 2016 07:27:07 GMT

Hi Ákos Halmai,

I did another test this morning with another shapefile containing more

polygons, and unfortunately the fix I did yesterday isn't working with

more than one polygon.

In GIS like QGIS, MapWindow or GRASS, they display only one polygon...

See sample in attachment.

I have to investigate further...

Regards,

Le 05/04/2016 16:50, Ákos Halmai a écrit :

GeoNet <https://community.esri.com/?et=watches.email.thread>

Error on shapefile generation... What's wrong ?
reply from Ákos Halmai
<https://community.esri.com/people/halmaia?et=watches.email.thread> in
/Developers/ - View the full discussion
<https://community.esri.com/message/599732?et=watches.email.thread#comment-599732>

Re: Error on shapefile generation... What's wrong ?

ÁkosHalmai — Thu, 07 Apr 2016 08:10:17 GMT

Hi,

I would like to check out the attachments, but there are no attachment in the comments, because (I assume) you used the answer function in your e-mail provider and in this case there might be no attachment support on the GeoNet. So you should attach the files with the GeoNet web-page inbuilt attachment support. The attachment is also missing in your previous comment, only the first contains one.

Ákos Halmai

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Thu, 07 Apr 2016 08:55:14 GMT

Hi Ákos Halmai,

Indeed, I was replying by email, but this is strange, I see the attachment in my previous post.

Anyway, please find another export (53503) with simplier polygons : first with 4 nodes, second with 3 nodes and third with 5 nodes.

The last one isn't displaying.

I'm currently checking the code step by step with these.

Regards,

Re: Error on shapefile generation... What's wrong ?

ÁkosHalmai — Thu, 07 Apr 2016 09:37:43 GMT

Hi,

There is the Arc-version of your files.

I think (I'm not sure) the problem somewhere with the offset: length + offset is not enough because there is a record number too.

H.Á.

Re: Error on shapefile generation... What's wrong ?

Jean-ChristopheBastin — Thu, 07 Apr 2016 11:51:47 GMT

Hi Ákos,

Thank you so much for your help !

I wasn't so far with my "+4 bug fix" in my previous post (+4 means index record size).

The +4 offset was put in the length of geometry object, not in the offset stored in index records.

You pointed to the right place, and I lost many time to test things that was already working.

It was just an issue in the way the index file was built.

Again, thank you very much, I'm definitly out with this bug now ! 🙂

Best regards,