topic Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice in File Geodatabase API Questions

CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

SimonWood — Thu, 31 Jan 2013 00:35:41 GMT

Hi,

Came across what looks to be a linux 64-bit specific bug when opening the same gdb twice (using different FileGDBAPI::Geodatabase instances) simultaneously - the CloseDatabase call is never returning.

Ive attached some code to reproduce this (just need to modify the path to the gdb to point to something valid).

On the same linux rhel6 (64 bit) machine, this code:
* works fine when built as 32-bit
* hangs forever on the first CloseDatabase call (debugger shows it is stuck in FileGDBAPI::Geodatabase::CloseGeodatabase() ), when built as 64-bit.

Its based on one of the samples, so Im just using "make" to build 64-bit and "make ARCH=32" to build 32-bit.. And Im using v1.3 of the API, but also happens with earlier versions.

I assume this is a fundamentally supported thing to do (?) - I can't see anything in the docs to the contrary..

Thanks,

Regards,
Simon

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

VinceAngelo — Fri, 01 Feb 2013 01:35:15 GMT

What do you gain by opening the same geodatabase twice in the same application?
Multiple queries and/or inserts from a single connection are supported (though multi-
threaded access is not). Given the additional overhead, I'd think this would be a
practice to avoid, just on general principle.

- V

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

SimonWood — Wed, 06 Feb 2013 03:28:18 GMT

Hi,

Yep there is no specific reason to open the same geodatabase more than once... really just a consequence of the implementation (fitting the FileGDB API into existing data structures made this easiest thing to do).

I will add in some reference counting code to handle this in my particular context.

Thanks.
Simon

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

SimonWood — Thu, 07 Feb 2013 20:40:05 GMT

Although having done that, it would still be good to avoid the hanging in this scenario to prevent occurences that are difficult to detect in client code (ie its not necessarily easy to resolve two paths to the same underlying file due to things like symbolic links, relative paths, alternative mount points, exotic file systems etc etc).

Cheers,
Simon

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DavidSousa — Mon, 03 Jun 2013 20:23:46 GMT

I can't imagine why this would only be happening on 64 bit Linux. There have been some investigations into this issue, but I don't think any insight has yet been gained.

Having said that, you shouldn't have to resort to ref-counting of the geodatabase object in your application. We are already doing that for you. In general, the geodatabase depends on the single instance model for objects of various types. For example, this includes the geodatabase and table objects. When a call is made to open a geodatabase or table, we first search an object cache to see if that object has already been instantiated. If so, rather than re-instantiating the same object, we add-ref the existing object and return a reference to it. This avoids needless overhead, but also ensures that the internal state of the object remains consistent.

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

ViksitAgarwal — Tue, 18 Jun 2013 06:29:27 GMT

I too observed this issue, but on Windows-7 64-bit OS with C++ FileGDB 1.3 VS 2010 API. I have a class that has Geodatabase as the member variable where in destructor of my class I am trying to close the Geodatabase and it gets destroyed when the instance of my class goes out of scope. I designed my class assuming that each Geodatabase object is independent and can be used independently in their independent multiple threads. But the above replies reveals that Geodatabase objects are pointing to the same memory internally, and hence leading to problems in our code. I think the same holds good for Table object as well. If the internal implementation of Geodatabase wants to reuse the same instance and using reference counting internally, then it seems there is a bug in Geodatabase implementation that destructing or closing one instance is causing problems in other instance. This problem is more evident in 64-bit.

Any help or suggestions on this issue would be of great help. Otherwise I need to use my own reference counting mechanism as used by simonwood74.

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

VinceAngelo — Tue, 18 Jun 2013 12:12:54 GMT

The library is documented as being NOT thread-safe. You'll need to structure
your multi-threaded code to take this into account.

- V

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DaveBrann — Tue, 18 Jun 2013 19:46:08 GMT

The library is documented as being NOT thread-safe. You'll need to structure
your multi-threaded code to take this into account.

- V

Sorry, I have search the entire File Geodatabase API documentation and can find no text that states the FGDB API is not thread safe. Can you point me to it, please?

Thanks,
Dave

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

LanceShipman — Tue, 18 Jun 2013 21:03:58 GMT

It is in the Unix read me file, but is missing from all of the others. It's been added in for the next release.

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DaveBrann — Wed, 19 Jun 2013 12:27:55 GMT

Thanks for the reply.

The UNIX README states "3) There is no multi-threading support at this time.". But this is a little vague to me. Does that mean an app can only have a single thread accessing the FGDB API? Therefore, effectively have an STA environment for accessing the API?

What I would like to do is have several worker threads, such that each worker independently creates the Geodatabase object over an FGDB database, and create an EnumRows, iterates over the table rows, and then deleted all objects it created. The above discussion indicates there is a problem doing this at the Geodatabase object level. But could I have a locking mechanism to ensure only one thread accesses the Geodatabase object at a time? This won't work if the Geodatabase uses thread local storage or is a COM object that requires and STA thread, I suppose.

Thanks,
Dave

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DavidSousa — Wed, 19 Jun 2013 15:03:58 GMT

The API does not use COM. There are no COM objects, no STA, just C++ classes. There is no use of thread local storage either.

The limitation on multi-threading is due to the use of LibXML by the API. LibXML has global state and cannot be used concurrently by more than one thread per process.

There might be certain workflows that don't make any use of LibXML during processing. In those cases, multi-threading might just work. But even then, if an individual database or table was being accessed simultaneously by more than one thread it might lead to trouble. It would be better to segregate access to tables to separate threads.

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DaveBrann — Wed, 19 Jun 2013 16:42:11 GMT

Thanks again for the information. It is very helpful.

In a post in the discussion, it was mentioned that FGDB code caches Geodatabase and Table objects and just reference count them if the same database and table are requested by an API user. If that is true, then it seems like it would not be possible to segregate access to tables in separate threads. Correct?

Thanks,
Dave

Re: CloseDatabase hanging on 64 bit Linux when same File Geodatabase opened twice

DavidSousa — Wed, 19 Jun 2013 19:37:26 GMT

It would mean using a separate thread for each table. That is under your control.

I am not sure this would work, but it's much more likely to work this way than it would if the same table was accessed by more than one thread.

This also assumes that whatever processing is happening is not making any use of LibXML in more than one thread at a time. The main use of LibXML is when the API is doing anything related to schema, e.g., creating a new table, describing an existing table, adding or removing a column, etc. All of those operations involve the use of XML. But XML usage in minimized if all you are doing is opening a table and getting its rows.