nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Slow performance reading FGDB

ahuarte

Hello, I'm using this wonderful API to read FGDBs. I'm checking the performance and I am not getting very favorable results compared to the same data in ShapeFile.

For a layer ~500,000 records, the FGDB reader needs ~5sg, but the SHAPE reader less than half.

The sourcecode is equivalent, but if it is true that I use FileMapping in the SHAPE reader and I load the DBF values only when needed. The FGDB reader is implemented in a pair C# and C++ .NET DLLs.

It In order to accelerate the FGDB API, I wonder if FileMapping is used internally in the API.

Is it possible to raise some of these optimizations?

1. Access the values ??????in a row, also for field index.
2. Capable of maintaining data in a row so disconnected from their 'EnumRows' father, to load data only when needed, and not have to read all the row values ??????forever. Or access the data using a byte stream recovered from the row, or something

Vind meer berichten getagd met

Accepted answers

All comments

ahuarte

Thank you very much for your repply!

The fact that the testing you have done is impressive. There are certain details that are noticed only with managing large volumes of data.

I look forward to seeing if implemented access to the attributes of a faster 🙂

VinceAngelo

Just to close up this thread, I did some experimentation with some custom
tools to measure access performance of a random-generated dataset...

First I generated a 100k row shapefile with roughly the same size .shp as
.dbf (50.8Mb and 49.1Mb, respectively). Then I used ArcGIS 10 to populate
it in a file geodatabase (66.7Mb).

Next I used a 'C' app to time how long it took to read the files (pure I/O,
without any semantic parsing). On my reference machine (a 4-CPU/4Gb RAM
laptop running 64-bit Windows 7) [and after an initial read pass to cache
the I/O], it took 220, 188, and 265 milliseconds to access the .shp, .dbf,
and a0000000a.gdbtable files (respectively). Each time represents ~1 micro-
second per 512-byte block (kicking the blocksize up to 256k with setvbuf
dropped all access times to under 100ms).

I then wrote a custom app with the File Geodatabase API, and applied the
millisecond timer to break down the processing times on a FGDB query into
three components -- Open (OpenGeodatabase + Geodatabase.OpenTable +
table.GetFieldInformation + various FieldInfo calls), Read (table.Search +
EnumRows.Next), and Get (row.Get*), plus a total:

FGDBa 0 1229 1673 2902

Parsing just the geometry data (without a change to the selection list)
was much faster:

FGDBg 0 1170 0 1170

As was limiting the selection list to just "SHAPE" (though, curiously, not
as fast as a "*" column list and just calling GetGeometry):

FGDBs 16 1122 48 1186

I then instrumented a shapefile reader with the same millisecond timer and
compared the access performance using five different access methodologies:
A) Bind query on all columns

Bind query on *only* the .shp component (simulating an empty .dbf)
C) Using getter by column number on all columns
D) Using getter by column name on all columns
E) Using a custom getStream function

Timing is in milliseconds, with values Open (includes column describe),
Read, Get, and Total --

10 cols:
shpA 0 2387 0 2387
shpB 0 1622 0 1622
shpC 0 2372 280 2652
shpD 16 2480 484 2980
shpE 0 2449 125 2574

To determine the impact of column count I reorganized the dBase attributes
into more columns of shorter length (to preserve the overall size), and then
re-ran the measurements with 20, 50 and 100 columns using the FGDB and
shape methodologies above:

20 cols:
FGDBa 0 1240 4096 5336
FGDBg 16 1310 94 1420
FGDBs 16 1404 0 1420
shpA 0 2480 0 2480
shpB 0 1638 0 1638
shpC 0 2434 405 2839
shpD 0 2355 1202 3557
shpE 0 2464 188 2652

50 cols:
FGDBa 15 2200 17628 19843
FGDBg 0 2090 31 2121
FGDBs 16 2105 16 2137
shpA 0 2574 0 2574
shpB 0 1638 0 1638
shpC 0 2399 736 3135
shpD 0 2577 6003 8580
shpE 0 2620 219 2839

100 cols:
FGDBa 0 3819 61669 65488
FGDBg 0 3370 46 3416
FGDBs 16 3417 16 3494
shpA 0 2652 0 2652
shpB 0 1638 0 1638
shpC 0 2840 1014 3854
shpD 0 2844 24846 27690
shpE 0 2620 485 3105

Since the time progression on both FGBDa and shpD suggest a "big O
N-squared" algorithm issue (doubling the columns increases duration by
roughly four times), I re-wrote my shapefile findColByName function
to use a circular search algorithm (each search starts with the column
after the last found position), and then re-tested (methodology F):

10 cols:
shpF 0 2299 447 2746
20 cols:
shpF 0 2465 468 2933
50 cols:
shpF 0 2780 808 3588
100 cols:
shpF 0 2574 2028 4602

In conclusion, it appears that the getter calls on attributes with the
FILEGDB API do have a significant performance cost (this is especially
true when there are a large number of attributes in the table). Fortunately
for your case, you have the option of recoding to delay the use of 'Get'
attribute accessors until they are actually necessary.

I've recommended that a circular search algorithm be used in the getter
and setter functions, and that consideration be given to overloading the
Row accessors using an integer radix (vice name) which should give uniform
access cost without regard to the attribute retrieval order.

- V

ahuarte

ok, thanks

I will try in ideas.esri.com

VinceAngelo

The problem with your #1-B example is that it ignores the possibility of NULLs. If you want
to submit an ER at ideas.esri.com, you'll need to add a bitmap of length (ncols+7)/8 bytes
to the front of the bytestream. I still don't think this will result in significant time savings.

- V

ahuarte

For example I've detailed.

Or an very configurable fast map viewer or a GIS format converter .... where the application does not know the input format of the geometries, and the format reader does not know the use geometries that will make them.

Not to lose, all I ask to see if feasible, is there a method on the Row that offers a byte[] of the attributes similar to what is the geometry.

Thank you very much

VinceAngelo

I can't think of a single application I've ever written that needed a full table scan for every query
(and didn't know exactly what attributes would be needed). FGDB would be much faster than
shapefile with a spatial subset, but because of your particular requirements, shapefile format
is going to be your best option.

- V

ahuarte

The cost of time I already know which are the n calls Row::GetXXX () via C#/C++, my SHAPE reader does not make a priori.

The problem is that in reading the geometry do not know what attributes are required in the subsequent use of it will make the application, therefore I load all or do I have everything necessary for that geometry.

In the painting of the map, the Renderer can use some of the attributes of the geometries, but a priori in reading of the element I do not know.

In the case of reading from SHAPE I only get the byte[] of the row of DBF, and if when the Renderer is painting and requires a value, then I parse the stream.

For heavy layers this behavior saves precious time (~6sec -> ~3.6sec for 500,000 records).

So suggested a similar approach it was possible for FGDBs.

Thank you very much

VinceAngelo

My son's doing fine now, thanks.

There is no reason to iterate the attribute list if you don't need the attributes. I doubt it's
required even if you do select all the columns, but I know it's not required if you select
only the geometry column in the Search request that generates the EnumRows iterator.
(If nothing else, you should specify the geometry column first.)

File Geodatabase format uses a compression algoritm on the geometries, but if the
"a000*.gdbtable" that corresponds to your table is larger than the .shp on the shapefile,
there is no chance for the FGDB to be faster in a full-table scan query (and even if it is
smaller, there's still the decompression cost to factor into the equation).

I'd recommend you modify the Search, then instrument your code with a millisecond
resolution timer, so you can see where the application is spending its time while reading
in the two different formats.

- V

ahuarte

The important thing, I hope you son gets better soon!

Regarding the rest, I know what to say. It is clear that the cost is reading the disc. It is also clear that the best optimization of a code is not to make unnecessary calls.

I do not know if I've managed to explain the particular casuistry.

I do not mean to be read data from disk twice, but that all the attributes of the row can be obtained in one byte[] or something.

Note that now I must make n calls C#/C++ to obtain the values â??â??of the attributes and then in the render of the geometry on a map are not required.

I think adding this functionality can be interesting, defer the interpretation of attributes, as read from the disk to a single byte stream to be interpreted by the client code when they are strictly necessary.


FgdbDataRow row = ...

// #0) Geometry.
Element tempElement = Element.FromShape( row.GetGeometry() );

// #1) Attributes.
element.pProperties = new Property[10];

// #1-A) -> Current Option.
for (int i = 1, icount = 10; i < icount; i++)
{
   object value = null;

   switch (m_oFeatureClass.GetEsriPropertyType(field))
   {
      case EsriPropertyType.esriFieldTypeInteger:
      {
         int valueT = 0;
         if (row.GetInteger(property.Name, ref valueT)) value = valueT; //-> ### 10 calls C#/C++ to Row::GetInteger!
         break;
       }
       ...
   }
   property.Value = value;
}

// #1-B) -> Proposed Option.
byte[] pMstream = row.GetAttributesBuffer(); //-> One call!
element.BufferAttribs = pMstream;
...
for (int i = 1, icount = 10, offsetPos = 0; i < icount; i++)
{
   // If I need de i-attributte.
   int value = BitConverter.ToInt32(element.BufferAttribs, offsetPos+=4);
   property.Value = value;
}

Thank you very much!

VinceAngelo

You haven't provided your code, and I haven't done performance benchmarking on FGDBAPI,
but I'm willing to wager my son (with a 103.5 fever and cranky as all get-out) that the real
cost is in reading the row from disk, not in the accessor function to copy the row members
from memory.

I'm quite fond of the ArcSDE 'C' API which uses row accessor functions *or* bind variables --
It's so much cleaner to organize a "copy" as:

 
while ((sr = SE_stream_fetch(stream1))== SE_SUCCESS) {
    if ((sr = SE_stream_execute(stream2)) != SE_SUCCESS) {
        /* handle write error */
}    }
if (sr != SE_FINISHED) {
    /* handle read error */
}

but the accessor design pattern is well-established (especially in Java, where bind variables
are considered poor form).

The only time that accessor functions are really inefficient is when transferring large BLOBs
(and CLOBs, NCLOBs,..., and geometries). The majority of your cost difference is in the fact
that you're *not* reading the .dbf component (and therefore doing half the I/O).

You can certainly file an enhancement request for an alternate method to access the row
buffer, but since ArcGIS doesn't even use bind variables with ArcSDE, it's probably going
to be a hard sell.

- V

ahuarte

I agree, this time nor does it seem slow :-), but if the user are accustomed to wait 2 seconds rendering a layer, with the format change this time is twice and the user does not mind the advantage of new format.

I think that the key is the "expensive" calling to FileGDBAPI::Row::GetXXXX() in order to recover the attribute values of a row, n fields -> n calls to these functions.

Most of the time the calls to these functions are unnecessary, but I have to make if you need to retrieve later any alphanumeric values �??of a geometry. If possible get a single "byte stream" of the attribute values of a row like your "ShapeBuffer" for the geometry, the time savings will be greater, the greater the number of fields (n fields -> the best one call, in the worst case n+1 calls).

Thank you very much for your replies!

VinceAngelo

Five seconds doesn't sound slow to do a full table scan on 500Mb of data. Can you provide
the code you're using in both cases?

There are a number of advantages of file geodatabases over shapefiles, among them:
+ Support for numeric nulls
+ Second resolution date type (vice day resolution of dBase)
+ Variable width attribute rows (vice dBase fixed-width)
+ Unlimited width string fields (vice 254 in dBase)
+ 64 character attribute field names (vice 11 in dBase)
+ Full support for all types supported by ArcGIS (NSTRING, UUID, CLOB, NCLOB)
+ SQL query access
+ Open access to spatial and attribute index queries

- V

ahuarte

Sorry my bad English!

~5sg -> approximately 5 seconds

The data is a polygon feature class, about 500,000 records with aprox 500mb shapefile (SHP+DBF+SHX).

I dont cache the data, I am reading the data as a IDataReader ADO.NET.

Speaking in the context of a map viewer, most of the time only need to read the geometry. If a user sets a color theme based on an attribute, then if that is necessary to read any of the attributes. But the "expensive" read the rest may be postponed. This would be possible if you could decouple row of your reader, or if it could get a "byte stream" of data to be treated as necessary

VinceAngelo

What are you using to read the shape file?

What is "~5sg"?

Is the data point, line, or polygon?

How large is the file that corresponds to the table in the FGDB? How large are the .shp and .dbf
files of the shapefile?

Are you saying that you cache 500K rows of geometry in RAM, and only access attributes when
you need them? What is your access criteria?

- V

Doe mee aan het gesprek

Aangemelde leden kunnen berichten plaatsen, updates volgen en meer. Nieuw hier? Registreer een gratis account.

Snelle Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.