Can Enterperise Geodatabase Handle Large Data Sets?

[Mark_Hotz]

I am having some issues with a large data set in our enterprise geodatabase, and would like to learn if this is simply a limitation of the technology, or if perhaps I have a setting incorrectly applied.

I have a large point data set (close to 3.8-million records) that we have been able to successfully use as a stand-alone file geodatabase to create heat maps.  I can even create an individual heat map from that one file geodatabase feature class and publish that directly to Portal (10.6.1) from ArcGIS Pro (2.3.2).  However, a manager here would like to be able to query the data itself using Portal, and create heat maps based on those queries, and with smaller data sets this does indeed work.

However, when I imported that large data set into our enterprise geodatabase (so I could get it ready for publishing), and then opened the attribute table to make sure all of the records were there, I suddenly started to run into ODBC driver connection errors, as per below:

I do have other smaller feature classes (linear mostly) that are in the 0.5-million records range and there are no problems opening these attribute tables and populating any of these with all records.  The plan is to eventually vector cache this data set to make it less cumbersome to use, but I will cross that bridge when I solve this issue first.

So does this ODBC Driver 17 failure mean that the enterprise geodatabase has limitations, or is there perhaps a way I can fix this?  I'm also wondering if I could just put this data set into a registered folder on our server and then add it to Portal that way instead.

Thanks

[George_Thompson]

I do not think that it is a limitation of the Enterprise Geodatabase. The error referenced is something along the lines that the connection between your ArcGIS client and the SQL Server DB was closed.

I have seen this in the past related to networking issues (routers closing ports after a specific amount of time), other DBMS errors.

Can you access the data in SSMS and see all the records?

If you close the ArcGIS client and re-open it, can you see all the records in the table?

Geodatabase‌

[Mark_Hotz]

George:

Thanks for the response...most helpful.  I had to test this with our IT manager here and it took a little longer :-).  We were able to access all of the data in our largest data set (9-million + records) using SSMS, and scroll through it without issue.  I also noticed that we only had 12 GB RAM on that server, so we then upped that to 32 GB.  

Closing either ArcGIS (10.6.1) or ArcGIS Pro (2.3.2) and then relaunching them doesn't help.  I did find some online help that indicates that there is an Enterprise Geodatabase setting somewhere that allows one to adjust constrictions, but this had more to do with file size.  Apparently the default setting constricts the geodatabase to 1 TB, but can be adjusted to be as much as 256 TB.  Our data is nowhere near 1 TB, but I'm wondering if wherever that setting is located if there are other constraints that can also be adjusted (like number of records etc.).

[Asrujit_SenGupta]

I believe the 1TB/256TB limit that you mentioned, is for File Geodatabase object size, not an Enterprise geodatabase.

File geodatabase size and name limits—Help | ArcGIS Desktop 

Table or feature class size: 1 TB (default), 4 GB or 256 TB with keyword

Is the issue observed from ArcGIS Client on any machine? Even if the ArcGIS Desktop\ArcGIS Pro is on the Database Server?

What is the version of SQL Server being used?

[Mark_Hotz]

Asrujit:

Thanks for the help...I think you're right about the File Geodatabase object size...I was just hoping I guess :-).  I'm the only one here with ArcGIS on my desktop, but I have not installed it on any other server.  We have it installed on a VM in case we have a consultant come here to do periodic work though.

Our version of SQL Server is 2016, but I have noticed that our enterprise geodatabase is set to use 2012 (probably because that was the latest setting available in ArcGIS at that time.

[George_Thompson]

Hi Mark - Thanks for that information. It seems that the issue maybe between the ArcGIS client and the SQL Server machine. That error is related to something closing the communication link between the client and the server.

What is the speed of the network that you are on (100MB, 1GB, etc.)?

Are you on wireless or wired?

I am also curious to see if the client has the same issues is installed on the SQL Server machine.

[Mark_Hotz]

George:

Our network speed here is 10 GB, and although we do have a wireless network this is mainly for consultants and staff using laptops for meetings, and for other mobile devices.  My workstation is definitely wired.  I'm not sure if I would be allowed to install ArcGIS on our SQL Server machine though...they have learned how some geoprocesses can drag the system down LOL.  I have a lot of software installation privileges here, but that is one of the more guarded servers.  I can ask though.

[George_Thompson]

I noticed on your other reply that it is set for 2012 compatibility, would it be possible to take a full backup, then update the compatibility to 2016?

Another option, would be to create a new EGDB in SQL Server, load the large data set and test from that new EGDB. That would let you know if it is specific to a geodatabase or SQL Server as a whole.

The last option would be to contact Esri Technical Support and work with a Geodata analyst on this issue to see if they can provide any insight.

[Mark_Hotz]

George:

This discussion got me thinking about upgrades...and then I discovered that our enterprise geodatabase has never been upgraded.  I thought it was automatic when GIS Server was upgraded...apparently not.  I'm in the process of upgrading this right now, and when done I will certainly explore the option of updating the compatibility to SQL Sever 2016.  I think the reason it was set to 2012 because that was the limitation of that version of enterprise geodatabase...perhaps with an upgrade some of these other issues will go away.  I will post the results here.

Thanks