Latest Contributions by VinceAngelo

‎06-13-2022

You can simulate large datasets ("millions" might not make the cut for "huge" -- I put it at closer to 50M points for "very large") with ten lines of SQL code. For example, in PostgreSQL (grabbed from my post in GIS SE ) ... CREATE TABLE example1 ( idcol serial NOT NULL, geomcol geometry NULL, CONSTRAINT example1_pk PRIMARY KEY (idcol), CONSTRAINT enforce_srid CHECK (st_srid(geomcol) = 4326) ); INSERT INTO example1(geomcol) SELECT ST_SetSRID( ST_MakePoint( (random()*360.0) - 180.0, (acos(1.0 - 2.0 * random()) * 2.0 - pi()) * 90.0 / pi()), 4326) as geomcol FROM generate_series(1, 1000000) vtab; CREATE INDEX example1_spx ON example1 USING GIST (geomcol); The same is possible with ArcPy of course, as a DA InsertCursor. Finding data is generally more domain-specific than a generic request on a web forum. - V

‎06-10-2022

SEQUENCEs are too slow for bulk-loading data rows. Esri uses "i-tables" to implement the equivalent in batches, with the name using the table_registry identifier as a suffix. They used to be documented, but I can't find where it moved in the current doc set. - V

‎06-10-2022

Shapefile uses right-hand rule*, but geodatabase uses left-hand rule*. Some storage formats have a required order (e.g., SDO, KML), but others (GML) are flexible. Since the sketch code originated in ArcView, and it was based on shapefile, it's likely that sketch would use right-hand rule, as does ArcObjects in general (and therefore ArcPy). The SgShape library used by all Esri tools allows for extraction in either order, but at geometry assembly it will flip rings, demote interior rings, and promote exterior rings (as necessary) so that the geometry is internally consistent with left-hand rule. The only time vertices are stored twice is if a CAD entity is present. - V ----- * The "right-hand rule" where, when walking the perimeter, the right hand is toward the inside (aka, clockwise on exterior rings and counter-clockwise on interior), not the "right-hand rule" where, when the right thumb is pointed up from the face of the ring, the curl of fingers indicates the vertex order direction (that's called "left-hand rule").

‎06-07-2022

No real changes to the geodatabase model infrastructure since then. Note that "precision" refers to both the vertex equivalence comparison tolerance AND the spatial reference coordinate storage depth (though at this point, *all* spatial references are HIGH precision). - V

‎06-03-2022

I've used a number of mechanisms to push contents from one database to another, but attempting this with different database vendors and such a large release gap and a different versioning model seems to drift into nightmare. For a production system, you really need to limit to no more than one complication (and different versioning models should be scored as two or three complications). Sorry. - V

‎05-15-2022

@JustinPence2 wrote: If we set up File Geodatabases, they would be stored on a Network Volume/Drive at my company's datacenter. Yeah, don't do that. Network drives have half the performance of local drives. If you're single-user, copy the FGDB locally, then upload again when done. It's a myth that relational databases are faster than local files. Relational databases have a more robust toolbox for SQL queries and data manipulation, but they're generally not faster because they have ACID overhead. Make no mistake: Having atomicity, consistency, isolation, and durability are *good* things, but they come with a price. The real proof is in the execution. Use FGDB (locally) and see if the performance is where you need it. Then optimize what is not up to spec. Preemptive optimization can be an antipattern. - V

‎04-26-2022

You're the only one that knows your requirements for the size of "hot spots". Obviously, too coarse and you get blocky, useless maps, and too fine and the processing never finishes. The only way to find out how your environment will perform is to try it. While Create Fishnet will populate a grid in a specific projection, writing your own polygon generation script has benefits. I've created global grid of 25 sq. km polygons in decimal degrees, using a coarse generalized landform to restrict polygons to within 60nm of land (using a fixed width, and adjusting height to get the polygon size within 0.13% of true area, based on geodesic calculation). I also integrated metadata in each polygon so that there were tessellations (each 25 sqkm cell had a 100 sqkm parent, and each 100 sqkm had a 400km parent, ... up seven levels). This allowed modelling at multiple scales, so that model validation could be quickly performed at 1600 sqkm, then the overnight job could run at 25sqkm. Since you're working with a smaller area, you could use a finer grid (6.25 sqkm and 1.5625 sqkm were possible, though at a significant processing cost, so you might only want to model limited areas [known butterfly "farms"] at that resolution). Your first task is to try generating a coarse grid, then to work the math to generate a less coarse one with the same alignment. Then you can select out the cells which are entirely over open ocean, and see what you can do with the rest. From there, it's a matter of data quality and processing time. - V

‎04-19-2022

I have no experience with either FME or Data Interoperability, so I'm the wrong person to ask. - V

‎04-17-2022

I'm more likely to use ArcPy to move geometries, but I've done this a half-dozen different ways in the past 27 years. Since any solution is likely to be specific to a given database/dataset, using the tools you feel most comfortable with is likely to be easiest. - V

‎04-15-2022

You'll probably want to reconsider this. Cross-database queries within a single database instance are an antipattern. Cross-server queries are pretty much worst case, in terms of performance and reliability. The best practice solution would be to stage a copy of the table(s) in a single database and transmit INSERT/UPDATE/DELETE messages from the primary database to the publishing database. In this way, you can take full advantage of the SQL engine to index and optimize queries, and you don't put the burden on the GIS engine to do the database's job. - V

‎04-04-2022

Long, long ago, Arc/Info had LENGTH field on LINE coverages, and AREA and PERIMETER on POINT and POLYGON (because they shared the ".PAT" suffix). Those properties were retained as methods on geometries in SDE, ArcSDE, and enterprise geodatabase, depending on the storage (and topology class). In the meantime, file geodatabase was created, and it exposed these meta-properties as fields, but the units are Cartesian, so they're not completely useful in all use cases. If you want to keep these phantom fields across datasets, it's often best to create your own fields to preserve them (but then you have to update them if the geometry changes). - V

‎02-24-2022

ArcGIS tools can read and write to PostgreSQL databases natively. No additional client library/configuration is required (unlike all the other supported databases). I just finished a seven year stint as a PG DBA with a group that didn't want to use enterprise geodatabase functionality, but did want hundreds of millions of rows of spatial data in ArcGIS Server services. This worked fine on a query basis. All the editing was done through SQL, or by uploading file geodatabase snapshots via FeatureClassToGeodatabase, then using SQL to drive updates. If you are using ArcGIS Enterprise to render the data, the difference between QueryLayer layers and EGDB FeatureClass layers is pretty subtle. But if you try to use an UpdateCursor or edit through ArcGIS on a non-enabled database, and you're used to an EGDB, you are not likely to enjoy it. I'm not going to argue that an EGDB isn't useful, just that it isn't required in all situations. You need to evaluate your situation. I've been using enterprise geodatabase since SDBE 1.2.1, and been a GIS DBA for over 33 years, so my view is somewhat unique. One colleague once commented that I could run every version of SDE from 2.0 to 9.3 simultaneously, on a toaster oven (but that wasn't true -- the toaster oven didn't have enough RAM for more than four active database instances). It all depends on what the application demands require [and, I should add, what the DBA is capable of supporting -- Sometimes this is just a bit of training away, others can grow into it on the job, working with geospatial types.] - V

‎02-23-2022

The difference in this case isn't "enterprise geodatabase" vs. "regular database", it's "has a spatial index" vs. "doesn't have a spatial index." IMHO, you should always use a native geometry type before constructing geometry on the fly with each query (with the usual low row count caveat, but if you only have a thousand features, why bother with a database?). A GIST index is going to outperform a (float,float) index, and can be leveraged in ST_DWithin queries. Once you hit the "enterprise geodatabase" vs "non-enabled database", then the difference is more subtle, mostly due to not having an extent property stored in metadata. - V

‎02-23-2022

Quoting my profile page: Note: Please do not try to message me directly or tag me in questions; just ask a question in an appropriate community. I saw the question. If I had something to contribute, and the time to do so, I would have. Tagging me slows my response, and uses the time I would have used answering questions. - V

‎02-16-2022

If a field of type geometry exists in the column list, even if it's always NULL, then TableToTable is not the right tool. - V

My Ideas

Latest Contributions by VinceAngelo

Re: requesting huge spatial data

Re: SQL geodatabase dbo.i tables?

Re: SDO_GEOMETRY polygon: Vertices are clockwise in ArcMap, but counterclockwise in SQL

Re: Precision vs. XY Resolution

Re: Geodatabase synchronization between oracle and MS-SQL databases

Re: Performance File Geodatabse vs Enterprise Geodatabase

Re: Creating Fishnet for North America? Where to start?

Re: Join two SQL Databases and Share the Resulting Layer in Enterprise Portal

Re: Join two SQL Databases and Share the Resulting Layer in Enterprise Portal

Re: Join two SQL Databases and Share the Resulting Layer in Enterprise Portal

Re: Shape_area and shape_length field disappear when exporting to a .sde

Re: Geodatabase vs database effective differences in ArcGIS Enterprise

Re: Geodatabase vs database effective differences in ArcGIS Enterprise

Re: How to store Feature Class in certain schema

Re: Unregister a Feature Class

Re: Error Connecting to Enterprise Geodatabase (SQ...

Re: Precision vs. XY Resolution

Re: Best practices for programmatically mirroring ...

Re: Best practices for programmatically mirroring ...

Re: Best practices for programmatically mirroring ...

Coordinate Reference Systems