Latest Contributions by MarcoBoeringa

Projected and Geographic coordinate systems and coordinate system transformations can really drive you nuts... :confused: And yes, you are opening Pandora's box... Nonetheless, I strongly recommend you to take serious time to familiarize yourself with the basic concepts, as the consequences of inappropriate and incorrect usage can be devastating (yes, not only on paper but also literally, I once worked a short period for an oil company, and they had a special introduction to coordinate systems for new employees working in GIS that started with an oil platform going up in flames due to wrong usage of coordinate system information - this was no joke!) Read as much as you can from different sources, ultimately, things should start to clear up a bit. As a starter: The ArcGIS Help - What are map projections? I also recommend you to read all other sub-topics under the same heading, look in the Help TOC (Table of Contents) on the left of the Online Help window for the sub-topics. Also, this Ordnance Survey UK document, although partly focused on the UK situation, gives a nice introduction, and basic reminders and concepts, of Coordinate Systems: A guide to coordinate systems in Great Britain And these ITC webpages (some Dutch examples, ITC is an international institute but located in the Netherlands): http://kartoweb.itc.nl/geometrics/index.html And the Wikipedia pages: http://en.wikipedia.org/wiki/Geodesy http://en.wikipedia.org/wiki/Coordinate_system I can also recommend you these two pages by Eric Gakstatter, although you need to have some basic understanding first, and know some GPS jargon... See the Wikipedia links below. Nightmare on GIS Street: Accuracy, Datums, and Geospatial Data Part 2: Nightmare on GIS Street �?? Accuracy, Datums, and Geospatial Data Wiki links: CORS Continuously Operating Reference Station DGPS Differential GPS GNSS Global Navigation Satellite Systems RTK Real Time Kinematic WAAS Wide Area Augmentation System ... View more

Land Platting: Magnetic Declination & Why it Matters About applying a ground to grid correction (ArcGIS Help) About spatial adjustment About spatial adjustment transformations About spatial adjustment edgematching About spatial adjustment rubbersheeting ... View more

I have made one last attempt to get closer to Keith's data and working process, by densifying the features one more time, inserting a vertex each 0.1 meter. I used a selection of about 10% of the original dataset, so there is only 1806 polygons left here. This resulted in a dataset with the following characteristics. Results for densified layer Polygons: 1806 Vertex count Total: 4883810 Average: 2704 Min: 23 Max: 34119 Std: 3505 As you can see, the resulting dataset now has on average even more vertices per feature than Keith's dataset (2704 versus 2127). I than copied this data to a SDE_BINARY dataset using the Copy Features tool by setting the proper configuration keyword, and back from there to GEOMETRY. This last step is very close to what Keith is doing, as my initial tests were GEOMETRY to GEOMETRY mainly, and not specifically from SDE_BINARY to GEOMETRY. Since this step involves a conversion of data types, it might slow down things. I also copied from GEOMETRY to GEOMETRY again for comparison: GEOMETRY TO GEOMETRY results: 1806 polygons copied in 59 sec 1806 / 59 = 30.6 features written per second 4883810 / 59 = 82776 vertices written per second. GEOMETRY TO SDE_BINARY results: 1806 polygons copied in 37 sec 1806 / 37 = 48.8 features written per second 4883810 / 37 = 132K vertices written per second. SDE_BINARY TO GEOMETRY results: 1806 polygons copied in 43 sec 1806 / 43 = 42 features written per second 4883810 / 43 = 114K vertices written per second. CONCLUSIONS: Well, "surprise"... or maybe not so. But the writing to and from GEOMETRY to SDE_BINARY, both ways, is actually faster than from GEOMETRY to GEOMETRY, despite a needed conversion. Anyway, looking at the figures ranging from 30.6 to 48.8 features / second max, I do have to contest Vince's "two-orders-of-a-magnitude" to low performance figures with the 7 to 9 features Keith reported. It seems likely from these figures, that a maximum gain of about 5-6x is a more realistic figure based on real world data (albeit densified). This is assuming ArcGIS and geoprocessing framework usage, not se_toolkit or ArcSDE Command Line tools, of course. And test hard-/software configuration as in this post in this same thread. Lastly, Keith's dataset has more attribute fields than mine, that may contribute to lower insertion figures too. The databases are on virtual servers.  The destination server is configured as two Intel E5-2260 @ 2.2 GHz CPUs, Windows Server 2008 Standard Edition, 64 bit, 16 GB RAM. I guess this needed to be Intel E5-2660 2.2 GHz. According to this benchmark page, they should be about double as fast as my Core i5 2320. That would realistically put them in the "one order of a magnitude" faster than the current loosy 7-9 features / s score, so maybe 80-100 "2000+ vertex" features / s should be achievable. ... View more

Perhaps what I was doing was exposing my naivete and relative inexperience when dealing with spatial data in "hard core" fashion.  Here's how I got my numbers: I queried SDE_LAYERS to get the layer_id for the feature class I wanted to examine, then I simply queried the corresponding f table to take the average of numofpts column.  That is the number I reported as being the average number of vertices for a feature in that class.  Did I misunderstand the content of the f table? No, you didn't. You interpreted the data right. You undoubtedly know this already, but be aware that GEOMETRY and GEOGRAPHY in SQL Server don't use a f table, features are stored straight in the base table. I have the following stats for the layer in question: 366k features numofpts: avg 2127, min 13, max 144,969 (!!!), standard deviation about 5086, median 1023. The feature with 145k vertices is Monroe County, FL. Still a huge number of vertices per feature by any measure. ... View more

Deprecation is public declaration of the intent to stop supporting something, so deprecation "at 10.2" means it won't be around with whatever follows 10.2.  Deprecation "after 10.2" would mean two more releases, which is not the case. Ah, well, there you go if something is not your mother tongue... but anyway, the explanation you gave, is how I interpreted the deprecation plans too, and what I intended to say in my post: 10.2 as the last release to include the ArcSDE Command Line tools and C/Java API. ... View more

Finally, the ArcSDE 'C' API will be deprecated with the release of 10.2, so long term plans for toolkit use should be limited. There is a little bit more breath here, it is supposed to be deprecated after 10.2 according to these deprecation plans, so with the release of ArcGIS Professional Application (ArcGIS 11 probably). You should really determine where your I/O bottleneck is, since there isn't any clear indication of why your database is so slow. - V x2, but maybe also give some more information about the configuration details. What hardware is the server? What network bandwidth do you have (still 100Mbit/s or Gigabit?) etc... ... View more

Hi Vince- Thanks for the extended example.  Since asc2sde is not geodatabase-aware, I assume that I'll have to find a way to register the results (and get them into the desired feature datasets) separately? 10.1 extended upon the number of tools you can use to administer a (geo-)database, and you can register datasets or enable geodatabase functionality easily from the toolboxes and geoprocessing tools available. Also, you can use Query Layers to access the data in read-only mode directly, without the need of registering the data with a geodatabase. See also my PDF here: http://forums.arcgis.com/threads/83644-quot-The-ESRI-Geodatabase-Framework-quot-PDF?p=295462&viewfull=1#post295462 and the "Future" PDF here: http://forums.arcgis.com/threads/83644-quot-The-ESRI-Geodatabase-Framework-quot-PDF?p=303021&viewfull=1#post303021 ... View more

Hi- Thanks Vince.  Is it possible to accomplish the same task, at similar speeds, using the geoprocessing tools provided with ArcGIS Desktop? You probably already saw it, but have a look at my post in the other thread you posted in: http://forums.arcgis.com/threads/87986-Spatial-View-so-slow?p=313416&viewfull=1#post313416 ... View more

Just for the sake of it, I decided to do a little test. I densified the dataset I had by using the Densify tool to create a dataset with a vertex every 0.5 meter. I guessed this would get me close to Keith's dataset, but I still lag behind with an average 540 vertices per feature: Results for densified layer Polygons: 18968 Vertex count Total: 10247488 Average: 540.25 Min: 4 Max: 18956 I will also copy the original dataset, which had the properties as listed in the previous post: Original dataset Polygons: 18968 Vertex count Total: 535334 Average: 28.22 Min: 4 Max: 2106 Once the densification process itself was finished, I used ArcGIS 10.1 and the Copy Features tool to create a new Feature Class from the existing densified one, curious to know how much time that would take, and compare this with the writing of the data during the densification step, and the copying of the original dataset: Copying of the original non-densified dataset: 18968 polygons densified in 18 sec, so 18968 / 18 = 1035.8 features written per second, and 535334 / 18 = 29741 vertices written per second. Results for Densification step: 18968 polygons densified in 9 min 2 sec, so 18968 / 542 = 35.0 features written per second, and 10247488 / 542 = 18906 vertices written per second. Results for Copy Features step of densified dataset: 18968 polygons copied in 1 min 54 sec, so 18968 / 114 = 166.4 features written per second, and 10247488 / 114 = 89890 vertices written per second. The test configuration was: Acer Veriton desktop: - Core i5 2320 3.00GHz quad core processor - 6 GB RAM (never used completely during the process) - 1TB harddrive capable of up to 180 MB/s, connected on SATAII, but never or rarely exceeded read/write IO of over 10 MB/s during the process - ArcGIS Desktop 10.1 SP1 - SQL Server Express 2012 SP1 - Dataset using SQL Server GEOMETRY storage - No versioning on the datasets written! "CONCLUSIONS" (take with a pinch of salt ;)): This is definitely not server hardware, but it isn't a low spec laptop either, nor is it a shared server bogged down by other processes running on it... I am surprised to see how close the Densification step is to Keith's results: - Keith had 7-9 "2100-vertex" polygons written per second, with 14700 to 18900 vertices per second. - The Densification step is 35.0 "540-vertex" features written per second, and 18906 vertices per second - The Copy Features step is 166.4 "540-vertex" features written per second, and 89890 vertices per second - Original dataset copy is 1035.8 "28-vertex" features written per second, and 29741 vertices per second - Vince achieved 113 "2100-vertex" pseudo random circles written per second, and +/- 240K vertices per second Of course, the simple "Copy Features" step outperformed the Densification by a factor 4-5, but it still shows there is a real penalty in writing big polygons with many vertices, especially if additional computational or IO steps are involved, and it may actually be that Keith's hardware is simply running at its max when importing this data... Vince's results on his laptop were even better than the Copy Features results on my desktop writing his pseudo random "2100-vertex" circles (in terms of vertices, not in terms of records / features), but that was using the se_toolkit. It may be that the overhead of ArcGIS is the problem here, and in addition my real world dataset contains a couple of extra non-system text fields with data in it, needing to be written too. PS: 1) Writing to SDEBINARY using the Copy Features tool gave the following results: 18968 polygons copied in 1 min 18 sec, so 18968 / 78 = 243.2 features written per second, and 10247488 / 78 = +/- 131K vertices written per second. 2) I now also attempted writing to GEOMETRY using Copy Features, to copy to a Feature Dataset using another Coordinate System and even another Datum, thus forcing re-projection. Results are: 18968 polygons copied in 3 min 51 sec, so 18968 / 231 = 82.1 features written per second, and 10247488 / 231 = 44361 vertices written per second. As you can see, the re-projection slows the copying down by about 1/2 of the original speed compared to copying without re-projection (166.4 features / 89890 vertices per second). This again makes it likely the figures Keith supplied may actually not be that abnormal... although still on the low side of things. ... View more

Vertices per second is a very misleading metric.  I wouldn't trust it for much. Databases do work in transactions.  Each ROW is processed in total, which includes LOBs and strings.  A one-vertex GeoNames point could have 8-10k of data in a single row, while a dense 2k vertex polygon without attributes could easily use less.  It is critical to make sure all comparisons are for roughly equal row contents, otherwise the difference in real work (I/O) processing each row will warp your expectations.- V I fully realized that, but rows / features by itself is also a bad metric, as your results show: 1-vertex point features load over 15x faster per feature than huge 2100-vertex pseudo random circles... Not taking into account the rare possibility of users storing whole images, Office documents or whatever in LOBs in a row of a geographic feature, that is a significant difference not to be ignored either... Saying you have bad loading speeds of just a few features per second, while omitting the fact that each polygon has thousands of vertices, is therefore also not a very consistent question. Like you say, it is mixture of a whole bunch of factors at play, including feature count and size of features in terms of number of vertices. I really think Keith still needs to answer the question why they have average 2100 vertex polygons in their datasets... Unless this data is from extensive boundaries of big counties, and actually represents 1:1000 or 1:5000 detail in reality, and in practice must be combined in geographic overlays and other geoprocessing etc. with similar high quality, large scale data, I see no reason to maintain such huge features at scale 1:100.000. The data could be generalized. I don't think having 2100 vertex polygons by itself is impossible or bad practice, there may be good reasons to maintain it like that, but it sure isn't standard, especially not in the context of datasets having "millions" of polygons. I just ran some statistics again against a sample of the large scale photogrammetry based datasets I wrote about in the other thread. This is well maintained real world data: Polygons: 18968 Vertex count Total: 535334 Average: 28.22 Min: 4 Max: 2106 If you look at the graph below, most of the polygons have <100 vertices (vertical "Frequency"). The other image shows an example of the data, including some extra line layers, to give some impression of the level of detail. [ATTACH=CONFIG]25957[/ATTACH][ATTACH=CONFIG]25956[/ATTACH] ... View more

In all cases the data are (supposed to be) WGS84.  We are relying on the Esri tools to correctly detect and handle that during data migration. ... The issue we face now is that writing data to the new databases is unbelievably slow- only a few features a second when writing polygons averaging 2,100 vertices per feature- and we have millions of features that have to be written.  We can't possibly get the job done in a time frame that allows us to maintain our regularly daily data update production schedule.  (Joe alluded to this in the first message in this thread;we are pretty sure now that the GEOMETRY data type is the culprit based on other threads that we've read.) Vince (and Keith): you are noticing that ArcGIS may actually be performing a full re-projection including datum transformation during the load using these 2100 vertices / feature datasets with "millions" of features? What would be the consequences of that during the data load? I have no real-world experience with these numbers to share... but I would sure love to hear the answer! I also wonder what happens if the data is just loaded like it is, with the projection of the Feature Class set to that of the data, to avoid re-projection. EDIT: I now see Vince reported 1700 features as point features, so loading 1700 individual vertices / second on a laptop with encryption into SQL Server GEOMETRY... and 113 features per second with 2100 vertices per feature = +/- 240K vertices per second for the generated circles. The data load of 14700-18900 vertices / s on real server hardware may not be that unexpected than... depending on the hardware, network and other possible issues (reprojection going on). ... View more

For a polygon feature class that has about 25 attribute columns, we are getting data copy rates of only 7 to 9 features a second whether we use Feature Class to Feature Class, Copy Features, or Append.  The polygons are based on 1:100,000 scale data at the county-level size (average of about 2,100 vertices / feature) to give you a sense of the level of detail. ... Of course there is the basic issue of whether you have the business key data on which you are doing the SQL joins indexed properly- that can have an enormous impact on performance, particularly when you have thousands or tens of thousands of featured involved. So you're saying you have 10K+ features comprising an average of 2100 vertices per feature at scale 1:100.000??? :eek: Is this data stream-digitized from imagery, with never a proper weeding out of unnecessary vertices? I can't imagine any polygon / line needing up to 2100 individual vertices to define its form at scale 1:100.000. Even at scale 1:1000, it would be highly undesirable to have such huge amounts of vertices per feature... For a highly detailed and high quality 1:1000 base map here in the Netherlands of the national highway system, where I was involved in the re-design of the photogrammetry based workflow and database migration some ten years ago, a test dataset had an average of about 27 vertices per feature..., minimum 4, maximum 1026, but that was a rare exception. Writing to an old, but dedicated and further unused Sun Sparc Ultra-2 single 100 MHz processor Unix server, and storing in SDE_BINARY, resulted in data load speeds of about 120 features / second, so about 27*120=3240 vertices / second. (Oracle 😎 Again, we're talking >10 years ago here... Your data loads at 14700 to 18900 vertices per second. I leave it to others to comment if that is a normal speed right now with the configuration details you posted... but you really may need to reconsider your workflow for collecting and storing this data... ... View more

I have to little experience with MySQL and ArcGIS Viewer for Flex to guide you further on this. I think you will need to wait for some responses by other forum members. ... View more

Unfortunately, there is no "direct" way of using your MySQL database in ArcGIS, as MySQL is a non-supported database platform in the ESRI product line. This means you can not create ESRI enterprise geodatabases in a MySQL database. You might consider converting to a PostgreSQL database, if you want to take full advantage of all options the ESRI enterprise geodatabases have on offer, and stay with Open Source. Other than that, I guess you will be stuck with exporting the data to some file format the ArcGIS Viewer for Flex supports, possibly shapefile or maybe through some more steps using ArcMap to a file geodatabase... Be aware that shapefiles may truncate field names though due to limitations in the file format. ... View more

Since the primary purpose of this key is to serve as a lookup, I strongly encourage you to review your methodology, to see if you can assign a sequence-based key for joins after the edited data has migrated back into the primary tables (using the slow UUID values for update ONCE, rather than stacking the deck against join performance ALWAYS). - V So essentially you are suggesting to Cole: - To use GlobalID and GUID columns during the check-out "low-volume" field editing to establish and allow setting of relationships. - Convert these relationships to a less costly type of relationship ("sequence based key") upon importing into the main enterprise geodatabase using some other - maybe using a 32 bit integer? - type of field, to store the same relationship, possibly automated using database triggers & stored procedures? - Web applications, if not used off-line, should use the less costly sequence based key, while the off-line tablet applications should use the GlobalID & GUID... ... View more