Latest Contributions by MarcoBoeringa

As I understood we do not need to store rasters in SQL DB and should use filebase system. But would we have any troubles with multiple access from web if for example 10 or 20 people would request same image mosaic? I have heard about some file system locks or something like it. You are not going to access the file system directly from the outside world, ArcGIS for Server and the webservices you define for it, will be handling all this. This shouldn't be an issue, this is what ArcGIS for Server is all designed for, taking care of such multiple access against webservices. Also how to manage in filebase db attributes? Can I do select and insert operation from script/ written from scratch app without running ArcGIS. The problem that I can't image how to work with filebase system. I really think you should start delving into the vast ArcGIS Help system concerning raster management options. A good starting point could be the following pages: Raster data organization What is a mosaic dataset? Especially start reading about the ArcGIS Mosaic Dataset datatype. Realize first that Mosaic Datasets are a kind of "wrapper" around huge collections of raster datasets stored on a file system. They allow you to properly manage such collections, and do all kinds of fancy things like color balancing between images, and run dozens of other functions as well. The nicest thing about this all, is that these functions operate "on-the-fly", so they don't affect or touch the original data, but the functions (for example the color balancing), is done "on-the-fly" during display. ... View more

Yes. We have ArcGIS For Server for workgroup, but we need to create raster mosaic with thousand of rasters. The limitation of SQLExpress is 10GB. As to the bold remarks, you probably want to read Vince's (ESRI) remarks in the thread post linked below (e.g. "Raster support in databases is very nearly only for backward compatibility at this point."). Directly storing huge amounts of raster data in an enterprise database / RDBMS, instead of using Mosaic Datasets referencing the original raster data files on a file system, is not a recommended practice anymore (if it ever really were). ESRI's strategy for the future is raster datasets outside the enterprise database, but referenced through a Mosaic Dataset, allowing flexible management of the rasters, while not burdening the enterprise database / RDBMS with the task of storing the actual data. http://forums.arcgis.com/threads/102936-Postgress-PostGIS-Rasters-in-ArcGIS-not-working?p=371748&viewfull=1#post371748 This is also similar to other trends in the database world, with file storage for huge amounts of unstructured data outside the RDBMS, but "managed" and referenced inside the RDBMS / data warehouse. ... View more

The SDO Geometry versus ST_Geometry versus Compressed Binary versus whatever storage format for geometries discussion is something ESRI has been avoiding mostly in the past ten years or so... As to best practices, performance of software / hardware systems etc., the System Design Strategies Preface gives guidance and lots of stuff to read, some others here on the forum may give more specific answers as regards the Oracle Spatial storage. ... View more

I gave up on SELECT a.polyid, p.pointid FROM f_pnts p, f_pols a WHERE SDE.st_intersects(p.point,a.poly)=1; after ten minutes.  When I dropped the polygon count by an order of magnitude, this "find points which overlap polys" query ran in 14.56 minutes. ... Note:  The PostgreSQL optimizer works differently, so the equivalent query on SDE.ST_GEOMETRY in PG 9.1 returned in <45ms against 216k polygons, no matter the order, and completed in <50ms against 1.34m polygons. Looking at these figures, wouldn't it be better to say the Oracle optimizer isn't working at all for this particular case of the order of the datasets input to the the SDE.st_intersects command?, whereas the PostgreSQL optimizer, correctly considers optimal order for processing the command? Is this an Oracle issue, or does ESRI need to adjust the ST_GEOMETRY implementation for Oracle, to have the Oracle optimizer optimally process the command? ... View more

I've noticed the horrible response for Geometry verses SDEBinary.  Has ESRI said how long SDEBinary will continue to be a storage option? A more interesting question is if there are technical reasons why SDEBINARY could no longer be supported in the future in SQL Server. In case of Oracle, the LONGRAW field type that stored SDEBINARY was deprecated by Oracle, so that set a clear endpoint. The more interesting (naive) question is therefore if the field type used by SDEBINARY in SQL Server, is set to go away. Part of that answer lies probably in the below quote from this Help page and additionally, see this one for more explanation: "In ArcGIS 9.3 and lower releases, ArcSDE Compressed Binary storage in SQL Server is stored as an image data type. New data created using ArcSDE Compressed Binary storage in SQL Server at ArcGIS 10 and later releases is stored as a varbinary(max) data type." varbinary is explained here in Microsoft Technet pages: binary and varbinary (Transact-SQL) This datatype was one of the data types that is replacing ntext, text and image data types set for deprecation, see this page: ntext, text, and image (Transact-SQL) As can been seen from the quote above, image was the field type ESRI used for storing SDEBINARY at ArcGIS 9.3.x and below. So ESRI already anticipated and accommodated the deprecation of this datatype by Microsoft by switching to varbinary(max) at 10.0, so technically the option for storing SDEBINARY doesn't seem to be going anywhere soon... Of course, the main reason ESRI has been pushing for a shift to "native" Geometry storage or ST_Geometry, is the better accessibility by third party (CAD) software to the data, and the abilities to easily define views etc. on the RDBMS side including spatial data (no separation of geometries and attributes in different tables - feature and business - for a single Feature Class, no "inaccessible" Compressed Binary storage). Another more technical reason may lie in this latter remark, as maintaining the referential integrity between feature and business table is more challenging (see this Help page and the remarks about the triggers set to manage this) But of course, if your business / organizational workflow won't involve anything other than ESRI software accessing geodatabases, than these advantages are mute... There is a known bug NIM089293 from a year ago dealing with the draw time for the GEOMETRY spatial type verses the draw time for SDEBINARY. Unfortunately, it's severity is "low."  In my experience that usually means it's not going to be fixed. This issue should probably not have been logged as a "known bug" at ESRI, as, as Vince explained already, this is not an issue ESRI is responsible for, but Microsoft. If it should be logged as a bug (which is already disputable since true bugs are generally code errors, not performance issues which would more realistically classify as "enhancement requests"), it should be logged at Microsoft. ... View more

Marco, I would not classify a Query Layer as a "view" at all.  It is QUERY.  A query is an expression which retrieves data from tables.  A view is a database object that stores an SQL query in such a way to reduce the overhead of parsing the query. The Query Layer stores the query and rendering rules against either a table or a view, and reapplies that query each time it needs to Thanks for that addition that clarifies it further. I already put the "view" notion of the Query Layer in quotes, and tried to emphasize the rendering aspect of it. (note that views cannot be registered with the geodatabase, just with ArcSDE, which is what that message means). Made some changes to accommodate that. Going to log out now... ... View more

I would say the next focus should be on registering views, not creating them. Agree Wait right there -- This is the key misunderstanding.  Query Layers are a way of accessing spatial databases without ArcSDE.  There is no correlation between views and Query Layers, other than that they need to be spatial to be rendered. Maybe it is better to describe the difference between a Query Layer and a Database (Spatial) View as the following: - Database Views are Views defined by a SQL expression that are stored in, and managed by, the RDBMS. A database view thus is persisted at the RDBMS level. - Query Layers are "somewhat" similar to Views, as being defined by a SQL expression, but only exist within the context of an ArcMap session. They are persisted at the ArcGIS for Desktop application level, within an ArcMap session/document. Query Layers do a lot more though than storing a SQL expression, they inherit all the properties of normal layers like legend symbology, labeling, min / max display scale and so on. Like normal layers, you can save them as a *.lyr file outside ArcMap, on any drive you have available. In essence, a Query Layer is a "view-on-a-view", or maybe I should better say a ""view"-of-a-view" (since it defines symbology and display too), just like you can create views based on views (queries based on queries) in a database / RDBMS. But remember: a Query Layer does a lot more in the context of ArcMap, as per the layer settings described above... They certainly can be registered with an enterprise geodatabase, if you a) have an enterprise geodatabase, and b) choose to do so. Vince, you are ignoring an important distinction here that may help in making it more understandable for others, see also the quote below from this sdelayer command page: "Note: ArcSDE commands do not interact with the geodatabase system tables. Therefore, if your layer is registered with the geodatabase and participates in geodatabase functionality, such as networks or topology, do not use the sdelayer command to administer the feature class. Instead, use the ArcGIS tools and wizards." In essence, there are 3 registration levels on top of each other. Depending on which option you use, a spatial layer will be "known" and useable within a certain context: 1) - Registering a a database view or table in the native database / RDBMs system tables only. This is by defining the database view through either the RDBMS's tools like SQL Server Management Studio, or(!) creating the view using the CreateDatabaseView_management geoprocessing tool. Data registered as such can be used and accessed as Query Layer only, and not participate within a geodatabase and its functionality. 2) - Registering (a possibly predefined) database view or table in the ArcSDE System Tables only that form part of the ArcSDE Repository using the ArcSDE Command Line Tools like sdelayer (This option will no longer be possible once the ArcSDE Command Line Tools are retired, unless ESRI comes up with a replacement, as Vince also pointed out in the first quote I included above). Data registered like this is recognized "more or less" as "normal" Feature Classes, in the sense of showing proper icon (Point, Line, Polygon), but still can not participate in all advanced geodatabase behaviour, as it isn't registered with the geodatabase. This option registers data in the following tables:     TABLE_REGISTRY (or sde_table_registry)     COLUMN_REGISTRY (or sde_column_registry)     LAYERS (or sde_layers)     GEOMETRY_COLUMNS (or sde_geometry_columns) 3) - Registering a database table in the ArcSDE System Tables AND(!) in the Geodatabase System Tables that form part of the ArcSDE Repository by using the Register with Geodatabase menu option within ArcCatalog. This option gives you the full geodatabase functionality, including the ability for the data to participate in advanced geodatabase behaviour. NOTE: This option is not available for database views (see remark Vince below also), because the data need to be editable to add crucial fields like ObjectID and other geodatabase maintained fields that form part of geodatabase Feature Classes, something which can not (generally) be done with views, e.g. some view types may be non editable at all, like those summarizing data by grouping. Therefore registering with the geodatabase may not be available for all data sources. This option registers data in the following tables: (see also this Help page: Registering a table with the geodatabase)     GDB_ITEMS     GDB_ITEMRELATIONSHIPS *** and also(!): ***     TABLE_REGISTRY (or sde_table_registry)     COLUMN_REGISTRY (or sde_column_registry)     LAYERS (or sde_layers)     GEOMETRY_COLUMNS (or sde_geometry_columns) Only registering with options 2) or 3) will cause the crucial metadata to be stored in the ArcSDE System Tables, and hence the proper icon (Point, Line, Polygon) being shown immediately in ArcCatalog, and the ObjectID and spatial extent of the dataset being "known". Note that for layers registered with all three options (1,2,3), all layers can also be accessed as Query Layers, but the functionality will be curtailed compared to accessing a layer registered with option 3) and accessed as a normal Feature Class. Query Layers offer less functionality than geodatabase Feature Classes. A list of the functionality available is here: ArcGIS functionality available for database tables that are not registered with the geodatabase ... View more

Another point is, is that by design, hitting the ArcCatalog Preview button, retrieves the entire dataset, as it defaults to full extent. So it will take a long time whatever way the data in the RDBMS is registered or stored if the accessed dataset is large or the defined SQL query to access it complex. It also remains unclear in the posts if the faster response times accessing Query Layers when adding the data through the Add Data button as per this link, isn't caused by simply selecting "Use Spatial Reference extent" as the default extent for guessing a suitable extent for a layer that isn't registered with ArcSDE nor the Geodatabase, which would circumvent the need to plough through the table to calculate an extent on the fly. I also wonder at what stage some sort of caching on the local machine might play a role when accessing any of these layers. ... View more

Raster support in databases is very nearly only for backward compatibility at this point.  There isn't much point in new development to support an archaic use pattern -- Files in a filesystem are much more scalable, adaptable and open than trapping rasters in databases - V X2, this played in the back of my mind when writing the above post, and I should probably have added this, but the OP asked about a situation where the data is already added to the database. Now we're on it, I guess it is good to mention Mosaic Datasets (with the preffered option to use referenced datasets not residing in the RDBMS but on the file system), as the primary new alternative option for using and managing raster datasets in ArcGIS. ... View more

I seems to me rare that if you are thinking in storing raster images in a spatial database and consume those images using ArcGIS, ORACLE is the only available option. Jose, Have a look at these Help pages if you haven't so: How raster data is stored and managed Raster datasets and raster catalogs in a geodatabase in PostgreSQL I think you're right about the PostGIS Raster format currently not being supported (but QGIS seems to be the only GIS now with "out-of-the-box" support according to this PostGIS FAQ, so ESRI is not the only party not yet having implemented support) But the quote above is only true as regards reading the native raster data format of the RDBMS itself. ESRI does support rasters in databases, but in their own format (albeit with possibility to either use a native RDBMS field type for raster storage, or use the more generic ST_Raster but depend on extra configuration for ST_Raster support). There are pros and cons to this approach: con is that you can not read data already loaded in an existing database using the RDMS's tools. Pro is that ESRI can more easily deliver a consistent user experience across different databases / RDBMSs, without relying to much on individual database features, and can probably optimize the raster data format for usage in ArcGIS. ... View more

Now, being the updates are reflected in the sde.default and its versions by the �??reconcile version�?� tool, the �??communities�?� in the SQL server doesn�??t match the one on the ArcGIS Number of records of the �??communities�?� in the SQL is 781 Number of records of the �??communities�?� in the ArcGIS is 788 ... Which tools can be used to have Number of records of the �??communities�?� in the SQL and in ArcGIS equal? That process is carried out by the Compress operation, step 2 in your ModelBuilder model. If the Compress failed to do that, than there are still versions or replicas (which as William correctly stated are also technically versions) that haven't been reconciled & posted properly. ... View more

You could also consider creating "shadow" Feature Classes in the TOC (Table of Contents) of ArcMap with the appropriate legend (symbols). Then create a legend based on these while excluding all the other layers you want to display and make the legend non-dynamic to retain the shadow layers, then deselect the shadow layers in the TOC to make them invisible. ... View more

Eric, Thanks for getting back at lightning speed. After reading your explanation I looked for the 171 value and embarrassingly enough, it is a point that I did not see previously. But solid explanation, very helpful. my Next time, you might consider setting the drawing order of the points with their colors, to reflect the range from low to high by using Symbol Levels. This way, the high value would have been obvious, as drawn last and on top of the others, instead of being hidden: What is symbol level drawing? Marco ... View more

There is also the ArcGIS X-Ray tool that can report and manipulate geodatabase structure: X-Ray for ArcCatalog (ArcGIS 10.2) ... View more

A lot of improvements on the geodatabase side have been made in the lastest ArcGIS releases like renaming and modifying columns and making the sde command line tools available via tools. The possibility of creating proper spatial views however, still needs attention before deprecating the current tools. It would be more correct to replace the highlighted text with: "A lot of improvements on the geodatabase side have been made in the lastest ArcGIS releases like renaming and modifying columns and adding (geoprocessing) tools that can substitute functionality of the sde command line tools." The new tools do not rely on the sde command line tools for their implementation in ArcGIS. ... View more