Latest Contributions by MarcoBoeringa

Robert, As someone who is active in the OpenStreetMap community, but not working for ESRI, I do have a couple of remarks regarding your concerns. Note that I speak fully on personal terms here as a community member! I am also not a legal representative of the OpenStreetMap community: ESRI only copies / displays the existing data, OpenStreetMap data content is not curated or owned by ESRI (or for that matter anyone). It is a community project residing under the OSMF (OpenStreetMap Foundation). So for any questions regarding OpenStreetMap content, you likely should not be contacting ESRI, but use one of the communication channels of OpenStreetMap (forums, mailing lists etc., e.g. OpenStreetMap Help Forum , OpenStreetMap: Geo News you can use! ,  GIS - OpenStreetMap). If you have concrete evidence that transmission lines, substations or other infrastructure in your region was directly copied / imported from data that does not conform to the Open Database License (ODbL) of OpenStreetMap, then you should contact the Data Working Group of the OSMF (OpenStreetMap Foundation). They take care of illegal imports not conforming to the ODbL. That said, although I do not know the details, "There is an indication that one of our lines was added 12 years ago." seems a claim that needs much better substantiation. There is a lot of highly detailed data in OpenStreetMap, often carefully crafted from detailed aerial imagery, that may seem similar to other datasets, but has been independently digitized by community members. In addition, legal imports do take place after consent of data owners and if the data's license is ODbL compatible. See the OpenStreetMap Import Guidelines for the recommended practice. On the one hand, I understand your concern regarding Homeland Security, but on the other hand, it is not that you can hide powerline infrastructure. Even if it does not appear on the map, high voltage lines and substations are visible from miles away due to their size, and can easily be found on the ground or identified from aerial imagery as available almost everywhere. So, as a personal opinion, I don't think removing power infrastructure from OpenStreetMap, would add any security to these vital infrastructure elements. In addition, there are many countries around the world, e.g. here in the Netherlands where I live, where powerlines and substations are a formal part of official government produced topographic maps, and have no special status. OpenStreetMap is a community project, it can only thrive in an atmosphere where people accept that there may be local country specific differences in content added to the map (actually, the underlying database), but that any rendering in a map has to be a neutral, not country specific, representation, and may include something like powerline infrastructure. While there are a few known exceptions, like Google showing different content and specifically country borders to users in some countries, it is virtually impossible to create a styling that takes care of all sensitivities in each country. It is already hugely complex to render a map as detailed as OpenStreetMap is, let alone take care of >200 country specific styling wishes. The only realistic way that the visual map on www.openstreetmap.org therefor can differ in content, is if the local - country specific - community has broad consensus to not map specific content. I actually personally only know of one such case: In Israel, there seems to be broad consensus among the local community, to not map any military infrastructure (which is being mapped in other countries), and remove anything added by community members not aware of this local consensus. In your case, if you feel action is needed, you would need to convince the local US chapter of the OpenStreetMap community, to have consensus about not mapping powerline infrastructure in the US in OpenStreetMap. That will be a tough call I think, but is the only realistic way. ... View more

Late response, and you probably solved your issues by now, but what is "lots of vertices"? Thousands, millions? If you get over 100k vertices per polygon or line geometry, things can slow down dramatically. E.g. in a custom Python scripting and generalizing workflow, I've been processing data with records of up to 1.7M vertices per polygon in a PostGIS database generalizing them. I discovered by monitoring pgAdmin, the database management software of PostgreSQL, that it could take half a day for a single record to be processed when dealing with geometries > 1-2M vertices. I don't know if it is acceptable for your workflow, but by dicing them to 100k vertex limit, processing times were dramatically improved. E.g. for one medium long running dataset (although not containing that 1.7M monster polygon), it reduced processing times from 1h23m to just over 4-5 minutes. Dicing may not always be acceptable, but in my case it was, and it helped solving the performance issue. ... View more

Hi Megan, My remarks were more of general note and warning to anyone reading this thread, not to specifically criticize your work. I understand the difficulties with this type of work, and also the cost involved with analysis, and how all needs to be balanced. Sorry to have side stepped the question a bit, and not actually help you out with your problem, but good to read you managed to solve it yourself. ... View more

Actually, one of the first things I did when receiving that data set of Radon measurements, was to simply load it in ArcView, and set a graduated color legend. That simple act of "exploratory data analysis" already showed me the kind of random nature of the measurements of the dataset, with a confetti of light and dark colors abounding. One thing to keep in mind as well is the importance of a sense of "scale" of the phenomena you are trying to measure. E.g. if you wanted to interpolate a DEM based on height measurements in a coastal dune area, than clearly sampling one point per square kilometer isn't going to deliver results, and you just get random noise. Dunes generally vary on scales of dozens of meters (some exceptions like the ultra large coastal dunes in the Namib desert), hence you need to sample on a much finer scale. Ground surface height is also generally not a phenomena well suited for something like (kriging) interpolation, as ground surface can vary abruptly as well (ridges, canyons etc.). In general, it is the smoothly varying data like groundwater levels in sandy soils, pollution spreading in aquifers, ocean or the atmosphere, that shows strong spatial autocorrelation and is most suitable for interpolation. That said, there have been some interesting developments in the past 20 years or so, also with the incorporation of new interpolation techniques in Geostatistical Analyst like "Emperical Bayesian kriging", "Indicator kriging", and "Aerial interpolation", that allow you to deal with less "standard type" of datasets (An introduction to interpolation methods—Help | Documentation ) ... View more

As a nice anecdote about how things can go wrong if you forget that some data sets are unsuitable for interpolation: About 15 years ago, due to my Kriging Interpolator extension being available on ArcScripts, I was contacted by a fellow working for an environmental agency in one of the Nordic countries here in Europe. They had this fantastic data set with more than 25000 measurements of Radon gas in basements of houses across the country, including detailed data about the age, construction and condition of the house, and on what soils it was located if I remember it all well. All-in-all a treasure trove for a statistician. Radioactive Radon gas is a health thread, and they wanted to have a country wide map showing the regions with high and low radon in houses. Ergo, they wanted to interpolate the data, at least, that was their first idea of solving this issue. So why did he contact me then?... Turned out the data showed almost 90-100% nugget variance, and hence virtually no spatial auto-correlation, which they didn't understand. They had this fantastically detailed data set, and now they couldn't interpolate it?! After receiving the data, I could confirm their observation, virtually no spatial autocorrelation between Radon measurements. Even measurements in houses separated by only a small distance, could vary wildly in Radon. So, this set me thinking, why is this? The problem is, Radon is highly correlated with the specific condition of the house: on what soil type is it located, what building materials have been use, how is the ventilation status of the basement? None of these factors are ones that necessarily have high spatial autocorrelation, with smoothly varying changes. In fact, things like soil type and construction often vary abruptly: a soil or rock type is often abruptly changing from one spot to another, when geological processes like sedimentation, uplifting, volcanic activity etc. have changed the origin of the underground layers. The same holds true for human influenced factors like building construction. As a consequence, Radon in houses can vary abruptly and wildly as well in any random selection of buildings, even in the same town or even the same street. All of this precluded interpolation as the scientific and statistical method to analyze the Radon data! And the 90-100% nugget variance showed it. Yet the fellow that contacted me, had a hard time digesting this: a fantastic data set of Radon measurements across the country, and now they couldn't interpolate it to a country wide map? I ultimately suggested to him, to use traditional statistical methods with software like R and SPSS, to establish correlations between all the factors and conditions they had so zealously collected, and the corresponding Radon measurements in each house. After all, this huge data set with all its measurements and ancillary data, still was a treasure trove for a statistician! After establishing correlations, they could then use this knowledge to assess risk factors and regions at risk based on things like a geological soil map. If one or more soil types showed to be highly correlated with high Radon levels, simply classifying a geological soil map based on the statistically determined relations, might give a coarse, but statistically sound, indication of regions "at risk". The same could be done for building construction: if some types of buildings were shown to be at risk, classifying the countries buildings based on the statistically determined risk factors, could give them an indication of the status of all buildings in the country. ... View more

Megan White‌, I may not have to tell you this, because you may already be fully aware of it, but I am always highly wary of seeing people "automate" kriging interpolation... Kriging, by its very nature, is an interactive process that IMO should not be automated. It is actually the process of exploring your data using semivariograms, crossvalidation and other tools like the ones available in Geostatistical Analyst, that make kriging a superior interpolation method over most other ones, and allow you to achieve scientifically sound quality results. ...or discover that your data set is unsuitable for interpolation! - I still quite often see people trying to interpolate stuff even in the light of exploratory data analysis showing that they shouldn't interpolate the data. Some data sets inherently cannot be reasonably interpolated, e.g. when there is almost no spatial autocorrelation between points, and likely need other kinds of statistical analysis using tools like R or SPSS. People often seem to forget this, in a desperate attempt at all cost to create a surface covering data set from a set of point measurements. Doing away with all of that by "automation" may lead to erroneous results. I also think the kriging interpolation options of Spatial Analyst are unsuitable for truly scientific results (that is quite a thing to say for me, because I was the original author back in the days of ArcView 3.x of the "Kriging Interpolator" extension that build upon the Spatial Analyst tools to provide a more user friendly interface). I especially say this in the light of the Geostatistical Analyst, that provides far superior options for exploratory data analysis and validation of interpolation results. ... View more

If you are working with a File Geodatabase, like you do, then setting NULL means using None in Python, there is no NULL concept for setting values in a File Geodatabase to None / NULL, you should use None instead as in the code example. Remember: You are writing Python code here, which is not the same as defining SQL statements for use in ODBC. If you are working with an Enterprise Geodatabase and writing DML SQL statements and sending them through e.g. ODBC, then yes, you can set the values to NULL, in a DML SQL statement. ... View more

None is more or less the Python equivalent of NULL in a database. You are writing Python code, not SQL, here. Unless you are executing SQL through ODBC with something like pyodbc, and emmitting queries like <FIELD> IS NULL you generally would only use None in Python code. If you want to insert empty strings instead of None, you could simply use row[0]='' ... View more

To add to what Robert Scheitlin, GISP wrote, historically, it is actually a Google designed data format (Protocolbuffer Binary Format). It is now widely used by big name companies like Google, Mapbox and now ESRI to effectively distribute vector based geographic data, e.g. in Vector Tiles, and is the main data format for distributing OpenStreetMap data. One of the main benefits is that it is a very compact data format, and is fast to process with currently available tools. ... View more

Definitely don't expect any miracles from a VM. A VM is just another virtual computer, and as such limited by the same constraints that are valid for your own desktop. A virtual processor is still tied to a core somewhere on the globe, and those cores may even be worse in single thread processing than your local PC if you are unlucky. Only when your geoprocessing tool supports parallel processing, could you see benefits of some VM with many virtual cores or GPU based processing. But unless you're willing to pay a significant sum, a truly powerful VM may be out of reach, unless your university offers specialized computing services for students. Maybe cutting up your raster data set in smaller units, and compute those individually (in parallel), on multiple cores of your current machine, or multiple computers at the same time in your department, is the best bet to get results faster. ... View more

If these rail and bicycle path lines are in one and the same layer, you likely need to set the Symbol Levels to get the data to display and export right. I recognize the issues you describe, but setting Symbol Levels has always proven the reliable way out of these issues. Note though, that web services do not support this properly, so if your ultimate goal is to create web services, you may need to choose different types of symbology, or separate the classes in their own layer. Also, using symbol levels adds extra drawing cycles to ArcGIS, meaning display times will suffer. Usually not an issue if your main goal is to export to high quality cartographic PDF output, but it may be an issue in other cases. ... View more

Hi Adam, Note that you will not see the actual geoprocessing tool listed in pgAdmin. Instead, the ArcSDE application is what handles the SQL communication in the background with PostgreSQL. To access that information in pgAdmin: - Click on the database in the left pane - Click on the Dashboard TAB in the right pane if it isn't already visible - In the Server activity pane on the bottom, choose Sessions if it isn't active - In the displayed list of sessions, look for your user name and / or IP address in the User / Client columns and a session with "ArcSDE" in the Application column. - Check that the State is active. In the example below, only the postgres user has an active status, but this was just an example screenshot. Idle sessions should be ignored, they are no longer active. - Click on the little black arrow in the third column of the display of an active session to open a drop down that shows the Details of the active session. - You can now see the SQL being executed at this moment. Note that the Server activity details pane is not automatically refreshed, use the refresh button in the top right of the details pane to update it. It is easiest to do all of this on a database not used by hundreds of other users, as you will immediately see the proper user / IP address, instead of potentially hundreds of sessions in the Server activity pane. ... View more

You really should open up pgAdmin, the database manager tool of PostgreSQL, to truly see what is going on. E.g., if you deleted only a selection of the rows of a (very) large table, and the Delete Row tool triggers a VACUUM FULL on your table, then PostgreSQL may have spend the excess 34 minutes re-writing the entire table in the background to remove dead tuples / rows and optimize the table by re-claiming disk space. Some database operations are inherently costly, and may not show progress (although it would be a good thing if geoprocessing tools in applicable cases and where possible showed at least a warning message if such situations are occurring). Note that in case you intend to delete all rows of a table, the Help justly recommends using the Truncate Table tool instead, which will be must faster / instantaneous. ... View more