Latest Contributions by ScottFierro2

If you have an advanced SQL DBA you can put lots of the work on the DB side. We use partitioned tables, column store indexes, FILESTREAM, and I am sure some other DB side tricks to manage datasets that climb well over 1 million. Takes some effort and we break or push the edges of almost everything out of the box ESRI says it supports but in doing so we maintain performance speeds on the front end for users and actually make some things more manageable for us on the back end. I'd agree with mikedmanak‌ that we use all offline for collections and that's been a key piece of managing the system especially since we could more easily push our statewide .tpk files for high resolution imagery we collect. ... View more

We are in the process of growing this but currently you can view last 3 years on our networks in our TIMS application. From this link at the bottom under Safety select the year and then in the blue bar by map you can run various queries against it. https://gis.dot.state.oh.us/tims/map We are working to roll nearly all of our crash data and front end analysis work into a separate piece of this app (pending summer release time) that will provide not only more in depth public facing access but will also have secured content for our internal statewide stakeholders. Our biggest issue is always ensuring we've fully scrubbed out any PII or SPI sorts of data and only release what's determined to be publicly allowable. ... View more

We currently use Collector and also use AirWatch. Sounds like roughly the identical scenario and conundrum we had. Thus far we haven't seen a way to make this work. Our workflow was to first pre-load all the iPad Mini 4's 128GB with the .tpk's. Since we need the entire state of Ohio that's not feasible as there's not enough storage space for the entire cache, which is roughly 320GB. Thankfully we have the state subdivided into 12 districts and the largest cache for the single biggest district was around 70GB and it allowed us to include a little bleed over imagery along boundary lines for full coverage. With the pre-loaded maps, each morning the crew come in and select the work areas for that day and download the Feature Classes (SQLite DB) with related tables for offline use. At end of collection connect back to WiFi and sync. Bit clunky but the limitations of the existing technology it seems. We can say that we have also configured Collector to use FILESTREAM on the backend for storage of attachments in SQL Server and this has been honored without issue with Collector's offline usage. From recent conversations with ESRI we have also identified that the ESRI preferred workflow means that once the user is done with a particular work area DO NOT have them remove the offline map but instead have them download the new work area leaving the old on the device. This workflow may be ideal if you are using a single user -> single device setup. We have multiple users -> multiple devices as our setup making the ESRI preferred workflow not possible (usually 4 iPads in a shop with between 7-15 users per shop where on any given day any user could grab any of the 4 devices). The reason it's not possible for us would be tied to device storage limitations. We have already seen where not removing the existing map caused the storage to fill up once a device had circulated enough different users and work areas. Our workflow to have each user remove the offline map at the end of each day seems to work fine 99% of the time. We have had Collector deployed as a Production collection system for roughly 9 months and to date have had 1 instance where the users data collected for the day was fully lost due to syncing issues that appear to be tied to the removal of maps each day (per exhaustive digging with ESRI). We have seen 1 other anomaly with this scenario where in maybe 3-5% of usage the users are not able to sync. The workaround has been to have the users perform a hard close of the Collector app (double tap Home and swipe app up to close; presumably clears local devices cache) then re-launch the app and sync. This has been successful every time except the 1 previously mentioned. Will be keeping an eye on this thread though as it'd be interesting to see if anyone has found a way to get this working. ... View more

Ahh ok so first thing I see is in your local you are using a database connection that could be navigated to within ArcCatalog by going to the built in "Database Connections" section (inside your local copy listed as the "local variables"). Can't guarantee this is or isn't the issue but we have found some odd behaviors when using the ESRI built in DB connections. You could consider setting up a local folder location and copying the sde connection into the local path (via Catalog)then re-sourcing your model using that connection instead. We solely use this workflow so after building a new DB connection in Catalog we copy all sde connections into D:\Arcgisserver\Data folder. This makes the connection string universal regardless of which user is publishing rather than the user specific paths that ESRI dumps way down in C:\User\<username>... when utilizing the built in "Database Connections". Maybe give that a shot because I think ESRI reads it differently so your DB.schema.table name may be preserved. IF this doesn't work then I'd suggest going into your model that resides in that v101 folder and checking to see the pathways being used for your inputs. I'd be willing to bet you will find that rather than the published model pointing to your valid SDE connection and table that exists within the v101 folder it's either now using %packageWorkspace%\tablename. You can edit the model and navigate to the v101 .sde file and table and the published model will re-source to use your full shared folder path, save the model and its fixed. Sadly, can't explain why these occur just know how to quickly identify them and correct them when they do. We probably have upwards of 200 GP's across 3 environments and would say we have around 20ish that we know must be modified after publishing and have made many attempts to correct these issues with no luck yet. ... View more

Actually, just looked closer at it appears that your process is attempting to make a call using your .sde. Sorry I didn't look more closely at the error was focused on your mention of your script. Step 1: Is that .sde registered with the new server environment and does it validate? Step 2: If yes, navigate to the same path I mentioned above (also seen inside the error message) and confirm that there is a copy of your .sde connection inside that v101 folder? Step 3: If yes to #2, if you navigate to the location via ArcCatalog and launch that copy of your .sde connection does it work? Can you see the layer in question? Step 4: If yes to #3, begin investigating your model and/or script that exists within v101 and look for publishing issues as described above in 1st response. ... View more

Can't speak to whether you are doing anything wrong or not but can say we have had a slew of issues with ESRI modifying code at publishing. Still have yet to fully pinpoint the cause but in some cases it attempts to re-write input paths and parameters fields to use things such as the %packageWorkspace% call that goes wrong when it sets that as g_ESRI_variable_1. In most cases our anomalies we see are tied to a call that in desktop is something like arcpy.GetParameterAsText(0) that gets converted into arcpy.arcpy.u'D:\<whatever pathway>'. No clue why it modifies to the double arcpy call causing the pathway to break but have seen it a few times. Best thing I could suggest would be from your server navigate to your servers system directory path (can be found in Server Manager under the "Site" tab). From within the arcgissystem folder navigate to arcgisinput; NCDOT_Tools; SelectAlamanceCountyTool.GPServer; extracted; v101; <find your script wherever it might be in here>. Launch your script into whatever IDLE or Text Editor you prefer and compare the published script version to your desktop script version. Any discrepancies you can correct and save here to your published version. Attempt to re-run. I know this is less than ideal and at times this practice has allowed us to identify a slight issue in the original desktop script we can modify and then things work. Other times we simply have to document which services are known to need to be modified post publishing due to what seems to be a publishing bug. I can say this issue seems to arise more when we use a hybrid of model builder with python scripts. In pure python processes I can't say we have noticed this issue. ... View more

That server log isn't based on the server environment still having an active service it is based on a user making a request. Despite having removed the service there could still be users that have the removed service stored into an MXD or Story Map (as examples, plenty more they could have it stored in). This means when they launch these items, requests are still made to your environment for the service that no longer exists. Just to ensure things on your end are cleaned up because it's actually somewhat common to have residual folders/files from doing a delete you can scrub your NTFS locations as well. From Arc Server Manager go to "Site" -> "Directories" and you should be able to see the location for your server instance folders. You will go into each of the 4 folder listed here and look for the removed service (note: you can skip the cache folder unless you had built a cache for the service in question). From here navigate into the "Directories" folder and go into the "arcgisjobs" folder. This should look familiar as it's going to list folders and services you see in Server Manager. Navigate to the correct location and ensure there is nothing still there for the removed service. Do the same thing by going into the "arcgisoutput" folder and ensuring the service is removed. Last you will go into the "arcgissystem" folder. In here go into the "arcgisinput" and find the services potential location and be sure everything is deleted. Last still in the "arcgissystem" folder go into the "arcgisuploads" folders and click the "services" folder and same as the rest navigate to anticipated location and ensure everything is gone. Lots of steps but ensures your server is cleaned up and there's no residue left from the delete and you are already clearing out the logs for this. Still won't prevent users from being able to make requests which will cause error logs but it will give you peace of mind that you've eliminated things on your side. ... View more

sboden-esristaff‌ thanks for clarifying and we took a few minutes to test the workflow and confirmed everything worked as expected. Great work to ESRI for this and this will drastically improve and assist in user management of our 500+ and growing AGO user base on our end. Thank you for streamlining this process. ... View more

So right and by administer I guess what exactly is it they need to be able to do? Delete a service or change published parameters? Most published parameters can be edited by a user with publishing rights. So now it's just down to the ability to start/stop a service or delete it? If it's either of those it sounds like stuff you could download the server admin toolkit (https://www.arcgis.com/home/item.html?id=12dde73e0e784e47818162b4d41ee340) nest an admin level account inside of some python and then build out tools for each group to use. More effort on your end obviously but seems like a doable solution since it's not a native option in ArcServer. I was thinking more down the line of custom AD groups in which users are granted a role but even that is limited because ArcServer lacks the custom defined roles such as what is in ArcGIS Online. So even if you built a custom AD group populated with users and locked that folder down with the AD group if that group is assigned to the admin role on the server they get it on everything regardless. So it'd be multiple ArcServers with different users given admin or as Jonathan said consider an SOI. ... View more

So a start would be applying folder level permissions within Manager. http://server.arcgis.com/en/server/latest/publish-services/linux/about-gis-server-folders.htm ... View more

Agree with Rebecca Strauch, GISP‌ and that mirrors our workflow which we have finished with testing and moving to production. We took a 10.2.2 and a 10.3.1 DB both on SQL 2012 boxes and did backup/restore to SQL 2016 boxes that we have now gone to a high availability cluster on the SQL side with failover and DB side replication across the 2 VM's in the cluster and post restore we performed the upgrade to 10.5. The only caveat for anyone going this route do the restore to 1 VM perform the upgrade by direct connecting to the VM directly or you can use the clustered DNS connection but not have the DB replicated to the second server and once the upgrade is done to the first server then use SQL to replicate the DB to the next server because attempting to upgrade with active clustering in place doesn't work. We also have spun up an extra 10.5 DB in the new SQL environment from scratch as we had a need for pulling 1 schema out of the 10.2.2 DB into a DB of it's own. All DBO schema for us. So can't guarantee our configs are the same and for most they won't be as we are constantly pushing and breaking ESRI limits or back dooring ESRI processes from the SQL side of things but can say it does work. Can read into some of this SQL clustering configs though for any interested here: https://msdn.microsoft.com/en-us/library/ms189910.aspx  ... View more

rastrauch‌ I will try to keep this short without deep diving the technical details. Essentially, we have 2 AD forests, 1 for internal users 1 for external users, and we have a need to be able to authenticate users from either domain. The limitation to this has always been at the ArcServer level in which only 1 identity store can be used and ESRI staff has stated that it can read 1 global catalog which in our case = 1 AD forest. The hang up with this was that in order to address issues with locking down services on the ArcServers the only viable option is supporting 2 mirrored environments. Well that solves your issue at the ArcServer end but then introduces a considerable amount of work at your front end applications along with addressing how your single endpoint to the user base (a single website URL) simultaneously supports users from either forest through a combination of secured and unsecure services across your 2 ArcServer environments. Ideally the single ArcServer environment would simply address authentication across both. So we are in the final stages of testing our solution that makes this works and expect to roll out a production solution shortly. I will be writing up a paper that walks through building out an environment that can support these dual forest authentication issues and the logic/reasons behind it. The purpose to this post was the only situation in which we found issues with our test to occur is with the use of an internal AD service account not being able to authenticate but all user accounts authenticate just fine. Expectedly, since this is a non-ESRI supported configuration we anticipated speed bumps and the service account access into the REST endpoint for a secured service was the only issue we encountered. In our digging into things the only explanation is that within AD the service account doesn't have normal attributes (first/last name, email, etc.) populated. We confirmed this issue by building a generic service account into the internal AD configuring permissions to a secured service letting it fail, then populating every field that is populated for our typical users and the service account got in. Not something that's a show stopper we can populate the fields on service accounts, not ideal but its a workaround solution so happens, but it brought us to a good general knowledge point yet to be answered, what are the minimum required attributes? Simply due to time we aren't going to systematically walk through testing of populating various configurations of attribute fields in AD to solve this so took it to ESRI and then here. This means there are specific attributes that ESRI is expecting at authentication but nobody could tell us which attributes. The use case would be for scripted processes that we run nightly being able to leverage the secured REST endpoints all via the service account. ... View more

That's right it's all about the local host executing the task not where the script is doing work at. All that work you covered within the script itself. So the permissions issue seems to be your AD account having the local desktop permissions to execute python.exe which your Network or System Admin should be able to allow those permissions and limit to just that executable. ... View more

OK, so I found an old example out on a DEV server and have screenshots of all the tabs where we had used direct python call instead of running a batch file. You can check these for any variances but it seems like that double quotes issue would have been it. Given that the script runs fine in your IDLE but fails from Task Scheduler then if all your settings are similar to what's below and nothing stands out that fits a permissions issue I would have to lean towards your AD account lacks the permissions to run the python executable itself. Process of elimination dictates that the script works and you can execute the script from the IDLE and if your Task Scheduler is configured right then the only missing link between these is how Task Scheduler calls the Python executable to run the script. ... View more

OK, that looks like the correct configuration and appears as though you have the double quotes correct around the argument line. As I said we do this all via batch files now so been a little bit but try this. Ensure you have double quotes at both the begin and end of each line (program & argument). Give that a shot and see what happens but I think that will solve your run issues as I am almost certain that is the correct format. If it doesn't then there is something else still missing. ... View more