Latest Contributions by RoyceSimpson

We have reverted back to 10.2 and are experiencing normal, stable behavior. We discovered that at AGS 10.2.2, our Netapp shared storage appliance was keeping files open and never freeing them up (probably most from the tile cache folders and config-store), so over the course of about 4-5 days, the Netapp would have around 180,000 open files on each of both our AGS servers.  At some point the Netapp reaches a limit on what it can handle per server for open files and it starts freaking out.  That in turn causes the AGS servers to go wonky and eventually crash. I'm currently in communication with esri support on this so hopefully there will be some solution. ... View more

I downgraded one of our servers to 10.2, added it back to the site and now we have no locked files on the netapp for the 10.2 machine and a ton for the 10.2.2 machine.  That pretty much wraps this up.  Time to revert the other machine back to 10.2. Conclusion:  AGS 10.2 = 🙂 AGS 10.2.2 = 😞 (click emoji for more detail) ... View more

Welcome to my world.  Evidence is mounting to suggest this is a 10.2.2 issue.  I'm very close to reverting to 10.2.  This is a real bummer since we've just started to do some Collector/offline editing stuff, which requires 10.2.2. Esri, any word on this being a bug?  Fixed at 10.3? ... View more

did you get this resolved?  Same issue here. No resolution to the issue.  But, we are onto a new lead...  We are thinking that AGS 10.2.2 is not releasing files properly (or something like that)... such as our cached map service tile files, which are stored on a "netapp" network drive appliance.  We are noticing that the netapp is showing a constantly rising "open files" number for each server of our two server based AGS site.  Every other computer connected to the netapp is showing zero to a few open files at any given time.  The number goes up for a bit then goes back down.  For our AGS servers, the number just keeps going up and up and up.  As I write this, AGS server 1 is at 60,000+ open files and server two is at almost exactly the same, for a total of about 120,000 open files for the two servers.  Our netapp sys admin has been on the phone with netapp tech support and apparently the netapp doesn't like it when that number gets to (I'm not sure of the exact number but we'll say...) 260,000. So, we are watching those numbers and waiting.  In the past it has taken about 4 or so days to have this "config-store" crash.  That meshes with what we are seeing with the netapp open-files thing.  We'll see. Anyway, we are looking at two choices... downgrade to 10.2 and see if that clears this up or move all our shared files (config-store and cached tiles) off to a different form of shared storage, such as Compellent. I'll keep this thread updated as we go. ... View more

On Friday, we found the our AD ArcGIS Sever account had been removed from our server admin group.  We added it back in and haven't had the "config-store" communication issue yet.  Will update this thread in a week or so with a "yay or nay" on this being the fix. This didn't fix the issue.  We continue to have this issue about every 4 days or so.  I'm beginning to think this isn't a config-store issue but some sort of issue with Server 10.2.2 is getting all gummed up and not serving up map service cache tiles in a timely manner.  I've been monitoring our web server and prior to the servers being disconnected to the config-store, the web server's "active requests" start going through the roof.  What that means is, when someone fires up one of our map service maps, the requested tiles are not retrieved and the request stays "active" on the web server.  The only fix has been to reboot our AGS servers.  Once that is done, the "active requests" goes to a very low number and the site is peppy again.  Until a few days pass an this issue rears up again. Also, one of the AGS errors that comes up when this happens is: "SEVERE May 31, 2014, 5:33:49 PM Failed to get the configuration of the server machine 'COL.LC.GOV'. Too many open files" Time to open a ticket. ... View more

On Friday, we found the our AD ArcGIS Sever account had been removed from our server admin group.  We added it back in and haven't had the "config-store" communication issue yet.  Will update this thread in a week or so with a "yay or nay" on this being the fix. ... View more

Maybe you can check what the differences in the dev/test environment is compared to your production environment with your admins.  They could have possibly applied an OS patch to dev/test that was not applied to production (or vice/versa). Do you know if your dev/test servers are absolutely identical to your production servers?  I ask because my organization builds our servers at different times so there could be slight differences (Not an ideal situation for a controlled environment) between the environments which could explain why dev/test is working as expected and you have a problem with only your production environment. Do you have a heavy load on the production servers when this issue occurs? Load is never at a point where I'd think the servers would struggle to communicate properly with the config-store but I'm monitoring things very closely to make sure of that. I'll also look more closely at our test/dev env vs our production env.  This is a very tough nut to crack. ... View more

Do you have a development environment that can mimic the setup of your production environment with multiple machines?  This is what my organization did so we could try to flush out multi-server environment issues in the development environment so it would not impact production users.  I would suggest getting EDN licenses for your development environment to save cost. Good point.  Yes we have a dev/test env setup with "identical" servers and AGS configurations except that their config store is \\netapp\gisserverdata\arcgisserver_test\.  We have seen no issues with that "test" ags site. ... View more

I've never actually tried the local thing myself, it was just an idea to check for network issues. I was thinking you wouldn't change any kind of configurations or publish/change any services during the time of the testing, in which case the config-store data would all stay the same, I think? In which case, would the 2 ArcGIS Servers still need to be able to modify the config-store directory? It could certainly break the site for a few minutes when you first test it, but you would know immediately I would think. You are correct that this kind of thing is not supported or recommended though according to this documention. http://resources.arcgis.com/en/help/main/10.1/index.html#/Expanding_from_one_GIS_server_to_multiple_GIS_servers/015400000597000000/ Along that line of reasoning... if the config-store is not being actively modified by map service changes or what have you, then it shouldn't be an issue in terms of the servers communicating with it at a time like the other morning at 8AM when nothing "change oriented" was going on.  However, I think that ArcGIS Server actively communicates with the config store on a regular basis and one of the outfalls of the servers not being able to communicate normally is this:  On every occasion that this has occurred, if I try to go to our map service rest endpoints such as "http://maps.larimer.org/arcgis/rest/services", I'm presented with an ArcGIS Server login screen.  Nothing I put in there for login/password works.  So, for some reason, when communications with the config-store go down, the rest services don't know who is or isn't allowed to see them and respond by putting up a login screen. In any event, I do think that for testing purposes, if need be, I could copy the config-store folder to the c drive of one of the two servers, then repoint the config folder setting in Manager to that, making sure the other server can see the first server's C drive.  If all goes well with that setup, then I can assume there is some issue with communicating with the netapp.  If the same thing happens, then I can assume the issue is with ArcGIS Server itself. ... View more

I'd check the Windows Server Event Logs during the time of the communication errors as well. Might be some kind of permissions issue as well. For changing the config-store location, I should have clarified a bit better; I was referring to having a local copy of the whole directory on each server, so in that case I would just copy-paste the whole config-store directory that is on the network location to each server on a local drive for each server. I've checked the windows server event logs as well, with nothing unusual to report. Regarding the config-store thing... wouldn't what you mentioned above break the site?  Don't both servers need to be able to interact with a shared config-store location?  If I copy that folder to each server locally and each server only sees the copy on their respective machines... wouldn't that break how "the site" works in terms of the two machines managing the shared services? ... View more

I know you said you checked with your admins, but still, the first thing I would check is some network activity logs during the time it starts to give this message, if it is really occurring on both ArcGIS Server boxes at the exact same time then it is almost guaranteed to be some kind of network issue with connecting to that network appliance. Maybe they recently started testing IPv6 or the DNS server was updated or anything along those lines. I would try an nslookup of that server (arete.lc.gov) from each ArcGIS Server box when it says it can't connect to the config-store. You could probably test this hypothesis by switching to a local config-store for 1 week on both servers and see if the problem stops. I think they have checked the network activity logs but I'll affirm that.   Also, the issue only seems to happen to one server or the other.  As you can see by that log screenshot, the affected server is "Arete".  The errors for that particular instance started at 8:00AM and not until 8:14 does the other server "Col" start throwing errors. I've thought about doing the "local config-store" thing but am playing the waiting game for now.  If I do choose to put the config store locally, what is the procedure for that?  Just point to the new location in Manager and make sure the other server can see that location?  Or do I need to do something more drastic like copy the config-store folder from the netapp to the local space first, then point to that in Manager? ... View more

prompting the user to save the file locally will work. var file : FileReference = new FileReference() file.save(myPDF.save(Method.LOCAL),"my.pdf"); I too am looking for a better workaround, everyone likes to see the PDF before they print or save. I would encourage everyone who is experiencing the issue to vote on the bug nasha001, THANK YOU! The below code solves this issue for me. Good riddance! Alert.show("PDF creation complete. Please choose a destination folder for your PDF file.", "Success!", Alert.OK | Alert.CANCEL, this, alertListener, null, Alert.OK); private function alertListener(e:CloseEvent):void { if (e.detail==Alert.OK) { var file:FileReference = new FileReference; file.save(_pdf.save(Method.LOCAL),"mypdffile.pdf"); } } ... View more