topic Re: ArcEnterprise 10.8 Service Definition File Size Limit in ArcGIS Enterprise Questions

ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Thu, 09 Jul 2020 17:32:43 GMT

Hello - I don't think that this section from the Enterprise help

Manage hosted feature layers—Portal for ArcGIS | Documentation for ArcGIS Enterprise

at 10.8 is entirely accurate:

"Allow attachments

You can allow editors to attach images and other files to individual features in a layer in a hosted feature layer. This is useful, as it allows you to associate documentation or photos to specific, relevant features. For example, a code compliance officer might attach a photo of the code violation for a specific address point, or a building inspector might include a PDF of a permit for a building feature.

Each hosted feature layer view inherits the attachment setting of the hosted feature layer from which it was created. By default, all view users can see the attachments inherited from the hosted feature layer. To control who has access to these attachments, you can hide the attachments in the hosted feature layer view.

Each file you attach to a feature can be a maximum size of 2 GB. To attach files larger than 2 GB, you can use the Upload Part and Add Attachment operations from ArcGIS REST API to do a multipart upload. . . . . . "

What I have found is that when sharing or overwriting a hosted feature layer via the 2.5.x ArcGIS Pro sharing module to Enterprise 10..8, the maximum service definition file size appears to be 2GB.

In my case, I have a point layer with some 3600 records and pdf attachments. The largest attachement is about 9MB. However, the total size of the service definition file exceeds 2GB, and the sharing module overwrite now fails with a 99999.

Specifically, the:

## arcpy.UploadServiceDefinition_server('C:/ArcProProjects/PortalUpdates/Environmental/' + lyr.name + '.sd', 'My Hosted Services', "","","","","","OVERRIDE_DEFINITION","SHARE_ONLINE","PUBLIC",
## "SHARE_ORGANIZATION","")

fails because I think that the entire sd file size cannot exceed 2GB.

Has anyone encountered this?

Re: ArcEnterprise 10.8 Service Definition File Size Limit

AnibalMmartinez — Mon, 13 Jul 2020 13:43:25 GMT

Buen día David Coley‌, proba subir en ventana incógnito de google chrome.

Saludos,

Anibal

Telecom Argentina

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Mon, 13 Jul 2020 19:45:15 GMT

Grasias Anibal - es posible. Pero luego tendria que usar la api . .

And I didn't really want to do that because the file does publish to AGOL using the Pro sharing module and I didn't want to have to re-write this part of my workflow unless necessary

Re: ArcEnterprise 10.8 Service Definition File Size Limit

JonahLay — Mon, 13 Jul 2020 21:14:45 GMT

Hi David,

I'm posting my response here as well to keep this post up-to-date. There is no file size limit for SD files. If your attachments are below the 2GB file size limit and you still see an error, there is likely a different issue here. We would need to see your python script to look into the problem.

Thanks,

Jonah

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Sun, 12 Dec 2021 02:01:32 GMT

Hi Jonah - yes thanks. And yes, that's what I thought as well re SD files. I'd never read anywhere about a size limit for those. Yes, the largest attachment (in this case pdf's) is about 9.5Mb. I like to work in the IDLE for Pro - here is a snippet of my code:

def stageSDDraft(mp,lyrName,summary,lyrs,descript,folder,tags,sdfolder):
 sddraft_out = "C:/ArcProProjects/PortalUpdates/" + sdfolder + "/" + lyrName + ".sddraft"
 draft = mp.getWebLayerSharingDraft("HOSTING_SERVER", "FEATURE", lyrName, lyrs)
 draft.portalFolder = folder
 draft.tags = tags
 ##draft.allowExporting = True
 draft.summary = summary
 draft.description = descript
 draft.overwriteExistingService = True
 draft.exportToSDDraft(sddraft_out)
 return
try:
 portal = arcpy.SignInToPortal("https://ags3.myorg.net/portal", "userName_portal", "userNamePw")
 aprx = arcpy.mp.ArcGISProject('C:/ArcProProjects/PortalUpdates/PortalUpdates.aprx')
 for m in aprx.listMaps("FloodAttachments"):
 print("Map: " + m.name)
 for lyr in m.listLayers():
 desc = arcpy.Describe(lyr)
 if lyr.name == 'ElevationCertificate':
 print(lyr.name)
 lyrList = []
 lyrList.append(m.listLayers(lyr.name)[0])
 stageSDDraft(m, lyr.name, "The Elevation Certificate point layer contains the addresses of homes and businesses that have a documented finished floor elevation that is above the Effective \
 FEMA designated base flood elevation.",lyrList,"The Elevation Certificate point layer contains the addresses of homes and businesses that have documented finished floor elevation \
 that is above the Effective FEMA designated base flood elevation. The certificate is accessed as a layer attachment.",
 "EnvironmentalLayers","Cadastral,Environmental,Services","Environmental")
 print(lyr.name + " Draft Created")
 arcpy.StageService_server('C:/ArcProProjects/PortalUpdates/Environmental/' + lyr.name + '.sddraft', 'C:/ArcProProjects/PortalUpdates/Environmental/' + lyr.name + '.sd')
 print(lyr.name + " Service Staged")
 arcpy.UploadServiceDefinition_server('C:/ArcProProjects/PortalUpdates/Environmental/' + lyr.name + '.sd', 'My Hosted Services', "","","","","","OVERRIDE_DEFINITION","SHARE_ONLINE","PUBLIC",
 "SHARE_ORGANIZATION","")
 print(lyr.name + " Service Uploaded")
 message = message + "\n" + "1. " + (lyr.name) + " Service Uploaded"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It's pretty straight-forward. However, you are probably right about there being another issue. I have been overwriting this feature layer weekly for a couple of years because of how the attachments are constructed.

The attachments are pulled in from another system of record and then we use a GP to dump them into another directory before attaching them to the points via a match table.

I am going to try setting this up with a new layer, new SD file, etc and see it works.....

Re: ArcEnterprise 10.8 Service Definition File Size Limit

ShilpiJain — Tue, 14 Jul 2020 15:37:46 GMT

Hi David,

Could you try the following as well? This might help us identify the issue.

Try the same workflow from Pro UI instead of Python.
Try uploading the SD directly to portal from portal home app.
Check the logs on server while using Python script and share the logs, if possible.

Thanks,

Shilpi

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Tue, 14 Jul 2020 16:50:45 GMT

Hi Shilpi - I did try using the Pro UI with the same result. 99999 fail code at the upload stage. I will try adding as an update later today.

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Wed, 15 Jul 2020 18:30:53 GMT

This is the portal error i receive:

Failed to add or update item '1cab88c8bef743baab32716aa17e9724'. Error writing to file: D:\arcgisportal\content\items\1cab88c8bef743baab32716aa17e9724\FloodCert.sd

Note this is an entirely new layer (but the same data and attachments) copied to my file gdb and added to my pro project's map and so in this case is not an overwrite but a new write.

I think i've got either a corrupt attachment or attachments or there are communication errors between my portal, server and datastore (although all components test correctly). So as Jonah Lay‌ suggested earlier there are probably other issues going on with my environment or the layer.

Unfortunately, my org is really abusing the attachment capability. I don't think attachments should replace a document search system or repository.

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Thu, 23 Jul 2020 19:40:29 GMT

So it turns out that this is a result of invalid characters in the attachment file name and some invalid attachment types The Postgres datastore at 10.8 does not like certain characters in the name - Brackets, braces, dashes, periods can all cause issues.

In this case some invalid email attachment types ".msg" and some period "." characters in the file names caused an 'Object Store Exception' error. Datastore thought that was the file name.

Amazingly ArcGIS Online hosted this layer with no issue, i guess it's datastore can handle these things.

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Tue, 28 Jul 2020 14:35:31 GMT

interesting development Jonah Lay In this case, after removing bad naming conventions and invalid attachment types, the sd file will still not complete an overwrite with either the pro sharing module or the api. The pro module fails with the same 5 9's code while the api featureLayerCollection.manager.overwrite(data_file=sdFile) method fails with

Expecting value: line 1 column 1 (char 0) after a certain amount of time. (@ 6 minutes on portal, ~ 21 minutes to agol)

this suggests a bad name or type of attachment, but if that were the case I would think the layer would fail it's .add and .publish when adding as a new layer. On the other hand, there are issues with size and hosted feature layers:

BUG-000127663 Exporting a hosted feature layer containing large attachments fails if the size of the
exported file geodatabase exceeds 2 GB.

To be clear, other service definition files / layers with attachments (with standard attachment names and types) that are less than 2Gb in size don't have any add, publish, overwrite or pro sharing module issues.

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Tue, 11 Aug 2020 21:18:25 GMT

Sad to say this is still an issue at Enterprise 10.8.1 and ArcGIS Online. The:

sdFile = os.path.join(sdPath, "FemaElevationCert.sd") #item[:-4]

sdItem.update(data=sdFile)

fs = sdItem.publish(overwrite=True)

method continues to fail as does the overwrite method from the feature layer collection manager"

portalFLC.manager.overwrite(sdFile)

whatever it is, for sd files with attachments that are larger than 2Gb can be added and published as new items through the api, but cannot be overwritten. The json error remains:

Expecting value: line 1 column 1 (char 0) after a certain amount of time. (@ 1 minute on portal, ~ 21 minutes on agol).

Jonah Lay‌ thought you might want to know

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Wed, 26 Aug 2020 18:37:47 GMT

So I gave up on this. I began by setting up a virtual directory in my IIS and tying that to a sever directory. I pushed all my documents into the server directory and set up a link field in arcade. If I need to link to more than one doc for any given feature, I'll use arcade for that too, most likely in a when statement. Sure is alot easier than trying push all these attachments as part of an sd file to the datastore.

Re: ArcEnterprise 10.8 Service Definition File Size Limit

GayathriVijay — Tue, 15 Sep 2020 22:01:05 GMT

Hi David,

I was able to successfully publish and overwrite an SD with attachments greater than 2GB from Pro using python window. I was also able to publish the data from scratch from Pro and also overwrite the same. I would suggest you contact Esri support and give your data, as well as the information about the errors/warning you see in the server logs.

Thanks,

Gayathri Vijay

Re: ArcEnterprise 10.8 Service Definition File Size Limit

DavidColey — Thu, 17 Sep 2020 17:40:53 GMT

Yeah I have done all that. Naming convention of the pdf attachment documents becomes very important when the total size of the service definition file causes the file to upload the json in parts. When a json part is encountered that contains some invalid character at the beginning of the part - like an ampersand, the initial upload does not necessarily fail, but all of my overwrite attempts do.

Now that I understand that, I understand why my large service definition files with a 100s to 1000s of attachments, would fail upon overwrite, but not in the same place.