J'ai souvent rencontré des questions concernant le téléchargement des pièces jointes (par exemple, photos et documents) à partir des couches d'entités ArcGIS Online. L'objectif est de fournir une solution alternative à ce qui est actuellement proposé.<\/P>
<\/P>
Premièrement, la seule façon dont les pièces jointes sont incluses lors de l'exportation des données via ArcGIS Online est lors de l'exportation en tant que File Geodatabase. La raison en est que le File Geodatabase est le seul format qui prend en charge les enregistrements liés. Par conséquent, la réponse la plus courante pour accéder aux pièces jointes des couches d'entités en dehors d'ArcGIS Online est d'exporter la couche d'entités en tant que File Geodatabase, puis d'extraire les données à l'aide d'un script comme celui-ci<\/A> via ArcGIS Desktop. Il existe également un script de synchronisation<\/A> disponible qui peut synchroniser un File Geodatabase local avec une couche d'entités hébergée. Ces solutions fonctionnent bien, mais elles dépendent d'ArcGIS Desktop, et dans le cas de la première option, nécessitent également de télécharger le File Geodatabase (qui peut augmenter en taille à mesure que vous y ajoutez des données) avant d'exécuter le script.<\/P><\/P>Pour cette raison, j'ai créé un script utilisant la nouvelle ArcGIS API for Python<\/A> qui fonctionne indépendamment d'ArcGIS Desktop. En fait, il fonctionne directement avec un Web GIS, donc vous n'avez même pas besoin de télécharger un File Geodatabase pour accéder à vos pièces jointes. Il peut être relancé régulièrement et ne télécharge sur le disque que les nouvelles pièces jointes si elles n'ont pas été téléchargées auparavant.<\/P><\/P>Que fait ce script ?<\/STRONG><\/P><\/P>Crée un dossier dans lequel les pièces jointes des entités sont stockées (par exemple Attachment Downloads)<\/LI>Dans ce dossier, crée un sous-dossier pour chaque couche dans la couche d'entités spécifiéeSi la variable AttachmentStorage est définie sur 'GroupedFolder', les pièces jointes sont stockées au format ObjectId-AttachmentId-OriginalFileName dans un seul dossier<\/LI>Si la variable AttachmentStorage est définie sur 'IndividualFolder', un nouveau dossier est créé pour chaque entité avec une pièce jointe (nommé selon l'ID de l'objet de l'entité), tandis que les pièces jointes sont stockées au format AttachmentId-OriginalFileName<\/LI><\/OL><\/LI>Si la pièce jointe existe déjà sur le disque, elle ne sera pas téléchargée à nouveau<\/LI>Résumé des téléchargements (nombre total de téléchargements et taille totale des pièces jointes) fourni en sortie finale dans la console<\/LI><\/OL><\/P>Comment s'installer ?<\/STRONG><\/P><\/P>Suivez le guide Install and set up<\/A> de l'ArcGIS API for PythonSi vous utilisez Anaconda, vous pouvez utiliser le fichier .yml pour créer l'environnement approprié<\/LI><\/OL><\/LI>Si vous utilisez le fichier Startup.bat, modifiez-le (dans un éditeur de texte) pour référencer les dossiers appropriés<\/LI>Une fois que vous avez ouvert un Jupyter Notebook, collez le fichier .zip ci-joint dans un emplacement accessible (et décompressez-le)<\/LI>Ouvrez le fichier DownloadAttachments.ipynb via Jupyter Notebooks et exécutez-le (cliquez dans le code et appuyez sur Ctrl+Enter)<\/LI>Le script s'exécutera sur la couche d'entités publique spécifiée et téléchargera les pièces jointes dans le dossier Downloads spécifié<\/LI>Un fichier journal associé est créé dans le dossier Logging chaque fois que vous exécutez le script<\/LI><\/OL><\/P>Notes :<\/STRONG><\/P><\/P>Mettez à jour la variable FeatureLayerId pour exécuter sur votre propre couche d'entités<\/LI>Spécifiez votre PortalUserName, PortalPassword si la couche d'entités est sécurisée<\/LI>Les noms des dossiers sont filtrés pour ne montrer que les caractères 0-9 et a-Z<\/LI>Ce script n'a pas été testé avec ArcGIS Enterprise (seulement les couches d'entités hébergées dans ArcGIS Online), mais il devrait quand même fonctionner<\/LI>Le code est également disponible via le dépôt GitHub Esri developer-support<\/A><\/LI><\/UL><\/BODY><\/HTML>
Pour cette raison, j'ai créé un script utilisant la nouvelle
Que fait ce script ?<\/STRONG><\/P><\/P>Crée un dossier dans lequel les pièces jointes des entités sont stockées (par exemple Attachment Downloads)<\/LI>Dans ce dossier, crée un sous-dossier pour chaque couche dans la couche d'entités spécifiéeSi la variable AttachmentStorage est définie sur 'GroupedFolder', les pièces jointes sont stockées au format ObjectId-AttachmentId-OriginalFileName dans un seul dossier<\/LI>Si la variable AttachmentStorage est définie sur 'IndividualFolder', un nouveau dossier est créé pour chaque entité avec une pièce jointe (nommé selon l'ID de l'objet de l'entité), tandis que les pièces jointes sont stockées au format AttachmentId-OriginalFileName<\/LI><\/OL><\/LI>Si la pièce jointe existe déjà sur le disque, elle ne sera pas téléchargée à nouveau<\/LI>Résumé des téléchargements (nombre total de téléchargements et taille totale des pièces jointes) fourni en sortie finale dans la console<\/LI><\/OL><\/P>Comment s'installer ?<\/STRONG><\/P><\/P>Suivez le guide
Notes :<\/STRONG><\/P><\/P>Mettez à jour la variable FeatureLayerId pour exécuter sur votre propre couche d'entités<\/LI>Spécifiez votre PortalUserName, PortalPassword si la couche d'entités est sécurisée<\/LI>Les noms des dossiers sont filtrés pour ne montrer que les caractères 0-9 et a-Z<\/LI>Ce script n'a pas été testé avec ArcGIS Enterprise (seulement les couches d'entités hébergées dans ArcGIS Online), mais il devrait quand même fonctionner<\/LI>Le code est également disponible via le
very cool! i'd love to see this land in https://github.com/Esri/developer-support
Hi John - more than happy to push this into a GitHub repo! I'll PM you to progress.
Script has now been merged into the Esri developer-support GitHub repository!
LOVE this script, it has huge potential to streamline my workflow. One question though- how would I need to modify the script to change the naming convention of individual attachments to include a value from a field rather than the objectID?
In my case, each feature has a SerialNumber field, and I'd love the exported attachments to be named (SerialNumber, AttachmentName) rather than (attachmentId, attachmentName)
Hi Josh,
Glad you find the script useful! The names of the files are set on lines 117:
fileName <SPAN class="operator token">=</SPAN> <SPAN class="string token">'{}-{}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>attachmentId<SPAN class="punctuation token">,</SPAN> attachmentName<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN></SPAN>
and 131 (depending on whether you are using individual or grouped folders):
fileName <SPAN class="operator token">=</SPAN> <SPAN class="string token">'{}-{}-{}'</SPAN><SPAN class="punctuation token">.</SPAN>format<SPAN class="punctuation token">(</SPAN>currentObjectId<SPAN class="punctuation token">,</SPAN> attachmentId<SPAN class="punctuation token">,</SPAN> attachmentName<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN></SPAN>
However in order to include attribution in the file name the code would need a decent amount of rejigging. The reason for this is that only Object ID's are returned from the Feature Layer as in link 92. A subsequent query would need to be made to get the attribution.
featureObjectIds <SPAN class="operator token">=</SPAN> featureLayer<SPAN class="punctuation token">.</SPAN>query<SPAN class="punctuation token">(</SPAN>where<SPAN class="operator token">=</SPAN><SPAN class="string token">'1=1'</SPAN><SPAN class="punctuation token">,</SPAN> return_ids_only<SPAN class="operator token">=</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN></SPAN>
The reason it's written this way is to ensure the script can deal with a large dataset. It's a good suggestion so I'll consider enhancing the script if I get a chance. But it's unlikely to happen anytime soon. Please feel free to contribute to the script in GitHub if you manage to figure it out!
Mikie
This script is very helpful. Is there a similar example for adding attachments to a hosted feature layer?
Hi Robert - not that I'm aware of. Ultimately you would have to structure the attachments so they can be linked to specific features in the dataset (like how they are downloaded through this script). There is documentation on adding attachments below, and some logic from this script could probably be used:
Layer Attachments - ArcGIS API for Python
arcgis.features.managers Module
Hey Michael I was able to get a script running to add attachments from a folder to a hosted feature layer based on an attribute value. I used your configuration and then used the dataframe object to match files etc. Very basic but it works for me! Thanks for pointing me in the right direction.
AddAttachments with API for Python
Very nice Robert - thanks for sharing. Definitely one to add to my bookmarks!
Hi Michael Kelly
Thanks a lot for the script, it's highly appreciated, as the save as "FileGDB", "Replica" and download from Survey123 interface doesn't work for the concerned service.
Yet we still run into issues.
1. The "GroupedFolder" as AttachmentStorage doesn't seem to work (individual folders are created, and no objectid attached to the filename).
-> Not so important, we just added the objectid to filename also in the "IndivualFolder" storage part.
2. The feature Service contains about 5000 points with about 10000 pictures. The script runs fine up to ~1000 pictures (in ~400 folders), but then stops.
Here are the errors:
RuntimeError Traceback (most recent call last)<ipython-input-1-8bd11ca8f5f2> in <module>() 117 if not os.path.isfile(newAttachmentPath): 118 logger.info('The size of the current attachment being downloaded is {}MB'.format((attachmentSize/1000000)))--> 119 currentAttachmentPath = featureLayer.attachments.download(oid=currentObjectId, attachment_id=attachmentId, save_path=currentFolder) 120 #Rename to ensure file name is unique 121 renameFile(currentAttachmentPath, newAttachmentPath)
~\Anaconda3\lib\site-packages\arcgis\features\managers.py in download(self, oid, attachment_id, save_path) 38 desired_att = [att for att in att_list if att['id']== attachment_id] 39 if len(desired_att) == 0: #bad attachment id---> 40 raise RuntimeError 41 else: 42 att_name = desired_att[0]['name']
RuntimeError:
Do you have an idea? Before that error, we ran also into timeouts. So we are not quite sure if it's a connection issue. Or are there simply too many features? Is there a way to tell the script to do the first 1000, pause, resume where it stopped for another 1000 etc until it's finished?
Your help is highly appreciated,
Annina
#survey123 download
Hi Annina,
I haven't tested for a dataset with that many features, and it could potentially cause problems. A quick fix might be to alter line 92.
featureObjectIds <SPAN class="operator token">=</SPAN> featureLayer<SPAN class="punctuation token">.</SPAN>query<SPAN class="punctuation token">(</SPAN>where<SPAN class="operator token">=</SPAN><SPAN class="string token">'1=1'</SPAN><SPAN class="punctuation token">,</SPAN> return_ids_only<SPAN class="operator token">=</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN></SPAN>
This is where the original query to the feature layer is made and returns all the relevant features. When the where clause is set to 1=1, all features are returned. You could alter this to something like below to run the script for parts of the dataset:
featureObjectIds <SPAN class="operator token">=</SPAN> featureLayer<SPAN class="punctuation token">.</SPAN>query<SPAN class="punctuation token">(</SPAN>where<SPAN class="operator token">=</SPAN><SPAN class="string token">'OBJECTID < 1000'</SPAN><SPAN class="punctuation token">,</SPAN> return_ids_only<SPAN class="operator token">=</SPAN><SPAN class="token boolean">True</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN></SPAN></SPAN>
Given that the Feature Layer won't export, something may be corrupted internally. Judging by the error message above, the script has a problem with an attachment ID which is 'non existent' which has been return through the attachment query. If you could share the Feature Layer with me, I may be able to identify issues with the script and/or Feature Layer in question. If this is possible, my ArcGIS Online username is mikie.kelly and you could share it with me via a group without making it public.
Kind regards,
Good stuff, thanks for making our lives a little bit easier. Here's my question though - my colleague has shared a feature class with our organization and is asking me to complete this task for her. However, when I put my user credentials in the script and the feature name I can't seem to find the feature (probably because it belongs to her user credentials?). Is there a way to adapt this script for this type of enterprise level task, or do I have to own the feature class to run this script?
Hi Daniel,
Once the Feature Layer has been shared with you it should work. Can you access the Feature Layer by inserting the Item ID to the URL below and logging in?
https://www.arcgis.com/home/item.html?id=InsertItemIdHere
Is the database you are accessing hosted in ArcGIS Online, ArcGIS Enterprise (Data Store) or in an Enterprise Geodatabase? I have not tested this script with Enterprise Geodatabases.
Thank you for the quick response Michael!
I did figure it out – I was trying to use the name of the feature layer (ArcGIS online). However, I was able to find the layer using the id in the http address (a long hex decimal ID). Is that normal and is there a place to find that id anywhere in the layer properties or only in the address bar? I’ve highlighted these in the attached pdf.
Second, we’d like to name the pictures with a field from the actual feature layer (rather than the ATTACH table). It looks like the script accesses the feature layer properties through the ‘gis’ and the ‘.content’ and ‘.get’ methods. Can you direct me to the resources to better understand this approach, and how I might name the folders & images with a field of our choice?
Thank you kindly,
Daniel
Yes the URL is typically where you source the unique Item ID. An Item Name is not always unique so cannot be used for finding a specific item.
See my comment above in relation to attribution. In short, it is not currently possible with the script in its current state, and would need some further development to get this working. It's something I'll consider as an enhancement, but unlikely to happen in the short term.
Thanks again Michael,
I’m going to have to figure this out… I’ll forward a solution to you!
Daniel Aragon | Technical Specialist - GIT / Stream Restoration | Michael Baker International
daniel.aragon@mbakerintl.com<mailto:daniel.aragon@mbakerintl.com> | 720-479-3184
165 South Union Blvd., Suite 1000 | Lakewood, CO 80228
Thanks Daniel - feel free to push any enhancements to the GitHub repository! Also, I fixed the hyperlink in my previous comment.
Hi Michael,
That worked! We managed to download all the attachments. It helped also to identify that "non existen"-object. Thanks a lot for your help.
PS: Possibly that layer was corrupt. That's also why we needed the attachments to download, store and republish properly 🙂
Glad to hear it Annina
Thanks Michael Kelly for this workaround. This has been so useful
This may sound really stupid, but is it possible to schedule the script to run every 10 minutes or so?
Thanks so much,
Gee
Hi Gee,
Yes it can definitely be run on a scheduled basis, you can check out the following blog which goes into further details: Scheduling a Python script or model to run at a prescribed time
Thanks MKellyesri-ireland-ie-esridist
I'm not sure how I can set this up in python as I'm using jupyter notebook to run this.
So, is it possible to create an execution schedule inside jupyter notebook? Or, could you please shed some light into how I could run this entire process within python?
You can download as a .py file via Jupyter notebooks as below:
Once you do this, it can be used within Task Scheduler. If you are using the Python environment installed with ArcGIS Pro, the Python path will be something like:
C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python.exe
Hi MKellyesri-ireland-ie-esridist,
I get the following error when I run it in python. Any thoughts?
Thanks again
Really really appreciate the help....
Traceback (most recent call last): File "C:\Users\Geethaka\Downloads\Download Attachments\DownloadAttachments.py", line 59, in <module> fileHandler = logging.handlers.RotatingFileHandler('{}/{}.log'.format(SaveLogsTo, logFileName), maxBytes=100000, backupCount=5)AttributeError: module 'logging' has no attribute 'handlers'
I commented out the following lines of code to get it working
Pretty sure these only affect the creation of the log files
"""fileHandler = logging.handler.RotatingFileHandler('{}/{}.log'.format(SaveLogsTo, logFileName), maxBytes=100000, backupCount=5)formatter = logging.Formatter('%(asctime)s %(levelname)s %(relativeCreated)d \n%(filename)s %(module)s %(funcName)s %(lineno)d \n%(message)s\n')fileHandler.setFormatter(formatter)logger.addHandler(fileHandler)"""
Glad to hear you got it working.
In answer to your question, it might be just that you have to import logging.handler: Why do Python modules sometimes not import their sub-modules? - Stack Overflow
Yep, that was it
Thanks so so much for the help
Hi Robert Weber,
The link seems to be broken. Could you please share this again.
This would be super helpful
Thanks,
The page has been moved - see this link. In case this happens again, you can go to the root of Roberts GitHub account here.
Thank you for posting this, it seems like after an update something broke. I believe the method featurelayer.attachments.download is returning a list instead of a string.
TypeError Traceback (most recent call last)\spotted_lantern_fly.py in <module>() 134 currentAttachmentPath = featureLayer.attachments.download(oid=currentObjectId, attachment_id=attachmentId, save_path=featureLayerFolder) 135 #Rename to ensure file name is unique--> 136 renameFile(currentAttachmentPath, newAttachmentPath) 137 downloadCounter += 1 138 downloadSizeCounter += attachmentSize
\spotted_lantern_fly.py in renameFile(currentAttachmentPath, newAttachmentPath) 43 #Rename file - ensure new attachment path does not exist already 44 if not os.path.exists(newAttachmentPath):---> 45 os.replace(currentAttachmentPath, newAttachmentPath) 46 logger.info('{} being renamed as {}'.format(currentAttachmentPath, newAttachmentPath)) 47 else:
TypeError: replace: src should be string, bytes or os.PathLike, not list
I've done some research but cant find figure out how to provide a string. Ive tried replace, rename, renames Is this something anyone else is experiencing and any ideas on how to fix?
Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.
