Export files from file server using URL path

[shildebrand]

I recently wrote this script to download files from a file server using a folder path.  For example the "LINK" field has values such as this: "\\wapgis61\Scans\101-005\101-005.pdf".  We have recently changed the source data to to reference a virtual directory which uses URL's.  For example, the path referenced earlier is accessed with a URL like this: http://wapgis61/Scans/101-005/101-005.pdf .  Does anyone know how to modify this script to extract files that a referenced with a URL instead of a UNC path?

import arcpy
import os
import shutil

fc = arcpy.GetParameterAsText(0)
dest_path = arcpy.GetParameterAsText(1)
print(fc)
cursor = arcpy.SearchCursor(fc)
for row in cursor:
    field = "LINK"
    link = (row.getValue(field))
    head, tail = os.path.split(str(link))
    print ("Downloading: " + tail)
    shutil.copy(link, dest_path)

print("Download Complete")

[JakeSkinner]

Hi @shildebrand ,

You can use the requests module to do this.  Below is an example:

import requests

# Variables
url = 'https://some.server.com/test.pdf'
exportLocation = r'c:\temp\test.pdf'

response = requests.get(url)

with open(exportLocation, 'wb') as f:
    f.write(response.content)

[shildebrand]

Thanks Jake!  Do you know how to reference the a feature class field that has the URL in it?  The script below returns an error showing that the "LINK" field is an invalid URL.  I'm guessing I probably have an incorrect syntax?

import arcpy
import os
import requests

fc = arcpy.GetParameterAsText(0)
dest_path = arcpy.GetParameterAsText(1)
print(fc)
cursor = arcpy.SearchCursor(fc)
for row in cursor:
    #url format = http://wapgis61/Scans/101-005/101-005.pdf
    url = "LINK"
    print ("Downloading: " + url)
    response = requests.get(url)
    with open(dest_path, 'wb') as f:
        f.write(response.content)
    
print("Download Complete")

[JakeSkinner · Accepted solution]

Try the following:

import arcpy
import os
import requests

fc = arcpy.GetParameterAsText(0)
dest_path = arcpy.GetParameterAsText(1)
print(fc)
with arcpy.da.SearchCursor(fc, ['LINK']) as cursor:
    for row in cursor:
        print("Downloading: " + row[0])
        response = requests.get(row[0])
        outputFile = row[0].split("/")[0]
        with open(dest_path + '\\' + outputFile, 'wb') as f:
            f.write(response.content)
del cursor

[shildebrand]

Thanks Jake, with a slight modification to your suggestion, it worked!  Just changed the index number to 5 in row 13.  Thanks for your help!

import arcpy
import os
import requests

fc = arcpy.GetParameterAsText(0)
dest_path = arcpy.GetParameterAsText(1)
print(fc)
cursor = arcpy.SearchCursor(fc)
with arcpy.da.SearchCursor(fc, ['LINK']) as cursor:
    for row in cursor:
        print("Downloading: " + row[0])
        response = requests.get(row[0])
        outputFile = row[0].split("/")[5]
        print(outputFile)
        with open(dest_path + '\\' + outputFile, 'wb') as f:
            print(f)
            f.write(response.content)
del cursor