topic Re: Detect broken links with arcpy? in Python Questions

Detect broken links with arcpy?

TracyDash — Fri, 07 Mar 2014 18:59:01 GMT

Hi,

I adapted some code I found online to search for broken (moved or renamed) hyperlinks in a shapefile. It should write all broken links to a text file, but instead it is writing ALL the hyperlinks to the text file. Any advice on what to change? I know extremely little about python.

Also, I have arcmap 10.1

thank you!

import arcpy
import os

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW"]

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        if os.path.exists(Roll):
            pass
        else:
            f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","a")
            f.write(Roll + os.linesep)
            f.close()

Re: Detect broken links with arcpy?

JamesCrandall — Sat, 11 Dec 2021 22:31:20 GMT

Hi,

I adapted some code I found online to search for broken (moved or renamed) hyperlinks in a shapefile. It should write all broken links to a text file, but instead it is writing ALL the hyperlinks to the text file. Any advice on what to change? I know extremely little about python.

Also, I have arcmap 10.1

thank you!
import arcpy
import os

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW"]

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        if os.path.exists(Roll):
            pass
        else:
            f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","a")
            f.write(Roll + os.linesep)
            f.close()

In other words, it is evaluating everything as "broken".

This may seem like a silly question but is "PROJECT_DW" the correct field that contains the paths?

Also: would the evaluation fail if there are blank spaces in your "Roll" string variable? Maybe you should attempt to deal with that possibility:

Roll = str(row[0])
Roll = Roll.strip()

Re: Detect broken links with arcpy?

JoshuaChisholm — Sat, 11 Dec 2021 22:31:22 GMT

Code looks (really) good to me. A few questions:

Are the hyperlinks to a drive you are now connected to?

Are the hyperlinks pointing to the internet? If so, try this code:

import arcpy
import os
from urllib2 import urlopen

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW"]

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        try:
            urlopen(Roll)
        except:
            f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","a")
            f.write(Roll + os.linesep)
            f.close()

Re: Detect broken links with arcpy?

TracyDash — Fri, 07 Mar 2014 19:42:24 GMT

jamesfreddyc:

That's a good thought- I was just thinking that 'pass' wasn't working

That's not a silly question, but yes it is the correct field

and thanks for the idea, tried but eliminating spaces didn't make a difference...

hua17:

1. yes, the hyperlinks are on a drive I'm connected to. The drive can be a bit flaky at times (well, just when the computer falls asleep), but I think it should be OK...

2. no

Re: Detect broken links with arcpy?

JamesCrandall — Sat, 11 Dec 2021 22:31:25 GMT

Can you post an example of how the hyperlink looks? Maybe it is a character evaluation thing that isn't working as expected.

Also, just for kicks, print out "Roll" to verify it


import arcpy
import os

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW"]

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])

        ## print out the link to verify it
        #try to print it like this first:
        print "{0}".format(row[0])
     
        #try to print it like this too:
        print str(Roll)

        if os.path.exists(Roll):
            pass
        else:
            f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","a")
            f.write(Roll + os.linesep)
            f.close()

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 13:46:35 GMT

Sure, here are examples of types of hyperlinks we will search through:

folders: F:\LP04\COPP DUNN PLANTATION
pdfs: F:\MSL\EMAIL SURVEYS\STANDARD PACIFIC\DUNN PLANTATION\L65 GBBF 10-30-03.PDF
dwgs: F:\LP04\COPP DUNN PLANTATION\DWG\116-03-295 LT65.DWG

If spaces are an issue, than I'm pretty much screwed. There is way too much data to change all the names..

Tried the print function- again displayed ALL the hyperlinks instead of just the broken ones (I entered some fake links for testing).

Re: Detect broken links with arcpy?

JamesCrandall — Sat, 11 Dec 2021 22:31:28 GMT

Sure, here are examples of types of hyperlinks we will search through:

folders: F:\LP04\COPP DUNN PLANTATION
pdfs: F:\MSL\EMAIL SURVEYS\STANDARD PACIFIC\DUNN PLANTATION\L65 GBBF 10-30-03.PDF
dwgs: F:\LP04\COPP DUNN PLANTATION\DWG\116-03-295 LT65.DWG

If spaces are an issue, than I'm pretty much screwed. There is way too much data to change all the names..

Tried the print function- again displayed ALL the hyperlinks instead of just the broken ones (I entered some fake links for testing).

I tested for spaces in the filename and the os.path.exists picked it up just fine. I am wondering if the path is not constructed well enough though.

My best guess is that you are not able to apply string literal to the evaluation. For example, this would evaluate as "else":

roll = 'H:\exists\broken\dfBA D.csv'
if os.path.exists(roll):
    print "roll exists: " + roll

else:
    print "roll failed"

However, this would evaluate as "exists":

roll = r'H:\exists\broken\dfBA D.csv'
if os.path.exists(roll):
    print "roll exists: " + roll

else:
    print "roll failed"

The difference being that the "roll" variable is set with an "r" in front of the actual string.

Re: Detect broken links with arcpy?

JamesCrandall — Sat, 11 Dec 2021 22:31:30 GMT

See if this works -- replace "\" with "\\" in your path string formation:


import arcpy
import os

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW"]

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        Roll = Roll.replace("\\", "\\\\")
        if os.path.exists(Roll):
            pass
        else:
            f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","a")
            f.write(Roll + os.linesep)
            f.close()

This is also valid way to deal with it too:

roll = roll.replace('\\', r'/')

Re: Detect broken links with arcpy?

JoshuaChisholm — Mon, 10 Mar 2014 15:26:21 GMT

I think James might have found the problem. If so, it's a pretty similar problem to this forum page.

I'm not sure if the replace functions will work. As soon as python reads in the string from the cursor (Roll = str(row[0])), it sees any \'s as special characters. For example, it sees "Path\to\newfolder" as "Path[tab character]o[new line character]ewfolder". I don't think the replace will work, because it no longer thinks \ is in the string.

One (lame) workaround might be to create a new field in the attribute table (maybe called "PROJECT_DW2") and replace "\" with "\\" (using VB).
Something like this:
[ATTACH=CONFIG]32082[/ATTACH]

Then run your original script with "PROJECT_DW2" instead of the original path.

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 16:49:39 GMT

That is a great thought. I had no idea that could be an issue- unfortunately I did what Joshua said and still ALL the hyperlinks showed up.

Is it possible to blame my sometimes occasionally flaky network? I usually don't have problems during the day, just when my computer starts up and wakes up from sleeping...or possibly some sort of other path issue?

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 16:50:09 GMT

Forgot to add, also tried James idea and still all showed up.

Re: Detect broken links with arcpy?

JamesCrandall — Sat, 11 Dec 2021 22:31:33 GMT

Forgot to add, also tried James idea and still all showed up.

Your issue is related to how Python is handling the escape characters ("\") in your path names. As Joshua corrected my post, you will not be able to overcome your problem by simply attempting to string.replace() method on the values, and I have not found any other way to deal with this. For now, you'd have to calculate a new field like Joshua suggested.

Edit: I noticed you said you tried Joshua's suggestion but I think you may have missed the full solution as it worked for me. That is if I repalce my test string to this:

Roll = "H:\\exists\\broken\\dfBA D.csv"

It will get correctly evaluated in any test logic:

if os.path.exists(Roll):
    print "roll exists: " + Roll
else:
    print "roll failed"

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 17:27:50 GMT

Well, I added a new field and recalculated it so all \ became \\. Still, every single hyperlink showed up on my text file when I ran the original code after doing that (and I did change the field name). So not sure where to go from here.

Re: Detect broken links with arcpy?

JamesCrandall — Mon, 10 Mar 2014 17:44:14 GMT

Well, I added a new field and recalculated it so all \ became \\. Still, every single hyperlink showed up on my text file when I ran the original code after doing that (and I did change the field name). So not sure where to go from here.

Maybe another dumb idea but make sure you can see the F: drive from where you are executing the Python script. I stay away from mapped drives and opt for using UNC paths for everything file/directory related (which also has an issue with character limits).

Sorry I cannot help any more on this one as I am not sure what else I could do.

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 17:46:18 GMT

Yes, will do. Our server can be strange at times.

But thank you so much for all your help!

Re: Detect broken links with arcpy?

JoshuaChisholm — Sat, 11 Dec 2021 22:31:36 GMT

Ok, two more dumb questions:
1) Have you been clearing/deleting the "C:\Users\GIS\Documents\brokenlinks.txt" file before each time you've run the script. You are using the append parameter ("a"). This means every time you run the script all additional broken links (or whatever this script is spitting out) will be appended to the end of whatever already exists.

If it were me, I would organize my script a little differently:

import arcpy
import os

fc = r" C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_DW2"]

toTxtFile=""

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        if os.path.exists(Roll):
            pass
        else:
            toTxtFile+=Roll + os.linesep

#write to file
f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","w") #note that "w" will overwrite the file if it already exists
f.write(toTxtFile)
f.close()

2) Can you provide a sample (copy and paste) of the kind of links that should be working but are appearing in the brokenlinks.txt file anyway.

Re: Detect broken links with arcpy?

TracyDash — Mon, 10 Mar 2014 19:06:01 GMT

1. Yes I have

Also- tried your script (thanks!!) but it still produced the same result.

2. OK I attached two files- the first is a piece of the attribute table and the second is a piece of the txt file (if you scroll down on the text file there are thousands of more hyperlinks). It's a bit funny because some fields are blank (could this be an issue?? I realize now that I should have mentioned that sooner.)

-also I forgot to run the second field I made with two backslashes. It has the same result, however. Sorry

Re: Detect broken links with arcpy?

JoshuaChisholm — Sat, 11 Dec 2021 22:31:38 GMT

Ok. I'm really stumped.

Let's try another few small changes to the script:

import arcpy
import os

fc = r"C:\Users\GIS\Documents\TEST 2-25\project_test.shp"
fields = ["PROJECT_D2"]

toTxtFile=""

with arcpy.da.SearchCursor (fc,fields) as cursor:
    for row in cursor:
        Roll = str(row[0])
        if not os.path.exists(Roll):
            toTxtFile+=Roll + os.linesep

#write to file
f = open(r"C:\Users\GIS\Documents\brokenlinks.txt","w") #note that "w" will overwrite the file if it already exists
f.write(toTxtFile)
f.close()

Secondly, let's also try picking out a few hyperlinks from brokenlinks.txt that you are 100% sure actually exist (copy and paste into windows explorer). Then let's opening python (Start > All Programs > ArcGIS > Python [2.7] > IDLE (Python GUI)), type in import os, and try those selected hyperlinks manually. For example:

os.path.exists(r"F:\Lp04\PROV NORTHGLEN\DWG\SHEET 1.DWG")

Just in case, let's test these two lines too:

os.path.exists(r"F:")
os.path.exists(r"C:")

Let me know the results.

Re: Detect broken links with arcpy?

TracyDash — Tue, 11 Mar 2014 11:40:47 GMT

1. code yielded same results as before

2. tested true (also tried a fake hyperlink which tested false)

3. true

Re: Detect broken links with arcpy?

JamesCrandall — Tue, 11 Mar 2014 12:04:54 GMT

2. tested true (also tried a fake hyperlink which tested false)

This is because the string literal is forced. The problem is that because you have stored your file paths/links with single "\" backslashes, Python is evaluating these as escape characters.

If you were to replace:

os.path.exists(r"F:\Lp04\PROV NORTHGLEN\DWG\SHEET 1.DWG")

With:

os.path.exists("F:\Lp04\PROV NORTHGLEN\DWG\SHEET 1.DWG")

Then it will evaluate as false.

Additionally, I tested these:

These evaluate as True:

os.path.exists("F:\\Lp04\\PROV NORTHGLEN\\DWG\\SHEET 1.DWG")

os.path.exists("F:/Lp04/PROV NORTHGLEN/DWG/SHEET 1.DWG")

I didn't find any way to apply a .replace method on the stored paths you have because it will always incorrectly evaluate the second backslash in each stored path and fail. You will have to correct the source data with some other means as far as I can tell. Although I'd really like to see a Python solution.