As-Built/Record Drawing management

[dolson_farmington]

Hi all-

It has been 9 months since I started my role as the sole GIS staff for a small city. Prior to this role, I was a GIS analyst in environmental consulting. Many of the skills I learned there apply to my current role, but there are several new challenges I'm navigating as the sole GIS contact for the city. 

One of these challenges is linking the city's record drawings to our spatial data. I know many cities opt to contract this work out or use software outside of ESRI's. Other cities use things like the Plans and Drawings Solution. 

My question is: what does your local organization use, what are the pros and cons, and what is the cost associated? 

We have an Enterprise deployment, so I would like to get as much use out of it as possible and avoid spending additional money on external software for this effort, but am aware that most cities do opt to outsource this type of task. 

Any an all response is appreciated!

[Bud]

We have:

• A feature class that has a polygon for the extents of each project.
• And a non-spatial table that is a list of files with a hyperlink field. The table is related to the polygons via a common PROJECT field. The relationship is 1:M. One polygon to many non-spatial table records.

It’s archaic and messy, but I suppose it works.

We considered using linear referencing instead, using a roads centreline feature class and a linear referencing table. But we figured it would be too fragile and too difficult to manage when a road needed to be split due to new roads being added to the network.

You could also try asking in one of these other communities:

• AEC (Architecture, Engineering and Construction) Questions
• Water Utilities Questions

@MErikReedAugusta might find this interesting.

[MErikReedAugusta]

@Bud , I'm starting to think you're spying on my team, given that this is now twice you've nailed something we're semi-actively working on!  😛

We're actually just starting to dig into this over here, too.  There's a system we've inherited, but it's not entirely functional for our department, and we're likely going to be rebuilding from the ground up.

There's a polygon FC in our Enterprise called "RecordDrawings" that (in theory) has all the plans relevant to the Storm Sewer, Sanitary Sewer, and Water systems.  (The first of those collections is the one that's least-functional at the moment).

There's also a tool floating around that a few people in the Sanitary & Water groups have that'll read a selected polygon on that feature class, go find the georeferenced document on the server, and plop it into the map for them to look at.  Very few of our georeferenced Storm files are currently available, though.

Some pitfalls we've seen with this solution:

1. The georeferenced drawings & the server are only available if you're in the office.  They can't currently be viewed from the field.
2. Sanitary/Water and Storm are in two different departments, each with separate servers holding drawings.  Makes it difficult to collaborate and is part of the reason the Storm stuff is behind-the-times on this.
3. When the plans were digitized & georeferenced, the extents weren't always the best.
   1. Some include things like elevation graphs that sat below a linear map; it'll make you think a drawing is going to cover your area, but it turns out it's a street over, instead.
   2. Some were somehow digitized with a bounding box around the rotated polygon.  So instead of being 1 mile long by 100 ft wide angled northeasterly, the polygon is a square maybe 1.5 miles to a side and oriented precisely parallel to the latitude.  This means it gets pulled for all kinds of queries that it has nothing to do with.
4. The current tool to pull the georeferenced file into the map was written in the Desktop days, which means it uses the mapping library.  That's been wholly replaced with the mp library, so the script needs to be rewritten.  (Not actually a huge issue, but something that needs to be done during our Pro conversion).
   1. This tool was written by a contractor, and our Water/Sewer folks were planning to pay that contractor again to do the conversion.
   2. I have yet to see the contractor's script, but I don't think it's all that complicated.  I wrote a quick proof-of-concept to see if I could replicate the functionality, and it took me all of a day, including the time to familiarize myself with the relevant bits of the mp library.

We're currently in the early conceptual stages for field-accessible options.  One is using simple attachments (attaching the drawing directly to the extent feature) and the other is hosting in a cloud server and including a hyperlink in the attribute table.

Our current main takeaways at this stage:

1. Trim your documents down to just the portion that contains the actual map & Assets, and only then georeference it.  You don't want excess clutter causing you to pull documents that have nothing to do with what you're researching.
   1. Counterpoint to this: The page area outside the map often includes valuable data that someone reading the plan will need.  So you have to balance these two conflicting issues.
   2. Best thing would likely be extent polygons that represent only the core map component, but that reference georeferenced files that contain the whole page.  In practice, this means extra steps to create & maintain.
2. Figure out a solution that lets all departments share the same data source(s).  It helps with collaboration and saves you on space & headache.
3. Consider where someone might be that needs to access this data.  Is it reasonable that they always be in the office to do so, or do they need access in the field, as well?

[Bud]

For what it’s worth, our drawings aren’t georeferenced. They’re just plain pdfs and images.

[Bud]

@MErikReedAugusta 

I'm starting to think you're spying on my team, given that this is now twice you've nailed something we're semi-actively working on!

I think I saw you mention storm sewer GIS data once. I figure, anyone who deals with sewer data likely faces the same challenges. And you've commented on a few of my posts, making me think you and I have similar roles in our respective organizations. 

When it comes to construction records retrieval (the subject of this post), that's a constant challenge. Our polygon and standalone table design works, but is clunky, so we're always interested in seeing how others do it.

A lesson learned is: Don't dump everything into your records retrieval system. Such as ancient hand-drawn plans and City-wide documents. They all pop up every time someone wants to retrieve records for a specific area. It's a pain to sift through all the stuff in there. Maybe less is more. Or have a separate system for the miscelaneous/old records. Or just filter them out for general use.

We get a fair bit of criticism about our records retrieval system. We've ended up putting a LATEST_DRAWING hyperlink field in our sewer FCs. Some users like that better when it comes to retrieving the latest construction drawing for a given sewer asset.

[dolson_farmington]

@Bud @MErikReedAugusta Great responses, thanks for such thorough input!

It sounds like in some ways, you both use strategies similar to the Plans and Drawings solution. I have always been curious to know whether that solution has been utilized by folks and if they thought it worked well. 

Your responses inspired a few follow up questions: 

• How are your documents named? Do you have any sort of system for naming each drawing that a.) makes sense and b.) doesn't require document name changes? Or, does document name not affect how you link your drawings to your spatial data? 
• Do you see value in separating out your storm, sewer, and water drawings? If so, how do you handle drawings that have combinations of any of the three? I'm personally of the belief that clicking on an area and getting all drawings of any category as a result is the way to go. Would love to hear thoughts on that.
• MErikReed--since you're in the process of rebuilding the whole linking system, I'm curious if you find it is worth the time and effort to link documents page by page manually? My thought is yes, as long as it only has to be done once (for a very long time at least). 
• Bud--since you say the current system is archaic and messy, I'm curious how you would approach rebuilding it if you were to start from square one. 

Thanks again for both of your responses! This is a huge undertaking with thousands of pages of record drawings to be linked, so I want to make sure I'm taking steps to ensure whatever strategy we use is a long-term one.

[MErikReedAugusta]

Right now, the majority of the documents we have on that feature class do not have human-readable names.  I haven't been able to really figure out the pattern to the naming scheme, though it feels like there probably was one back at the start of things.

My relatively loose plan for the new system would be to indicate that with an attribute that I can query out, but that may not work out in practice (many of the end users either don't have GIS software that can query like that, or aren't familiar enough with the software to do the query).  We're still workshopping this, because I think there definitely is value, but it increases the complexity rather a lot.

Right now that's more-or-less our plan, yes.  At minimum, generating the extent polygons would be individual pages.  If we're clever enough with how we go about splitting & georeferencing those files, though, I have a feeling I could automate at least some of the work to link back to a non-split document, if we go that route.

[Bud]

I think you should use human readable names as the file names. For example:

• Subdivision number
• Contract number
• Address
• Corporate file number

If there are data integrity issues and you need a plan B for finding a file, then it is very useful to be able to search your network folder via File Explorer for a given subdivision number, etc. Some users even prefer to search for files that way compared to using GIS to find files.

Also, for files that are not yet in your records system, if the file name is human readable, you can generate a list of the file names via File Explorer, Excel, Python, etc. and then use that list of file names as a starting point for data entry. Very handy. For example, maybe the subdivision developer submits drawings PDFs that have human readable file names. Use those file names to populate the GIS records in the attribute table.

@dolson_farmington and @MErikReedAugusta, if you work for government (like me), then feel free to reach out via a private message. Maybe I can get permission to show you our records retrieval system (pros and cons) in a Teams meeting. If that’s of interest.

[Bud]

By the way, my I.T. department has a Python script that they run as a nightly scheduled task on the GIS server to do the following:

1. Delete the rows from a “list of files” helper table. The table isn’t really part of the records retrieval system, it’s a separate read-only table that we use for occasional QC purposes.
2. Re-populate the table with a list of files that exist in a network folder (the folder that contains all the construction drawings).
3. I can use that table in ArcGIS Pro to check for files that are missing in the GIS data/records retrieval system.

I could look into getting the Python script and share it with you if you want.