topic Re: Points to line in Arcmap in Python Questions

Points to line in Arcmap

akshayloya — Mon, 03 Apr 2017 15:46:59 GMT

Hi,

I have a set of points (as shown in the picture). I want to join them and assign unique id to them. How this can be achieved using python or Model Builder on Arcmap 10.5?

And join the like the below picture :

Thanks in Advance.

Re: Points to line in Arcmap

DanPatterson_Retired — Mon, 03 Apr 2017 17:15:31 GMT

You could... but how many do you have? 10's, 100's 1000's? and is this a one-off? because there are a variety of clustering point data and the spatial statistics toolset has some, but implementations in python/scipy/etc can be used as well depending on the complexity of the problem. I guess you would have to weigh the time to do this manually versus the time to implemented in an automated fashion.

Re: Points to line in Arcmap

AbdullahAnter — Mon, 03 Apr 2017 17:44:40 GMT

I 'd prefer to read this thread Create a line from points using attributes , I think you will get good answer.

Re: Points to line in Arcmap

XanderBakker — Mon, 03 Apr 2017 18:08:45 GMT

May we assume that the points do not yet contain an identifier (like "CS-1")? does the process has to figure out which points to combine into a single line? If so, that would be the first challenge. The second challenge is to define in which order the points need to be connected. For the first challenge it would help to know what licence level of ArcMap or Pro you have (Advanced would allow the usage of tools like Near). Are you willing to share (a part of) your data?

Re: Points to line in Arcmap

akshayloya — Tue, 04 Apr 2017 07:03:07 GMT

Yes you're correct. How can I attach my data here? Couldn't find any option.

Re: Points to line in Arcmap

akshayloya — Tue, 04 Apr 2017 07:12:15 GMT

Sorry I didn't get it.

Re: Points to line in Arcmap

akshayloya — Tue, 04 Apr 2017 07:14:36 GMT

I've shared it. PFA.

Re: Points to line in Arcmap

XanderBakker — Tue, 04 Apr 2017 12:30:56 GMT

I notice that the only attribute the points have, is a text field with a numeric value (probably depth).There is no "line identifier" and one should apply a distance calculation to define which points are to be combined into a line. In order to write a process that will create the lines, there are a number of challenges that you will face and some data quality issues that I noticed:

The red circle top center shows two points that have very different values. The white point probably has a wrong value. In the lower part the ellipse shows a larger distance (up to 150 meters) between the points that would probably need to be connected or are part of the same line.

At the same time the profile lines sometimes are close together which will make it harder to distinguish the lines and assign points to the correct line.

... and in some case the lines cross:

This will complicate the process.

Now the order of the points (FID) seems to follow the order in which the data was collected and that will help in defining the lines:

We can do something with direction and distance between consecutive points to define the lines.

Re: Points to line in Arcmap

DanPatterson_Retired — Tue, 04 Apr 2017 13:19:02 GMT

Xander... keep on the inter-point direction idea, it should at least enable you to figure out the major groupings. For each grouping (maybe not all), a translation and rotation about the X or Y should facilitate point sorting, hence connection.

Re: Points to line in Arcmap

XanderBakker — Sun, 12 Dec 2021 02:33:41 GMT

I just played with just using the distance (not the direction) to define where to start a new line, and the results are not too bad I guess. See code below:

import arcpy
import os
arcpy.env.overwriteOutput = True

fc = r'C:\Esri\GeoNet\GIS_RIVER\23_elevation.shp'
fld_memo = 'TxtMemo'
fc_out = r'C:\Esri\GeoNet\GIS_RIVER\3D_lines_v06.shp'
fld_id = 'LineID'
fld_zmin = 'Z_min'
fld_zmax = 'Z_max'
id_format = 'CS-{}'
tolerance = 150

flds = ('OID@', 'SHAPE@', fld_memo)
lst_data = [[r[0], r[1], float(r[2])] for r in arcpy.da.SearchCursor(fc, flds)]

# create output featureclass
sr = arcpy.Describe(fc).spatialReference
ws, fc_name = os.path.split(fc_out)
arcpy.CreateFeatureclass_management(ws, fc_name, "POLYLINE", None, None, "ENABLED", sr)
arcpy.AddField_management(fc_out, fld_id, "TEXT", None, None, 25)
arcpy.AddField_management(fc_out, fld_zmin, "DOUBLE")
arcpy.AddField_management(fc_out, fld_zmax, "DOUBLE")


# insert cursor
flds = ('SHAPE@', fld_id, fld_zmin, fld_zmax)
cnt_id = 0
with arcpy.da.InsertCursor(fc_out, flds) as curs:
    lst_lines = []
    lst_pnts = []
    for i in range(1, len(lst_data)):
        print("i:{}".format(i))
        lst_cur = lst_data[i]
        lst_prev = lst_data[i-1]
        pntg_cur = lst_cur[1]
        pntg_prev = lst_prev[1]
        dist = pntg_prev.distanceTo(pntg_cur)

        if dist <= tolerance:
            # add to existing list
            pnt = pntg_cur.firstPoint
            z = lst_cur[2]
            pnt.Z = z
            lst_pnts.append(pnt)
        else:
            print(" - create and store line and add point to new list...")
            # create and store line and add point to new list
            if len(lst_pnts) > 1:
                polyline = arcpy.Polyline(arcpy.Array(lst_pnts), sr, True, False)
                zmin = polyline.extent.ZMin
                zmax = polyline.extent.ZMax

                # lst_lines.append(polyline)
                cnt_id += 1
                line_id = id_format.format(cnt_id)
                curs.insertRow((polyline, line_id, zmin, zmax, ))
            else:
                # not enough points to create line
                pass

            pnt = pntg_cur.firstPoint
            z = lst_cur[2]
            pnt.Z = z
            lst_pnts = [pnt]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The resulting 3D polylines also contain Z min and Z max values. The order of the ID's will follow the order of the points, not necessarily from the north to the south.

I will attach the resulting shape file for you to have a look and inspect if this comes anything near to what you are looking for.

Re: Points to line in Arcmap

DanPatterson_Retired — Sun, 12 Dec 2021 02:33:44 GMT

Good... got it..

For your first line... just want to confirm some results..

I just took the first line you produced in the array (a), did a sliding distance to confirm relative Interpoint spacing and determine the length.

The points are obviously are in order as expected, however, if they weren't but could be assumed to form a line, you can use the einsum distance calculation to calculate a rolling distance, then resort and reorder the input points.

The other simplest way is once the point array is pulled, you do a lexicographic sort, then do the rolling distance... same thing.

I will explore some more, but I like your approach to attain the groupings.

fc = r"c:\data\xander\3D_lines_v06.shp"
a = arcpy.da.FeatureClassToNumPyArray(fc, "*", explode_to_points=True)

a
Out[4]: 
array([(0, [3149179.089548039, 915975.5614984678], 0, 'CS-1', -34.0, -11.0),
       (0, [3149246.2167146187, 915900.8027559514], 0, 'CS-1', -34.0, -11.0),
       (0, [3149309.089548039, 915823.5614984678], 0, 'CS-1', -34.0, -11.0),
       ...,
       (317, [3161297.4812808945, 700683.7889148083], 0, 'CS-318', -75.0, 14.1),
       (317, [3161211.598822891, 700631.6329313102], 0, 'CS-318', -75.0, 14.1),
       (317, [3161127.88806113, 700576.5761440248], 0, 'CS-318', -75.0, 14.1)], 
      dtype=[('FID', '<i4'), ('Shape', '<f8', (2,)), ('Id', '<i4'), ('LineID', '<U25'),
       ('Z_min', '<f8'), ('Z_max', '<f8')])

a0 = a[a['LineID'] == 'CS-1']
xy = a0['Shape']

d = art.e_dist(xy[:-1], xy[1:])

print(d)
[[  100.47   200.01   300.53 ...,  1099.95  1201.24  1299.93]
 [    0.      99.6    200.11 ...,   999.48  1100.77  1199.46]
 [   99.6      0.     101.27 ...,   899.98  1001.33  1099.95]
 ..., 
 [  899.75   800.24   699.93 ...,    99.76   201.43   299.71]
 [  999.48   899.98   799.62 ...,     0.     101.85   200.01]
 [ 1100.77  1001.33   900.81 ...,   101.85     0.      99.85]]
lgn = art.e_leng(xy)

lgn
Out[18]: 1304.1020133011878‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Points to line in Arcmap

akshayloya — Tue, 04 Apr 2017 15:06:57 GMT

Amazing. Would try this.

Re: Points to line in Arcmap

akshayloya — Tue, 04 Apr 2017 15:07:24 GMT

You're amazing. Thank you

Re: Points to line in Arcmap

XanderBakker — Tue, 04 Apr 2017 15:50:20 GMT

The logic I have used, will probably result in points that will not be included in any line and currently there is no mechanism included, to register those points to investigate what is going on for those cases. This logic could be included on line 59 to store those points in a separate featureclass for quality control.

Re: Points to line in Arcmap

DanPatterson_Retired — Sun, 12 Dec 2021 02:33:47 GMT

Xander... I have only begun looking at this, but since you mentioned that it appears that the points are in sequence, I exploded your polyline file to get at the points and then began to try to simplify the distance and angle calculations to see if that data could be readily used to find the break points in the dataset.

Here is some food for thought...

from numpy.linalg import norm
import numpy as np

fc = r"c:\data\xander\3D_lines_v06.shp"
a = arcpy.da.FeatureClassToNumPyArray(fc, "*", explode_to_points=True)

# just take the first 2 polylines as an example and get the xy coordinates

a0 = a[a['FID'] <= 1]   # first two polylines
xy = a0['Shape']        # xy coordinates for above

# distance and its first order difference calculation

d = norm(xy[1:] - xy[:-1], axis=1)
d_diff = np.diff(d, n=1, axis=0)

# first order difference in coordinates for angle calculation

diff_1 = np.diff(xy, n=1, axis=0)  # first order difference in coordinates
ang = np.arctan2(diff_1[:,1], diff_1[:,0])
dang = np.rad2deg(ang)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The break points are readily detected, but the issue remains where lines cross or the end points are closer than the Interpoint spacing. The above could simplify the Interpoint distance calculations and offers the angle of incorporating angles in the analysis at the same time.

d   # interpoint distances in a sequential point list ... note 3127.751 ...
array([ 100.474,  99.595,  101.273,  100.026,  101.059,  99.358,  100.664,
        99.884,  99.705,  100.603,  99.765,  101.850,  99.847,  3127.751,
        102.403,  100.232,  101.770,  99.748,  99.731,  100.276,  100.402,
        99.814,  100.313,  100.225,  100.627,  103.806])

d_diff   # interpoint first order difference ... note the break point
array([-0.878,  1.678, -1.247,  1.032, -1.701,  1.306, -0.780, -0.179,  0.899,
       -0.838,  2.085, -2.003,  3027.905, -3025.348, -2.172,  1.539, -2.022,
       -0.017,  0.545,  0.127, -0.588,  0.499, -0.089,  0.403,  3.179])

# the angle of point pairs in the sequence (degrees)... note the transition between
# the lines at -150
dang 
array([-48.079, -50.855, -40.900, -52.817, -43.300, -49.899, -48.370, -52.989,
       -45.406, -52.674, -47.031, -42.119, -56.951, -150.087,  126.719,
        111.634,  105.997,  117.997,  111.050,  115.547,  109.158,  114.364,
        110.313,  114.775,  104.025,  98.100])‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The interesting thing to note is that sometimes the distances can be used solely as in this example, but it would be possible to obtain two sets of points using the distance and the angles, using the transition pair (dist, angle) to split the pairs. Once the set is split, then you can pull out the coordinates and connect the lines.

I have tested some with randomized points from the data set and found that the triplet of (coordinates, Interpoint distance and angles) works pretty well to partition the data and form the lines. It worked better than kmean or kmean2 etc, Other options that I have looked at include the following and the are alright, except they fail when line spacing is in the same order as point spacing, or if lines cross or almost meet at their end.

(see other options

# from scipy.cluster.vq import kmeans2, kmeans, vq

# from scipy.cluster.hierarchy import dendrogram, linkage

# from scipy.cluster.hierarchy import fcluster )

We should probably branch this off as a discussion, but with your code replicated in that branch and the sample data

Re: Points to line in Arcmap

akshayloya — Fri, 07 Apr 2017 11:15:53 GMT

Hi, any idea on how I can join the last points to the polygon (So that it touches the river bank). Secondly now I've elevation point details and width of the cross section line, how can I calculate area under each cross section.

Re: Points to line in Arcmap

XanderBakker — Fri, 07 Apr 2017 11:57:44 GMT

Hi Dan, That is indeed very cool. +1 for that! I just wished I had some more time available to really dig in...

Re: Points to line in Arcmap

Sai_AnandPeketi1 — Tue, 25 Apr 2017 22:01:53 GMT

Xander,

Is there any possibilities that we can join all the point because some point are falling out of the line. It cause problem while extend the line up to River bank.

You can see the above example that when we extend the line feature its not touch the boundary of the river banks What I means that, Can we do like creating a line feature by nearest geometry instead of FID.

Thanks for help.