One way I think that might work:
1. Call IProximityOperator::ReturnDistance to find the distance between the polyline and polygon.
2. Call ITopologicalOperator::Buffer with the distance as the input to create a polygon.
3. Find the touch point on one geometry to the boundary of the buffer polygon. This will be the nearest point of the geometry.
4. Using the same steps or call ReturnNearestPoint to find the nearest point on the other geometry.
I don't think that this code is ideal (see the comments for its weakness) but it seems to get the job done.
private void ClosestPointsOnTwoGeometries(IGeometry geometry1, IGeometry geometry2, out IPoint pointOnGeometry1ClosestToGeometry2, out IPoint pointOnGeoemtry2ClosestToGeometry1)
IProximityOperator proxOp1 = (IProximityOperator)geometry1;
IProximityOperator proxOp2 = (IProximityOperator)geometry2;
double distanceBetween = proxOp1.ReturnDistance(geometry2);
//Make the distance slightly greater than half the distance so that the two buffers will intersect
double distanceToBuffer = (distanceBetween / 2.0) + (distanceBetween / 1000000.0);
ITopologicalOperator buf1 = (ITopologicalOperator)((ITopologicalOperator)geometry1).Buffer(distanceToBuffer);
var buf2 = ((ITopologicalOperator)geometry2).Buffer(distanceToBuffer);
//Note, you would think that two polygons being buffered would result in a polygon, but it actually
//results in an error. Thus until I can figure out why that is, I'm returning 1 dimension
//(esriGeometryDimension.esriGeometry1Dimension) instead of 2. Hoping to resolve this, but this is probably
IPolyline intersectionLine = (IPolyline)buf1.Intersect(buf2, esriGeometryDimension.esriGeometry1Dimension);
//Take the average of the intersection line (which under most circumstances is very short)
IPoint middleOfIntersectionPoint = new PointClass();
IPoint fromPoint = intersectionLine.FromPoint;
IPoint toPoint = intersectionLine.ToPoint;
middleOfIntersectionPoint.PutCoords( ((fromPoint.X + toPoint.X) / 2.0), ((fromPoint.Y + toPoint.Y) / 2.0));
pointOnGeometry1ClosestToGeometry2 = proxOp1.ReturnNearestPoint(middleOfIntersectionPoint, esriSegmentExtension.esriNoExtension);
pointOnGeoemtry2ClosestToGeometry1 = proxOp2.ReturnNearestPoint(middleOfIntersectionPoint, esriSegmentExtension.esriNoExtension);
1) you can get Distance IProximityOperator:: ReturnDistance
2) buffer on polyline A with Distance from step1
3) test with IRelationalOperator:disjoint on every ISegment of polyline B with buffer from step2 -> create list of Segments that intersect (IRelationalOperator:disjoint = false)
4) buffer on polyline B with Distance from step1
5) test with IRelationalOperator:disjoint on every ISegment of polyline A with buffer from step4 -> create list of Segments that intersect (IRelationalOperator:disjoint = false)
6) Find distance absolute min of two loops (if frompoint and topoint return same distance you have paraller line)
MinDistance (Polyline (listSegmentsB).ReturnDistance(listSegmentA).FromPoint, Polyline (listSegmentsB).ReturnDistance(listSegmentA).ToPoint
MinDistance (Polyline (listSegmentsA).ReturnDistance(listSegmentB).FromPoint, Polyline (listSegmentsA).ReturnDistance(listSegmentB).ToPoint
7) using ReturnNearestPoint using Point from dist min of step6