Using Viewshed and Observer Points for visibility analysis—Help | ArcGIS for Desktop or viewshed 2, you can convert you polyline to observer points to a maximum of 16 points as described in the help topic
An overview of the Visibility toolset—Help | ArcGIS for Desktop the whole help topics is pretty self-explanatory
I know that toolset, i am searching for a method to limit the visible areas of my route, other than testing it uncountable times for a variety of polyline combinations
If I understand correctly, you are trying to find a line route that would be the least visible in your extent. One question that arises is visible from where? Viewed from a specific location? Viewed from every location in your extent? How are you defining where the viewer is for this analysis?
Chris Donohue, GISP
The approach i have thought it is when, i am using the polyline as observer point, i will have the less visible areas. I cannot think of any other approach for that task. But with that approach i also have to test every polyline i create, and i am looking for a more efficient way.
An you are limited to 12 observer points as in the link, should you choose to use the points from the polyline. The calculation is very computationally intensive. You should be able to rule out tracks of land setting them to nodata prior to running a test, for locations on a west facing slope aren't going to be see anything on the otherside of the hill on the east facing side, hence if you polyline is on that side, there is no need to test for the first case.
I would use your known observation points (the places you don't want to see your path from), not your unknown proposed path, for the viewshed analysis. Find the visible areas, or most often visible, and then place your line. After you've invested the time to calculate all (or a sample of) visibility from observation points, you are set up to use the resulting rasters as a cost surface (i.e. calculate the least cost path, where cost = number of times visible).