Esri has their own installation folders for their pythons to make sure they are using the version of numpy, matplotlib etc that that their shipped and supported python tools (there are oodles of these in the Geostats toolbox) are expecting.
On Windows this is especially problematic because of .dll conflicts that can crop up because unlike unix, it's not all about the path; .dlls are registered with Windows - it can get very complicated and messy (and disastrous) hence the term ".dll hell."
A term you hear sometimes is "Python stack" for a python install with a known set of modules (at certain versions) that are packaged to play well together. Esri is currently building a Python stack with a set of popular third-party modules ready to go (and compatible with arcpy) that includes scipy, pandas, and netcdf, among others. They have released one for ArcGIS Pro 1.0 / Python 3.4 and one will be available for Desktop 10.3.1 / Python 2.7.x at user conference time. I really like this approach as it allows me to write a script tool against a standard Python distribution that Esri ships with their software, without asking the user to deal with Python software installs.
Enthought ($) and Anaconda (open-source) are python systems that are set up to allow easy install and update of third-party packages, even if you do not have admin access. Both of these (and python itself) are moving toward the support of something called "virtual environments" which are setups that allow you to install a stack of packages for a certain purpose, and maintain a separate stack of packages for another (for example, a stack compatible with ArcGIS 10.2 and one compatible with 9.3).
