Some hate it. Some love it. So, yes, please make it an option.
Also, it would be nice if it had the ability to pan continuously for a long distance on release (like Google Earth), or to glide to a halt in a stickier fashion (like Bing Maps and Google Maps). Both of these map apps are hugely successful, so there are enough people out there who enjoy glide/swipe (kinetic) panning. (BTW: if the user stops panning before releasing the mouse, map should not glide at all (as is done in Bing Maps, Google Maps, etc.)
Kinetic panning is particularly useful for large areas. Similar concept to free-spinning mouse wheels for zipping through long documents - less effort to move across large areas.
Also, rather than just a Boolean, a numeric factor could be set to determine glide time (0 = no glide effect).