Basically all a silverlight app requires is:
1) A html page that contains and loads the Silverlight Plugin.
2) A .xap file which is the Silverlight app (this is usual the output in the ClientBin folder of your website when you build it).
3) Optionally (but highly recommended!) the silverlight.js file which helps users install the plugin if they don't already have it.
Also as Ali mentions, the mimetype for the .xap file is important. As long as this is set, you can host your app on any webserver that can simply host html, javascript and .xap files (meaning there's no requirement on using IIS, ASP.NET, .NET etc). Of course if your website contains ASP.NET pages and/or services that the silverlight app communicates with, it will require an ASP.NET webserver like IIS.