Chris,
As a 9.3 user, you can upload and download files, like MXDs and LYRs. These files work as they do on your desktop. The person who uses them must have access to the data they reference. However, it can be a way to share these files with others who you know have access to the data. You can also add references to your services running on ArcGIS Server (if you have one).
Yes, you do need 9.3.1 to create layer packages. Layer packages extend the above in that the data is included with the package so if someone downloads it, they'll get the data as well. Layer packages can only be uploaded and downloaded. There is no ability to "web enable" them at this time. Thus, layer packages don't take the place of ArcGIS Server. You still need ArcGIS Server to create map services that you want to view on the web. We are working on "web enabling" layer packages so they can be displayed as a layer on the web. When we release this feature, it will likely be a premium feature (for fee) of ArcGIS.com.
Hope this helps,
Mike