Use IDataset.Delete. I would imagine you can't delete feature datasets unless they are empty though.
IDataset pdataset; pdataset = (ESRI.ArcGIS.Geodatabase.IDataset)featureClass; if (pdataset.CanDelete() && pdataset != null) pdataset.Delete(); // Throw exception
Where is the feature class in use? If it's in your map document then you probably need to disconnect the layer from its data source (IDataLayer2.Disconnect) and delete it from the map.