I maintain a library using the ESRI Flex API and I know that error all too well.
Granted, this error will ONLY occur in Flex/FlashBuilder, I can successfully link my library files in IntelliJ and FDT withouth issue. Go figure.
Anyway, in the Properties for your main project, add your library file (Add Project), make sure Link Type is RSL (runtime shared library).
FlashBuilder will compile and link to the library swf.
Now, here is the kicker. If you make any changes to your library file at all that force it to rebuild, you will need to remove the library project from your Main project and re-add it following the steps above.
You'll need to do this every single time you update your library project.
It's a horrible workaround, but I do it on a daily basis. I am really not a fan of Flashbuilder at all.