I would be cautious. The bulk of your join should be fine, but I have experienced where there are multiple structures on a single parcel and sometimes they share an address and sometimes they are separate addresses. It sounds to me like your join may only take one structure into account. If a home were to have an unattached garage, would your join be a one to many? Think of apartment complexes and mobile home parks. In our case, the Parcel IDs were supposed to be unique, not recycled and specific to a single parcel polygon; however in some cases there were multipolygons that shared an ID - it was a single record but graphically showed multiple polygons, each with the same Parcel ID.
As long as your methodology for data clean up after your join is thorough, you should be fine.
Since you are at your beginning stages, another workflow process to consider might be ESRI's Address Data Management solutions; https://solutions.arcgis.com/local-government/help/address-management/
You might want to also consider a multipoint feature. So, this would be an environment where you would have a graphic point to represent the driveway entrance, another point for centroid of the structure and another one for the main entrance to the structure and one for the side entrance to the structure and one for the unattached garage; but would be a single record in your table. While this workflow sounds intense, it may make sense for a base for your feature as you work through issues in years to come.
