To what coordinate system was your base map generated?
What coordinate system are you using for the data in Survey 123?
Are differences generally in the same direction and distance for all points?
Coordinate systems need to match or you will have a tranformation error. These errors are typically in the same direction and distance.
Are you manually placing the point locations or using the internal GPS of a device? Many GPS units in mobile devises can have substantial errors dependent upon time of day and satellite positions. These errors will typically be more random in direction and distance. I have seen 10-15 m errors from iDevices.