You are on the right track. Here is how I did it. Not perfect.
Create a large txt (or note) field outside the repeat. i did text so users can copy and paste it.
Add a calc joining the fields from the repeat join(", ", ${SpeciesList})
Set a constraint on this join field (not the field) - the trick here is to remove the last entry since it is the one you are trying to add currently.
So like this not(contains(substr(${AllPlantsCheck}, 0, string-length(${AllPlantsCheck}) - 3), ${SpeciesList}))
--If you do not do this it immediately adds to the list then complains it is already there.
--(note my plant codes are 4 chars. so if i remove the last 3 chars it will not find what I just added)
It can be funky if you go backwards and then change something but it still seems to work ok.
I even built on exceptions to the rule. If it contains a XXXXX generic code then the repeat is allowed.
Like contains(${SpeciesList}, "XXXX") or not(contains(substr(${AllPlantsCheck}, 0, string-length(${AllPlantsCheck}) - 3), ${SpeciesList}))
Now it sounds like you may want to compare across multiple repeats? If so make a join() field for each repeat then make a master field that joins all of the joins together. Then check against that. (Confused yet )
Hope that helps. Works slick for me.