Can Python be used to find duplicate datasets?

[GailMorrison]

Can Python be used to compare datasets?  We have 2 directories stuffed with datasets. I need to sort thru the directories to determine which dataset is a stand-alone vs. a duplicate in a 2nd directories. Any ideas on how to go about this?

[DanPatterson]

depends on what you are comparing, but it is part of the standard module in all versions of python

filecmp — File and Directory Comparisons — Python 3.8.3 documentation 

moved to Python‌

[JoeBorgione]

We have 2 directories stuffed with datasets.

What is your definition of dataset in this context?  Are these shape files?  Are they feature classes within geodatabases?  When you say duplicates do you mean the same name?

[DanPatterson · Accepted solution]

As Joe suggests, it could get real difficult if you have multiple data types and aren't familiar with coding (using arcpy.da.Walk) and several of the tools in this toolset

An overview of the Data Comparison toolset—Data Management toolbox | Documentation