File Checksums
Duplicate File Detective can compare the contents of files by computing the file's content checksum. This checksum is a numerical representation of the file's contents derived through a series of mathematical computations - a process known as hashing.
When the file contents criterion is enabled in the Comparison Options window, you can choose from a range of hashing algorithms:
- CRC32 - Quick checksum (32 bits)
- ADLER32 - Quick checksum (32 bits)
- MD5 - Medium strength, slower checksum (128 bits)
- SHA1 - Stronger, slower checksum (160 bits)
- SHA256 - Even stronger, slower checksum (256 bits)
- SHA512 - Even stronger, slower checksum (512 bits)
- Perceptual - Matches visually identical image files (see below)
Generally speaking, the "stronger" the checksum / hashing method, the more likely it is that two files compared with the resulting checksum will be identical. Stronger hashing algorithms are also generally a bit slower than weaker ones.
Note that file content hashing algorithms such as MD5 and SHA are extremely unlikely to produce false positives (e.g. mistakenly identify two files as being identical to one another when they are actually different). Even the smallest differences in file contents will (with overwhelming probability) result in completely different hashes due to a cryptographic concept known as the avalanche effect. If you must be absolutely certain that two files are identical, use the byte-for-byte content match confirmation option.
Perceptual (Visual) Comparison
Unlike the checksum algorithms above, which match files whose bytes are identical, the Perceptual basis compares the decoded image content of picture files. It matches photos that look the same even when format conversion, recompression, or resizing has altered the underlying file data - cases where a conventional checksum would never match.
Notes on perceptual comparison:
- File size matching is disabled automatically when the perceptual basis is selected (visually identical images routinely differ in size).
- Matched photos may differ in quality or detail - review results carefully before deleting.
- Copies, not lookalikes. Perceptual comparison finds copies of the same picture. Images that have been cropped, retouched, watermarked, or genuinely re-oriented are treated as different pictures and are not grouped. Photos whose orientation is recorded in EXIF metadata are turned upright before comparison, so a copy that had that rotation applied to its pixels still matches the original.
- Images carrying too little visual detail to identify - solid colors, blank scans, lens-cap shots - are excluded from matching rather than grouped together.
- Applies to image files. JPEG, PNG, TIFF, BMP and GIF are supported out of the box; HEIC, AVIF, WebP and camera RAW formats require the corresponding Windows codec extension to be installed. The related image metadata criterion offers an alternative way to match photos by embedded metadata such as date taken and camera model.
Additional Content Options
- Byte-for-byte content match confirmation - Validates each candidate match at the binary level. Slower, but provides absolute certainty that matched files are identical.
- When hashing zip files, enumerate and hash the files they contain - Compares zip archives based upon the files inside them, rather than the raw archive bytes. Use the adjacent Set Password link to supply a password for encrypted archives.
Tip: Duplicate File Detective provides a File Hash Calculator feature that you can use to experiment with the computation of file checksums.
See also: