File Hash Caching
Duplicate File Detective includes powerful file hash caching technology that makes subsequent comparisons against cached files much faster.
Hashing is by far the most expensive stage of a duplicate file search. Computing a content checksum means reading every candidate file from beginning to end, so performance is governed almost entirely by disk - and sometimes network - I/O. The file hash cache attacks that cost directly: before hashing a file, Duplicate File Detective checks whether it already holds a hash for it. If the cached entry carries the same size, creation date, and modification date as the file on disk, the stored hash is reused instead of being recomputed - and an expensive full read is avoided entirely.
The payoff is largest exactly where duplicate scanning hurts most: repeatedly scanning the same set of search paths, containing large, relatively static files. Photo libraries, video archives, virtual machine images, design assets, and departmental file shares are prime candidates. Re-running a duplicate file search over that kind of data with a warm cache can be dramatically faster than the first pass.
The file hash cache can even be persisted to disk so that multiple discrete sessions can benefit from it - close Duplicate File Detective, come back tomorrow, and the cache is still there. You control how large the cache is allowed to grow and the minimum file size worth caching (small files are cheap to re-hash, so they're skipped by default). When the cache fills, entries are pruned by a combination of age and file size, deliberately preferring to retain the large files whose hashes were most expensive to produce.
A live statistics panel reports how many entries the cache holds, the smallest, largest, and average entry sizes, how many cache hits the current session has scored, and roughly how much memory the cache is using - everything you need to tune the settings for your own data. And you can clear the cache outright at any time when you know its contents are no longer relevant. Note that the hash cache applies only to projects that compare files by content.