Caching

Duplicate File Detective provides a file hash caching subsystem that can improve duplicate search execution performance in many scenarios.

When Duplicate File Detective is configured to compare file contents, it will generate content checksum values for files which require it. From a performance perspective, the creation of file checksums (e.g. "hashing") is the most expensive (e.g. the slowest) operation in the duplicate analysis process chain. File hashing requires that files being analyzed are read in full, resulting in performance that is primarily driven by disk (and sometimes network) I/O.

Hash caching can help mitigate the performance impact of file hashing operations for subsequent project runs by storing hash results in memory (and optionally on disk) for later re-use. Before a file is hashed during a duplicate search, Duplicate File Detective queries the cache to see if a hash value for the file already exists. If a cache entry already exists and has the same creation date, modification date, and size as the current file, its hash value can be re-used (as opposed to being recomputed). The resulting performance gain can be dramatic, particularly when the files in question are large.

Hash caching will be most valuable (e.g. provide the greatest performance gain) in scenarios where duplicate search paths are being re-used, and the files these search paths contain are relatively large. For example, if you are regularly searching a common set of user directories for duplicate files (by comparing file contents), the file hash cache can dramatically speed up subsequent searches, particularly when those files are semi-static (e.g. change infrequently).

For projects that do not use file content comparison, the hash cache is not used.

Cache Configuration

The Enable file hash caching option is the master switch for the caching subsystem. The remaining options are:

• Maximum number of hash cache entries - The hash cache is limited by size (a specific number of cache entries) so as to constrain its usage of host system memory. Larger values grant the cache more room to grow, while also increasing memory consumption (minimum 100, maximum 9,999,999; default 4,096).
• Minimum cache candidate file size (in KB) - Files smaller than this value will never have their hash values cached (default 50 KB). Small files are cheap to re-hash, so caching them provides little benefit.
• Persist cache on disk between sessions - When engaged, the cache will be saved to (and loaded from) disk upon program exit and start, respectively.

When the hash cache becomes full, it will remove (or prune) a percentage of existing entries in order to make room for new ones. Because the expense of file hashing is directly proportional to a file's size, the hash cache prefers larger files (the preference is proportional to the file size). Therefore, entries will be removed based upon a combination of age and corresponding file size.

Users can also clear the hash cache explicitly via the Clear Cache Now button. This might be useful in cases where you know that existing cache entries no longer have value (e.g. previous duplicate file search paths are very unlikely to be used again in the future).

Statistics

The statistics area of the hash cache shows a range of related metrics: the number of entries in the cache, smallest and largest entry sizes, average entry size, session cache hits, and approximate memory used.

Users that become familiar with how hash caching operates may find these metrics useful in fine-tuning available settings.