Skip to content

dataset_find_duplicates

Perceptual hash duplicate detection for image datasets.

Inputs

  • dataset_path
  • algorithm: phash, dhash, whash, avg
  • hamming_threshold
  • across_splits: when false, train/val/test splits are compared only within the same split

Outputs

  • duplicate groups
  • total duplicate count and percentage, counting only extra copies beyond each group's representative image
  • total image count and analysis duration

Examples

{"dataset_path":"~/data/mini_yolo"}
{"dataset_path":"~/data/retail","algorithm":"dhash","hamming_threshold":3}
{"dataset_path":"~/data/factory","across_splits":false}

Common Errors

  • dataset not found
  • empty dataset

dataset_inspect, dataset_validate