Reliable duplicate image finder?

[TwoToTango]

Ok, I'm sitting here with about 45 GB of
images on my hard drive. You know,
landscapes, portraits of inspiring people,
that sort of thing. Definitely no porn, oh
no no no.

About half of it is nicely sorted into thematic
and topical galleries. About half is a complete
hodgepodge.

I've used some shareware/freeware duplicate
image detectors in the past, but none of them
have been able to cope with this many thousands
of files. I'm talking about the programs that
don't just look for exact matches, but detect
duplicates that have been cropped, resized,
resampled, and/or captioned.

Does anyone know of a dup finder that can
handle gigabytes at a time? Commercial,
freeware, shareware, anything.

Recommendations appreciated.

[Albert]

I was thinking along the lines of a perl script comparing MD5 hashes until I read your requirement that it should be able to cope with cropped and resized images...

This might be too much to ask.

When you crop or resize an image you lose the information that makes such a comparison possible. If there was software that did this it would surely be based on some kind of heuristic and would only be an educated guess at best.

To understand how difficult this is, try to imagine how you would say FOR CERTAIN that two pictures were "the same" even though one had been cropped or resized... they might look roughly the same, but what about the palette, the brightness adjustment, colour filters, skewing, etc etc etc...

Not to mention site branding...

[Cynthetiq]

I use ACDsee 4.0 and 5.0 has a dupe image finder.. works great!!!

[TwoToTango]

Originally posted by Albert
If there was software that did this it would surely be based on some kind of heuristic and would only be an educated guess at best.

That's right. As I said, I've tried several
such programs (sorry, don't have pointers
here at work, might be able to dig up a
name or two when I go home). I believe
they operate by taking color samples at
various positions relative to the size of the
image, and calculate some measure of
closeness (perhaps weighting toward the
center?) Usually there's some way to tune
the sensitivity.

They actually work pretty well. Occasionally
you'll get a false positive if there are two
shots of the same chick in the same pose --
er, that is, of the same beautiful natural
vista.

The only trouble is that the ones I've tried
start to choke when the file count hits a few
thousand. I'm hoping there's one that doesn't.

[TwoToTango]

Originally posted by Cynthetiq
I use ACDsee 4.0 and 5.0 has a dupe image finder.. works great!!!

I bought a copy of 3.1 long ago. Guess I
should look at an upgrade.

You've tried it on a large collection?

[Cynthetiq]

Originally posted by TwoToTango
I bought a copy of 3.1 long ago. Guess I
should look at an upgrade.

You've tried it on a large collection?

file count of total files or total dupes? i was able make about 1,500 dupes from about 5,000 pictures...

[TwoToTango]

Looks like about 125,000 files all told. As
I mentioned, maybe half that is already
hand-groomed for dups and categorized.

Still, even if I can only do five or ten thousand
at a pop, that's progress.

[yotta]

gqview can detect visualy simmalar images. However, it is a linux program.

[txd]

ImageDupe does quite a good job. It can take quite a while to search through thousands of images so I broke it down into sections to scan (I had a pretty good idea where the duplicates were) and once all sections were complete I ran it on all files to double check

[TwoToTango]

txd, yotta, thanks for the pointers. I'll
check those out.

Cynthetiq, I looked at the user manual for
the latest ACD See and it looks like their
dup detection is for identical files only, not
for visual similarity. True?

[Cynthetiq]

Originally posted by TwoToTango
txd, yotta, thanks for the pointers. I'll
check those out.

Cynthetiq, I looked at the user manual for
the latest ACD See and it looks like their
dup detection is for identical files only, not
for visual similarity. True?

visual similarity? meaning same picture different file name? it picks up each file and allows side by side comparison to make sure that they are in fact the same file and then you can decide which to dispose...

[thebigk]

I needed to clean up my picture collection and searched to find this thread...glad to see I'm not the only one with some catching up to do!

ACDsee crashed on my 35,000+ collection...I Googled around and found Dup Detector, a freeware utility that finds dups and similar files - it successfully detected, for example, that a picture with a watermark was actually the same as a picture without one.

[TwoToTango]

Originally posted by thebigk
...I Googled around and found Dup Detector...

Thanks for resurecting the thread and for
posting this tip. I just grabbed it and am
giving it a try. I pointed it at one of my
facials directories -- 17,857 images -- and
it's chugging away. We'll see what happens.

Facials? I meant historic monuments. Sorry.

[TwoToTango]

Originally posted by TwoToTango
...it's chugging away. We'll see what happens....

Choked. Pfui.

[charliex]

have you tried imatch 3 ?
http://www.photools.com/

[TwoToTango]

Originally posted by charliex
have you tried imatch 3 ?
http://www.photools.com/

I have now, thanks to you, and I think I
probably owe you a beer.

I ran the visual similarity search on about
4,700 images. The results were overly
generous, but there are about half a dozen
algorithm tweaks that I haven't tried playing
with yet.

It did a good job finding cropped, resized,
and color-adjusted similarities; it did a bad
job identifying images with captions or logos
added/removed.

I also ran the exact duplicate search on
17,000+ images, and the program, just
like Tiny Tim on the last page of A Christmas
Carol, did NOT die.

A tip of the hat to charliex!