{"id":94,"date":"2009-04-10T15:04:00","date_gmt":"2009-04-10T15:04:00","guid":{"rendered":"http:\/\/www.roysac.com\/blog\/wp-admin\/2009\/04\/de-dupe-files-script-tool-for-windows-32bit\/"},"modified":"2014-03-26T05:03:25","modified_gmt":"2014-03-26T13:03:25","slug":"de-dupe-files-script-tool-for-windows-32bit","status":"publish","type":"post","link":"http:\/\/www.roysac.com\/blog\/2009\/04\/de-dupe-files-script-tool-for-windows-32bit\/","title":{"rendered":"De-Dupe Files Script Tool for Windows 32bit"},"content":{"rendered":"

You can skip the background story about the why, when and how I decided to write this little File De-Duping script<\/strong>, if you are not interested in it and jump right the section about how the script works, what it does, where to download it, how to install and de-install it and the source code of it as well. The tool is Freeware<\/strong>, but any donation (money<\/a> or goods<\/a>) and\/or simply a ???Thank You<\/em><\/strong>??? (e.g. via the comments section at the end of this post) are appreciated nevertheless. <\/p>\n

Important Node: I updated the script and this post, because of some bugs that I found and issues with the 3rd party tool ???touch.exe???. I had to remove it and come up with another solution for the problem that it solved. I also added some nice stuff, so it was more than just bug fixing :).<\/p>\n

The Background Story<\/h3>\n

Who did not have the problem yet to have hundreds of any type of files, text files, documents, spread sheets, images, videos and others in one directory with the a high chance that you have duplicates, identical files that only have a different file name. I often have to deal with duplicate images that I downloaded from the internet. <\/p>\n

I cannot always remember, if I downloaded a particular image already or not and go by the slogan, archive \/ save \/ back up first, sort later, because there will be nothing to sort, if you don???t save a copy and go back to the place where you found them at a later time, to learn that the stuff isn???t there anymore for any reason. Maybe even the whole web site is gone the way of the Dodo<\/a>. <\/p>\n

The original file names are often useless, either to generic, like image1.jpg, logo.gif<\/strong> or simply 1.jpg, 2.jpg<\/strong> etc. or they are long and cryptic without any meaning, like: 3104458219_cc0dfd3980_o_d.jpg<\/strong> or something like that. So you end up giving the files your own name and thus make it highly likely that you download the same file at a different time again and give it a different file name than you did the first time around. Voila, duplicate.<\/p>\n

I looked at a number of tools and options and found some that were decent, but all of them had always something (or didn???t have something) to make things that should be semi automatic to a manual time consuming ordeal. What I wanted is something that finds duplicates, does not delete them straight out or have me right then go over each dupe found to make a decision about it right away. I also wanted to be able to verify that the dupes found are really identical so it must be easy to know quickly which file is a duplicate of another file. <\/p>\n

Non of the solutions that I tried delivered on all those aspects so de-duping of files was painstaking and time consuming, if I did it or I simply would not do it at all and carry a bunch of dupe garbage around .<\/p>\n

The idea for this simple script of mine came when I did the sorting<\/a> and inventory of<\/a> the SAC art pack releases<\/a> back in December last year and January this year<\/a>. I uploaded files, like MODs<\/a><\/em><\/strong> that I converted to Mpeg-1 Audio Layer 3<\/a>, short MP3,<\/em><\/strong> and other files to my file sharing account at Mediafire.com. I learned that Mediafire.com<\/em><\/strong> has a de-duper function build into their system that prevents user from uploading the same file twice to their account. You can upload it a second time, but you will get the message that the file already exists and that your new upload was being deleted. I got that message and first thought that it was an error on Mediafire???s<\/em><\/strong> part. <\/p>\n

What dupes? There were no dupes, I thought. Yeah, there were some SID Adlib music<\/a><\/em><\/strong> files that had the same size and all, but there where like a dozen of them, all with a different name and even different prefix, indicating that the files were created by different musicians. Also the dates were different. Being close to send an email to Mediafire<\/em><\/strong> customer support, I decided to listen to the songs that could be potential duplicates to the file that was rejected (Mediafire didn???t tell me the file name or shared URL of the original file). And there it was, the same tune with a different file name and different file dates by the same artist. <\/p>\n

I was curious about how Mediafire<\/em><\/strong> noticed that the files are identical and did some research. They are using the MD5 Check Sum<\/em><\/strong> value of a file. Chances are astronomical that two files that are not byte by byte identical will have the same MD5<\/em><\/strong> check sum value. That???s a smart, fast and easy way to find dupes and the idea of writing a de-dupe script that does exactly what I want it to do was born.<\/p>\n

This is enough of a background story I believe. Let???s get busy with the Script itself.<\/p>\n

How the Script Works<\/h3>\n

The script detects duplicate files within a directory. Duplicate files are files that have the same MD5 Check Sum value.?? Two DIFFERENT<\/strong>\/NON IDENTICAL files having the same MD5 Check Sum<\/strong> is not impossible, but highly unlikely. This allows the script to detect duplicate files regardless of their file name or other characteristics, such as “date created” or “date modified”.<\/p>\n

The tool scans all files within a directory. It does not include<\/strong> files in sub directories<\/strong> of the processed folder. <\/p>\n

If a duplicate file is found, it will be renamed by by appending the original file name as prefix with an _ as separator, which is also used to replace the “.” that indicates the file extension of the original file name (other “.” in the file name itself remain). At the end of the file name is the string [DEDUPED] added.<\/p>\n

<\/a> <\/p>\n

Example<\/h3>\n

For Example aFile1.EXT<\/em><\/strong> and bFile2.EXT<\/em><\/strong> are identical. After the script was executed, one of the two files will remain as it is and the other one is being renamed. Which file will be considered the “original<\/em>” is determined by which file was found first. The script sorts the files by name first, before it de-dupes them.<\/p>\n

In this example bFile1.EXT<\/em><\/strong> would be considered the original and bFile2.EXT<\/em><\/strong> will be renamed to aFile1_EXT_bFile2[DEDUPED].EXT<\/em><\/strong>. This makes dupes appear right after the original, if you sort the directory by file name. To be able to filter the dupes to copy\/move them away or to delete them, use the copy, move or del command in MS DOS. For example “DEL *[DEDUPED].*<\/em><\/strong>” would delete all duplicate files found and renamed by the script.<\/p>\n

More New Stuff<\/h3>\n

The script creates two files by default in the processed directory:<\/p>\n