Skip to main content

Finding Unique Files

I've posted before about finding duplicate MP3 files. The other day, I found myself needing to do the opposite; this time with pictures. Pictures are a little easier to identify (at least in my case) because an MD5 over the entire content of the image is sufficient to identify images (MP3s require only hashing the non-ID3 portion of the file). Here's what happened:

After a small mishap with my photos, I needed to do a partial restore from backup (a nightly rsync -a --delete). I restored some files and then ran fdupes to remove any duplicates. However, since the mishap involved moving photos between folders, renaming some files, and deleting others, I wasn't sure if I had restored all the affected photos. To find out, I used a little shell foo:
$ find /pictures/ /backup/pictures/ -type f -exec md5sum {} \; > md5sums
$ sort md5sums | uniq --check-chars=32 --unique
This results in a list of files that only exist in either the primary or backup location. I expected to find some files that I had failed to restore properly. Instead, I was surprised to find some of my pictures were not backed up! Further investigation found that my backup drive was full and that my cron emails were being deposited in the spam folder...

The moral of this story is that finding unique files is just as useful as finding duplicates and that it can help you determine that both your restoration was successful and that your backup is complete.
Labels:

Popular posts from this blog

Python on Android

Note: This post is out of date. If you'd like to run Python on your Android device, please see my Android Scripting Environment project. Here's an early Christmas present for all those Python fanatics (self included) out there! With a lot of help from my friends (thanks Manuel and Thomas !) I managed to install Python 2.4.5 on my G1. It's still rough around the edges, but I think it's a good start. Klaus Reimer has a nice overview of how to cross-compile Python . My instructions borrow a lot from his. Download and build the Android source . These directions assume that you have installed the source to /android_src . Download and build the Python 2.4.5 source . These directions assume that you have installed the source to /python_src . Make copies of python and pgen for use later in the build process then clean up. $ cd /python_src $ cp python hostpython $ cp Parser/pgen Parser/hostpgen $ make distclean Apply the following patch to the Python source. diff -r -c -b P...
Read more

Email Injection

Not so long ago, I ran a wiki called SecurePHP. On that wiki, there was one particular article about email injection that received a lot of attention. Naturally, with all the attention came lots of spam. As a result, I disabled editing of the wiki and content stagnated. Still, the email injection article remained popular. About a year later, the server that hosted SecurePHP died and I never had a chance to hook it all back up. I saved the article though and I'm reposting it now. It may be a bit old (I've been away from PHP for a long time), and I didn't write all of it, so feel free to leave comments about needed updates and corrections. Though this article focuses on PHP, it provides a lot of general information regarding email injection attacks. The PHP mail() Function There are a lot of ways to send anonymous emails, some use it to mass mail, some use it to spoof identity, and some (a few) use it to send email anonymously. Usually a web mailform using the mail() funct...
Read more

Review of Fable III

Damon says : Fable III is the most disappointing sequel since The Kingdom of the Crystal Skull . Laura says : I was disappointed. The worst part was that the game was okay - maybe a 6/10 or 7/10 thanks to the Darkness Incarnate quest. It wasn't great, but it wasn't so poor that I felt I could legitimately hate it. The graphics are fun if cartoony, the sound is good, the gameplay is easy , the customization is almost non-existent, and the story is mediocre at best and boring at worst. Oh, and for some of the achievements you need an Xbox Live account, which annoys me to no end. The environment in Fable III is as rich as ever. Plenty of different regions, all with different climates, peopled by various citizens/denizens/enemies that change as you play. As always, the people of Albion are incredibly chatty, but since Lionhead seems to have supplemented the new stuff they recorded with all the random NPC comments from Fable II, there is enough variation to not driv...
Read more