Skip to main content

Finding Unique Files

I've posted before about finding duplicate MP3 files. The other day, I found myself needing to do the opposite; this time with pictures. Pictures are a little easier to identify (at least in my case) because an MD5 over the entire content of the image is sufficient to identify images (MP3s require only hashing the non-ID3 portion of the file). Here's what happened:

After a small mishap with my photos, I needed to do a partial restore from backup (a nightly rsync -a --delete). I restored some files and then ran fdupes to remove any duplicates. However, since the mishap involved moving photos between folders, renaming some files, and deleting others, I wasn't sure if I had restored all the affected photos. To find out, I used a little shell foo:
$ find /pictures/ /backup/pictures/ -type f -exec md5sum {} \; > md5sums
$ sort md5sums | uniq --check-chars=32 --unique
This results in a list of files that only exist in either the primary or backup location. I expected to find some files that I had failed to restore properly. Instead, I was surprised to find some of my pictures were not backed up! Further investigation found that my backup drive was full and that my cron emails were being deposited in the spam folder...

The moral of this story is that finding unique files is just as useful as finding duplicates and that it can help you determine that both your restoration was successful and that your backup is complete.

Comments

Popular posts from this blog

Android Recipes and Snippets

I've put together a small collection of Android recipes. For each of these recipes, this is an instance of Context (more specifically, Activity or Service) unless otherwise noted. Enjoy :)

Intents
One of the coolest things about Android is Intents. The two most common uses of Intents are starting an Activity (open an email, contact, etc.) and starting an Activity for a result (scan a barcode, take a picture to attach to an email, etc.). Intents are specified primarily using action strings and URIs. Here are some things you can do with the android.intent.action.VIEW action and startActivity().Intent intent = new Intent(Intent.ACTION_VIEW);
// Choose a value for uri from the following.
// Search Google Maps: geo:0,0?q=query
// Show contacts: content://contacts/people
// Show a URL: http://www.google.com
intent.setData(Uri.parse(uri));
intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
startActivity(intent);Other useful action/URI pairs include:Intent.ACTION_DIAL, tel://8675309Intent.ACTION_CALL…
Read more

Email Injection

Not so long ago, I ran a wiki called SecurePHP. On that wiki, there was one particular article about email injection that received a lot of attention. Naturally, with all the attention came lots of spam. As a result, I disabled editing of the wiki and content stagnated. Still, the email injection article remained popular. About a year later, the server that hosted SecurePHP died and I never had a chance to hook it all back up. I saved the article though and I'm reposting it now. It may be a bit old (I've been away from PHP for a long time), and I didn't write all of it, so feel free to leave comments about needed updates and corrections. Though this article focuses on PHP, it provides a lot of general information regarding email injection attacks.

The PHP mail() FunctionThere are a lot of ways to send anonymous emails, some use it to mass mail, some use it to spoof identity, and some (a few) use it to send email anonymously. Usually a web mailform using the mail() function …
Read more

XBee ZNet 2.5 Wireless Accelerometer

I managed to put together a wireless accelerometer the other night using my two new XBees, an Arduino XBee shield, an XBee Explorer USB, an ADXL330, and some Python. I struggled a bit with some of it, so here's what I learned:

First, a parts list.
XBee 2mW Series 2.5 Chip AntennaArduino XBee (with XBee Series 2.5 module)XBee Explorer USBADXL330I'm not sure exactly what the specs are on the XBee that comes with the Arduino shield. But, it is definitely a series 2.5.

The first thing to do is to configure and upgrade the firmware on your XBees. To do that, you'll need X-CTU (for the firmware upgrade at least, but it's also nice for configuration) which, unfortunately, is only available for Windows. But, it works fine from VMware. First up, the XBee we'll hook up to the computer to read incoming data from the accelerometer:
Plug one of the XBees into the Explorer (it's also possible to do this from the Arduino shield by shifting the two XBee/USB jumpers to USB and remo…
Read more