Small Update: aufs3 and progress

As I mentioned in my last post, I was playing with aufs3, which basically allows to eject the CD even when it is currently in use. This works fine in low memory mode and – due to other properties of aufs – actually uses less memory. This is actually quite nice.

The downside is that after ejecting the CD, no data can be read from it anymore, obviously. This makes ejecting the CD in low memory mode generally a bad idea, unless you don’t need it anymore, for example right before the computer is switched off. The trouble was, that regular – where you can eject th disk without negative side effects – did not work anymore. Luckily I found a fix.

Apart from that, progress was slow, since I took a week off, I still fixed a few minor bugs.

Eject and RAM usage

So I recently commited a patch to svn that claimed to allow “ejecting the CD in lowmem mode”, well… it didn’t. I just booted the wrong mode.

The good news is that I kept looking and found Aufs. I won’t go into detail on its capabilities, just the one I use. It basically allows to cache part of the CD image into the memory, so the CD can be ejected. if your system has enough memory, you will be able to continue using any program that you ran at least once prior to ejecting the disk (e.g. web browser, CD burner), but you won’t be able to run other programs, since they were not cached and are therefor not accessable. If your system has less than abou 256MB of memory, GuARD will simply crash, since not even all things to keep the system running can be cached.

This is actually worse than the docache option currently used, which stores everything in the RAM (requires about 500MB RAM). With this you can eject the CD and still use everything, no matter if you already started it or not. Unfortunately the docache options seems to be broken (will check it out later).

Nevertheless, you will always be able to eject the CD, so this works on shutdown/reboot as well. So this would actually be an upgrade. The RAM usage is also significantly lower than in regular mode (about 20%). I was even able to boot the amd64 version with 64MB of RAM, it was so slow that it was pretty much unusable, but it booted.

Until I looked into the docache problem, I won’t commit any changes, since systems with a single optical drive will most likely experience trouble when trying to burn CDs/DVDs. If I found a clever way around it, it may be possible that I don’t need to distinguish between lowmem and regular anymore.

Undelete from FAT file systems

Finding a tool to undelete data from FAT filesystems has proven quite difficult. This is weird because FAT has been around since the ’80s (FAT16) and is still used on many USB drives and digital cameras (FAT32).

I found one tool that seemed perfect: TestDisk, but it has no scriptable undelete feature. It features a text-based menu to select actions but supports far more that just undeleting, thus simply using it instead of a newly written script is out.

I ended up with the direct approach, I only tested this manually and this post is actually my recipe to implement it:

1. Get basic info about the filesystem

Before you can start undeleting files, I needed to know a bit about the filesystem, I ran the following command

fsck.vfat -v /dev/sdd

where /dev/sdd is obviously the device I want to check. The output will look something like this:

dosfsck 3.0.12 (29 Oct 2011)
dosfsck 3.0.12, 29 Oct 2011, FAT32, LFN
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "MSDOS5.0"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
      4096 bytes per cluster
        34 reserved sectors
First FAT starts at byte 17408 (sector 34)
         2 FATs, 32 bit entries
   7716352 bytes per FAT (= 15071 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 15450112 (sector 30176)
   1929028 data clusters (7901298688 bytes)
63 sectors/track, 255 heads
         0 hidden sectors
  15462400 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/sdd: 805 files, 1464916/1929028 clusters

The relevant data here is cluster size and start of the data area (highlighted above).

I need these values later, when I need to calculate the actual positition of a file in bytes (or clusters), so let’s give them a name:

bs=4096
start=15450112/$bs-1=3771

The -1 in the starting position is there, since the start of the data area is the first block we need to include when counting blocks, thus we want to skip those before.

2. Know your enemy (aka get a list of deleted files)

Neither have I found a way to let fsck display the deleted files, nor do I intend to write this function myself (at least for now), so I use TestDisk to list them (Luckily listing files is scriptable):

testdisk /log /cmd /dev/sdd advanced,list,recursive,fullpathname

This created a log with all file names and some additional info, lines starting with an “X” mark deleted files. So I only need the lines starting with “X”, also I can ignore directories since this would actually just create an empty directory. This can easily be done using grep:

grep "^X-\s*[0-9]\+\s-" testdisk.log

If you look at the log you see that the first entry is the position is clusters, while the second entry states the permissions. These permissions start with “d” if it’s a directory, otherwise it’s “-”. The “^” means the start of the line. What the expression does is it display all lines starting with X followed by zero or more empty spaces and at least one number, these number(s) are followed by an empty space and a “-”.

Let’s pick one of these lines as an example (I used my music player to test this, so it’s a music file):

X1860341 -rwxr-xr-x     0      0   6757812  3-Jan-2011 19:41 /MUSIC/Kreator/Live Kreation (disc 1)/Kreator - 07 - Phobia.ogg

Here the relevant info is (in order of appearance): starting cluster with leading “X”, the size in bytes and the file name. Let’s define some more values:

pos=186041+$start-1=1864111
size=6757812/$bs=1650 (rounded up)
fname="/MUSIC/Kreator/Live Kreation (disc 1)/Kreator - 07 - Phobia.ogg"

Again the -1 to include the first block. Alternatively I could have defined start with -2 instead of -1.

With this we have all info we need to undelete that file, now we just have to do it.

3. Restore it

To restore the file I use the dd command. It’s a useful tool that I use to create disk images as well.

The key to success is that you can tell dd to skip the first n blocks and then copy a specific number of blocks. Of course, you can tell dd how big these blocks should be.

We still need a target, let’s just say it’s an arbitrary directory /path. So let’s stitch together what we have and also create the correct directory structure for our file:

mkdir -p $path${fname%*/}
dd bs=$bs skip=$pos count=$size if=/dev/sdd of=$path$fname

This will create the directory /path//MUSIC/Kreator/Live Kreation (disc 1) and copy the file Kreator - 07 - Phobia.ogg into it.
We’re done, the file is undeleted.

So far so good. For the actual script I’ll have to add the usual error handling and wrap a loop around it to restore multiple files at once and above all do some more testing.

Interface and tools

Although it has been a little quiet, I’m still working on GuARD.

For v0.7 I’m working on improvements of the interface as well: I already posted the new “mounted filesystems” bar the last time. In addition you finally get a clock in the upper right corner, something which is not essentially but can be pretty annoying when it’s missing. I also plan to display some important system info on the desktop itself (e.g. location of Windows installations), though I’m not sure yet what info to include.

Of course the tools will get improvements, too. Apart from the usual bugfixes I already mentioned scalpel a few weeks ago. On top of that I found out that the filesystem check utility for FAT filesystems actually includes an undelete tool, so it will get a GUI tool as well, maybe I’ll merge ntfsundelete and “fatundelete” into a single undelete tool that checks the filesystem type by itself.

The backup will receive new features as well, such as creating and restoring compressed images using fsarchiver.

Development is rather slow right now due to limited free time, but I hope to get all this done by christmas.

Improved automounting

Up to now internal and removable drives were simply mounted without some kind of message. Now, you still don’t get a message but something even better: an additional bar on the left keeps track of all mounted media. You can see it in the picture below.