Thursday, March 6, 2014

Finding Shell Metadata

Finding Shell Metadata

In my last post I talked about Shell32 in Windows and gave some background on how it works. In this post I want to talk about how we can leverage Shell in MetaDiver. With the latest MetaDiver you can choose to discover additional Metadata.I have also provided a basic example using Windows PowerShell.

If you need a refresher on Shell I have a short writeup here.

How MetaDiver finds Shell Metadata

In the latest MetaDiver all the items you expect to see are predefined while the items we don't care about are left out. This has a few benefits. It allows you to only see relevant fields and it also allows us to control the formatting and naming of the fields.

You may want to find Shell items that aren't listed. There are some Shell property items like free space and space used that we aren't grabbing. Also, if you have a shell plugin installed and want to find additional fields. The Shell plugin could be a 3rd party PDF or something you wrote. Differences in codecs for file formats like Videos, Audio and Images can alter FolderItem lists as well. We have tried to identify all of the relevant ones for you, however you may want to turn on discovery of new items and see if you find items. Lucky for you this is really easy to do in MetaDiver as you can see below.

To have MetaDiver find unknown items just check the box next to "Shell: Find and add unknown columns".

What is MetaDiver actually doing to find Shell items?
Once enabled it MetaDiver to go through every file in the path you selected and run Shell32 per file to look for undefined items. It goes through each file and enumerates every Shell FolderItem, matches against what is predefined in MetaDiver and if it is new it adds it.

The wheels in your head are now telling you this could be slow, and you're right. It's going to be much slower. That is why it is now optional.  If you are processing 2000 files and lets say each file has on average 50 but we are checking for 400 then 2000 x 400 is a fair amount of work at 800,000. I don't know of another tool that will use Shell in this way to dig out Metadata.

Using PowerShell to find Metadata

You can use PowerShell to enumerate property items with Shell32 as well with some limitations. I have had to write a lot of error handling in to MetaDiver to work smoothly with Shell,Com and ActiveX, so good luck with PowerShel or Perl.

Here is an example using Shell in PowerShell to enumerate Office MetaData.

What is happening in the PowerShell script:

Creating the instance of Shell:
 $obj = New-Object -ComObject Shell.Application

Enumerating the Shell folder item (key). This gives the field name such as (Create, Size, FileName, etc..):

Then getting the FolderItem value for the FileName:

Hypothetically lets say we know the index value for the FolderItem. Let choose Create Date and that it is 4. Then we could just do

Windows PowerShell also supports Remoting. This creates interactive sessions or connections to multiple remote machines at once. I have yet to try this in PowerShell but I'm intrigued at the possibilities for gathering vast metadata over an enterprise, especially for E-Discovery and Incident Response.


As you can see Shell is very interesting and there is a lot more that can be done beyond Shell FolderItem. Shell is used everywhere in Windows!

I hope you find this post useful and informative!

Wednesday, March 5, 2014

MetaDiver - What is Shell

I want to continue to explore some of the powerful options in MetaDiver 1.1.1.

What is Shell?

I'm going to back up a bit and talk a little about what I mean when I talk about Shell and why I talk about it so much.

Shell refers to the Windows Shell API functions. In windows this is the shell32.dll. The ability to access the API has been in Windows a while (at least Windows Vista) and it is used in Windows Explorer, File Properties, Office, just about everywhere.. You start to get a hint as to why i find it so interesting and useful. You can do a lot more than just read information about a file using Shell, you can create files and alter Metadata as well. We only care about reading and preserving however!

If a program like Office has a dll registered with Shell you can pull properties about that file. This works for Office Documents, Windows Shortcuts, URL's, Video (MP4, AVI, etc..) and MP3 files (ID3 Tags) and sometimes Adobe PDF's as well as others.

You can write your own shell handlers and plugins. These are what you see in the context menu when you right click on a file or directory in Explorer. Those are shell plugins! They are all talking via Com (Component Object Model). has user comments talking about Shell32.dll, all of the comments are basically correct.

I've used MetaDiver to discover as many Shell attributes (or as you see them columns) as possible about a file. I've found over 400 at this point I think. The number can vary by the version of windows you are running and the applications you have installed.

I'll have more on this in my next post.

I hope you find this post useful and informative!

MetaDiver - Generating File Hash Lists

In MetaDiver (1.1.1) you have more control over what it does.

Today's post is an example showing how you can easily generate a File Hash List.

File Hash List Example

For this example you select a source like you normally would. Then you go to the menu bar and select Options and then Processing Templates and select "File Hash List". What this does it set the check box options for you. You can do this on your own as well!

This what you see before as you select the template.

After you set the template you see the check boxes have changed! I used red arrows to show the options changed.

  • Shell and 3rd party libs are turned off.
  • Hashing is turned on. The template sets both MD5 and SHA1 hashing on by default.
  • Current output is ItemPath, FileName, FilePath, Extension, MD5,SHA1. I can easily add file mac times in the next version if someone needs it.
  • Output can be changed or altered if you like, just post your comments to the MetaDiver forum.
As you can see it's easy to generate a File Hash List in MetaDiver!