FileTreeView – a SequioaView-like Application

This post has moved to http://www.scottleckie.com/2009/06/filetreeview-%e2%80%93-a-sequioaview-like-application/

I've long been a huge fan of the SequoiaView application released by Technische Universiteit Eindhoven, which displays disk utilization in a beautiful squarified cushion treemap format.
This was released in 2002 and does a great job of showing exactly what's eating the space on your disk, but it has one major drawback; if you point it at a 2TB volume with a million files, but you only want to see what's taking the space in a small corner of the disk, it reads the entire volume before displaying what you originally asked it to.
So, I decided to write a C# alternative to SequoiaView, partly to help us find the big files in specific folders really quickly, and partly just as an exercise.

You can download the setup from here. Do be aware that this requires the dotNet Framework 2.0 or higher and, while the setup is supposed to go fetch this if required, it does seem a little flakey. In other words, it’s probably safer to ensure you have the .Net Framework 2.0 or higher installed before starting.
You also need to install the Microsoft Data Visualization Components, which are available here.

Source

I’m going to be uploading the source, plus a background article, to CodeProject shortly. I’ll come back and update this link then.

Using FileTreeView

Refer to the following picture for each of the components within FileTreeView;

1. Enter the path or folder that you want to display here, or press the "..." button to browse, then press "Go"
2. The number of folders and files found so far will be displayed here. You can press "Cancel" if you get bored and it will display what it has so far
3. This rather nifty set of colours (actually seven little label controls) allows you to set the node colours
4. By default, the names of the files and folders will be displayed at all depths (which means from the top folder down to the deepest folder). Use this slider to de-clutter the display by displaying labels only to the depth of your choosing
5. This is the TreeMap control, that displays all discovered folders and their contents, grouped by the relative size of each folder
6. Each folder or file is a node. You can hover over a node to see its details, double-click to drill down or right-click for more options
7. Right clicking a node allows you to display it in Windows Explorer, open it, or zoom in and out of the tree structure

Licence

The FileTreeView application is open source, under the CodeProject Open License. Note that the Microsoft Data Visualization Components are free to use for non-commercial use only. See here for specific licence terms.

Help

If you have any suggestions or issues, please post a comment to this article.

Microsoft Data Visualization Components on Windows 7

this post has moved to http://www.scottleckie.com/2009/05/microsoft-data-visualization-components-on-windows-7/

I’ve been playing around with the Data Visualization Components recently (looking to incorporate the TreeMap control with a SequioaView-a-like disk space analyser) but ran into problems getting the toolkit installed on Windows 7 RC1. Running the setup from the official page at http://research.microsoft.com/en-us/downloads/dda33e92-f0e8-4961-baaa-98160a006c27/default.aspx gets stuck looking for .Net Framework 1.1.4322;

Of course, being Windows 7, .Net 3.5 is already installed which should include .Net 1.1 but it looks like the Components installer is hopelessly confused. I couldn’t get the installer to believe we had something better than 1.1 already installed and I didn’t want to try and hack .Net 1.1 on top of Windows 7.
So, all I really needed were the “bunch of files” that come in the Component setup so I installed the package on an XP machine and copied them across to a folder on Windows 7, then referenced the DLL by hand.
I doubt there’s any .Net developers out there, running the RC of Windows 7, who doesn’t have a spare XP machine lying around too(!) but, just in case there is, I’ve placed the Zip here for your convenience. The licence states that it’s OK to distribute for non-commercial use and doesn’t say it needs to be in the original MSI format, so I don’t see a legal problem with this.

Auditing access to a file on Vista

This post has been moved to http://www.scottleckie.com/2009/04/auditing-access-to-a-file-on-vista/

Every so often, when I open Outlook, it tells me that my PST file was not closed properly and it’s being checked for errors. It seems from a spot of googling that this isn’t an Outlook problem as such, more that another app opened my PST and did not close it properly.
Task Manager or Anvir Task Manager (excellent app) shows that, whoever the culprit is, it’s not running the next time I start Outlook. My suspicions are firmly on Skype, especially as there are a ton of postings on it keeping the PST open, but I’ve checked the obvious settings and they don’t seem to be the culprit.
Right, I thought, let’s turn on file auditing and I can see which process accesses the PST file… Wow! What a faff this is, in Vista. Prior to Vista, auditing was really easy but now things have got way more “powerful” (according to Microsoft) or “difficult” (according to me). What is particularly galling is that you can even go and enable access audits on a specific file, and Vista won’t give you a hint that File System auditing isn’t even enabled! So, after a bit of digging around, this is how you audit access to a file;

Turn on File System auditing

You need to use a horrid app called AuditPol.Exe – the semantics of this are a bit too esoteric for me, but here’s what you need to enable file system auditing;

1: D:\Windows\system32> auditpol /set /subcategory:"file system" /success:enable /failure:enable

Note that the whole “auditpol” command is on one line.

Enable Auditing for a file

Now, go and right click the file you want to be audited, right click and go to properties. Go to the “Security” tab, and then click on “Advanced”. Now that you are in the Advanced dialogue go to the second tab, which should be called “Auditing”. You may have a button marked “Continue” if you are still using UAC – if so, click this and accept the next dialogue;





Finally, you get the auditing dialogue. Clock on “Add” and enable the auditing that you’re looking for. If you just want to know what / who opens the file, you just need “List folder / read data” auditing.





Very important; the “Name” entry defaults to your user ID, which means that it will only audit access by you. If you want to record access by anyone (and remember services may be running under the SYSTEM or other account) then you need to click the “Change” button and set it to audit access by “Everyone”.

Checking the event log for audit records

Last stage; you can check the Windows Event Log (via the “Administrative Tools” group or by running EventVwr.exe) to see audit results. Check the Windows Logs / Security log for event ID 4663, or the Task Category of “File System”;

Remember to disable when you fixed your problem

Best advice suggests that, once you’ve got to the bottom of your file access problems, you should disable auditing. To do this, follow these steps in reverse order; remove the audit options from the file, then disable auditing with AuditPol.ex;

1: D:\Windows\system32> auditpol /set /subcategory:"file system" /success:disable /failure:disable

Anti Virus...

Greetings from Hamburg, Germany...
Over here on business; meeting tomorrow and flying back on Friday.

The good news? I flew from Edinburgh (EDI) to Heathrow (LHR) T5 and from there to Hamburg but, because I still have a BA gold card I got to visit the "First Lounge" for the (ahem) first time. All in all, it was a similar experience to the old T4 First experience but they have a separate area for the champagne bar. And so, this morning, I was relaxing with the Times (GoBama being very appropriate) and some canadian bacon, some scrambled eggs and a glass of Krug.

Where was I? Oh yeah; anti virus products...
A while ago (two years and five days to be exact) I bought a Kaspkersky Protection for Three PCs bundle from PC World. I had my main PC plus the ancillaries so I would easily use the three licences, right?
So, here we are, two years on and my Kaspersky rollout is precisely one. In the meantime, I moved my backend services to Linux, and I installed Clam and AVG and all was good...

Then, yesterday, I got a message that my main PCs AV (Kaspersky) was out of date and that updates were no longer possible.
WHAT? No Warning?
What about my three licences? OK, this is my fault; if I never installed it anywhere else, what do I expect?

I'll tell you what I expect; goddarn warning! I don't expect to go from 100% one day to 0% the next...

So, here's where I'm at...

Clamshell is doing great at protecting my mail server.
AVG does fine protecting my "throwaway" laptops
And I just installed Avira Free on the workstations I care about.

My question to you is this; why should I install anything above Avira for my main workstations?
(Kaspersky fans need not apply)

Backing up VMs

A lot of my production systems (web, email, dev) are now running on VMs.

I've been worried about my belts and braces approach to these VMs and I have now combined a mix of scripting (DOS) and syncing (SyncToy 2.0) to ensure that I have local copies of the VMs that I can restore at a few moments notice.

I'm still working on a few points, most notably the unpredictable nature of PLink, and will post a how to shortly...

Finding duplicate files

At work we have a problem. We've got an expensive Storage Area Network (SAN) which isn't expensive purely because of the servers and disks, but also because we back up a lot of that data every single night.
Now, imagine that (like us) you are an ISV. So, we have a nightly build which results in a code base of around 900MB. Some of these make it to testing. Some of these make it to release candidate and some of these actually become a release.
Our main file server has it's storage on the SAN so even taking into account the fact that most releases never hit the customers, we have issues with duplicate files on the server /SAN. Consider an overnight build...

We end up with a build (21390) in the build area. Good for the Devs....
Testers then copy it to their own area...
It gets released so we maintain a "gold" copy of the released code.
The Service Desk have their own library of released code.

With a typical release, we end up with at least four copies of the 900MB code; auto-build, test, gold-release, support-library.

SAN is one of the most expensive resources that I need to account for, so I looked around for a utility that could help me. I was hampered on a number of fronts;

1. Vitally, most of the dup utilities did not expect to run against shares with ~100,000 folders and millions of files
2. They seemed to be aimed at people who wanted to delete dups. I don't want to do that; I want to investigate failures in process and fix that
3. Their view of a duplicate did not match mine, which is; Suffix, Size, MD5 hash

So, I decided to write my own!

The app searches a folder or share, and then breaks down files into the following sets;

• All large files (as defined by you, when you run the program)
• Sets of files that have the same suffix, and are the same length
• Sets of files that have the same suffix, length and md5 signature

These sets are then displayed in to the user, where you have the option to save the details to a CSV file.

Using this, we have been able to discover duplicate files (obviously) but also failures in process. For example, when files are sent to us as part of a support incident, we were copying them to several different folders. We were also storing builds in one folder and then testing by copying to another. This app identified where people were not following the correct process and were copying files to secondary folders.

You can download the app from here. I'll be posting the source to CodeProject shortly.

S.