Seven Million Files on NTFS

I had occasion to look into performance problems on a Windows 2003 server the other day. The server is running IIS and generally runs along tickety-boo. Then someone started to delete some files from an NTFS volume and the server practically locked up. That's when I was called in.

I quickly discovered that the one volume had approximately seven million files files on it. This amount of files will pose probems: even if the application(s) using the volume can cope, explorer.exe won't fair so well.

I've never been particularly impressed with Explorer's performance when large file operations are required, wheter that's working with a small number of large files, or a large number of small files. In fairness though, deleting seven million files is bound to take some time. That said, it shouldn't appear to lock up the entire system.

Any component of an operating system has its limits. Seven million files appears to be beyond what explorer on W2k3 can cope with by default. The application was also generating long files names with the dfferences in the names at the end of the filenames which poses an overhead for 8.3 filename generation. It is of course possible to tweak NTFS performance using fsutil to turn off 8.3 filename generation, but I favour leaving things at their defaults unless there is a compelling reason to do otherwise.

Debug logging had been left enabled in the application and was generating the files. It was not required in normal operations so was turned off. A quick format of the volume blew the files away quicker than a traditional delete could ever do. Log file generation reverted to a more respectable level and normal operations were resumed.

If you want to know more about the innards of NTFS, here's some good starting points:

What is NTFS?
How NTFS Works
NTFS Documentation
NTFS.com
Linux NTFS Wiki

B