Death to .DS_Store

December 24th, 2011
Spencer Nielsen Follow snielsen42 on Twitter

The “.DS_Store” file is an abomination and must be stopped. You know what I’m talking about. I regularly rant about how this annoying file gets in your way, dirties things up and just screws with your stuff in general. Today I decided to do something about it. Before we get to that, lets quickly review what it is and why it sucks.

What Is It?

The .DS_Store is a Finder metadata file created primarily by Mac OS X’s Finder.app. Because of the dot (“.”) prefix it is typically not visible in many file browsers and most Mac OS X users are probably not aware of it. It is regularly created when the Finder accesses filesystem directories. It contains directory information about icon locations, view options, silkscreen configuration and the like. The functionality that it provides is moderately useful, but becoming less and less relevant over time. In any case, a long time ago the horrible decision was made to store that Finder metadata in an actual file (.DS_Store) in the filesystem within the relevant directory. We have been paying for it ever since. Over time more and more metadata relating to files and the filesystem has been added to Mac OS X, but thankfully those have been stored in saner places (extended attributes, etc). For the time being though .DS_Store is still here with us and still causing trouble. What’s so harmful about the file you might ask?

Cross-OS Cleanliness

The .DS_Store file is only useful if you are accessing the filesystem with the Finder. What if you aren’t using the Finder though? What if you aren’t even using Mac OS X at all? At best the files then become wasted space that is ignored. At worst they are horrible clutter that dirties up your file browsing experience. It doesn’t take much online searching to find countless forum posts from Windows users asking what these files are or the innumerable removal scripts that Linux users have authored to clean them from their directories. On Mac OS X, if you insert a USB flash drive, mount a network volume or zip up a directory to give to a friend, more likely than not, .DS_Store files have hopped on for the ride. Whoever has to interact with those directories next that doesn’t use a Mac is going to have to deal with those in some way even if it just having to consciously ignore them. Eventually a power-user facing flag was added that would disable .DS_Store file creation on network volumes (“defaults write com.apple.desktopservices DSDontWriteNetworkStores true“) which was a good move. There are still however, plenty of reasons why Mac OS X users hate having them on their own systems.

Source Control Gunk

All the programmers in the room, raise your hand if you have ever added “.DS_Store” to your .gitignore file or equivalent source control ignore configuration? What’s that? That guy in the back who didn’t raise his hand? Oh, he was sleeping…there’s his hand. The contents of the .DS_Store file are always changing and so even if you did happen to check the .DS_Store files into your repository (furthering it’s unholy propagation) it is always going to be showing up as a changed file and gunking up your commits with changes and data that have absolutely nothing at all to do with your code. I have to wonder how many global man hours have been wasted in configuring source repos to ignore these files or cleaning them out of the repos after someone YET AGAIN added them in their latest commit.

Permissions Playtime

Because the .DS_Store file is so inconspicuous and usually innocuous, a lot of the time we simply forget it is there. Unfortunately when doing things like mass permission changes and the like, the .DS_Store file will unintentionally come along for the ride. Suppose you tighten up the permissions on a directory that has a .DS_Store in it. After a while you decide to go through an manually clean it or lower the permission levels on the contents again. You had better remember that you also raised the permissions on the .DS_Store file as well or you might be puzzled why you don’t have permission to delete the directory or why the Finder isn’t saving your directory options. It is no show stopper. Any competent command line monkey can figure it out and fix it. But again, the few minutes it takes to fix it should never have been necessary. Technology is supposed to save time, not destroy it.

So What Do We Do About It?

Those were just three examples that I thought of off the top of my head. Leave your own .DS_Store horror/annoyance stories in the comments and I will add any more that don’t generally fall under one of those three examples. So they are a time consuming annoyance, easy enough to whine about. But what to DO about it? The other day I again ranted on twitter about them and it was suggested that maybe someone could write a daemon that would listen to fsevents and nuke them on creation. I kept thinking that the evil should really be stopped at the source. The Finder. I wondered aloud what it would take to patch the Finder so that it wouldn’t even create them anymore at all. I decided that the creation of .DS_Store files needed to be stopped. Like Jack Ryan in Patriot Games I decided that “I will make it my mission in life”. For an afternoon at least. So in the interest of cleanliness I came up with a dirty hack. I call it DeathToDSStore. I have documented my investigation and development process below.

Who’s Behind This!? I Want A Name!

I initially suspected that it was solely the Finder.app that was responsible for .DS_Store creation. After a little investigation it appeared like that was pretty much true but there might be other things out there that also create them. My first lead was the DSDontWriteNetworkStores setting that causes the Finder to not create the files on network mounts. I needed to find out where the code that interacted with that setting lives. Examining /System/Library/CoreServices/Finder.app with strings came up empty. It must be in a framework somewhere. Poking around /System/Library/PrivateFrameworks revealed the culprit: DesktopServicesPriv.framework. Ok so, .DS_Store authoring functionality is very likely in this framework. Clients of this framework in /System/Library/CoreServices include Dock.app, FileSyncAgent.app, TimeMachine.menu, backupd and the Finder.app. For the time being I was just going to concentrate on the main offender.

AKA Property Stores

I started perusing DesktopServicesPriv.framework‘s symbols using nm (piped into c++filt) to see what kind of things I had to work with. Hmm, FSVolumeInfo::ShouldWriteNetworkPropertyStores()? That sounds like it could be directly connected to DSDontWriteNetworkStores. Perusing around a little more it appears like .DS_Store files are called “Property Stores” internally. There is even a C++ HFSPlusPropertyStore class present. I had just found what I was looking for. Time to start poking at things.

I Wanna Be A mach_star

I pulled out rentzsch‘s awesome mach_star injection/overriding code and started overriding various routines in that class. For some reason I wasn’t able to use dlsym() to get the address of any C++ member functions. I tried all sorts of permutations of the mangled name that nm was giving me for stuff but nothing seemed to work. “Well, I guess I could always just plug the raw address in.” I thought. So I used vmmap Finder to get the base address of where the DesktopServicesPriv.framework‘s __TEXT section was residing in memory. If ASLR (Address Space Layout Randomization) is in play then this value can be different every launch. (EDIT UPDATE: comex reminded me about the _dyld family of routines and so this is now obtained manually at runtime instead.)

nm gave me the offsets of the functions I was targeting and so it was simple enough to just add them together to get the location of the functions I needed to override. It would obviously be a lot cleaner to be able to determine the address from within the binary at runtime but I couldn’t convince dlsym to give me anything and this was working (unfortunately it adds an external dependencies on Apple’s Developer Tools (nm) and BSD.pkg(c++filt)).

Gotcha!

After a bit of trial and error I found that overriding HFSPlusPropertyStore::FlushChanges() with a function that simply did nothing, successfully prevented the creation of .DS_Store files on both Snow Leopard and Lion. On first inspection it doesn’t appear that overriding this function has any unintended behaviors and appears to accomplish the objective of halting .DS_Store creation quite nicely! Or about as nicely as you can when you are doing something like live patching the Finder.

(Side note: I noticed that I am not the only one using mach_star to live patch the Finder. The Dropbox daemon is doing it as well every time the Finder launches)

There’s An App For That

So I packaged my work up into an app (you can also run it through the command line under sudo with the “-silent” option) and published the source on github. In the near future I am going to add a launchd option so that it will run every time the user logs in or any time the Finder is run (EDIT UPDATE: The option to install a Launch Agent has now been added). Now that I have a solution I am going to put my money where my mouth is and start running without .DS_Store file creation full time. Hopefully I wont run into any major issues (I will report them here if I do) and I am pretty sure that I wont miss their functionality (At least by having new ones created. Existing disk images and CDs with custom layouts will still be read just fine). So far the result is pretty positive! Is this safe? Eh…. The safety/cleanliness/forward compatibility of patching OS components has been discussed and debated ad nauseum. You probably already understand the risks involved. Things look pretty stable but I make no promises. Go ahead and try out the DeathToDSStore.app or build your own from source if you want and let me know how it works out for you. Remember that you need to have the Developer Tools installed, BSD.pkg installed and admin authorization. Death to .DS_Store!

El Capitan Update

In OS X El Capitan, Apple introduced a new feature called System Integrity Protection (SIP) which prevents filesystem and runtime modification of the operating system. Unfortunately, this prevents the mechanism that Death To .DS_Store uses from working on the Finder process. You can disable SIP to get things working again, although a better choice might be to head over to https://bugreport.apple.com/ and file a radar. You can optionally request that it be duplicated against my bug (radar://17348033) that I officially filed with Apple over the issue. I am still holding out hope that eventually Apple will do the right thing and fix their mistake.

Sierra Update

Slight compatibility update for 10.12. __pthread_set_self() changed to _pthread_set_self() (thanks @iccir). Everything else functions the same as it has. You might need to enter your password once to authorize Developer Tools Access and there is still a race condition during login between the time when the Finder first flushes .DS_Store files to disk for open windows and when the Finder actually gets patched. Once it’s patched though, it still correctly (and harmlessly I might add, after running it full-time for almost 5 years) suppresses .DS_Store creation by the Finder.

One interesting point of note in macOS Sierra is that .DS_Store files are now completely invisible to the Finder, even when you have it configured to show invisible files. A quick check with the Terminal proves that they are still there, but for some reason the Finder will now not display them under any circumstances. This behavior change is a bit confusing because all of the above problems with .DS_Store files still persist. Out of sight, out of mind?

107 Responses to “Death to .DS_Store”

  1. Mike M Says:

    There’s another app that does exactly that: SyncFF

    http://www.limitpointstore.com/products/syncff/

    I spent quite a while posting on various forums, experimenting with Apple/shell script, and trying to figure out how to write a script to do what this app does. In the end, it was well worth the $10.

  2. Mike M Says:

    Mark,

    There’s another app that does exactly that: SyncFF

    http://www.limitpointstore.com/products/syncff/

    I spent quite a while posting on various forums, experimenting with Apple/shell script, and trying to figure out how to write a script to do what this app does. In the end, it was well worth the $10.

  3. Mark Harrison Says:

    Hi Mike,

    Thanks for the tip. I like what I see with SyncFF with one caveat: Are folder creation dates that are older than the modification date change preserved or lost to the “restored” modification date? What is your experience?

    Many of my folders have been built up over time and preserving that record matters to me as well.

  4. Mike M Says:

    It works as described. Basically, for any folder/path, the app will drill down through the hierarchy to all of the deepest subfolders and files, and set the modification date for every folder to the most recent modification date of any files/folders contained within it. This corrects the issue of folders having a modification date much more recent than any file or folder contained within them – a problem that was frequently caused by the invisible .DS_Store files contained within them.

    In reference to creation dates: those are separate from modification date, and aren’t changed by this tool. It only analyzes the modification dates of the hierarchy, and updates folders date modified to the most recent modification date of any file/folder contained within them.

    If you are interested in pushing the creation date of any folders/files to their modification date (i.e. make the date modified match the date created, regardless of whether or not the file has actually been modified), you can follow the info I got in a forum post about this very issue at:

    https://apple.stackexchange.com/questions/421140/set-file-created-modified-timestamp-to-the-earlier-of-the-2-for-a-ton-of-files

    This will preserve modification date of any files/folders that are (somehow, possibly due to user error, or deliberate editing of timestamp data) older than their created date. But for any files with a creation date older than the modification date, the modification date will be set back to the creation date.

    I hope this helps. It’s certainly taken a significant amount of my “free” time to figure it all out. It’ll feel more worth it if the solutions I found can also help others out of the same predicament I was in – feeling utterly powerless to manage my file system and keep things organized by *actual* modifications made, oppressed by the invisible, brute force tyranny of .DS_Store files in macOS Sierra. 🙂

  5. Mike M Says:

    Spencer,

    Thank you so much for this app! It’s a great fix for the Finder .DS_Store file issue. Unfortunately, however, when other apps use the Finder API to browse the file system, .DS_Store files are still being created. So anytime I need to browse to open or save something from another app, the folders I browse through will have the .DS_Store file generated in them. 🙁

    Is there any way that your DeathToDSStore app could be modified so that it might also prevent other apps using the Finder API to browse the file system would also be prevented from creating .DS_Store files?

    I should also note that I haven’t restarted my system since I ran your app. So maybe it does address this issue, but only after a restart?

    I’ll post an update later on whether or not a restart resolved the issue.

  6. Mike M Says:

    Also, for anyone who has noticed their desktop icon positions have all been shuffled to a default cluster as a result of purging .DS_Store files, I found this app (very basic, and free) that lets you save and restore icon positions.

    http://www.parker9.com/DesktopIconManager2.1.0.zip

  7. Josh Says:

    (In response to Mike M’s comment about desktop icons)

    As a part of my war on .DS_Store files, I prefer to keep hidden files visible at all times. This makes the desktop challenging, but I started using this method: https://superuser.com/a/950897/191730 (delete the hidden files from the desktop, then set the folder to be read-only)

Leave a Reply

Entries (RSS) and Comments (RSS).