Death to .DS_Store

December 24th, 2011
Spencer Nielsen Follow snielsen42 on Twitter

The “.DS_Store” file is an abomination and must be stopped. You know what I’m talking about. I regularly rant about how this annoying file gets in your way, dirties things up and just screws with your stuff in general. Today I decided to do something about it. Before we get to that, lets quickly review what it is and why it sucks.

What Is It?

The .DS_Store is a Finder metadata file created primarily by Mac OS X’s Finder.app. Because of the dot (“.”) prefix it is typically not visible in many file browsers and most Mac OS X users are probably not aware of it. It is regularly created when the Finder accesses filesystem directories. It contains directory information about icon locations, view options, silkscreen configuration and the like. The functionality that it provides is moderately useful, but becoming less and less relevant over time. In any case, a long time ago the horrible decision was made to store that Finder metadata in an actual file (.DS_Store) in the filesystem within the relevant directory. We have been paying for it ever since. Over time more and more metadata relating to files and the filesystem has been added to Mac OS X, but thankfully those have been stored in saner places (extended attributes, etc). For the time being though .DS_Store is still here with us and still causing trouble. What’s so harmful about the file you might ask?

Cross-OS Cleanliness

The .DS_Store file is only useful if you are accessing the filesystem with the Finder. What if you aren’t using the Finder though? What if you aren’t even using Mac OS X at all? At best the files then become wasted space that is ignored. At worst they are horrible clutter that dirties up your file browsing experience. It doesn’t take much online searching to find countless forum posts from Windows users asking what these files are or the innumerable removal scripts that Linux users have authored to clean them from their directories. On Mac OS X, if you insert a USB flash drive, mount a network volume or zip up a directory to give to a friend, more likely than not, .DS_Store files have hopped on for the ride. Whoever has to interact with those directories next that doesn’t use a Mac is going to have to deal with those in some way even if it just having to consciously ignore them. Eventually a power-user facing flag was added that would disable .DS_Store file creation on network volumes (“defaults write com.apple.desktopservices DSDontWriteNetworkStores true“) which was a good move. There are still however, plenty of reasons why Mac OS X users hate having them on their own systems.

Source Control Gunk

All the programmers in the room, raise your hand if you have ever added “.DS_Store” to your .gitignore file or equivalent source control ignore configuration? What’s that? That guy in the back who didn’t raise his hand? Oh, he was sleeping…there’s his hand. The contents of the .DS_Store file are always changing and so even if you did happen to check the .DS_Store files into your repository (furthering it’s unholy propagation) it is always going to be showing up as a changed file and gunking up your commits with changes and data that have absolutely nothing at all to do with your code. I have to wonder how many global man hours have been wasted in configuring source repos to ignore these files or cleaning them out of the repos after someone YET AGAIN added them in their latest commit.

Permissions Playtime

Because the .DS_Store file is so inconspicuous and usually innocuous, a lot of the time we simply forget it is there. Unfortunately when doing things like mass permission changes and the like, the .DS_Store file will unintentionally come along for the ride. Suppose you tighten up the permissions on a directory that has a .DS_Store in it. After a while you decide to go through an manually clean it or lower the permission levels on the contents again. You had better remember that you also raised the permissions on the .DS_Store file as well or you might be puzzled why you don’t have permission to delete the directory or why the Finder isn’t saving your directory options. It is no show stopper. Any competent command line monkey can figure it out and fix it. But again, the few minutes it takes to fix it should never have been necessary. Technology is supposed to save time, not destroy it.

So What Do We Do About It?

Those were just three examples that I thought of off the top of my head. Leave your own .DS_Store horror/annoyance stories in the comments and I will add any more that don’t generally fall under one of those three examples. So they are a time consuming annoyance, easy enough to whine about. But what to DO about it? The other day I again ranted on twitter about them and it was suggested that maybe someone could write a daemon that would listen to fsevents and nuke them on creation. I kept thinking that the evil should really be stopped at the source. The Finder. I wondered aloud what it would take to patch the Finder so that it wouldn’t even create them anymore at all. I decided that the creation of .DS_Store files needed to be stopped. Like Jack Ryan in Patriot Games I decided that “I will make it my mission in life”. For an afternoon at least. So in the interest of cleanliness I came up with a dirty hack. I call it DeathToDSStore. I have documented my investigation and development process below.

Who’s Behind This!? I Want A Name!

I initially suspected that it was solely the Finder.app that was responsible for .DS_Store creation. After a little investigation it appeared like that was pretty much true but there might be other things out there that also create them. My first lead was the DSDontWriteNetworkStores setting that causes the Finder to not create the files on network mounts. I needed to find out where the code that interacted with that setting lives. Examining /System/Library/CoreServices/Finder.app with strings came up empty. It must be in a framework somewhere. Poking around /System/Library/PrivateFrameworks revealed the culprit: DesktopServicesPriv.framework. Ok so, .DS_Store authoring functionality is very likely in this framework. Clients of this framework in /System/Library/CoreServices include Dock.app, FileSyncAgent.app, TimeMachine.menu, backupd and the Finder.app. For the time being I was just going to concentrate on the main offender.

AKA Property Stores

I started perusing DesktopServicesPriv.framework‘s symbols using nm (piped into c++filt) to see what kind of things I had to work with. Hmm, FSVolumeInfo::ShouldWriteNetworkPropertyStores()? That sounds like it could be directly connected to DSDontWriteNetworkStores. Perusing around a little more it appears like .DS_Store files are called “Property Stores” internally. There is even a C++ HFSPlusPropertyStore class present. I had just found what I was looking for. Time to start poking at things.

I Wanna Be A mach_star

I pulled out rentzsch‘s awesome mach_star injection/overriding code and started overriding various routines in that class. For some reason I wasn’t able to use dlsym() to get the address of any C++ member functions. I tried all sorts of permutations of the mangled name that nm was giving me for stuff but nothing seemed to work. “Well, I guess I could always just plug the raw address in.” I thought. So I used vmmap Finder to get the base address of where the DesktopServicesPriv.framework‘s __TEXT section was residing in memory. If ASLR (Address Space Layout Randomization) is in play then this value can be different every launch. (EDIT UPDATE: comex reminded me about the _dyld family of routines and so this is now obtained manually at runtime instead.)

nm gave me the offsets of the functions I was targeting and so it was simple enough to just add them together to get the location of the functions I needed to override. It would obviously be a lot cleaner to be able to determine the address from within the binary at runtime but I couldn’t convince dlsym to give me anything and this was working (unfortunately it adds an external dependencies on Apple’s Developer Tools (nm) and BSD.pkg(c++filt)).

Gotcha!

After a bit of trial and error I found that overriding HFSPlusPropertyStore::FlushChanges() with a function that simply did nothing, successfully prevented the creation of .DS_Store files on both Snow Leopard and Lion. On first inspection it doesn’t appear that overriding this function has any unintended behaviors and appears to accomplish the objective of halting .DS_Store creation quite nicely! Or about as nicely as you can when you are doing something like live patching the Finder.

(Side note: I noticed that I am not the only one using mach_star to live patch the Finder. The Dropbox daemon is doing it as well every time the Finder launches)

There’s An App For That

So I packaged my work up into an app (you can also run it through the command line under sudo with the “-silent” option) and published the source on github. In the near future I am going to add a launchd option so that it will run every time the user logs in or any time the Finder is run (EDIT UPDATE: The option to install a Launch Agent has now been added). Now that I have a solution I am going to put my money where my mouth is and start running without .DS_Store file creation full time. Hopefully I wont run into any major issues (I will report them here if I do) and I am pretty sure that I wont miss their functionality (At least by having new ones created. Existing disk images and CDs with custom layouts will still be read just fine). So far the result is pretty positive! Is this safe? Eh…. The safety/cleanliness/forward compatibility of patching OS components has been discussed and debated ad nauseum. You probably already understand the risks involved. Things look pretty stable but I make no promises. Go ahead and try out the DeathToDSStore.app or build your own from source if you want and let me know how it works out for you. Remember that you need to have the Developer Tools installed, BSD.pkg installed and admin authorization. Death to .DS_Store!

El Capitan Update

In OS X El Capitan, Apple introduced a new feature called System Integrity Protection (SIP) which prevents filesystem and runtime modification of the operating system. Unfortunately, this prevents the mechanism that Death To .DS_Store uses from working on the Finder process. You can disable SIP to get things working again, although a better choice might be to head over to https://bugreport.apple.com/ and file a radar. You can optionally request that it be duplicated against my bug (radar://17348033) that I officially filed with Apple over the issue. I am still holding out hope that eventually Apple will do the right thing and fix their mistake.

77 Responses to “Death to .DS_Store”

  1. snielsen Says:

    @Rick, probably not. The app is not meant to be kept open after you use it to patch the Finder. In fact, ideally you just use it to enable the launch agent and never run it again!

  2. a Says:

    Thank you so much, I’ll post again if there are issues.. I’m on Yosemite 10.10.3 using totalfinder.binaryage.com, expandrive.com and mudflatsoftware.com for cross-network data management. Since the first two apps I’m using (plus the DeathToDSStore app now) integrate tightly with Finder, it will be interesting to see if they all play nicely. Do you think there would be any issues running Asepsis and DeathToDSStore together? How would they interact? Everything is currently syncing great, except for the all the Mac turds lying around!

    I’ve also reported the bug 😉
    https://i.imgur.com/LHqeXMM.png

  3. snielsen Says:

    @a There shouldn’t be any problem running both at the same time. Asepsis patches libc and redirects file interaction to a centralized .DS_Store holding pen. DeathToDSStore suppresses their creation by Finder.app.

    In practice, DeathToDSStore works well except since 10.9 there is a significant race condition involving when Finder.app is launched and when the DeathToDSStore launch agent is kicked off. This means that when you log in, the Finder has the opportunity to write a .DS_Store file for every currently open Finder window before the launch agent (which patches the Finder and suppresses .DS_Store creation) gets run. I haven’t had any time to investigate any sort of mitigation for this yet but want to get around to it sometime.

  4. a Says:

    There doesn’t seem to be a problem with running ExpanDrive, TotalFinder and DeathToDSStore and I’ve eliminated Mac turds from my life, ahh fresh air! How much would it cost to update DeathToDSStore?

  5. snielsen Says:

    @a DeathToDSStore is a hobby for me and so it is more of a time issue than a money issue. Ideally Apple will finally surprise us in 10.11 and fix their most glaring design bug :)

  6. a Says:

    So basically DeathToDSStore makes Asepsis unnecessary because it’s blocking the creation before Asepsis pens the files? I’ve noticed that Asepsis sometimes doesn’t work as expected or is not updated frequently. Also, the issue of the race condition is not a big deal since I never restart, and when I do I’m happy to wait until everything loads. Worst case, I’ll have to remove a DSStore on the Desktop (my default folder when TotalFinder opens). Does the race condition pose any stability or data corruption issues? Have you considered making a donation page or selling this app to fund maintenance?

  7. a Says:

    I posted before I noticed your reply, thanks for the feedback. I’m subscribed and look forward to future updates as your time permits.

  8. Evgeni Says:

    Should this app be installed separately for each user on my Mac?

  9. snielsen Says:

    @Evgeni If you install the launch agent then it will activate for every user on your Mac when they log in so there is no need to run the app or initiate the install procedure for your other users.

  10. Tricky Says:

    I can tell this even crazier. I also find a lot of ._ files in my directories. Like if I have a file named myimage.png then I can be sure I’ll soon have a ._myimage.png, and to make it worse the .DS_Store files often also fall through this making me stuck up with a .DS_Store file and a ._.DS_Store file.
    I recently fixed up a small app to remove all .DS_Store plus all “._” files, and I effectively won 15 GB in the process. Not to mention that I recently had to clean up my GitHub repositories as whatever I do to ignore them, a few of them always manage to sneak in.
    When I was Mac user for a short time I did find these files when I took my memory stick to work, and I always wondered what they were for and how they got there (as on work I had to use Windows), I found out later it was my Mac doing it. And I really wonder how much time I’m going to need to clean all servers hosting a website of mine because of some .DS_Store files.
    I’ve seen this http://asepsis.binaryage.com recommended on several sites, so I hope this is the solution. I like Mac, it’s the only reliable operating system I’ve ever come across, but these .DS_Store files are really a cancer in this otherwise great product.

  11. Josh Says:

    As of 10.11 Beta (15A234d) not only are .DS_Store files still created, but Asepsis is blocked by System Integrity Protection.

    On Github, darwin said:

    “AFAIK Asepsis under El Capitan with System Integrity Protection enabled is not possible.

    I’m going to stop developing Asepsis and supporting it under El Capitan.” ( https://github.com/binaryage/asepsis/issues/30 )

    After testing DeathToDSStore in 10.11 beta via cask, the creation of .DS_Store files seems to be halted. Does this mean it’s now our only hope?

    And as a follow up: What’s in these files anyway? Is there anything besides the fact that I like viewing that particular folder’s contents as list view, and may have added an icon to a file?
    These both seem like really trivial reasons to enforce .DS_Store files, even against user and OPSEC wishes. Someone, please tell us there’s more.

  12. snielsen Says:

    @Josh I haven’t had time to test Death To .DS_Store on El Capitan yet but I would actually be surprised if the mechanism that we use to inject code into the Finder is still allowed under the new security model. I will try to find some time this week to do some tests. Sometimes the Finder is funny about when it flushes the files to disk and so you might want to let it sit for a bit to see if it actually kicked in. Also, on 10.9/10.10 at least there is a race condition that allows the Finder to write the files to any open window before the patch kicks in during the login process so you might see a couple written here and there every time you reboot.

    If El Capitan is indeed the end of Death To .DS_Store, then I think it is time to starting filling radars, requesting they get dup’d to radar://17348033 and harassing Tim Cook’s twitter account :)

    To answer your question, they are just Finder metadata. Label info, icon positions, view defaults, icon associations etc. Moderately useful, but not vital information that clearly should be stored somewhere other than the filesystem.

  13. a Says:

    I reported DS_Store to bugreport.apple.com in April 2015 and got an email in August 2015. They closed my bug report saying it was a duplicate issue for bug 3479171, so I guess it’s already in the system. Did one of you file that :)

  14. Wowfunhappy Says:

    For El Capitan, why not just turn off System Integrity Protection? The extra security doesn’t seem all that necessary for a power user who knows what he/she is doing.

    Will this tool work for ._ files as well? Those bother me a lot more than .DS_Store, because there’s a heck of a lot more of them!

  15. Blue Says:

    I always just use: http://annoyingfilebegone.typhoonsoftware.com/ because it runs in my menubar and cleans these files automatically. I keep movies on a removable hard drive and it annoys me to see the mac system files when I plug the drive into my tv and browse.

  16. Roger Says:

    .DS_Store seems to be back in El Capitan and possibly worse than Yosemite. Any ideas?

  17. snielsen Says:

    Turning off System Integrity Protection is an option which will allow Death To .DS_Store and Asepsis to work again. Your system won’t be any less secure than Yosemite. If you are confident that your system’s outer defenses are good enough to keep any bad actors out, then SIP won’t make any security difference. It only mitigates the damage that something can do once it is already in your system and has somehow gotten privilege escalation.

    That said, there has never been a better time to file a radar about this issue at https://bugreport.apple.com :)

  18. Mike Kormendy Says:

    I’m not exactly sure what to put in the bug report that would get any sort of attention, or at least address the issue correctly. Should I say that SIP is screwing things up? Should I say that SIP needs to be augmented? Should I say that we need a native authorized option to turn off .DS_store files in the system settings?

    What do you think would work best?

  19. snielsen Says:

    @Mike Normandy The best thing to write is that Finder metadata should not be stored in the filesystem inside .DS_Store files because the cons of doing so far outweigh the pros. You can also simply ask that your bug be duplicated to mine (radar://17348033) and not write anything else. Bug duplication counts do actually matter within Apple when weighing priority.

  20. whoknows Says:

    I wish Windows ignored dotfiles… grrrrrrrrrr

  21. Greg Says:

    Hi,

    On OSX 10.11 I’ve turned off SIP, and now I want to install DeathToDSStore. I’m not able to find any instructions on how to install. Do you have any instruction I can follow? Thank You.

  22. snielsen Says:

    @Greg If you run the app you can optionally install (or remove) a launch agent that will kick off every time you log in.

  23. Greg Says:

    Hi Spencer,

    I’m really a ‘neophyte’, not even sure how to run the app? When I type >pip install DeathToDSStore

    Get the message:
    “Could not find a version that satisfies the requirement DeathToDSStore (from versions: )
    No matching distribution found for DeathToDSStore”

    Is there another way to run the app?

    Thanks in advance

  24. snielsen Says:

    @Greg No worries :) So there is a download link above to the pre-built app:

    http://www.aorensoftware.com/Downloads/Files/DeathToDSStore.zip

    Just unzip it and double-click to run.

  25. Lex Blagus Says:

    I am unable to find the bug report on https://bugreport.apple.com and/or radar://17348033 Would you please provide a direct link/uri so I can file a +1 to fix this?

  26. snielsen Says:

    @Lex Blagus You can’t actually look up bug reports other than the ones you file yourself on bugreport.apple.com. To +1 for the fix, you can simply file a new bug and just mention in the description field that you want your new bug to be counted as a duplicate of bug 17348033.

  27. Lex Blagus Says:

    Very nice, thanks a lot. I hope we fix this, .DS_Store is really a abomination

Leave a Reply

Entries (RSS) and Comments (RSS).