I've previously expressed my desire to and difficulty with using git-annex and thought it fair to provide an update.
I gave it another shot by starting completely fresh and being more intentional about the data I'm managing with it and it's been a much more positive experience.
I suspect my previous problems were due to accidentally copying other git repos into the git-annex repo which seems like it could be problematic.
Now I'm using it for a good amount of data and it's been awesome. I can have a representation of all of my data (but not the actual data) on my laptop and just pull specific files down when I need them and drop them when I'm done. I can also use the system to provide redundancy for my data rather than having to store it on my NAS. Or, I can put in YouTube URLs to save them for later without having to download them. Great stuff. Highly recommended.