I wouldn't think of myself as an audiophile, and I certainly don't think my tastes are so sophisticated that I'd see much of a difference between $100 headphones and $1000 headphones. I could be wrong about that. But I do notice a difference between $10 headphones and $50 ones.
I just got a pair of iGrado headphones -- they're the cheapest headphones Grado makes, but they're still at least as good as the Skullcandy ones I had for about a year, and a lot better than the $5 Walmart headphones I sometimes get.
And having been listening to them for a little while now, I can see the audiophiles' case.
While it's certainly nice, entertaining, even, to listen to my favorite songs piped out of my cell phone's speakers, or out of a cheap pair of earbuds I've got on hand, it doesn't compare to listening over good speakers.
Music, like all good art, has a transformative, transporting power. It can, in the right circumstances, scoop you out of the complex, ambiguous narrative of your daily life and temporarily cradle you in the bubble of its private little story.