In my experience, some musicians / groups require several listenings before their achievement becomes obvious to the listener. It's not necessarily the case that they're DIFFICULT, but frequently that they are CAGEY, i.e. that they write / play music that must be listened to and reflected on before the listener can make a final judgment.
If you like Pet Sounds, chances are you liked it on first or second hearing.
Here are a few works that I consider cagey:
* Mew, anything by this group.
* The Mothers of Invention, "We're Only in It for the Money"
* The Bonzo Dog Band, "Keynsham"
* John Cale and Terry Riley, "The Church of Anthrax"
Certainly my subjectivity plays a major role in selecting these albums. But it's often the case that I had to take some time to familiarize myself with what was going on there before I fully appreciated them.
So my rephrasing of the initial question is "Do you prefer music that appeals to you instantly or music that takes a while to do so?" Both may be considered "good." But music that takes a while to grow on you may at first seem inconsequential or even bad, but it may be considered "better" or “good” once you've taken the trouble to acclimatize yourself to it.
I think what we often do as music writers is to encourage our readers (music listeners) to take a second (or third….) look at music that they may have bypassed. It’s a win-win approach.