Do you keep a list of books to read? Whenever someone mentions a book that sounds interesting or an author references other books, I note it on a list I keep in my email drafts. I have the same experience as you if I only begin the task of finding books once I'm actually ready to start reading.
I would give yourself permission to give up on books fifty pages in. But also I would say if you're interested in a topic, sometimes you have to force yourself through a book that's academic. The more knowledgeable you are beyond the basics, the more interesting options you'll have.
Don't limit yourself to books. Lots of websites and blogs will have information you want. I think for non-fiction it's easier for writers to sharply capture what they want to say in whatever format is most appropriate rather than having to fulfill some pre-conceived book format with a made-up storyline, required page count, etc. Plus online resources will both mention books sometimes, and you'll become more well-prepared for the topics.
And +1 on Glenstache's suggestion.