@gavint, I don't remember the name of the name of the book because it was a long time ago.
I am glad you recognize the gap between our gut estimate and evidence-based methods. The book I read had what seemed to me like a third approach, which I summarize briefly here for you. The main ideas I got from it were:
1. State to yourself as clearly as possible the interviewee characteristics you want to find and to avoid. Define your goals, in other words.
2. Prepare and ask open ended questions to allow those characteristics to surface. For example, "What happened when you most recently experienced a difficulty in the workplace?" rather than "Have you ever experienced a difficulty in the workplace?" The second version is "closed" because it asks only for a yes/no answer, choosing among options defined by the interviewer. The first version is "open ended," because the interviewee decides what to say.
3. The questions need not be directly about the characteristics you're interested in, they just need to allow the interviewee to talk about incidents that relate to the characteristic. You as the interviewer listen thoughtfully to detect the underlying behavior patterns. For example, the open ended question above could give evidence about the interviewee's problem solving approach, and might reveal something about their approach to resolving conflict; you don't have to directly ask "How do you resolve conflicts with your boss?" or "How do you solve problems?"
4. Questions about past experiences help because they trigger real world examples, which are much more true to life than answering general or hypothetical questions.
5. Layer your questions to get a fuller picture on key criteria. Layering means asking followup questions, or asking a second question to get a different angle on some aspect that you identified in step 1. For example, if hands-on problem solving is a key factor in your trade, and the answer to the question about a difficulty at work was "I had to build a chair, but we didn't have a hammer for the nails," a followup question might be "Oh, what did you do?" If the answer to that was "I used my shoe to pound in the nails," maybe you're satisfied. If the factor you're interested in not problem solving but time management, perhaps you add another followup, such as "How much longer did it take than a hammer would have?" If the factor you want to know about is attitude toward the business owner, perhaps you follow up with "How did you feel about that?" and "How often were tools missing at that workplace?... In that workplace, what was the most important thing that should have been improved?"
6. You can prepare 2 or 3 layers of questions but you don't need to use all of them. Just use enough to evaluate the interviewee with respect to your underlying goals.