Traveling was so exciting but after a few months we experienced a lot of travel fatigue. If you are traveling in poorer countries, you will be haggled relentlessly and people will always be looking to scam you. It became exhausting trying to connect with locals when they all had a "my sick uncle" type of sob story..
It depends on where you are meeting these locals. If they're just approaching you on the street, then yes, I can see that most would be panhandlers.
We've recovered from our Travel Fatigue in many places around the world. We typically travel for 6-9 months straight and then pause for a break for 1-3 months. In each place we've stopped, we've managed to hook up with locals from our own socio-economic background. The trick is to join clubs or events that you have something in common with.
One of our first experiences was with a Spanish immersion homestay in La Paz, Mexico. Our hosts were well-to-do people, very well-traveled and lots in common to talk about, but they were a bit older. I spent all my time with their grandson playing guitar and hanging out with him and his girlfriend at their K-Pop dance studio (Korean Pop is huge in Mexico!)
In Costa Rica, a fellow motorcyclist invited us to stay with him and his family. It turned out he was in the same profession as I was, so we talked shop alot. One evening, we were watching football on TV. When CR beat Mexico in one of the games (pretty much all the countries in Latin America hate Mexico because they're so good in football), we spilled out into the streets honking our horns and celebrating with everyone.
Then on another break, we spent two months in Medellin, Colombia, so my wife joined an AcroYoga club that met in the local park several times a week. There we were able to meet a lot of locals that we hung around with after, had them over for dinner at our apartment, etc.
We've stayed in Chiang Mai, Thailand for a couple of seasons now (over six months in total) and whenever we're there we've joined hiking clubs, photography clubs, yoga. Thai is a difficult language to learn, but professional people that have the income and free time to pursue these interests most often speak English as well.
Just because a country may be poor as a whole doesn't mean that there aren't people of similar backgrounds as you living there. You just have to go to the right places to meet them.