You asked if we've ever lent money to a friend. I have, and I've borrowed money from family. The latter was actually for a house. Currently, my parents own the mortgage on our home. This was a mutually benifical thing however, and not because I couldn't afford a regular loan. I had a regular loan, which I couldn't refinance to lower rates because it was no longer owner occupied. I bemoaned this fact to my dad (just making conversation, with no agenda) and he offer to lend me the money. Guaranteed ROA on his money (and he liked the fact that my interest would stay in the family and I'd eventually inherit half of it back), and a lower rate for me.
As for the friend, she was also my college roommate. A few times, I lent her money (IIRC, it was maybe a couple hundred dollars, or perhaps up to $500, each time). Usually it was to tide her over until her student loan payment arrived. She always paid me back as promised or earlier.
In general, my approach to asks that I'm not willing to entertain is, "I'm sorry, we don't have an spare cash." This is, strictly, true. None of my money is "spare". It's purpose may be to pay rent, or to go into an investment account for future retirement needs, but it is not "extra".
Also, you call this person a friend, but then also a colleague. Do you hang out socially at all? If yo've lent him money, has he paid it back in a timely manner? I think the next time he asks, I'd say, "Bob, we've talked about this before. I'm sorry, but I won't be lending you any money, and frankly, the asks make me uncomfortable. My answer isn't going to change, so please drop it." This is assuming you haven't actually lent him money in the past, which may be incorrect since you mention he "exploits" you monthly It's also assuming we aren't talking about asking for $5 to pay for lunch and then repaying you the next day. If you've already lent him money, "Bob, I know we've done this in the past, but I can no longer lend you money. I'm sorry, but I need all of my cash, so the loans have to end. " And then the next time he asks, tell him you've discussed this, your answer isn't changing, and you'd appreciate it if, for the sake of your friendship, he drops it. At that point, if it affects the relationship, that's on him, not you.
This is also a reason that I am careful about how I speak about our finances, especially with people to whom I'm not super close or who I feel are somewhat likely to see dollar signs when they look at me. If they don't know you are squirreling away money, they don't see you as a future lender.