From your description, there are good reasons to say “No:” this project leads you away from your strategic focus on new industries, and threatens an excellent relationship/referral source.
But before you say “no” to the client, have that chat you mention with your former manager. Because it’s likely you could have a conversation that would end up with the manager telling you to go ahead with the work, and the two of you developing a relationship of mutual referrals. You are becoming a niche specialist, so you’ll start winning lots of this very specific work. You’ll also start to hear about projects outside of your specialty that you can refer to them.
In your sit down with the manager, start by explaining that a former client reached out to you, asked you to do the work, but that you’re reluctant—not just because of the non-compete—but out of your personal sense of not wanting to disrespect your past relationship or hurt your future one. A small percentage of folks hearing something like this will be too unhappy or “non-competey” to go further. But the majority, even if momentarily unhappy with the client, will understand this as an opportunity to broaden a constructive relationship with a colleague who has just proven themselves technically in demand and honest to a fault.