Have you ever ridden a horse beyond a leisurely walking trail ride? Actual horseback riding, when done correctly is exercise. You use a lot of core strength and leg strength to stay on and upper body strength working with the horse's mouth.
I wouldn't call my trail riding actually leisurely. We tend to go for 3-5 hours at a time, and if conditions permit will do maybe 1/3 trotting and a bit of cantering. Even the walking is usually fairly brisk, and I do have to spend a bit of effort holding her back from wanting to trot all the time. (Which is nice, since I spent two years rehabbing her from a serious leg injury - the sort where the vet said she'd probably never be able to do more than hobble around the pasture.)
While I do find that there's some isometric work in the legs, it's just using particular muscles that don't normally get much use, and I'm trying to adjust my normal routines to strengthen those areas. (But as with the OP, I'm exercising to ride better, not riding to exercise.) I don't find that I use much upper body strength at all: maybe this wouldn't be the case on a different horse, but she needs only the slightest hints from me.
I don't say that it's not possible that some horse riding might not be exercise, just that this isn't, even though my friend that suckered me into it said it would be (and still says it is). Maybe it's just perspective: for years I would go trail "riding" with her and her friends, except that I would walk & run while they rode.