I would never take out a loan for a horse, but I mostly trail ride. My previous two horses cost $800 and $1000 to buy, and my next one will be in a similar price range. Showing can get insanely expensive. Once you hit a certain level, horses run in the tens of thousands, then you board somewhere very expensive, pay for lessons and training rides, show fees, trailering, grooming, braiding, entry fees, and more.
Leasing horses is quite common. I've half-leased several during the time I haven't owned horses and financially it's been much better than owning. HUGE benefit - horse get sick or injured and it is not your problem. You can typically end the lease at any time, so if horse is lame and will need a couple months of recovery, you can move on to a rideable horse. On a half-lease, you usually pay a flat rate, and then get to ride three days a week. The owner pays for the hooves to be trimmed and for vet bills, unless you agree otherwise. The down side is that you don't get to make the decisions for the horse, and can't just go do whatever you want with them.
I also at one point got to ride a much nicer horse than I could have afforded at the time. He was an older show horse, with a ton of experience and ability - the kind who sold for over $75K when he was in his prime. I was able to become a much better rider during the time I leased him and then when his owner retired him, I no longer had to pay anything. In contrast, I owned an elderly horse that I paid for for years after he could no longer be ridden.