You don't have to move to a mountain town for access to the outdoors. We live in Boise and our local ski resort is just 30 minutes up the mountain. We get in 40-50 days of skiing per season. There are hundreds of miles of hiking and biking trails that start a few blocks from our house. Boise isn't unique in this regard, the same can be said of Ogden, UT, Spokane, WA and other cities in the West. You can still buy houses in/near these cities for under $400k.
Yes, moving to a HCOL area will set back your plans. You have to decide for yourself if it's worth it, no one here can make that value judgement for you.
I would also recommend renting for the first year. Durango isn't just a mountain town, it's also a tourist destination. Make sure you're okay with the influx of people during peak seasons. Also, you haven't mentioned your significant other: what do they think about living in a small tourist town? Do they have experience with this? I grew up in a small-ish tourist destination and I don't miss the high prices, crowded restaurants, or driving a long distance to do more than basic shopping.
One final thought that may not even be on your radar. Review the Colorado Wildfire Risk Map for the area (
https://co-pub.coloradoforestatlas.org/#/). Durango is surrounded by high risk/high intensity burn zones, which will get even worse with climate change. IMO, towns like this are primed to burn. Don't buy a house in the forest, make sure there's more than one escape route, and check fire insurance rates before you make an offer.