I love outdoor activities as do most of the people I know. I've seen three lifestyles around this:
1. industry job [fun, low pay, lots of time off]
2. non-industry job in a good location [less fun, medium pay, less time off, but you are able to ski/surf/bike/raft right where you are]
3. non-industry job in a big city [even less fun, high pay, less time off]
You are doing #1. That works if you are okay with having to go full dirt bag to save any money and you'll always be short on cash, but big on time and smiles. You can save/invest and get somewhere financially, but it will be slower and your end point will be a lot lower than the other options. That said if you are super cost-efficient [ie. living in a van, limited partying and eating out, etc..] you don't need a lot of money to live your life. The problem is the 40 year old you better still be okay with this since it will be hard to change directions later.
I had a friend that was a bank manager in Whistler - option #2. She made a decent income. Lived a lower cost lifestyle and enjoyed the amazing location she was in. She didn't have months off each year, but if she had a morning free she could go bike, ski, swim, hike, etc.. and get back to something else with minimal hassle because she traded career progression at the bank with living someplace she loved.
I'm doing #3...I've lived in decent, but not my ideal places. I've made a lot of money and taken a lot of time off [1-4 months a year], but the downside is I have to leave where I am to get to the places I really want to be. The upside is if I had been more efficient with my spending I could have been retired at 40 with no issues and had a ton of my life left to do fun stuff. I found MMM late so I'll be more like 51-52 when I FIRE.
Looking back on things I'd either do what I did, but save more earlier or do option #2 and make a decent wage someplace amazing and work a bit longer. The lack of financial security and required very low spending of option #1 don't appeal to me. You have to look at your own options and decide what trade offs you are willing to make. I don't think there is a wrong answer here "if" you are honest with yourself.
On the GF front if she is absolutely "The One!" then it makes sense to try and make it work while pursuing your own goals with some inevitable compromises along the way. That said from my perspective at nearly 50 I can tell you the planet is chock full of amazing ladies so if your current GF is not a good life match with where you want to head don't feel like it's her or nothing. You will meet someone else awesome and if you pick well she'll be better aligned with your life plan.
Since you sound unsure about some of the key issues here I'll leave you with one last thought. If in doubt take the path that limits you the least. If you go back to school and/or the big city to get a high[er] paying job you can always go back to rafting or some other outdoor rec job. OTOH if you spend 10 more years not getting where you want to go and then try and break out into a professional gig it will much much harder.