So, I do not work a manual labor, paid job – but I did switch from a completely sedentary position (think 3 hours of commuting per day, plus 8-9 hours at work on a computer) to my current role as a SAHM who works very part time from home. I went from being almost completely inactive for 12+ hours 5 days a week, to being on my feet all the time caring for a baby, doing chores, walking, lifting, bending, etc. I would not go back to my old job and schedule even if they paid me triple what I was making before. I can't comment on the financial piece, but I can weigh in on the lifestyle/activity level switch.
Bracken Joy has some good points – when your day is filled with moving about, be prepared for compensation in the fitness department. You are unlikely to have the desire or energy to want to go train for a marathon or hike a mountain when you are on your feet all the time. This is perfectly fine from a health perspective FYI – lots of moderate daily activity is extremely healthy (and the bio norm for the human body) and so you won’t need to be hardcore working out at the gym 4x a week the same way a sedentary office worker should.
Be prepared for a lot more aches and pains – manual labor lifestyles are great, but they do wear on you, especially if you do a lot of lifting/bending/crawling around. Once you sustain an injury, it can be harder for it to heal appropriately since your job depends on physical movement that can easily exacerbate the problem. A serious injury can take you out of the workforce completely for an indefinite period of time, unlike sedentary office jobs where most workers can continue to function and make money. Go into this with your eyes open and definitely look into carrying temporary disability insurance.
Finally, watch out assuming that a physically rigorous lifestyle will automatically give you satisfaction. Many physically demanding jobs can be just as tedious and boring, or more tedious, than computer work. When I switched to staying home, I had this bucolic vision of how awesome my life would be and how productive and fulfilled I would feel having such a flexible and active lifestyle. Some of my vision certainly came true, but I was shocked at just how boring, repetitive, and downright annoying things could be sometimes. Sometimes I do just want to laze about, and I can’t. On days where I feel sick or really tired, having a physically demanding lifestyle can really suck far more than when I could just cuddle up in my office chair and stare at a screen.
At the end of the day, I wouldn't change a thing. I am far healthier and happier living this way. But you asked for considerations so I figured I'd speak frankly about some potential downsides.