I have to chime in here. Physical labor itself is not necessarily wearing. When I was growing up, our next door neighbor in his 80s was constantly gardening and doing projects around his house. He was in incredible shape for his age. He used to shovel coal for steamships as a young man. He died in his 90s.

Physical labor can become debilitating when there is injury that is not allowed to heal completely or when there are excessive hours compounded by stress. Aside from that, it's probably very good for you.