One of the big problems with people in settled populations taking seasonal farm work is that it is seasonal and it is mobile. Mobile means moving to where the work is on a regular basis and paying for accommodation where the work is. If you are poor but have a house to live in, your income has to go to supporting that house (you can't risk losing it) and you can't afford to pay twice for accommodation, unlike a migrant worker who is only paying for accommodation where the work is. Another problem is that if you are on benefits, you will lose them for the period you are working, and the bureaucracy involved makes coming off benefits only to go back onto them after a few weeks a chancy business: if you are living at subsistence level you can't afford for your benefits to be delayed or not paid at all. Again, not a problem a migrant worker has. Plus, you are moving away from your social support networks, another factor that does not apply to the migrant worker. A significant rather than marginal increase in pay is needed if a worker is to pay twice over for accommodation while working and is to risk not being able to get back onto benefits when the job ends or not having a support network to hand if something goes wrong.
Farm work is backbreaking, yes, but so are many other jobs which local workers fill. It's not the backbreaking nature of the work that's the problem, it's the seasonal and mobile nature of it.