They look for signs of the definition of "substantial ties". For my move to the USA / Move back, it was the date I crossed the border with my family both times. (not for a short vacation) I had been working on a TN visa in the USA for 4 months prior, but the date that my husband and kids joined me was the defacto date. I would have had a lot of work to prove any other date.
For you, the way around it would be if the family on "visited" for a couple of weeks then went back to the other country, and you stayed at hotels during that time, even if you had an apartment studio rental or something else.
They take the 20/20 hindsight view to confirm if the move date matches the intent of what eventually happened (longer residency, getting bank accounts, a residence, etc).