It has to do with your short term memory of course. A study done on how many number digits humans can remember had these results:

Ericsson, Chase and Faloon (1980) decided to see how far this "recoding" idea could be pushed. They had an undergraduate student memorize random strings of decimal digits an hour a day, 3 to 5 days a week, for more than a year and a half. (Presumably they paid him well for this effort!) At the end of this period his memory span had increased from 7 to 79 digits. In other words, he could repeat back a string of 79 random digits immediately after hearing it without any error. His long-term memory for the digits also improved. By the end of the experiment, he often remembered many sequences from previous days.

He did this by relating the numbers to mile times since he was an avid runner. If you want to remember phone numbers you need to come up with a memory system to relate those numbers to something your brain can access more easily.

Here is the link if you want more information.