The ball is in the buyer's court. The buyer likely can withdraw or ask to renegotiate during the appraisal contingency period, but until the contract is modified in writing, nothing has changed. If there is an appraisal contingency written into the contract, what does the contract say about what happens if the property does not appraise? At what point must the buyer decide? 21 days?
Does your agent have a copy of the buyer's loan approval? I would never rely on the buyer's assurances that they have loan approval. My guess is the buyer is being a bit of a jerk. Without full loan approval, he can't move forward with the purchase and there is no closing in three weeks. Have the inspection contingencies been removed? No point in going any further in price negotiations until all the issues are on the table.
In your shoes, I would tell the buyer to remove the appraisal contingency or move on. If the last unit like yours was priced at $357k and went under contract in a day, you should not have a problem finding a new buyer, unless your property is in poor condition.