I am not a lawyer.
It's hard to tell based on your description, but here are a couple of points:
1. It may depend on your jurisdiction. I'm assuming you're in the United States.
2. Money into escrow doesn't matter. It's the contract that matters.
3. Verbal doesn't matter. It's the written stuff that matters.
So, you shouldn't have removed the addendum verbally, and they should not have accepted your offer verbally. Since it is not clear that both parties agree on the contract, there is no meeting of the minds, which is a condition of a valid contract in the US. So in my opinion you don't have a contract. (Even if you did, the house isn't yours until the terms of the contract are met and all the papers are signed at closing. I suspect you know that and are just speaking loosely.)
What you need to do is basically figure out if they've accepted your addendum or not. If they accept your addendum and want to include it as part of the contract, they should (1) sign their copy of the addendum and (2) get a copy of their signed addendum back to you via the realtors. If they don't want to include it, they should tell you that and if you agree, you should (1) sign their counter offer and (2) get a copy of your signed counter offer back to them via the realtors.
Basically, with real estate contracts, you offer and counteroffer until either (a) one side breaks off negotiations, usually via going non-responsive on the other party, or (b) you have a chain of offers and counteroffers that both parties have signed on the last page of the last counteroffer.