You would do better to stand your ground. If it’s already gone to an agency because the original creditor has given up on the debt, the longer you hold them off, the more likely they are to settle for less. I won’t ask about why or if you owe the money; that’s your business.
If you have instructed the agency not to contact you by telephone, you’re halfway home. They rely on pressure and intimidation, and taking the phone away effectively pulls their teeth.