violett, I suspect you have missed a step that you need to do before you can take the collar off. I kind of remember having that happen to me too. I retraced my steps and found I needed to do something else first. It's been a few years since I played the game, so I don't remember exactly what it was though.
Heading out to the gardens!