The most common panic after someone shares a fantasy is the assumption that they are asking to act on it. Usually they are not. They are telling you something about their inner life.
Most fantasies are not plans
They work precisely because they are not real. Acting some of them out is disappointing for exactly that reason.
The theme matters more than the scene
Underneath most fantasies is something simple — being wanted, losing control, being someone else for an hour. The theme is usually more useful than the details.
You are allowed to say the fantasy is enough
"I love hearing about this and I do not want to do it" is a complete and reasonable position.
A fantasy about someone else is not a betrayal
It is close to universal. What matters is what a person does, not what appears uninvited in their head.