'I am off to my yoga class' = I am going to a place where I will roll out a mat and do some weird shapes.

But technically speaking if we are talking about the postures we perform in a yoga class, we should be calling it Asana class rather than Yoga class. But that's like pointing out that a hoover should be called a vacuum cleaner and ping pong is the name of the brand of a table for table tennis...pointing that out would be pedantic, wouldn't it?