When I go to turn my dimmer switch off, it feels hot. Is this a fire hazard?

Dimmer switches are rated for the wattage load that they are designed to carry. If a dimmer is operating at or close to its design load, it will heat up. IF you are concerned that it may be getting too hot, either reduce the wattage (put in lower watt light bulbs in the fixture that the dimmer serves) or replace the dimmer with a higher rated unit.