While you are not entirely wrong about the glitter – some very large-particle glitters can eventually work themselves out of soft silicone – there are plenty of safe, fine-particle glitters that are perfectly fine to use in silicone toy-making.
Contrary to what you see at several stores, glitter is not safe for silicone. Every movement causes micro tears in the silicone both inside and out. This will lead to tiny holes on the surface which collect bacteria and are impossible to clean without sterilization. Over time the dildo will break down. Look into investing in large particle dazzle pigments that are free of glitter.
The Dicktionary 