Google made a lot of interesting changes in its Quick Settings and Notifications drawer in Android Lollipop. One of these is the addition of dynamic toggles that don't clutter the drawer for everyone, but only appear once a user activates the corresponding option from Settings. This applies for example to the Hostpot and Invert Colors toggles. The problem is that once these toggles attach to your Quick Settings, there doesn't seem to be a way to make them go away, even when you switch the action back off.
User eak125 on Reddit tried to figure out if there was a timer attached to this behavior or if the toggle would be permanently stuck in the drawer. Turns out that by manually setting the date forward a month, the toggle would disappear. But it's safe to assume that if you enabled the Hotspot for example and you kept on using it, it wouldn't go away, unless you don't touch it for a month, then it should take itself off the drawer. Obviously, this doesn't affect the default toggles (Wi-Fi, Bluetooth, etc...), which will always be in your drawer regardless of how often you use them.
But you don't want to completely change the date of your phone if that Invert Colors toggle is annoying you to tears, do you? Reddit user Stark_Tony suggests that instead of setting the date forward, you manually go back a month and a day (or more), click whichever dynamic toggle you want to get rid off twice (to enable then disable it), then go back to today's date. Abracadabra, poof!