How you can clean them depends on the material you've chosen for your label or sticker product:
Polypropylene vinyl (or other plastic)
As they're waterproof, these stickers can be cleaned with a damp cloth and even a little soap if required. If this risks causing damage to the surrounding surface, you can wash over it with warm water and pat dry instead.
If the issue is just some dry surface dust, it may be easier to use a dry cloth (such as microfibre) to gently wipe it off.
Either way, make sure the edges are stuck down firmly before you do this, as any repeated friction could catch on the corners and cause peeling. Always start off gentle, and preferably wipe from the middle of the sticker outwards to avoid lifting the edges of the sticker.
Paper
Unfortunately paper is much more difficult to clean as it will absorb liquids.
If it's dry dirt such as dust, you can probably wipe this off with dry cloth (such as microfibre). A fluffy duster can work well too.
However, if it's embedded dirt or contaminants that's absorbed into the sticker, you will likely have to replace it.