I take it off about halfway through, to give the glue as much time as possible to loosen. I find that if I try after only a few minutes, the glue still holds.
Side question: When the glue is strong, does everyone slide the band off the end, or does anyone try to peel the end of the band up?
When I slide off, occaisonally a fold of the wrapper gets caught and sliced off. But whenever I've thought about trying to get the end up, it seems like it will do more damage as I'll have to practically pinch the side of the cigar...