I usually do. Every now and then, when I know I'm just doing a few minutes of goofing off at lower speeds, I won't bother. Like if I feel like just doing a few donuts in the gravel. My karts far too heavy to go over doing donuts in the gravel, believe me- I've given' 'er my best! On the other hand, If I'm doing high speed passes up my road, then yes! Gravel road, lotsa potholes, 45MPH+. Helmet mandatory. The worst that's ever happened to me was losing control on a particularly loose section and going into the ditch. Fortunately, I'd managed to slow down quite a bit first and nothing was hurt...
Oh, yeah- If the kids are watching, then I do my best to wear my helmet no matter what- gotta set a good example!