No worries, I don't care much myself

I'd still add a few ounces of oil (penetraiting or other if you like)
just to stop the rust from eating away the tiny section of the steering shaft that holds onto the steering wheel.
since in the end.. it's still rust eating away the steel eventually

and the steering shaft by nature has a pretty large hole in it (for the through bolt)
and what's left isn't exactly beefy in the first place.
And if it's ugly or not is less important than if the steering shaft cracks or not,
especially in rough terrain
(unsuspended bumps nicely transmit into the steering wheel, don't they?)
Again, I am not saying you must, not even saying you should..
..I'm just saying I don't see no harm in removing rust
it's not even restoration.. it's closer to maintenance really
(one bolt, a quick lick with the sandpaper and a squirt of oil, right?)
'sid