driver wobble is likely just the outer cup that wobbles, on a series40 that could mean one of your roller weights jumped out of it's track or is worn down,
it could also mean your cup got a heavy hit and needs to be fixed/replaced
or the sheaves for that matter...

unlikely but possible the crankshaft is bent or the bearings worn out that MUST be fixed/replaced!
driven wobble is likely caused by a bent jackshaft or worn out JS bearings
or again it took a heavy hit on the sheaves.. whatever's bent needs to be fixed/replaced.
now a single mm of wobble sounds not like much, and in fact is likely repairable rather than something that needs to be replaced.(whatever's cheaper frankly)
but as soon as something is not running true, it vibrates..
(reduced lifetime of bearings; if it transmits to the belt, that too can wear out prematurely etc..)
Different people accept different tolerances, based on tools available and costs to fix of course;
if something works as intended there's no immediate threat

But if you're unhappy/uncomforrtable with any wobble or vibration... then find the cause and fix it.
'sid