Somebody once asked Abe Lincoln how long a man's legs should be. "Long enough to reach the ground." was the answer.
Small tires are for smooth roads and light karts. Large tires, especially balloon tires, are suited for rough terrain and off roading.
You will "gain or lose torque/speed" if you leave the gear ratio the same but change tires. Better to choose the tire according to the size and type of vehicle, and the type of riding you plan to do, THEN gear your kart accordingly. You can have the same off-the-line and hill climbing speed and torque whatever the tire size, if you design the correct gear ratio.
Larger diameter tires roll over obstacles more easily, absorb shock and avoid bending wheels and suspension better, and grip better in sand and gravel where smaller tires would quickly bury themselves.
Smaller, lighter tires are best for smooth pavement karts, where diameter offers little advantage and the extra weight affects takeoff speed and sometimes (where fractional losses count in competition) offer better rolling resistance and aerodynamic profile.
Use one of the online gear ratio calculators and see what playing with tire size and gear ratio does.
