Did your rotor come with your caliper in a kit?
If not, I was thinking your caliper might be designed for a vented rotor that's a lot thicker; and this might be resulting in your caliper reaching its max limit of travel before the pads can apply sufficient braking force.
I know some racekart calipers require shims installed between the piston and backing plate of the pad to help compensate for rotor thickness & pad wear...