I have one of dem HF "drill press" thingys. I hate the SOB! its good for 1/4 inch bits but nothing more. The micro fractional HP motor has less power than an electric toothbrush. I've been tempted to just throw it in the trash on several occasions.
You can make it work, as you have done, but it works best if ya increase bit size in very small steps once ya get over 1/4.
Problem with my real press is it won't take bits smaller than 3/16, cuz it has a 5/8 chuck capacity. Its gear drive so it either hogs its way through or just snaps bits off. You are using the best drilling sauce!
Again your creativity overcomes!
Gear drive? Holy sh*t

no wonder your breakin' drill bits
Yup

these little bench top units are mostly good for drillin' smaller holes
I've been usin' that little guy for 10 - 15 years
...& it does pretty good up to ~5/16"
Speakin' of that, last week, after drilling them rollers out to 5/16", I noticed that the chuck kept jamming up
...while trying to change drill bits
It got to the point where it would only adjust to a little bit bigger than, 5/16"
...& then, just a little bit smaller, than 5/16"
Upon examination it looks like one of them (3) "grippers" inside of the chuck ain't slidin' too good
Yup, I lubed the crap out of it
...& worked on "manually" forcing he "gripper" to move, with its counterparts
...but, It still wasn't cooperating too much
So, I was a thinkin' Ima need a new chuck
...but, then thought "Hmmm, I wonder if there are any used drill presses available, here in Toledo, on the FB marketplace, for a good deal"
So, I looked & found a bench top unit with a supposedly bad cord ($30.00)
...the ad said "it will work if you squeeze the cord"
So, I sent 'em a message "Is this still available?"
...& kept looking
Then, I found an older stand up unit, supposedly works good "just don't need anymore" ($100.00 OBO)
...so, I sent 'em a "Is this still available?" message too
A couple of hours later, they both sent "yes, it is" messages
So, I went & looked at them
...& yup, I'ma gonna give both of these orphans a new home

They were pretty firm on the $30.00 for the bench top unit
It a Tool Shop 5- speed unit with a 1/2" capacity chuck

Another view

I was able to get the stand up unit for $75.00
It's also, made by Tool Shop & it's a 16-speed with a 5/8" capacity chuck

Another view
They are both older units
...but, work good
...& I have already put a couple of hours on both of them, while drillin' these rollers
