You know, having two daughters myself, I completely understand your position; the upside is their brother kinda looks like you: big, scary, covered in tats... he used to head a group who had fun beating the crap out of each other...
Having said that, I also see ridin_low07's point; some time ago, my 23yo daughter was having issues with her partner; turns out, he was having issues & didn't want her to suffer as a result; in his mind, he was doing her a favour...
I'm looking at the whole picture, up to & including him calling you: it's possible that meeting your family for the first time overwhelmed him & his "fight or flight" response kicked in, his reason for leaving may have merely been an excuse...
I believe that, yes, you should sit down with him, with your hands in your pockets & your temper in check &, at least, hear him out; if you don't like what you hear, then react; what he has to say may surprise you...
Just my 2 cents; I hope it all works out...