Motor mounted out of square to the frame does mean misalignment, which means chain walking off the sprocket as it rotates. Same way you'd try to take the chain off by pulling it to one side and slowly rotating the sprocket.
Apologies for negativity here, but that homeowner-grade rafter square is for purposes other than machining and millwrighting. It's not long enough or true enough and for many applications not compact enough.
Get a REAL straight-edge, metal or metal-edged not to warp or bend out of true, that will bridge across the entire width of that axle sprocket and still reach all the way across the other sprocket.
I've used a 1" wide aluminum yardstick all my life to align pulleys, sprockets, bearings, etc...... Floppies itself into tight spaces, then becomes a dead true edge when turned on its side. Angle iron and similar can't be depended on to be true straight.