Look to see if there's a little check valve on the fuel return line. Those primers are crotchety at best, and if the check valve's bad, that would explain the primer coming and going mysteriously. The valve usually looks like a little black square plastic gizmo spliced into the line that returns excess fuel to the tank.
You can test the theory by removing the fuel return line from the tank, and pointing it into a suitable container. Push the primer bulb in and hold it. With the other hand, put your finger over the end of the return line and as you hold it there, release the primer bulb. Then unblock the tube, push the primer and hold... Etcetera.