Where should "Open link in new tab" and "Open link in new window" take you exactly if you are making an a tag behave like a button?

In order for the a tag to behave like a link that you may tab to you have to have the href setting. You may then turn around and make the a tag behave like a button by having a JavaScript event tied to it which first calls a function that does something and secondly offers:

return false;

 
 

But, someone can still ride the href setting somewhere if they really want to in modern times. Hell, they can click the center mouse wheel to kick off the "Open link in new tab" in Google Chrome. You should plan for what happens when this behavior occurs.