I might just not get it, but what happens to your defense dice if you are attacked with an attribute or standing other than the one you picked for your attack?
Nothing. You use Defence Dice as normal. As others have said, its not meant to represent any specific type of defence, just how much ability you have to deal with threat of whatever nature.
If you want a reason for it being tied to an Attribute, think about it like this. A Skilled Swordsman is more likely to be calm and collected when in a fight if someone starts yelling insults at him, because of his skill and experience in what he's doing. However, an orator may be scared witless and be unable to respond to those insults at all.