Physical Damage is inflicted by melee attacks and some spells. This kind of damage can be sometimes blocked by the target's Shielding (impossible to block when inflicted by spells), and is always mitigated by the target's armor.
Magical Damage cannot be blocked, and the damage it inflicts is not reduced by the target's armor. The primary drawback to magic damage is that many creatures are immune to different types of magic damage, and therefore are not affected by it. Ghost, Pirate Ghost, Spectre and Phantasmare the only creatures that are immune to physical damage.