I am not sure about the formulas but damage is calculated with (stat*x)*(typedamage*y) or (stat*x)+(typedamage*y)
Where x y is secretsause to me but smaller numbers on typedamage seem to have a larger impact. In short +3 magic up is usually less than +3 magic attack up and both stats will stack almost always.
Magic% up is a percentage magic stat buff to your unit, based on the units base magic stat. Magic Attack Up is like someone stated a modifier to a certain attack type (every magic based attack or spell). What makes these so good is that every attack/spell is based on the units stat (mag/atk) and level of the skill. This Mag Atk Up is a modifier added afterwards or how I should say it.
....
Go Magic Atk Up if your character MAG is low at the moment (maybe because Esper resonance is low or equipments) because it directly strengthen your skills.
MAG got blocked by SPR ( just like how ATK got blocked by DEF),
While Magic Attack type got blocked by magic resistance ( just like how Slash type attack damage got blocked by Slash resistance).
Both are contributing in the overall MAGIC type damage, however please note that some job such as spellblade use Slash type attack with MAG scale, thus it target SPR but got reduced damage from slash res