Ownership of the actual fence dictates who is responsible for repairs unless a second party has caused damage to it.
Typically the side the post's are on is the ownership side. At least that is the way it was explained to me.
thats how ive always done it, the pretty side is always facing out of the owners side as common courtesy.