Don't file, grind, or other wise open the holes.
Assemble the upper to the lower, then take the weapon out to fire. You will see that eventually, through use, the upper will mate to the lower.
Each time out, the Take Down and Pivot Pins will gradually get easier to remove and install.