I'm not sure what the answer is, but my gut is telling me it's not the rifle (is this the only one you load .30-06 for?). It doesn't sound like a crown issue... those tend to just throw bullets everywhere every time.

Are you squaring up your dies? I'd look at bullet runout as the first easy thing to check. You might find some are perfectly concentric and some are .006" off center. That will cause a flyer. Have you put the loads over a chrono? Are they consistent? Are you neck sizing only? What brass / powder / primer combo are you using?

When you pull out the bolt, do you have even wear on all four sides of the locking lugs? The action may need to be trued up / blueprinted.

How's the trigger? Is it one of the old lawyer-proof kind?