Gotcha. I'm not trying to talk you into or out of anything, but there are quite a few good options out there in bolt rifles from dirt cheap to holy cow in price. I have a couple on each end of the spectrum, and they are a ton of fun to shoot. If you're wanting the ability to pull the trigger real fast, that obviously disqualifies a bolt gun.
One thing that might play in your decision is this. I have several semi-autos, and with the subsonic target ammo I regularly shoot they don't always cycle reliably. Just not enough energy to run the bolt. I've worked around it by changing recoil springs, etc, but it wan't my favorite task.