When I shot IDPA, I wore a Sweatshirt or a Windshirt, which is what I actually wear on an almost daily basis in the winter. You can find inexpensive, loose fitting short sleeved warm up covers with back and underarm vents (baseball) at the used sports stores for like $10.
If you have not shot out at CRC when the wind blows, you might want to toss a gun cover and sand goggles in your range bag. They became permanent additions to mine after my first sandstorm out there.