you're not radiating the cold into the object you're radiating the heat out of the object, directly reversing the process. It's the same reason your solar cells have diodes otherwise they'd charge during the day and discharge at night.
The circumstances have to be just right though. If you need solar refrigeration you're going to have to look into the modern icy ball projects that are going on. Unfortunately my patent/experiments into a multi phase icy ball have stalled due to lack of funding and time.