Gold and silver both ended the day down after a strong start.