It's a little ironic that he's telling people who paid $35,000 a plate for dinner that they'll have to fend for themselves