Where A-listers retreat once the Hamptons quiet down