The house was rebuilt in 1728, much in the style it is today. Various architects have been linked with the re-design of the house, including Hawkesmoor and Giacomo Leoni but this has not been established beyond doubt. In 1761, Horace Walpole visited the house and commented favourably on a statue of Inigo Jones “a niche on the noble staircase”. The statue – our statue now – is still there today. Later, George Byfield and Lord Burlington became involved in some design modification.
Later still, from 1834 to 1859, the house was occupied by Earl de Grey, who was the first president of the Royal Institute of British Architects. From 1912 to 1942 the house was owned and occupied by the second Viscount Astor. In 1942 it was requisitioned by the government and was used by the London headquarters of the Free French Forces.