The early history and development of Downshire House is not clear but it is understood that work undertaken for General James Cholmondeley c.1770 can be associated with the building of the present house. Since 1770 the house has been much altered. From 1798 the house was owned by Arthur Hill, 2^nd Marquess of Downshire until his death, by suicide, in 1801. It was the Marquess who commissioned Robert Furze Brettingham to carry out works on the house, hence Brettingham is often credited with designing the house. 

Mary (née Sandys) Dowager Marchioness of Downshire then owned the property from 1802 until 1836. She spent most of her time at Hillsborough Castle in County Down and Ombersley Court, but she died at Downshire House in 1836. It was during this period that the house became known as Downshire House.

The owner from 1912 to 1920 was Sir Stephen Herbert Gatty. He commissioned the architect Oswald P. Milne to create a formal garden to the north of Downshire House on the site where the Cedars had stood, the Cedars (another of Roehampton’s Villas) was demolished sometime after 1910. Brick walls, stone balustrades with pineapples and a hipped-roof summerhouse with paired columns all survive from Gatty’s garden and were restored during the redevelopment work in 2016.

By 1949 Downshire House was the last of the Roehampton Villas to be used as a private residence. In 1950 the buildings and grounds were purchased by the London County Council as part of the Alton West Development. Outbuildings were demolished but the LCC preserved the Georgian house and it has been in educational use since 1963.

The existing building is of three storeys, six bays plus a north wing with bay-window, probably an addition. The most notable interior room is the central hall with a black and white marble floor and a Rococo plaster frame over the fireplace. The stair-hall is all 20^th century, but in the south-west room there is an exquisite late 18^th century frieze. The long narrow central room, facing the garden, has shallow 20^th century bow-windows flanked by two shell niches of early 18^th century character – possibly reused.