Initially castle, built as a defensive bastion of the Republic of Siena, was transformed in the sixteenth century in residence by the owner Mino Celsi. The sixteenth century restoration is attributable to the architect Baldassarre Peruzzi, who certainly intervened in the design of the circular chapel and in the external arrangement of the terrace walls.
In May of 1554 the complex was destroyed by the Austro-Spanish imperial troops of Charles V. During the XVII century, at the same time as the restoration of the building, some of the wonderful gardens were built, an expression of Baroque culture. In 1802 the property passed to the Chigi family, who transformed the residence into a neo-Gothic manor.
A further restoration of the dwelling and the garden is due to the Princes Aldobrandini, owners of Celsa from the early twentieth century.
Despite the numerous interventions undergone over the centuries, the building retains its sixteenth-century character while retaining the original castle, only the south corner tower.