The village of Castiglione retains the remains of two walls. A first fortified perimeter was built by the Pisans in the tenth century and marked the boundary is the village situated on a hill, is the site located in the lower part. With the passage of time, the continuous pirate raids led the inhabitants to move all at the top, leaving uninhabited area near the river. During the 15th century, before the Aragonese and then the Sienese carried on the construction of the new walls that delimit completely the current village located around the Castle. In the walls of the castle there are the three gates of Porta San Giovanni, Porta della Madonna del Giglio and Porta Castello; in the lower part, the ancient Pisan walls survives the old northern gate, commonly called the Portaccia and two towers, one at via Colombo and the other nearby known as Lilli, while Tower are the remains of a third tower in via cows.