It was born as a fortified residence of the Count Bishop. It was a castle with strong walls and towers overhanging the rocks below. The tallest tower, called Fuso, stood near the entrance gate to control the area in front of the castle. Near the tower of Fuso, to reaffirm its independence, the Free City raised the mighty Torre del Candeliere, taller and thus more formidable. The tower of Fuso was turned into a gunpowder deposits and exploded in 1560. The castle, after the demolition of the its towers, took its present aspect at the behest of the House of Lorraine. The renovation works were completed in two times, first in 1744 and after in 1845, and turned the building into the most important and efficient hospital of Maremma of that period.