Scotland, Scottish Borders, Hermitage Castle. Hermitage water running in front of the Hermitage Castle. The castle has a somewhat colourful history with numerous stories of war, treason and murder being associated with the property. It was built in an area considered to be a key element in the English and Scottish wars for control over the Liddesdale and the border area.