This is the site of the original castle that overlooked the town from Vivers Hill. It was built before the Norman Conquest and became the property of the Stuteville family who assisted William of Normandy in 1066. It existed as a wooden structure until the early 14th century but now only the deep moat remains. A stone castle in Castlegate replaced it around the mid 14th century.
The moated castle site on Vivers Hill can be reached from many directions. It is above the town behind the church.