There is a motte and bailey castle on the south-east side of the main street, a ruined parish church and graveyard roughly forty metres to the south-west, and, facing them from the east side of the street, the remains of Delvin Castle itself. A motte and bailey, for context, is an early form of Norman fortification: an earthen mound topped with a timber or stone keep, paired with an enclosed courtyard below. That two such distinct castle forms survive within the same small town is unusual, and it points to a settlement whose importance outlasted any single phase of building.

Delvin was probably founded by the Nugent family in the late twelfth or early thirteenth century, part of the broader Anglo-Norman push into the Irish midlands. By the early seventeenth century it was being described as a former market town, the phrasing suggesting something already faded. The 1659 Down Survey map of Castletown parish captured it in a terrier, a written description accompanying the survey, as a place of ruined walls and small cabins, once the chief seat of the Earl of Westmeath. By 1682, Sir Henry Piers was still impressed enough by the physical scale of Delvin Castle to write about it at length, describing a large oblong square castle, high-raised, with a great round tower at each corner that he felt equalled, if not surpassed, the main structure itself. He called it a structure speaking ancient magnificence, and then noted, without apparent contradiction, that it was wholly waste, without roof or inhabitants. The title of Lord Baron of Delvin had by then passed to Richard Nugent, Earl of Westmeath, whose family Piers traced back to one of the first English conquerors to settle on the site.