Two short stretches of rough limestone wall, tucked into convent grounds and running up against what was once a town gate, are almost all that remains of a medieval defensive circuit that once ran for nearly one and a half kilometres around a pear-shaped settlement in south County Kildare. The town carries the name of a castle, yet no castle survives within it. Its Irish name, Diseart Diarmada, meaning Dermot's Hermitage, gestures back to something far older than either walls or castle.

The site's origins lie in 812 AD, when a church was founded here and attached to the monastery at Bangor. It was plundered repeatedly, in 842, 867, 1040, and 1106, that last attack probably consuming a wooden structure. The earliest physical remains still visible belong to a slightly later period: a round tower and a twelfth-century doorway. The town's medieval character took shape after the Anglo-Norman settlement, when Walter de Riddlesford was granted the territory around 1181. Between 1199 and 1216 he founded a priory of the Fratres Cruciferi, a religious order sometimes called the Crutched Friars, and a borough was established. A murage grant in 1295, a royal licence allowing a town to collect tolls in order to fund the construction of defensive walls, suggests the walled circuit may have been formalised around that time. Those walls originally enclosed an area roughly 550 metres by 375 metres, pierced by three gates: the Dublin Gate to the north, the Friary Gate to the south, and the Carlow Gate to the west. None of the gates survive. The castle that gives the town its modern name was built by Gearoid More, the eighth Earl of Kildare, and was in existence by 1507, though nothing of it remains above ground today.

Of the town wall, two sections are still visible. One lies in the convent grounds near the friary site, measuring roughly 13.5 metres long and 2.5 metres high. The other, just to the north, runs into the position of the old Carlow Gate and stands about 3 metres high over a length of 11 metres; at that junction there is a wide basal batter, the sloping thickening at the wall's base designed to resist undermining, as well as two bar holes, the sockets that once held the timber bar used to secure a gate. The course of the eastern wall was likely defined by the River Lerr, though its precise line is no longer clear.