Nothing about its present appearance obviously signals that this ground has been, at different moments, a medieval religious college, a fortified stronghold trained with artillery on its own town, and a roofless ruin. The layers have been so thoroughly absorbed into one another that only a single ornate chimney-piece and two surviving towers connect the building to its earlier lives.

The site began as 'Our Lady's College of Youghal', founded in 1464 by the Earl of Desmond. It was a collegiate institution, meaning a community of clergy organised around a shared rule and shared revenues, in this case a warden, eight ordained Fellows, and eight lay brothers. Beyond teaching, those Fellows served St. Mary's and seven other churches scattered across the Barony of Imokilly. That arrangement ended badly during the Desmond rebellion of the late sixteenth century, when the college was, in the words of one account, 'spoiled and well nigh demolished'. Into the wreckage stepped Richard Boyle, the first Earl of Cork, one of the most energetically acquisitive figures in early modern Irish history. He acquired the property and around 1605 rebuilt it as his private residence. Then, in 1641 and 1642, with rebellion spreading again, he substantially militarised it, adding two large flanking towers, five circular turrets around the park, and an earthen artillery platform designed to command both the town and the harbour below. Boyle died there in 1643. By the middle of the eighteenth century the house had fallen into ruin, and a substantial rebuilding around 1782 erased so much of what remained that a contemporary observer noted hardly a vestige of the old college could be found. The Gothic wing came later still, in 1860, giving the building the outline it largely retains now.

The two towers from Boyle's fortified residence are the most tangible remnants, one of them folded into later construction. For a place that has witnessed the collapse of a Munster earldom, the ambitions of one of Elizabethan Ireland's great self-made magnates, and multiple cycles of ruin and rebuilding, the chimney-piece quietly holding its ground inside a still-inhabited house is a quietly strange kind of survival.