The present building, a four-bay, two-storey block with hipped roofs and a 19th-century look about it, is in fact the restored servant's wing of a far older and grander structure. The main house is gone, consumed by fire in 1887, and only a short section of wall on the north side survives from before that disaster. Yet embedded in the masonry of a square block to the southwest is a stone plaque bearing the initials IEG and the date 1578, a quiet signal that the ground beneath has been accumulating history for considerably longer than the current building suggests.

Those initials belong to John FitzEdmund FitzGerald, who is believed to have built a house on this site when he took possession of the See demesne, the lands attached to the Diocese of Cloyne, in the late 16th century. The site had long served as the Bishop's palace of Cloyne, and in 1718 Bishop Crowe replaced whatever stood there with a new residence known as the Manse House. That building survived for over a century before being leased out as a private residence in 1836, at which point the older portion of the palace was apparently demolished. The fire of 1887 then accounted for most of what remained, leaving the servants' quarters as the unlikely heir to several centuries of ecclesiastical occupation. At the northwest corner of the grounds, a shell of a fortified house adds another layer to the site's long occupation. Near the street entrance to the north, a hexagonal gate lodge of late 18th-century appearance survives in better condition, an unusual geometric form that tends to catch the eye of passers-by on Rock Street before they have even registered the grander story behind it.