Nothing of it is visible at ground level today, yet two separate maps from the mid-seventeenth century record it in considerable detail, showing a substantial crenelated building enclosed within a bawn, a defensive walled enclosure typical of Irish manorial complexes, complete with angle towers and a central gatehouse or tower midway along the foreground wall.

The palace was built by Hugh Mapleton, Bishop of Ossory, sometime between 1251 and 1260, at a place then known as Aghore, later Uppercourt. According to the historian Gillespie, Mapleton furnished it with fish-ponds and what the sources describe as "other necessary adjuncts of a manorial residence", suggesting a functioning episcopal estate rather than a purely ecclesiastical retreat. By 1640, the property had passed into secular hands, with a Robert Shee listed as proprietor. The Down Survey maps of 1655 to 1656, produced as part of a vast Cromwellian land-mapping exercise, captured the complex in two separate drawings, one in the Freshford parish map, one in the Crannagh barony map, each showing a fortified residence of considerable scale. The consistency between them suggests the palace was still a recognisable and significant structure at that point. Its end came quietly, in the late eighteenth or early nineteenth century, when Sir William Evans Morres constructed Uppercourt House nearby. A square castle that had stood adjacent to the episcopal residence was razed, its stones incorporated into the new house, and it is likely the palace met the same fate around the same time. The ground absorbed both structures without ceremony, and the new house rose partly from their bones.