While local tradition attributes its construction to one Carbery O'Shea, the castle's history is more firmly tied to the MacCarthy clan. Records suggest some form of residence existed here as early as 1398, when the annals note Taghd Mac Carthaigh's death at this location, though the current ruins likely represent 'the castle of Valencyen called Ballycarborow', mentioned in documents from 1569. Throughout the sixteenth century, the O'Connells occupied the castle as MacCarthy wardens; Morgan O'Connell of Ballycarbery notably served as High Sheriff of Kerry during Elizabethan times. Following the death of Daniel MacCarthy More, Earl of Clancar, in 1596, the castle passed to Sir Valentine Browne, but its fortunes took a dramatic turn when parliamentary forces slighted it in 1651-52 whilst fortifying Valencia Harbour.

The castle complex, which Leask dated to the fifteenth century, comprises a fine ivy-covered tower house surrounded by the remnants of a defensive bawn wall. The original bawn enclosed a substantial area measuring approximately 31.6 metres north to south by at least 35 metres east to west, though much of the southern and eastern walls of both the keep and bawn have been destroyed, presumably during the mid-seventeenth-century slighting. Early nineteenth-century watercolours by Daniel Grose show large masses of faced masonry lying on the south and east sides of the castle; these fragments, along with portions of the bawn's south wall, were unfortunately removed in the early twentieth century.

Today, visitors can still appreciate the western wall and western portion of the northern wall, which rise from a slightly battered base of 1.65 metres thickness to an average height of 4.2 metres. The defensive features include narrow vertical loops set in splayed lintelled embrasures, whilst the remains of an alure runs along the top of the wall. Near the northern end of the western wall, a short stair descends to a trapezoidal mural chamber that was later modified to include a garderobe. This modification may be connected to the construction of a large house that abutted the exterior of the bawn wall, probably in the eighteenth century. This house, which served as the Lauder family residence, was demolished in the early twentieth century but can still be seen in Grose's watercolours.