That is nearly all that survives of what was once the principal residence of the Blennerhassett family, a house known locally, along with the older fortification it absorbed, as "Puck's House". The rest, the north wall, the west wall, most of the south, has vanished entirely, leaving this single fragment beside a spring called Puck's Well and a coach road that still runs through the wood.

The Blennerhassetts came to this part of Kerry through a transaction that has a particular period flavour to it. On 14th August 1590, Sir Edward Denny of Tralee granted the adjoining townlands of Ballycarty and Ballyseedy to Thomas Blennerhassett of Flimby in Cumberland, in exchange for the annual payment of six pounds sterling and the rendering of one red rose each year at the festival of the martyrdom of Saint John the Baptist. Thomas's son Robert subsequently settled in Kerry, first at Ballycarty Castle, a Geraldine fort on the south bank of the River Lee. The Geraldines were the great medieval lords of Munster, and their forts were solid enough that later settlers frequently built on or around them. Ballyseedy itself had been one such fort, and when John Blennerhassett successfully petitioned in 1654, following the disruptions of the 1641 rebellion and the Cromwellian period, to reclaim the Ballyseedy estate, the family moved there and built their house adjoining and incorporating the old fortification. The structure that resulted, raised around 1627 according to local tradition, took the name Ballyseedy Castle. A 1689 legal agreement recorded by MacLysaght gives a vivid sense of the agricultural world attached to it: hundreds of sheep, seventy cows, thirty-six calves, stud mares, garrons, ricks of wheat and oats, all carefully enumerated in a bond between John Blennerhassett of Ballyseedy and Charles McCarthy More of Pallis.

The most prominent figure associated with the house is Colonel John Blennerhassett, known as "The Great Colonel John" and described as the father of the Irish House of Commons, born here around 1691 or 1692. He was still recorded as living at old Ballyseedy as late as 1756, but on inheriting the estate in 1708 he had already commissioned a new country house, Elm Grove, at the eastern end of Ballyseedy Wood. After his death in 1775, the old castle was abandoned and left to ruin. By around 1821, Elm Grove had been extensively remodelled and had taken the Ballyseedy name for itself, while the original house subsided quietly into the trees. Fifteen metres to the west of the surviving gable, a complex of other structures, locally described as a folly, may represent ancillary or courtyard buildings, and a 17th-century mill stands roughly twenty metres to the north on the south bank of the River Lee. Ballyseedy Wood became a public park in 2007, and the ruins can be found near its western entrance.