That peculiar geography was not lost on whoever first fortified it. Dún an Fheirtéaraigh is a promontory fort, a type of enclosure in which natural cliff edges do most of the defensive work and earthen banks and rock-cut fosses, essentially ditches, seal off the landward approaches. Here those defences are unusually elaborate: two separate lines of banks and fosses divide the promontory into distinct zones, each crossed by a narrow causeway. The western interior, now barely accessible, holds the ghostly impressions of at least nine circular hut-sites, their floors cut up to two metres down into bedrock, as well as thirteen or more oval hollows along the cliff edges whose purpose remains genuinely unclear; they may have been temporary shelters, though the question is still open.

The human story thickens considerably in the later medieval period. The Norman family of Ferriter had acquired extensive landholdings here by the end of the 13th century, and by the 16th century the cove below was recorded as a port of some importance. Sometime in the later 15th or 16th century the family built a tower-house directly onto the inner bank of the eastern defences, its eastern wall resting on boulder foundations at the base of the fosse. It was marked on maps of the period as 'Filliters Castle' or 'Castle Sybil'. The Ferriters remained there until the mid-17th century, when the family's most famous member, Piaras Feirtéar, a poet of considerable reputation in the Irish-language tradition, was captured and executed in Killarney in 1653. The castle was already ruinous by 1756, and a severe storm in 1845 brought down the main block, which had previously retained a stair turret and entrance in its south-west corner. What survives today is the north-west angle, standing roughly eleven metres high, along with fragments of the north and west walls and the lower courses of the south-east corner. The remaining masonry preserves details worth examining closely: coursed sandstone flags in sandy mortar, widely chamfered quoins, a double-splayed window loop in the west wall, two wall-cupboards, and beam holes that suggest a second floor once existed beneath the pointed barrel vault, now entirely collapsed.