The castle's history can be traced through various records from the late 1500s onwards, when it was home to the Leo family, Irish Catholics who maintained ownership through several generations. Richard Leas received a pardon in 1587, whilst Edmund Leos and his relative John Leos of nearby Dollagh were pardoned in 1600. When Edmund died in 1606, his son James, then aged 30, inherited the property. The estate comprised not just the castle itself but a substantial holding; the 1654-56 Civil Survey records it included a bawn, stone house, orchard and two mills.

The castle changed hands in the early seventeenth century when Sir William Parsons was granted Tullavin Castle in 1621, marking the transfer from the Leo family to Protestant ownership. By 1666, it had passed to Charles Ormsby. Architecturally, the structure follows a Type 1A tower house design, featuring a ground floor entrance leading to an entrance lobby with a spiral staircase on one side and a subsidiary chamber on the other. The building measures 9.6 metres by 5.4 metres and contains finely worked openings, though intriguingly, it appears to be only the end wall of what was intended to be a larger tower that was never completed.

Perhaps the castle's most distinctive feature is the sheela-na-gig carved on its outer wall at the southeast corner, one of Ireland's enigmatic female exhibitionist figures that appear on medieval buildings across the country. The structure includes practical defensive and domestic elements typical of its type, with window slits for defence, chambers across multiple levels, and a latrine accessible from the third storey passage. Both the castle and a watermill were significant enough to be depicted on the seventeenth century Down Survey map of Coshma Barony, testament to Tullovin's importance in the local landscape during this turbulent period of Irish history.