The tower house, built around 1410 according to dendrochronological tests on an oak floor beam, was constructed by the Tyrrell family along the 'Bealach an Tirialaigh', or Route of the Tyrrells. This strategic position on what was once the main road from Dublin to Galway gave both the castle and the modern village their names; the route was commonly known as 'the Pass', and by 1837 the Ordnance Survey Letters noted that this same pass had likely given name to nearby Passkilbride as well.

The Tyrrells themselves had a rather colourful origin story, if a 16th century account is to be believed. According to this tradition, the family name derived from the Irish 'Treaeth', meaning 'banished for murder or treason'. The tale distinguishes between 'old Tyrrells' who arrived with the Norman conquest and claimed kinship with Walter Tyrrell, the man who allegedly killed William Rufus, and 'new Tyrrells' descended from James Tyrrell, infamous for his supposed role in murdering the Princes in the Tower. By 1640, Thomas Tyrrell held the castle and was listed as an 'Irish Papist' in the Down Survey, whilst earlier records from 1623 show John Tyrrell possessed the castle along with 270 acres encompassing Brackland, Tyrrellstown, Kilmurry and Rathcollomkill.

The five storey tower house, complete with pointed vault over the second level, double stepped battlements and two circular bartizans, fell into ruin before being restored in the 1970s by Laurence Ginnell. During restoration, architectural elements were salvaged from other medieval sites; ogival headed lancets from Syonan Castle and three two light windows from the ruined church at Kilbeggan now grace the structure. A circular flanker from the original bawn still adjoins the tower on the southeast side, near where Richard Tyrrel and Piers Lacy famously ambushed and destroyed an English force under Christopher Barnewall in 1597. Today, the restored castle operates as a restaurant, allowing visitors to dine within walls that have witnessed six centuries of Irish history.