Fahan Castle, once granted to Sir John O'Doherty, met its demise around 1600 when it was partially dismantled, leaving behind only tantalising fragments of its former glory. By 1601, English observers noted the castle was "broken down" though they could still discern its original form; a square tower surrounded by a circular bawn, which served as a defensive wall. The castle stood opposite Inch Island in what was then O'Doherty territory, and despite its ruined state, it continued to shelter the Bishop O'Galthar.

The castle's exact location remained uncertain for centuries, though Castletown House, situated 400 metres south of Castletown Cottage, emerged as the most likely site. This theory gained credence in the 1940s when renovations at Castletown House uncovered intriguing architectural evidence. Workers extending the building discovered vaults or arches adjoining the north wall, and amongst the rubble lay a remarkable find; the head of a single light, ogee headed window, its surface punch dressed in distinctive 16th century fashion. The window fragment, with its sunken spandrels and rear chamfer, bore a bar hole at its top, suggesting it may have been repurposed as a door jamb after the castle's destruction. These vaults could well have been the remains of the barrel vaulted roof that once covered the castle's ground floor.

Archaeological investigations in 2007 sought to uncover more evidence of the castle during pre development testing at Magherabeg. Five trenches were excavated within the archaeological constraint zone, revealing layers of grey clay and loamy sand beneath the topsoil, along with modern pottery sherds, water services, and ceramic land drains. Despite these efforts, no trace of the circular bawn mentioned in historical accounts could be identified amongst the buildings shown on Ordnance Survey maps, and the testing revealed only natural, undisturbed stratigraphy. The medieval window head and a bullaun stone remain visible in the farmyard adjacent to Castletown House, serving as the sole physical reminders of this once significant O'Doherty stronghold.