The building itself rewards close attention: blocked doorways hidden behind refaced masonry, a large pointed arch inserted into the nave wall for a transept that was apparently never built, and a decorated transom stone whose ivy-leaf and knotwork carvings suggest it was salvaged from somewhere else entirely before being cut down to fit its current window.

The friary's foundation was set in motion by Edmund Butler, who obtained permission from Pope Pius II in 1461. Following Edmund's defeat in battle the following year, it was his eldest son James who became the effective founder, and who was buried here in 1487. According to the historian Carrigan, James had been living in concubinage with his relative Syve Kavanagh; after receiving a Papal dispensation allowing them to marry in 1467, he founded the religious house around 1468 to 1469 in reparation for what he regarded as a sinful past. The community quickly gained standing: by 1472 it had been placed directly under the prior general of the Observant reform and declared independent of the English provincial, and by 1479 Callan had become the centre of the Irish Observant Congregation. Before the Dissolution of the Monasteries in 1540, the friary was known for its learned community, its manuscript library, which reportedly held duplicates of rare books from Jerpoint Abbey, and its care of the poor. At dissolution, with William O'Fogarty as prior, the three-acre site passed first by lease to James Butler, Earl of Ormond, grandson of the founder, in 1541, and then into the possession of Thomas, Earl of Ormond, in 1557 to 1558.

The stonework carries deliberate echoes of Butler patronage elsewhere. The mouldings of the west doorway closely resemble those at the 15th-century Cistercian monastery of Holy Cross, another Butler foundation, and the sedilia in the chancel, a stone seat used by clergy during the Mass, similarly mirrors the arrangement at Holy Cross. This appears to have been intentional, signalling a connection between the Callan branch of the family and the main Butler line. After the Dissolution, the crossing-tower, the tall square structure separating nave and chancel, was adapted for domestic use: a fireplace and slop-stone were inserted at ground level, and flat-headed windows cut into walls that had previously held liturgical openings. Two grooves in the west arch of the tower, 2.95 metres above the ground, mark where a rood screen once divided the space. The friary was restored by the Board of Works in 1961.