Two of its most prominent architectural features, a west window and a west doorway, were not made for this church at all. They were salvaged from a medieval Augustinian abbey roughly three and a half kilometres away, and quietly incorporated into a later building, where they have sat ever since.

The abbey in question is at Fertagh, in the townland of Grangefertagh. It was an Augustinian foundation, and its church survived long enough for at least some of its stonework to be considered worth preserving. In 1799, the west window was removed from the Fertagh church and rebuilt into the Johnstown church, a process recorded by the historian William Carrigan in his 1905 history of the Diocese of Ossory. The west doorway made the same journey, though the precise date of its transfer is less clear. Medieval ecclesiastical stonework of this kind, with its carved surrounds and careful proportions, was sometimes relocated in the eighteenth and nineteenth centuries when older buildings fell into ruin and local builders or patrons wanted to retain something of the original fabric. The result is that Johnstown now holds architectural material from two quite different periods and two quite different purposes, layered together into a single structure.