What makes Toem worth a second look is the quiet accumulation of history beneath it, layer upon layer, stretching back at least to 1302, when a church here appears in the Plea Rolls, making it one of the earlier documented ecclesiastical sites in the county. The tower is all that physically survives of what was once a functioning parish church, and the headstones around it date only from the 18th and 19th centuries, leaving a long earlier chapter largely unreadable in stone.

The site's history is one of interruption and revival. By the time of the Down Survey of 1655 to 1658, a systematic mapping of Ireland carried out under Cromwellian administration, the church was already described as ruins. It was rebuilt after the Restoration, appearing in a church list of 1667 to 1670, and in the 1690s the White family carried out renovations. The Church of Ireland tower that now stands was constructed around 1805, possibly directly on the footprint of the earlier medieval structure. Local tradition adds a further layer: a monastery is said to have occupied the field immediately to the north of the churchyard, though no physical trace of it has been identified above ground. Whether that tradition preserves a genuine memory of an early religious community or is simply the kind of story that tends to attach itself to old ecclesiastical sites is difficult to say, but the elevated position of the churchyard, commanding good views in every direction, is exactly the kind of place early Christian communities favoured when choosing where to settle and build.