It is known principally as a site of early Christian pilgrimage, and that reputation tends to overshadow the fact that the island also carries the remains of a cashel, a type of stone-walled ringfort that served as an enclosed farmstead or defended settlement in early medieval Ireland. Where earthen banks were the norm on lower ground, builders in the west of Ireland frequently used the raw material closest to hand, and the result is a drystane enclosure that has endured, in varying states of completeness, for well over a thousand years.

The cashel form was common across Ireland from roughly the sixth to the twelfth century, functioning as the basic unit of a farming household with enough status or resources to invest in permanent enclosure. On an island like Caher, the sea itself provided a degree of natural protection, and the choice to build in stone as well suggests a community settled enough to think in permanent terms. The island's name is itself derived from the Irish cathair, meaning a stone fort, so the cashel is not incidental to the place but effectively its namesake and oldest visible feature. The pilgrimage tradition associated with the island's oratory and other ecclesiastical remains suggests that a small monastic or devotional community may have existed here alongside, or growing out of, an earlier secular settlement of exactly this kind.

The island is accessible only by boat from the Roonagh Quay area near Louisburgh, and landings are weather-dependent given the exposed nature of the bay. Visitors who do reach the island tend to focus on the early Christian remains near the shore, but the cashel repays attention in its own right, particularly for the way it occupies its ground and speaks to a layered, pre-pilgrimage history of settlement on one of Mayo's more remote Atlantic outposts.