The name comes from an island, Inis Pádraig, St. Patrick's Island, which sits 2.1 kilometres to the north-east in Dublin Bay. The graveyard and its 18th-century bell-tower occupy a site on the mainland, 720 metres south of Skerries and about 200 metres from the coast, and beneath the present ground level lies the levelled footprint of a medieval Augustinian priory. The Augustinians were a mendicant order of canons following the Rule of St. Augustine, and the priory here was not an original foundation but a relocation, the institution simply picking up its name and transplanting it to the shore.

The move happened in 1220, when Henry of London, Archbishop of Dublin, transferred the convent from St. Patrick's Island to this more practical mainland site. The old island name was kept, at least for a time, and a prior named Philip was still being recorded under the title "Prior of the Island of St. Patrick" as late as 1229. The priory accumulated considerable holdings over the following centuries: at its dissolution in 1537, the 3-acre site included roughly 1,000 acres of land, numerous messuages and cottages (a messuage being a dwelling with its surrounding land), a water-mill, and a rectory, all valued at £69 8s. 6d. The last prior, Peter Manne, died in 1537; his successor Philip Corre was granted a pension of £8 upon the dissolution that December. The property passed through several hands before being granted to Thomas Fitz Wiliams in 1578. By 1654-56, the Civil Survey of Dublin recorded only "the walles of the parish church" standing on lands belonging to the Lord of Thomond of Bunratty. The Hamilton family acquired the estate in 1720, and the 18th-century bell-tower that now marks the site dates from around that era of new ownership.

The graveyard is findable on the southern edge of Skerries, close to the coast road. The bell-tower stands at its centre and is the most visible indicator of the layered history underfoot. Medieval floor tiles were recovered from a grave 37 metres to the east-north-east of the tower, physical traces of the priory church whose walls had already disappeared before the Civil Survey was written. Visitors with an interest in early medieval island monasticism might also look north-east from the shoreline nearby, where St. Patrick's Island, the original home of the community, is visible in the bay.