This land grant, recorded in the Calendar of Ormond Deeds, suggests that an earlier earth and timber castle likely stood on the site before Walter's arrival. The original fortification may have been a ringwork castle, with the rock cut ditch visible to the west of the later stone castle possibly forming part of these early defences. When Walter constructed his stone castle, he appears to have followed the outline of this earlier ringwork structure.

The castle served as the administrative heart of the medieval manor of Arklow, which originally lay within County Carlow. Throughout its history, the fortress changed hands between English and Irish forces multiple times. In 1331, Lord Bermingham successfully defended it against an attack by the O'Tooles, though the Irish briefly captured it the following year before the English reclaimed and rebuilt it in 1332. The castle remained strategically important through the centuries; in 1594, its great hall hosted legal proceedings to settle disputes among local inhabitants, whilst various leases from the 16th and 17th centuries detail requirements for tenants to maintain and defend the castle walls, even specifying annual payments in barrels of herrings alongside monetary rent.

Today, the castle ruins consist of a circular corner tower and sections of curtain wall built on a rock outcrop overlooking the River Avoca. The tower, constructed from randomly coursed rubble stone and dating to the thirteenth century, once stood much taller than its current three floors, as shown in eighteenth century drawings. The surviving curtain wall runs southeast from the angle tower before turning south, varying in height from 4 to 8 metres. Architectural features include narrow loops with sandstone jambs, the remains of a large window with bar holes for glazing, and a pointed doorway with chamfered sandstone jambs on the second floor, possibly leading to a former wall walk.