Travellers who have moved on to a site in Hall Green in Birmingham said they were there to see the Pope.
Councillor Martin Mullaney said the families had moved on to a field next to Sarehole Mill, a childhood haunt of Lord of the Rings author JRR Tolkien.
He said they had told him they had no idea the event was ticket only.
Pope Benedict XVI is holding a special Mass in Cofton Park on Sunday 19 September, at the end of his four-day UK visit.
