How Tolkien did it
Tolkien’s masterstroke was the invention of hobbits, who are fantastic only because they don’t actually exist and because they are three feet tall and have hairy, toughened, feet but no beards; in all other ways they are resolutely Just. Like. Us. A story set wholly in The Shire would hardly be fantasy at all; it could be translated perfectly to certain regions of the U.K. at some point in the past, and that’s precisely the point. Hobbits are a signifier that we are in a fantastic world, and hence that we are reading an immersive fantasy, but they themselves contain nothing significantly fantastic, nothing magical.
Into this world, at once overtly fantastic yet utterly mundane, comes a genuinely fantastic intrusion: Gandalf the Wizard and his thirteen dwarves. They disrupt Bilbo’s life as profoundly as the appearance of vampires or zombies disrupts the lives of characters in a pure intrusion fantasy, but in this immersive fantasy world they are not regarded as extraordinary by Bilbo. Nor (and this is crucial) does the reader expect him to be taken aback by their existence—if Hobbiton were populated by the simple English countryfolk that hobbits stand for, and one of those were confronted by a wizard and thirteen dwarves, we would expect him to marvel and wonder, if not collapse in shock. But because Bilbo is three feet tall and has hairy feet, we perceive him as being of the same ilk as his intruders, and readily accept it when he instead (at first) is merely “confusticated and bebothered” by the dwarves and wishes they would go away and leave him alone.
This seems to me to be the reason why Tolkien, in the next chapter, can dispense with the long narrative in which the everyday gradually gives way to the fantastic, and instead accomplish it in two paragraphs: the first chapter already contains an effortless juxtaposition and contrast of the everyday and the fantastic, one which is possible only because Bilbo is a fantastic character who is adamant about being nothing of the sort.
Until, of course, the Took in him awakens. And then Tolkien can run the table on all the possible unambiguous fantasy structures: everyone leaves and goes off on a portal-quest. Here the opposition between the intrusion and portal-quest fantasies is not theoretical, but actual and sequential: the fantastic characters intrude into the mundane world, and when they leave to go back to their own, our hero Bilbo goes with them.
In The Lord of the Rings, this already radical conception is made even more so. The Edge of the Wild is dispensed with entirely. Or, rather, when we get to the thrilling sixth chapter, we realize that it is not located just west of Rivendell, but, among other places, in the back yard at Crickhollow. Indeed, the presence of explicit peril in the neighboring Old Forest contradicts the portrayal of the world in The Hobbit. And it is revealed not only that Bounders work to keep the perils of the outside world from The Shire, but that there is much more peril than the hobbits realize, because the Dúnedain in secret are doing most of the Bounders’ dirty work. In other words, the Edge of the Wild is nothing more or less than the borders of The Shire.
In LOTR The Shire has become what the critic John Clute, in The Encyclopedia of Fantasy, calls a polder, borrowing the term from the Dutch for a tract of farmland reclaimed from the sea and protected by dikes. Fantasy polders are areas of altered reality protected against the intrusion of outside evil by a well-defined and consciously maintained (and usually magical) border. Clute cites Bombadil’s domain within The Old Forest as one, and Shangri-La in Lost Horizon as another. But The Shire is in some ways an anti-polder. The alteration of reality within is not the presence of magic or the domination of a superior brand of it, but its absence, and the border is maintained through ordinary and not magical means. The adventure happens not when the polder is penetrated, but when it is left. Tolkien has in essence taken a fantasy trope and turned it inside out — at a time when it had hardly been invented. No wonder so many Tolkien imitations are ultimately unsatisfying; though they mimic all the accidents of the books, they miss their essences, which are so deeply embedded as to be almost indiscernible.
There’s no question that any film adaptation of The Hobbit that is designed to be a companion of The Lord of the Rings must be set in the Secondary World of the later story. Re-setting The Hobbit in the word of LOTR is something that must be accomplished on two levels: first, literally, in terms of geographies, peoples, and the like; and second, in terms of flavor and tone. Let’s deal with these in turn, because that’s the order we’ll encounter them as we tell the tale.