The title implies a question. It says, "that's where..." without saying where "that" is. The opening line answers: at the "big bang," if you will, of a song: "Somewhere in a burst of glory/[Where] sound becomes a song."
The next line contains the word "bound," which could imply three meanings. One is "obligated," as in "contractually bound," taking its meaning from the idea of "binding" something with, say, rope. Another is "inevitable," as in "bound to happen."
But since the overall metaphor is one of place, we can safely assume that, while these other meanings offer some shade of insight, the core meaning is "in the direction of," as in "eastbound train."
Then we shift to a speaking of time, not place, and yet we have: "When I see you smiling/ When I hear you singing... that's where I belong" [emphasis mine]. This time, the answer of "What place?" is, oddly, "This time." The lines in between, "Every ending, a beginning" might serve as the link; both places and times can start and stop. But then, "The way you turn and catch me with your eye," is not a time or place, but a way, a "how."
And we are left with the seemingly throw-away line in the repeat of the chorus-- "That's the way it is/ I don't know why"-- to explain that there is no explanation.
The song is meant to evoke a mood, a sense of being, irrespective of time and place, a situation that evokes the lush, enveloping aromas and smooth, fleshy softness of "lavender and roses." Any place in which the subject of the song is smiling, singing, or glancing at him-- that's where he belongs. Not in a specific place at all-- but with a person, where she (we assume) may be.
Then the music becomes more active and sprightly, and we shift focus altogether. Now, meet a character. He's a "spiny little island man." Like the "fine lady upon a white horse" in the nursery rhyme with "rings on her fingers and bells on her toes," this man also will "have music wherever he goes." He can play it on his "jingling banjo," or listen to it on his "radio" when he stops.
And now we have a place where this man is. On an "island," on a "dirt road," heading ("bound," if you will) for a "river where the water meets the sky." Some sort of delta, then, perhaps a tropical one. When he arrives-- not that arriving is the point-- he will be at a nexus of earth ("dirt"), "sky," and "water."
Right in the middle of the essential elements of creation, ready for the last one: energy (The ancients were not wrong that everything is made of earth, water, air, and fire. They just used the word "elements" and those common examples, instead of what we today would call "states of matter," respectively: solid, liquid, gas, and energy.).
So when the man gets there, it should not be long before his banjo and radio summon that fourth element, the energy that will explode in an "burst of glory," and organize raw "sound" into a "song" or "story." Since songs and stories are what he seeks, that is indeed where he belongs, and where he is bound.
Now, Simon does not play a banjo, although his guitars do "jingle" and chime more than they used to. And he is not "spiny," even if he is, well, not as tall as Garfunkel and so "little" in comparison to some. And he is not an "island man," in that he was not born on Manhattan, even though his office is there now. So he is not this character.
He just wants to hang out with him, by the river delta on an island, fishing in the sky for songs. There, and with whomever he is singing the song to. These are the places he feels he "belongs."
Again, this word has at least three meanings that could apply. Two are covered by Cole Porter in the lyrics from "Find Me a Primitive Man": "Not the kind of a man who belongs to a club/ But the kind with a club that belongs to him." So "belongs" could mean "joins as a member" or "is possessed by."
But there is a third meaning, that of "this jar belongs on that shelf." This is the meaning that relates to "where," to place. Simon (and it seems he is the speaker this time) has been around the world enough times to know that the place he is going is less important that the people he will meet there. Or the songs he will find there.
Where the people who can inspire songs are-- that's where he belongs.
Coming off the intense Capeman experience, Simon returned to the international musical thread of Graceland and Rhythm of the Saints on this, his 10th solo album. It went gold in the US and silver in the UK, cracking the Top 20 in both markets... and the Top 10 in Norway! It also did well in other European and English-speaking markets, and even made the Top 100 in Japan.
When You're the One was nominated for a Grammy, it made Simon the first musician to be nominated for Album of the Year five decades running. (It would take a different Paul-- McCartney-- another six years to match that mark.)
Next Song: Darling Lorraine