Small details, mostly from Mutsumi's memories, are mentioned in the Manga, mostly in Chapter 81. Some of the details are also revealed in the Anime, but since the Okinawa Island arc was skipped in the Anime, most of the clues are left out.
Mutsumi and Keitaro were good friends who met at the Hinata House because Mutsumi's Mother, Natsumi, worked there under Keitaro's Grandmother. At some point, Naru also came to live with them and they all became friends. Mutsumi and Keitaro often climbed trees, but couldn't come down unless Haruka(who was about 10 at that time) helped them. At some point they also crashed with a toy train, an accident Naru still has a scar on her left arm from.
At one point, Keitaro made a promise to Naru to go to Tokyo U together when they grew up, but three years later Naru(who was only two years old at the time of the initial promise) didn't remember the promise she made to Keitaro and Mutsumi "renewed" the promise that she and Naru would go to Tokyo U, causing Naru to be confused when Keitaro asked her 15 years later if she was the girl he promised, saying that "The person i promised... was a girl.". Granny Hina apparently heard the initial promise between Keitaro and Naru, and in Book 14 she calls and tells him that Naru is indeed the Promise Girl of his memories.
Keitaro's Memories are very foggy from that time, and he remembered he made a promise to a girl he remembers with Mutsumi's appearance(despite the fact that it was actually Naru) in a sandbox outside Hinata House. He also remembers having ran up the stairs to Hinata House, being dragged by the Promise Girl, and he and Mutsumi gets a feeling of Deja Vu when they both stand in the Sandbox.
The term "Fateful Sandbox" comes from the fact that both Keitaro and Naru's promise as well as Naru and Mutsumi's are made in the same sandbox outside Hinata House.