Adolescent elves do not have past-life memories while awake.
In the question, you wrote:
as adolescents, these memories begin to appear during their waking lives.
This is a misreading of the lore. First, we see:
During a young elf’s first few years, the memories evoked during trance are drawn not from current life experiences, but from the fantastic past adventures of the elf’s immortal soul.
As a young child, the elf has trance-memories of their past lives, but as they age, we see:
Parents of young elves and priests of Erevan Ilesere encourage the youths to explore these memories and talk about them with one another, but they aren’t to be discussed with adults until a memory of waking life first intrudes upon a youngster’s trance.
And later:
elves in adolescence learn how to use trance to evoke memories of their waking lives, giving them opportunities to reflect on the joys of the mortal world and to reinforce the principles of any training or practice undertaken while awake.
An elf in adolescence reflects on memories of their current awake life while in trance. They do not begin seeing memories of past lives while awake. Nowhere do we see that they have memories of past lives while awake. In adolescence, the memories of past lives are totally replaced by memories of their own waking lives.
Narratively, there are probably downsides to being a child.
Mordenkainen's Tome of Foes states:
During a young elf’s first few years, the memories evoked during trance are drawn not from current life experiences, but from the fantastic past adventures of the elf’s immortal soul.
The time in the life of an elf when these memories occur is childhood, pre-adolescence. As noted in the previous section, these past-life memories end when adolescence begins. So to have these memories, you must be a child. The difficulties of playing a child are explored in this Q&A: GM Considerations for a Human Barbarian Toddler PC
Should you decide to play a child, this is a world building tool for the DM, not really an ability you have.
All of the abilities you properly have are those racial traits listed in the race description. As such, this piece of lore about pre-adolescent elves is completely out of your hands, resting in the hands of the master of worlds, the DM. Which is to say, the ability is only as useful as the DM wants it to be.
There are no downsides baked into it, because it's not actually a racial trait that usually comes with the race. This is lore, not mechanics. It is balanced by the fact that the DM has total control over the shape these memories take.