The simple past and the past perfect imply slightly different things. In this case, I would definitely use the past perfect with had. The simple past just states that something happened in the past, with no specific implications about when:
The mailman left.
The prime minister resigned.
The ship sailed away.
The past perfect is used to indicate that something happened before another specific event in the past. We use it to relate two past events, like:
I rushed to the door, but the mailman had already left. (that is, before I got to the door)
They told me that the prime minister had resigned. (had resigned, because the resignation was before they told me)
The ship had sailed away by the time the police arrived. (the ship sailed, then later the police arrived)
Sometimes it doesn't make a great difference. For example, it would be normal and unremarkable to say "The phone stopped ringing before I could get to it" or even "Before I could get to it, the phone stopped ringing." But putting by the time in there makes it look like a specific comparison of times, so the past perfect feels more natural.
References:
PAST PERFECT
When is the past perfect exactly needed?
Past perfect and simple past compared