I am convinced that this is By the Waters of Babylon by Stephen Vincent Benet, a 1937 post-apocalyptic science fiction short story.
The story involves a father and son ("My father is a priest; I am the son of a priest. I have been in the Dead Places near us,
with my father—at first, I was afraid."), with the boy traveling "to the Place of the Gods—the place newyork [sic]", a post-apocalyptic Big Apple.
The father (per the above quote) is explicitly stated to be a priest, and it is implied that the son is in training for the priesthood.
The forbidden "Place of the Gods" lies across a river from the protagonist's home, which is to the west, implying that they may live in present-day New Jersey or Pennsylvania ("It is forbidden to travel east. It is forbidden to cross the river. It is forbidden to go to the Place of the Gods."). Considering that the boy identifies his tribe as the "Hill People" and that it is "eight suns' journey" to the forbidden place, a home in Appalachian Pennsylvania or upstate New York seems plausible.
The boy encounters many things that he does not understand but that a 20th century reader would,
There was also the shattered image of a man or a god. It had been made of white stone and he wore his hair tied back like a woman's. His name was ASHING, as I read on the cracked half of a stone. I thought it wise to pray to ASHING, though I do not know that god.
The boy wonders if this is a statue of a god, but we know that it is actually a patriotic statue of George Washington, an 18th century US politician who was known for wearing his hair in a ponytail.
Hot and cold water taps are explicitly mentioned:
All these things were magic, but I touched them and lived—the magic had gone out of them. Let me tell one thing to show. In the washing-place, a thing said "Hot" but it was not hot to the touch—another thing said "Cold" but it was not cold. This must have been a strong magic but the magic was gone. I do not understand—they had ways—I wish that I knew.