You can set control the placement of the extra x/y ticks labels via the extra x/y tick style:
extra x tick style={x tick label style={above, yshift=0.6ex}},
extra y tick style={y tick label style={right, xshift=0.3em}},

Alternatively, you can place the labels via a \node using the axis cs coordinate system:
\node [above] at (axis cs: 11.547,0) {\tiny$\frac{20}{\sqrt{3}}$};
\node [right] at (axis cs: 0,3079.2) {\tiny$\frac{16,000}{3\sqrt{3}}$};
Notes:
- As far as your question regarding overwriting the nodes, that is what
fill=white is doing. You can see the size of the nodes if you add draw=red. i.e., ticklabel style={font=\tiny,fill=white, draw=red}. This additional padding is controlled via inner sep, so setting this to zero via ticklabel style={font=\tiny,fill=white, draw=red, inner sep=0} you will see that the nodes are now only the size of the text and the overlapps is reduced (except of course when the actual text overlap other text).
References:
Code: extra x/y tick style:
\documentclass[10pt]{amsart}
\usepackage{tikz}
\usetikzlibrary{calc,angles,positioning,intersections,quotes,decorations.markings}
\usepackage{tkz-euclide}
\usetkzobj{all}
\usepackage{pgfplots}
\pgfplotsset{compat=1.11}
\begin{document}
\noindent \hspace*{\fill}
\begin{tikzpicture}
\begin{axis}[width=6in,clip=false,
axis lines=middle,
xmin=-25,xmax=25,
domain=-25:25,
xlabel=$x$,ylabel=$y$,
ymin=-3100,ymax=3100,
restrict y to domain=-3100:3100,
enlargelimits={abs=0.5cm},
axis line style={latex-latex},
ticklabel style={font=\tiny,fill=white},
extra x tick style={x tick label style={above, yshift=0.6ex}},
extra y tick style={y tick label style={right, xshift=0.3em}},
xtick={},ytick={},
extra x ticks={11.547},
extra x tick labels={$\frac{20}{\sqrt{3}}$},
extra y ticks={3079.2},
extra y tick labels={$\frac{16,000}{3\sqrt{3}}$},
xlabel style={at={(ticklabel* cs:1)},anchor=north west},
ylabel style={at={(ticklabel* cs:1)},anchor=south west}
]
\addplot[samples=501,domain=0:20,blue, ultra thick] {400*x - x^3};
\addplot[samples=501,dashed,domain=20:25,blue, ultra thick] {400*x - x^3};
\addplot[samples=501,dashed,domain=-25:0,blue, ultra thick] {400*x - x^3} node[right,pos=0.25,font=\footnotesize]{$y = 400x - x^{3}$};
\end{axis}
\end{tikzpicture}
\hspace{\fill}
\vskip0.25in
\end{document}
Code: \node placement
\documentclass[10pt]{amsart}
\usepackage{tikz}
\usetikzlibrary{calc,angles,positioning,intersections,quotes,decorations.markings}
\usepackage{tkz-euclide}
\usetkzobj{all}
\usepackage{pgfplots}
\pgfplotsset{compat=1.11}
\begin{document}
\noindent
\begin{tikzpicture}
\begin{axis}[width=6in,clip=false,
axis lines=middle,
xmin=-25,xmax=25,
domain=-25:25,
xlabel=$x$,ylabel=$y$,
ymin=-3100,ymax=3100,
restrict y to domain=-3100:3100,
enlargelimits={abs=0.5cm},
axis line style={latex-latex},
ticklabel style={font=\tiny,fill=white},
xtick={},ytick={},
extra x ticks={11.547},
extra x tick labels={},% <-- placed via a \node
extra y ticks={3079.2},
extra y tick labels={},% <-- placed via a \node
xlabel style={at={(ticklabel* cs:1)},anchor=north west},
ylabel style={at={(ticklabel* cs:1)},anchor=south west}
]
\addplot[samples=501,domain=0:20,blue, ultra thick] {400*x - x^3};
\addplot[samples=501,dashed,domain=20:25,blue, ultra thick] {400*x - x^3};
\addplot[samples=501,dashed,domain=-25:0,blue, ultra thick] {400*x - x^3} node[right,pos=0.25,font=\footnotesize]{$y = 400x - x^{3}$};
\node [above] at (axis cs: 11.547,0) {\tiny$\frac{20}{\sqrt{3}}$};
\node [right] at (axis cs: 0,3079.2) {\tiny$\frac{16,000}{3\sqrt{3}}$};
\end{axis}
\end{tikzpicture}%
\end{document}