If the center of Earth is at $(0,0)$, and the moon is at $(A, 0)$, and the observer is at $(0, R + h)$, then the line connecting the moon and the observer is
$ y = R + h - \dfrac{ R+ h}{A} x $
The distance between the origin (which is the center of Earth) and this line is
$ d = \dfrac{ R + h }{ \sqrt{ 1 + \dfrac{(R+h)^2}{A^2} } } = \dfrac{ A (R + h) }{\sqrt{A^2 + (R + h)^2} } $
And we want this distance to be greater than $R$. Hence we want to solve for $h$ the equation
$ A (R+h) \gt R \sqrt{ A^2 + (R + h)^2 } $
Square both sides
$ A^2 (R+h)^2 \gt R^2 (A^2 + (R + h)^2 ) $
From which,
$ (A^2 - R^2) (R + h)^2 \gt R^2 A^2 $
i.e.
$ (R + h)^2 \gt \dfrac{ R^2 A^2 }{A^2 - R^2} $
And finally
$ R + h \gt \dfrac{ R A }{\sqrt{A^2 - R^2} } $
which gives
$ h \gt -R + \dfrac{ R A }{\sqrt{A^2 - R^2}} $
