Maximizing the volume of a cone inscribed in a sphere

Substituting $r^2 = 2Rh - h^2$ into $V = \frac{1}{3}\pi r^2 h$ gives $V(h) = \frac{1}{3}\pi h (2Rh - h^2) = \frac{1}{3}\pi(2Rh^2 - h^3)$, a function of the single variable $h$ on the domain $0 < h < 2R$, which comes from requiring both $r^2>0$ and $h>0$.

Differentiating gives $V'(h) = \frac{1}{3}\pi(4Rh - 3h^2) = \frac{1}{3}\pi h(4R - 3h)$. Setting $V'(h) = 0$ yields $h = 0$ or $h = \frac{4R}{3}$; since $h=0$ is an endpoint giving zero volume, the critical point $h = \frac{4R}{3}$ lies in $(0,2R)$. Checking the sign of $V'(h)$ (positive before, negative after) confirms this is a maximum, so the volume is maximized at $h = \frac{4R}{3}$.