Circular orbits are stable if the effective potential has a minimum. Small perturbations lead to epicyclic oscillations. The condition for stability depends on the force law: gravity admits stable circular orbits while some power-law forces do not.
Your study of the effective potential in central-force problems gave you a powerful tool: the full three-dimensional problem of a particle in a central force field reduces to a one-dimensional problem in the radial coordinate, with an effective potential V_eff(r) = V(r) + L²/(2mr²). The centrifugal term L²/(2mr²) acts as a repulsive barrier at small r, and the combination with an attractive V(r) often produces a potential well — a minimum at some radius r₀. A particle sitting exactly at that minimum with the right angular momentum traces a perfectly circular orbit. Orbital stability asks: if you nudge that particle slightly, does it return toward r₀, oscillate around it, or fly away?
The stability criterion follows directly from the shape of V_eff. If r₀ is a minimum of V_eff — meaning d²V_eff/dr² > 0 at r₀ — then small radial displacements produce a restoring force, and the orbit is stable. The particle oscillates radially (epicyclic motion) while continuing to orbit, tracing a rosette-like path rather than a closed ellipse in general. If r₀ is a maximum of V_eff, any small perturbation grows: the orbit is unstable. This is exactly the same stability analysis you would apply to any potential energy curve in one dimension — local minima are stable equilibria, local maxima are unstable.
For the gravitational force F = -GMm/r², you can show that V_eff has a minimum for any value of angular momentum L ≠ 0, and that the effective potential is concave-up at that minimum. Circular orbits under gravity are stable, and small perturbations produce radial oscillations at the epicyclic frequency κ. For a gravitational potential, κ equals the orbital frequency Ω, which is why planetary orbits under pure Newtonian gravity close exactly: the radial oscillation period equals the orbital period, and the orbit returns to its starting point. This is a special property of the 1/r² force law — it produces closed elliptic orbits, as Bertrand's theorem formalizes.
For a general power-law force F = -kr^n, the stability condition becomes a constraint on n. You can derive from the effective potential that circular orbits are stable only if n > -3 — equivalently, the force must not fall off faster than 1/r³. Gravity (n = -2) comfortably satisfies this. A hypothetical force that decayed as 1/r⁴ would not: circular orbits would be unstable and any small perturbation would send the particle spiraling inward or outward. This explains why the specific power of gravity is not arbitrary from the perspective of stable planetary systems — slightly different force laws would not permit the ordered, persistent orbits we observe.
Perturbation analysis here is a microcosm of a general technique in classical mechanics and beyond: find the equilibrium, expand around it to second order, identify whether the quadratic term is positive (restoring) or negative (destabilizing), and compute the frequency of small oscillations. The same method applies to the stability of Lagrange points in the three-body problem, to the stability of fluid flow (Rayleigh's criterion), and to the small oscillations of any physical system around equilibrium. Mastering it for orbits gives you the template for all of those more complex analyses.