Mechanism design addresses the problem: given a desired outcome rule and agents with private information and misaligned incentives, design a game form (mapping message profiles to outcomes) where rational equilibrium play yields the desired outcome. It is the inverse of game theory: instead of analyzing games, it designs them to achieve social objectives.
Game theory, which you already know, takes a game as given and asks: what will rational players do? Mechanism design reverses the question: given the outcome you want, what game should you create so that rational players produce that outcome? This inversion is why mechanism design is sometimes called "reverse game theory." The designer does not control what agents know or want — those are fixed by the economic environment. What the designer controls is the rules of the game: who can say what, when, and how messages translate into outcomes.
The core challenge is that agents hold private information — their preferences, costs, or valuations — and have incentives to misrepresent it. Consider allocating a painting to whoever values it most. You could simply ask people their valuations, but the highest-valuer would exaggerate to ensure she wins, and the lowest-valuer might exaggerate hoping to resell. A mechanism must be designed so that truthful reporting (or at least behavior consistent with the desired outcome) is an equilibrium strategy. The constraint that agents will behave strategically, not obediently, is what makes mechanism design hard and interesting.
A mechanism formally consists of a message space for each agent and an outcome function mapping message profiles to allocations and payments. The designer's task is to find a mechanism where the Nash equilibrium (or a stronger solution concept like dominant strategy equilibrium) produces the socially desired outcome. For example, in a second-price sealed-bid auction, each bidder submits a bid, the highest bidder wins, and she pays the second-highest bid. The remarkable property is that bidding your true valuation is a dominant strategy — you cannot do better regardless of what others bid. This mechanism "implements" the efficient allocation (giving the object to whoever values it most) using the private information of bidders who have no incentive to lie.
The framework connects to constrained optimization in a specific way: the designer maximizes a social objective function subject to two types of constraints. Incentive compatibility constraints ensure that each agent prefers to report truthfully (or play the intended equilibrium) rather than mimic another type. Participation constraints ensure that each agent prefers to participate rather than walk away. These constraints limit what outcomes are achievable — not every socially desirable rule can be implemented when agents are strategic. The tension between what is socially optimal and what is incentive-compatible is the central theme of mechanism design, and it underlies practical applications from auction design and public goods provision to matching markets and regulatory policy.