A subgroup N is normal if gNg⁻¹ = N for all g ∈ G, equivalently gN = Ng for all g. Normal subgroups are precisely the kernels of homomorphisms. Only for normal subgroups does the set of cosets form a group.
From your study of cosets, you know that any subgroup H partitions G into left cosets gH and also into right cosets Hg. For a general subgroup, the left coset gH and right coset Hg are different sets — the element g is in both, but the rest of the members may differ. A normal subgroup N is one where this distinction collapses: gN = Ng for every g in G. The left and right cosets always coincide. Equivalently, conjugating N by any g (forming gNg⁻¹) gives back N itself. This is the normality condition: N is closed under conjugation by every element of G.
Why does this matter? Because normality is precisely the condition needed for the set of cosets to form a group in its own right. Try to multiply two cosets: you want to define (aN)(bN) = (ab)N. For this to be well-defined — meaning the product of any representative of aN with any representative of bN lands in (ab)N — you need N to be normal. Specifically, you need bNb⁻¹ = N so that the N's can "pass through" b. Without normality, the multiplication of cosets depends on which representatives you pick, so there is no consistent group structure. With normality, the quotient group G/N, whose elements are the cosets of N, is a genuine group with (aN)(bN) = (ab)N.
The deepest characterization of normal subgroups connects them to group homomorphisms. A homomorphism φ: G → H is a function preserving the group operation: φ(ab) = φ(a)φ(b). The kernel of φ is ker(φ) = {g ∈ G : φ(g) = e_H}, the set of elements that map to the identity. The kernel is always a normal subgroup of G: if φ(n) = e and g is any element, then φ(gng⁻¹) = φ(g)eφ(g)⁻¹ = e, so gng⁻¹ ∈ ker(φ). Conversely, every normal subgroup is the kernel of some homomorphism — namely the quotient map G → G/N sending g to its coset gN. So normality and being a kernel are the same thing.
The most important examples to internalize: every subgroup of an abelian group is normal (since ab = ba implies aH = Ha always). The center Z(G) — elements commuting with everything — is always normal. The alternating group Aₙ is normal in the symmetric group Sₙ. Non-examples are equally instructive: in S₃, the subgroup generated by a single transposition (say {e, (12)}) is not normal because conjugating (12) by (13) gives (23), which is outside the subgroup. The subgroup fails to be closed under conjugation, so left and right cosets differ, and no quotient group can be formed.