A boat is designed in such a way that it displaces water equal to upthrust experienced by it causing it to float(Archimedes' principle). 

When more passengers climb onto a boat than its maximum capacity, the total weight of the boat increases. This added weight causes the boat to displace more water. 

If the displaced water's weight is less than the total weight of the boat and passengers, the buoyant force decreases.

Once the buoyant force is insufficient to counteract the boat's weight, it can no longer stay afloat, leading to the risk of sinking.