Hong Kong's pro-democracy camp failed to win enough seats to regain some veto power in the city's 70-seat legislature in a closely fought and pivotal by-election.