The answer to that objection is very simple. When a plaintiff brings an action for damages he is saying to the court, "My character has been defamed and I want damages because of an attack made upon my reputation." If there is no damage to his reputation or no attack has been made, the fact that one small part of the allegation cannot be justified is no reason why he should ask the court to award damages.
For instance, we know how narrow is the line that separates the crime of false pretences from the crime of larceny. Suppose someone publishes a statement saying that a man has committed offences by stealing goods and obtaining goods by false pretences. Suppose it later turns out that although there were cases where he stole goods, strictly speaking, he did not commit the crime of obtaining goods by false pretences. If an attempt were made to justify that libel the plaintiff would be bound to win, because the defendant could not justify the charge. Great injustice might well be done in that way by allowing the plaintiff to get away with something when having regard to his reputation, he clearly ought not to succeed.