The social contract of tolerance doesn’t extend to those who are intolerant. They are outside the protection of Tolerant Society’s protection, and the Tolerant are free (and, in fact, strongly encouraged) not to tolerate them, because they aren’t covered.