The First Amendment claim was similarly an argument that his right to freedom of association was unconstitutionally infringed by virtue of the threat of criminal prosecution for mere non-criminal membership.
But he leaves unaddressed the most likely counter-argument: that a democratic state is a free association and freedom of association involves a right of exclusion.