What you describe being pro-labor, pro-regular people, pro-free speech and pro-business is rather conservative - this is the issue.
Labour's working class voters are basically small-c conservatives. The Labour elite sneers at these people, even openly calls them racists, then expects them to vote Labour come election, lol.