Matthew Gruter was among about 60 dark-clothed men -- some hiding their faces with sunglasses or hats -- who protested outside the New South Wales parliament this month.
Protesters shouted antisemitic slogans and brandished a banner reading: "Abolish the Jewish lobby" in the November 8 rally, according to television images and media reports.
Home Affairs Minister Tony Burke, who cancelled Gruter's visa Monday, said Australian Border Force officials took him into immigration detention early Tuesday.
The South African will be held until he gets a ticket to fly out of Australia or is deported, Burke told reporters.
"If you're on a visa, you're a guest in Australia," the minister said.
"If someone turns up for the purposes of just abusing people and wrecking the place and damaging the cohesion, we can ask them to leave, and Australia has asked him to leave."
Burke said Gruter had limited grounds for appeal and he was expected to leave the country "very soon".
Gruter, a civil engineer, moved to Sydney about three years ago and was a senior member of Australia's largest neo-Nazi group, the Sydney Morning Herald newspaper said.
"The bottom line here is we're setting the standard for Australia. We're setting the standard that multicultural Australia and modern Australia are the same thing," Burke said.
Australia's intelligence services have warned that growing numbers of people are being drawn towards neo-Nazi ideologies as extremist groups step up efforts to recruit new members.
Australia banned the Nazi salute and the display or trade of Nazi symbols last year.
