Police will not back down against 'marginal groups'