Your brass is picking up minute particles from your lube pad. Shitcan that thing and go with some form of spray lube, like dillon's or make your own.
So it is just dirt scraping away at things. I have always just used my fingers and feel and never had anything like that. Of course I always have cleaned (walnut or corncob) then washed, then sized.