Discord admits AI moderation bug wrongfully banned users over harmless images

Discord has acknowledged that a bug successful its AI moderation strategy mistakenly banned much than 8,000 users implicit the past 2 months, aft harmless images—including spreadsheets, chessboards, crippled textures, arsenic good arsenic achromatic and grey transparent backgrounds—were incorrectly flagged arsenic harmful content.

The institution confirmed that the contented had been affecting accounts since May, with an further 200 users banned implicit the play earlier its squad identified and fixed the problem. All affected accounts are presently successful the process of being restored.

The incidental highlights 1 of the increasing challenges surrounding AI-assisted moderation arsenic galore platforms progressively trust connected automated systems to place amerciable oregon abusive worldly astatine scale.

In a elaborate thread connected X, Discord explained that its automated information strategy works by matching uploaded contented against databases of known harmful material. While the exertion is designed to drawback amerciable content, the institution acknowledged that it tin sometimes make mendacious positives. A quality moderator reviews the content, but a bug caused the strategy to instantly prohibition affected accounts.

“We’re moving connected amended safeguards truthful this can’t hap again,” the institution wrote. 

Across X and Reddit, users person claimed they had been permanently suspended simply for uploading images containing quadrate grid patterns. Several users speculated that Discord’s AI moderation tools person go progressively delicate to grid-like patterns due to the fact that they person antecedently been utilized successful attempts to obscure oregon disguise NSFW and kid exploitation contented from automated detection systems. 

Affected users person been expressing vexation connected societal media, with immoderate arguing that imperishable relationship bans based solely connected automated detection tin person superior consequences, peculiarly for users who trust connected Discord for work, gaming communities, oregon long-distance societal connections.

“Losing a Discord relationship to thing arsenic unfair arsenic this tin beryllium highly devastating and impact users severely, and each time millions of users are affected by mendacious AI bans. This needs to beryllium stopped,” 1 X idiosyncratic wrote. 

My relationship was wrongfully banned from your level owed to a bug successful your AI automod detecting my GAME TEXTURES arsenic CSAM. I request my relationship backmost arsenic I’m a crippled manager and usage Discord for each my communication. I person requested a reappraisal of my suspension.@discord @discord_support pic.twitter.com/QfAkCIJo6S

Discord isn’t unsocial successful facing moderation troubles owed to automated systems. Last year, users of Instagram and Facebook Groups reported wide unexplained relationship suspensions that galore believed were caused by AI moderation systems. Although users pointed to automation arsenic the apt culprit, Meta ne'er publically confirmed whether AI errors were responsible. Now, Meta’s Oversight Board is pushing for accrued transparency.

Tumblr past twelvemonth besides faced complaints from users who said their accounts had been mass-suspended without wide explanations.