You need to monitor toxic content on your website. AI can help

Toxic content is everywhere on the internet, and companies that implement user-generated content run the risk of being host to scams, bigotry, and misinformation. Placing this toxic content can damage your brand image and damage consumer sentiment if not addressed quickly and widely.

Hire moderators to sift through every live stream, podcast, post, and gif, however, could bankrupt your business. There is simply too much content for humans to clean it all up. Another issue is that sifting through the worst of the internet can have detrimental effects on the health of your employees. In 2021, a judge reward a group of more than 10,000 former Facebook moderators an $85 million settlement after moderators developed PTSD at work.

Enter the market for content moderation solutions that, using AI or pairing it with humans, are helping to turn the tide in the war against toxic content on the internet.

Kevin Guo, co-founder and CEO of AI-powered content moderation company Hive, first saw potential business in automated content moderation when he and his co-founder Dmitriy Karpman were college students. ‘Stanford University. Guo and Karpman had created Kiwi, a video chat app that randomly matched users with strangers around the world.

Soon Guo found himself with what he called the “hot dog problem”, which would later be parody in the HBO comedy Silicon Valley. Simply put: the men were using the app to expose themselves on camera to a reluctant audience.

After determining that there was no solution to his problem, Guo decided to create a machine learning model that could identify and report “hot dogs” on its own. “I hand-labeled this set of images myself and it really could only do one thing, which was to tell if something was a ‘hot dog’ or not.”

Guo started selling his “hot dog” model on the side, but quickly realized there were more applications for a learning model that could identify and label objects in images and videos than mere nudity detection, so in 2017 he and Karpman shut down their apps. focus entirely on business activities.

Now, Hive offers automated content moderation services of all kinds, with models that can be trained to detect toxic content in text and audio in addition to images. These patterns are used by companies like Reddit, Giphy, and Vevo to detect and stop violence, hate speech, spam, bullying, self-harm, and other behaviors you’d rather not. not see on your website or app.

One of Guo’s early successes in content moderation came when live video Omegle and Chatroulette chat services contacted Hive to help clean up their content. Both companies became infamous in the early 2010s for their inability to deal with issues similar to the “hot dog” situation, so when they learned that Guo had cracked the code, they were intrigued.

“Now,” Guo says, “these platforms are 100% clean. We sample every video chat and we can report it as soon as something happens.” According to a case studyHive closes over 1.5 million Chatroulette streams per month.

Guo says his models are designed to be used without any human assistance or intervention, a particularly attractive aspect for large enterprises that need highly scalable solutions.

In October 2021, Microsoft announced that it had acquired Two Hat, a content moderation provider focused on the online gaming industry. Like Hive, most of Two Hat’s content moderation services operate without human interaction. In a blog post Announcing the acquisition, Xbox Product Services Vice President Dave McCarthy said Two Hat’s technology has helped make global communities on Xbox, Minecraft, and MSN safer for users through a highly configurable approach that allows it is up to the user to decide what they are and are not. comfortable with.

Other content moderation professionals, however, believe the real solution lies in combining what AI does well with human decision-making. Twitch, the global live streaming service for games, music and entertainment, creates internal programs that use machine learning in partnership with humans to flag suspicious and harmful content. While some content is banned platform-wide, such as nudity and violence, Twitch also allows streamers to customize content moderation specifically for their own channel.

A prime example of this personalization, according to Alison Huffman, director of Twitch Community Health Products, comes in the form of a recently released tool called Suspicious User Detection. The tool uses machine learning to identify users who have created a new account to circumvent being banned from a specific channel. The tool flags potential scammers and then lets creators make their own decision on how to proceed.

“We’re trying to combine the best of machine learning, which is scalable and efficient but imperfect at detecting nuance, with human review, which is less efficient but more nuanced and personal,” says Huffman.

“In this way, we’re using machine learning to provide creators with insights that help them make better security decisions faster, while leaving the final decision in their hands.”

Sherry J. Basler