Training a Moderator in 5 minutes (90% Accuracy)

Marketplaces need efficient moderation tools to ensure that listings are appropriate for the platform. A client of ours recently sought a solution for automating their marketplace moderation, and we delivered an AI system that achieved over 97% accuracy using just 300 examples and minimal training time. Here's how we did it.

Defining the Workflow

To begin, we created a new workflow. The goal was to determine if a marketplace listing should be upvoted ("true"), downvoted ("false"), or ignored ("none"). Our client wanted a straightforward classifier that would make a decision depending on the content of each listing.

Setting Up the Workflow

First, we created the framework for the workflow, which included defining three possible outputs for the AI:

True – the listing should be upvoted.
False – the listing should be downvoted.
None – the listing should be ignored.

In this case, the output options were straightforward. Since we were dealing with a classification problem, the AI model didn't need any creative responses, as might be necessary for tasks like writing posts or generating content.

Uploading and Managing Data

Next, we uploaded a CSV file containing 300 examples of various marketplace listings. Each example included a title and corresponding moderation rule. The beauty of our platform is that if you don't have a large dataset, we allow for synthetic data generation based on the few examples you have. This makes it easy to scale your training data when necessary.

Once the data was uploaded, we moved on to training the AI. The training process only took a few minutes to complete.

Achieving 90% Accuracy in Minutes

After the training concluded, the AI immediately reached an accuracy of 89.9%—close to 90%. Given the small dataset and minimal time invested in training this is great for a first version.

To further improve the classifier, we reviewed the predictions. For example, if the AI predicted "false" when the correct answer was "none," we could analyze why this happened and adjust the dataset accordingly. In many cases, errors might occur due to ambiguous data or inconsistencies in the original dataset, which is why iterative refinement can lead to even higher accuracy over time.

After improving our dataset and giving feedback, we were able to achieve 97% accuracy.

Continuous Improvement Through Feedback

One powerful feature of our platform is the ability to continuously improve AI models based on real-world feedback. As users interact with the marketplace, their feedback can help refine the model and boost its accuracy. Whether the AI is being used to classify listings, generate content, or any other use case, this continuous feedback loop ensures that the AI evolves and becomes more efficient over time.

Deployment Options

After achieving our desired accuracy, we moved on to deployment. Our platform offers several deployment options:

API Integration: This is ideal for apps and services that want to benefit from ongoing AI optimization. Through the API, users can make calls to the AI, and we gather data to refine the model further.
Prompt Export: For those who prefer a simple setup, we offer the ability to export the prompt. While this method requires users to input their data manually, it's a straightforward solution.
App Publishing: Users can publish their AI models directly on our platform, making them accessible to anyone who wants to use them. This allows the client to test the AI in real-world scenarios immediately.

Our client opted for a local integration of the AI, which allowed them to test the AI in real-world scenarios immediately.

Testing the Model

We then tested the AI with a new listing about pineapples. The AI correctly returned "true" based on the client's rules. This quick turnaround shows how easy it is to set up an AI moderation system and get it running with minimal effort.

Conclusion

In just a few minutes, our platform enabled us to help a client train a highly accurate marketplace moderation AI with only 300 examples. Whether you're managing a marketplace, creating content, or any other repetitive task, AI can be a powerful tool to streamline operations.

If you want to build your own workflow, we'd love to help. We are in public beta, so our platform is completely free to use.

You can schedule a demo here. Let's use AI to make your business even better.