Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.neuraltrust.ai/llms.txt

Use this file to discover all available pages before exploring further.

Toxicity detection is an essential pillar of modern content safety, helping platforms enforce community guidelines, comply with regulations, and ensure respectful interactions across user-generated content. TrustGate offers powerful plugin integrations with leading providers like Azure Content Safety and OpenAI Moderation API to analyze and filter harmful content in real time.

​
Why Detect Toxicity?

Toxic content can take many forms—hate speech, sexual content, self-harm encouragement, threats of violence, and more. Left unchecked, this type of content can:
  • Damage brand reputation and user trust
  • Violate regional and global regulations
  • Expose users to psychological harm
  • Undermine platform safety and inclusive experiences
Toxicity detection empowers organizations to take action before harmful content spreads, by flagging or blocking it according to configurable thresholds.

​
What TrustGate Offers

TrustGate supports two high-performance plugins for toxicity moderation:
PluginProviderContent TypesKey Features
Azure Toxicity DetectionAzure Content SafetyText & ImageSeverity level thresholds, multi-modal analysis, category-based filtering
OpenAI Toxicity DetectionOpenAI Moderation APIText & Image URLCategory-specific scoring, configurable thresholds, sub-category analysis
NeuralTrust Toxicity DetectionNeuralTrust APITextCategory-specific scoring, configurable thresholds, sub-category analysis
These plugins allow you to:
  • Analyze both text and images
  • Detect specific categories of risk (e.g., violence, hate, sexual content)
  • Customize response actions (block, warn, log)
  • Configure thresholds to balance sensitivity and false positive rates

​
Toxicity Categories

Typical categories analyzed include:
CategoryDescription
HateDiscriminatory or hateful language based on race, gender, religion, etc.
ViolenceThreats, graphic violence, promotion of harm to others
Self-HarmEncouragement or glorification of suicide, self-injury, etc.
SexualExplicit, suggestive, or otherwise inappropriate sexual content
HarassmentThreatening or targeted language aimed at individuals or groups
IllicitContent involving illegal activity or regulatory violations
Each plugin allows fine-tuned threshold controls, letting you define what gets blocked or flagged according to your audience and use case.

​
Example Use Cases

Toxicity detection is vital for:
  • LLM applications to prevent prompt abuse and jailbreaks
  • Messaging platforms for filtering hate speech and harassment
  • Education tech to keep learning environments safe
  • Public comment systems to block harmful or illegal content
  • Gaming communities for anti-toxicity and moderation automation
  • Customer support AI to intercept offensive or harmful messages

​
Response Format

Toxicity plugins typically return rich response data: 
{
  "analysis_results": [
    { "category": "Hate", "severity": 1 },
    { "category": "Violence", "severity": 0 }
  ],
  "is_blocked": false,
  "blocked_categories": []
}
This allows your system to act in real time—blocking, alerting, or logging—based on the specific risk levels.

​
Best Practices

  • Use moderate default thresholds and adjust with data
  • Monitor blocked content and false positives for tuning
  • Secure API keys and rotate regularly
  • Enable category-level logging to understand trends
  • Test with realistic sample data to validate coverage
Explore individual plugin capabilities: