Skip to main content

Plugins

Plugins are modular components that extend and customize AI Gateway's functionality. The plugin system follows a stage-based execution model, allowing plugins to process requests and responses at different points in the request lifecycle.

Plugin Architecture​

Execution Stages​

The plugin system operates in three main stages, each designed to handle specific aspects of request processing:

Pre-Request Stage​

The pre-request stage serves as the initial gateway for incoming requests, operating before any interaction with the Language Model (LLM) occurs. During this critical phase, the system performs essential request validation and transformation operations to ensure the incoming request meets all necessary requirements. This stage is particularly important for security and access control, implementing various checks and balances to protect the system. Common operations at this stage include authentication to verify user identity, rate limiting to prevent system abuse, and input validation to ensure request data meets expected formats and standards. This stage acts as the first line of defense and preparation, ensuring only valid and properly formatted requests proceed to the LLM.

Post-Request Stage​

After the LLM processes the request, the post-request stage takes over to handle the response processing pipeline. This stage is responsible for response modification and enrichment, allowing the system to enhance or modify the LLM's output before it reaches the client. The stage handles various aspects of response processing and analytics, including data transformation, content filtering, and response validation. Common examples include response transformation to modify output formats, caching to improve performance, and logging to track system usage and maintain audit trails. This stage ensures that the final response meets all requirements and contains any necessary additional information or modifications.

Error Stage​

The error stage provides robust error handling capabilities when issues arise during request processing. This critical stage is responsible for error conditions and failures management, ensuring that all error scenarios are handled gracefully and appropriately. It includes sophisticated error response formatting capabilities to ensure that error messages are consistent and informative. The stage also provides important fallback mechanisms to maintain service availability even when issues occur. Key features include error handling for various types of failures, circuit breaking to prevent cascade failures, and retry logic to automatically attempt recovery from temporary issues. This comprehensive error management ensures that the system remains stable and user-friendly even when problems occur.

Execution Flow​

Plugin execution follows a structured flow based on:

  1. Priority Levels

    Priority 1 (Critical)
         ↓
    Priority 2 (Essential)
         ↓
    Priority 3 (Optional)

  2. Execution Modes

    • Sequential: Plugins that must run one after another
    • Parallel: Plugins that can run concurrently

Plugin Types​

Sequential Plugins​

Plugins that require ordered execution:

  • Maintain state across requests
  • Depend on other plugin results
  • Handle critical security functions
  • Examples: Authentication, rate limiting

Parallel Plugins​

Plugins that can run independently:

  • Process requests without dependencies
  • Perform stateless operations
  • Handle non-critical functions
  • Examples: Logging, metrics collection

Plugin Inheritance​

Plugins follow a hierarchical inheritance pattern:

Gateway Level (Global)
       ↓
Service Level (Service-specific)
       ↓
Route Level (Route-specific)
       ↓
Consumer Group Level (Group-specific)
       ↓
Consumer Level (Individual)

More specific configurations override general ones.

Core Plugin Categories​

The AI Gateway implements four essential plugin categories, each serving a specific purpose in the request processing pipeline:

1. Security Plugins​

Security plugins form the foundation of API protection and access control. At their core, these plugins handle authentication and authorization, ensuring that only verified users with appropriate permissions can access the system. They implement sophisticated rate limiting and quotas to prevent abuse and ensure fair resource utilization across all users. Advanced IP filtering capabilities provide an additional layer of security by controlling access based on network origins. Comprehensive request validation ensures that all incoming requests meet security requirements and conform to expected formats, protecting against malformed or malicious requests. These security plugins work together to create a robust defense system that maintains the integrity and safety of your API gateway.

2. Traffic Management​

Traffic management plugins provide sophisticated control over request flow through the system. Advanced load balancing capabilities ensure optimal distribution of requests across available resources, preventing overload and maintaining consistent performance. The system implements intelligent circuit breaking mechanisms that can detect and respond to service degradation, preventing cascade failures. Flexible request routing allows for sophisticated traffic management based on various criteria, while traffic splitting enables advanced deployment patterns such as A/B testing or canary releases. Together, these plugins ensure smooth, reliable operation even under varying load conditions.

3. Transformation​

Transformation plugins modify and enhance the data flowing through the gateway. At their core, they handle request/response transformation, allowing modification of both incoming requests and outgoing responses to meet specific requirements. Sophisticated content filtering capabilities ensure that data meets policy requirements and security standards. The plugins support data enrichment to add valuable context or additional information to requests or responses. Advanced protocol conversion features enable seamless communication between different API formats and standards, making the gateway highly adaptable to various integration needs. These transformation capabilities ensure that data flowing through the gateway meets all technical and business requirements.

4. Observability​

Observability plugins provide comprehensive insight into the gateway's operation. They implement detailed logging and tracing capabilities that track requests through every stage of processing, enabling effective troubleshooting and audit trails. Sophisticated metrics collection systems gather performance data and usage statistics, providing valuable insights into system operation. Advanced analytics capabilities help identify patterns and trends in API usage, while continuous health checking ensures all components are functioning correctly. These observability features work together to provide a complete view of the gateway's operation, enabling proactive management and optimization.

Plugin Chain​

The plugin chain defines how plugins work together:

Chain Structure​

Authentication → Rate Limiting → Transformation → Logging
     ↑              ↑                ↑              ↑
 Security      Traffic Control    Processing    Observability

Chain Characteristics​

  • Ordered execution based on priority
  • Mix of sequential and parallel plugins
  • Configurable at multiple levels
  • Flexible and extensible

Best Practices​

1. Plugin Organization​

  • Group related plugins together
  • Maintain clear execution order
  • Consider plugin dependencies
  • Keep chains focused and simple

2. Performance Considerations​

  • Balance security and performance
  • Use parallel execution when possible
  • Optimize plugin chains
  • Consider resource usage

3. Security Guidelines​

  • Layer security plugins properly
  • Implement defense in depth
  • Handle failures gracefully
  • Maintain secure defaults

Next Steps​