Notification Handlers

What are Notification Handlers?

Notification handlers are integrations that deliver alerts from your monitoring system to your preferred communication channels. When monitors detect issues, notification handlers ensure the right people are informed through their preferred methods - whether that's email, Slack, SMS, phone calls, or other platforms.

Why Notification Handlers are Critical

Rapid Response

• Immediate alerts: Get notified the moment issues occur
• Multiple channels: Receive alerts through various communication methods
• Team coordination: Ensure the right team members are informed
• Escalation support: Route alerts to different people based on severity

Workflow Integration

• Existing tools: Integrate with tools your team already uses
• Automation: Trigger automated responses and workflows
• Documentation: Automatically log incidents in external systems
• Context sharing: Include relevant details with each alert

Reliability and Redundancy

• Multiple backup methods: Use several notification methods for critical alerts
• Failure tolerance: If one method fails, others can still deliver alerts
• Geographic distribution: Reach team members in different locations
• Device flexibility: Receive alerts on computers, phones, or other devices

How to Access Notification Handlers

Manage notification handlers through:

• Main dashboard → Notification Handlers section
• Sidebar navigation → Notification Handlers
• Direct URL: /notification-handlers

Available Notification Types

Email Notifications

• Purpose: Traditional email alerts to any email address
• Best for: Formal notifications, documentation, and team members who prefer email
• Features: Rich formatting, detailed incident information, attachment support
• Setup: Simply provide email address - no additional configuration needed

Slack Integration

• Purpose: Send alerts directly to Slack channels or users
• Best for: Team channels, real-time collaboration, and DevOps workflows
• Features: Rich formatting, channel mentions, threaded conversations
• Setup: Requires Slack webhook URL from your Slack workspace

Discord Integration

• Purpose: Deliver alerts to Discord servers and channels
• Best for: Gaming communities, development teams using Discord, informal communication
• Features: Embed formatting, role mentions, server-wide notifications
• Setup: Requires Discord webhook URL from your server

Microsoft Teams

• Purpose: Send notifications to Microsoft Teams channels
• Best for: Corporate environments using Microsoft 365, enterprise teams
• Features: Adaptive cards, @mentions, integration with Microsoft ecosystem
• Setup: Requires Teams webhook URL from your Teams channel

Telegram

• Purpose: Send messages via Telegram to users or groups
• Best for: Personal notifications, international teams, secure messaging
• Features: Instant delivery, cross-platform support, group messaging
• Setup: Requires Telegram bot token and chat ID

SMS (Twilio)

• Purpose: Send text messages to mobile phones
• Best for: Critical alerts, on-call engineers, when internet isn't available
• Features: High reliability, works without internet, immediate attention
• Setup: Requires Twilio account and phone number configuration

Phone Calls (Twilio)

• Purpose: Make automated phone calls for critical alerts
• Best for: Highest priority incidents, on-call escalation, critical system failures
• Features: Voice messages, immediate attention, hard to ignore
• Setup: Requires Twilio account and voice configuration

WhatsApp

• Purpose: Send messages via WhatsApp Business API
• Best for: International teams, regions where WhatsApp is primary communication
• Features: High engagement, multimedia support, global reach
• Setup: Requires WhatsApp Business API configuration

X (Twitter)

• Purpose: Post alerts as tweets to your Twitter account
• Best for: Public status updates, transparent communication, social media presence
• Features: Public visibility, social engagement, status broadcasting
• Setup: Requires Twitter API credentials and authentication

Google Chat

• Purpose: Send messages to Google Chat spaces
• Best for: Teams using Google Workspace, G Suite environments
• Features: Rich formatting, @mentions, integration with Google services
• Setup: Requires Google Chat webhook URL

Webhook Integration

• Purpose: Send HTTP POST requests to custom endpoints
• Best for: Custom integrations, automation systems, third-party tools
• Features: Flexible payload format, custom processing, unlimited integrations
• Setup: Provide endpoint URL and configure payload format

Internal Notifications

• Purpose: Display alerts within the monitoring dashboard
• Best for: Team members who frequently use the monitoring dashboard
• Features: Integrated experience, no external setup required, centralized view
• Setup: Enabled by default for all users

Push Notifications

• Purpose: Send browser push notifications to subscribed devices
• Best for: Desktop users, immediate alerts while browsing
• Features: Browser integration, desktop notifications, clickable actions
• Setup: Users subscribe through browser notification permissions

Creating Notification Handlers

Step 1: Navigate to Notification Handler Creation

1. Go to Notification Handlers section
2. Click "Create Notification Handler"
3. Select the notification type you want to configure

Step 2: Basic Configuration

Handler Details

• Name: Descriptive name for the handler (e.g., "DevOps Team Slack", "On-Call SMS")
• Type: Select the notification method (email, Slack, SMS, etc.)
• Description: Optional notes about the handler's purpose

Target Configuration

Configuration varies by notification type:

• Email: Recipient email address
• Slack: Webhook URL from Slack channel settings
• SMS: Phone number and Twilio configuration
• Webhook: Endpoint URL and authentication details

Step 3: Advanced Settings

Message Customization

• Alert messages: Customize the format and content of alert messages
• Variable substitution: Use placeholders for dynamic content (monitor name, status, etc.)
• Rich formatting: Use platform-specific formatting (Markdown for Slack, HTML for email)
• Language settings: Configure message language and localization

Delivery Options

• Retry settings: Configure how many times to retry failed deliveries
• Timeout settings: Set maximum time to wait for delivery confirmation
• Rate limiting: Control frequency to avoid overwhelming recipients
• Quiet hours: Suppress non-critical notifications during specified times

Step 4: Testing and Validation

Test Integration

1. After creating the handler, use the "Test" feature
2. Send a sample notification to verify configuration
3. Check that the message is received correctly
4. Verify formatting and content appear as expected

Troubleshooting Setup

• Check credentials: Verify API keys, tokens, and URLs are correct
• Network connectivity: Ensure firewall rules allow outbound connections
• Permission settings: Confirm the integration has necessary permissions
• Rate limits: Check if platform-specific rate limits are being exceeded

Configuring Specific Integrations

Setting Up Slack Integration

1. In Slack, go to your workspace settings
2. Navigate to Apps & integrations
3. Search for and add "Incoming Webhooks"
4. Choose the channel where you want to receive alerts
5. Copy the generated webhook URL
6. Paste the URL into your notification handler configuration
7. Test the integration to ensure messages appear in your Slack channel

Setting Up Discord Integration

1. In Discord, go to your server settings
2. Navigate to Integrations → Webhooks
3. Click "Create Webhook"
4. Choose the channel for notifications
5. Customize the webhook name and avatar (optional)
6. Copy the webhook URL
7. Configure your notification handler with the webhook URL
8. Test to ensure messages appear correctly in Discord

Setting Up SMS with Twilio

1. Create a Twilio account and verify your identity
2. Obtain a Twilio phone number for sending messages
3. Get your Account SID and Auth Token from Twilio console
4. Configure the notification handler with:
   • Twilio Account SID
   • Twilio Auth Token
   • From phone number (your Twilio number)
   • To phone number (recipient)
5. Test the integration to ensure SMS messages are delivered

Setting Up Email Notifications

1. Simply provide the recipient email address
2. Optionally customize the email subject and format
3. Configure any email filtering rules to ensure delivery
4. Test to confirm emails are received and not marked as spam
5. Consider adding the sender to your contacts or whitelist

Setting Up Webhook Integration

1. Create an endpoint that accepts HTTP POST requests
2. Configure the endpoint to process JSON notification data
3. Set up authentication if required (API keys, tokens, etc.)
4. Provide the endpoint URL in your notification handler
5. Configure any custom headers or authentication
6. Test the integration to ensure your endpoint receives and processes data correctly

Assigning Notification Handlers to Monitors

During Monitor Creation

• When creating monitors, select which notification handlers to use
• Choose different handlers for different types of monitors
• Consider the urgency and audience for each monitor
• Select multiple handlers for redundancy and broader reach

Editing Existing Monitors

1. Go to your monitor details page
2. Click "Edit" or "Settings"
3. Update the notification handler assignments
4. Save changes to apply new notification settings

Bulk Assignment

• Use bulk operations to assign handlers to multiple monitors
• Select monitors by project, type, or other criteria
• Apply notification handler changes to all selected monitors
• Review assignments to ensure appropriate coverage

Managing Notification Handlers

Editing Handler Configuration

1. Go to your notification handlers list
2. Click on the handler you want to modify
3. Update configuration settings as needed
4. Test the updated configuration
5. Save changes to apply updates

Enabling and Disabling Handlers

• Temporary disable: Turn off handlers without deleting them
• Maintenance periods: Disable during maintenance to avoid false alerts
• Team changes: Disable handlers for team members who are unavailable
• Re-enable: Turn handlers back on when needed

Deleting Handlers

• Before deleting: Ensure no monitors depend on the handler
• Reassignment: Update monitors to use alternative handlers
• Confirmation: Verify deletion won't leave monitors without notifications
• Documentation: Keep records of handler changes for audit purposes

Understanding Notification Content

Alert Information Included

• Monitor details: Monitor name, type, and target
• Status change: Current status and previous status
• Timestamp: When the issue was detected
• Response details: Response time, status codes, error messages
• Context: Additional information about the failure
• Links: Direct links to monitor details and logs

Message Types

• Down alerts: Sent when monitors fail
• Up alerts: Sent when monitors recover
• Maintenance notifications: Sent for scheduled maintenance
• Test messages: Sent when testing handler configuration

Customization Options

• Message templates: Customize the format and content of notifications
• Variable substitution: Include dynamic information like monitor names and timestamps
• Conditional content: Show different information based on the type of alert
• Branding: Include company information and styling

Best Practices for Notification Handlers

Handler Selection and Strategy

• Multiple channels: Use several notification methods for critical monitors
• Audience targeting: Send notifications to relevant team members only
• Urgency matching: Use more immediate methods (SMS, calls) for critical issues
• Geographic considerations: Account for time zones and regional preferences

Avoiding Alert Fatigue

• Appropriate thresholds: Set monitor sensitivity to reduce false positives
• Rate limiting: Prevent flooding recipients with too many alerts
• Grouping: Combine related alerts when possible
• Escalation procedures: Use different notification levels for different severity

Reliability and Redundancy

• Backup methods: Configure multiple notification types for critical monitors
• Handler testing: Regularly test all notification handlers
• Dependency management: Ensure notification systems are independent of monitored services
• Failover planning: Have backup notification methods ready

Team Management

• Role-based notifications: Send different alerts to different team roles
• On-call rotation: Update notification handlers for on-call schedules
• Vacation coverage: Adjust notifications when team members are unavailable
• New team members: Set up notifications for new team members

Troubleshooting Common Issues

Notifications Not Being Received

• Handler configuration: Verify all settings are correct
• Test functionality: Use the test feature to check delivery
• Authentication issues: Check API keys, tokens, and credentials
• Network connectivity: Ensure outbound connections are allowed
• Rate limits: Check if platform limits are being exceeded

Delayed Notifications

• Platform delays: Some services have inherent delivery delays
• Rate limiting: Too many notifications may cause delays
• Network issues: Temporary network problems can cause delays
• Queue processing: High volumes may create processing delays

Formatting Issues

• Platform differences: Different platforms support different formatting
• Character limits: Some platforms have message length restrictions
• Encoding problems: Special characters may not display correctly
• Template errors: Check custom message templates for syntax errors

Authentication Failures

• Expired credentials: API keys and tokens may expire
• Permission changes: Platform permissions may have been modified
• Account issues: Third-party accounts may be suspended or limited
• Configuration drift: Settings may have changed on external platforms

Security Considerations

Credential Management

• Secure storage: API keys and tokens are encrypted at rest
• Access control: Limit who can view or modify notification handlers
• Regular rotation: Periodically update API keys and credentials
• Least privilege: Use minimum necessary permissions for integrations

Information Disclosure

• Sensitive data: Avoid including sensitive information in notifications
• Public channels: Be careful with notifications sent to public channels
• Audit trails: Maintain logs of notification delivery and access
• Compliance: Ensure notification practices meet regulatory requirements

Advanced Notification Strategies

Escalation Procedures

• Tier 1 notifications: Initial alerts to primary responders
• Tier 2 escalation: Additional notifications if issues persist
• Management escalation: Notify management for severe or prolonged incidents
• Automated escalation: Use external tools to manage escalation timing

Context-Aware Notifications

• Business hours: Different notification urgency during business vs. off hours
• Service criticality: More aggressive notifications for critical services
• Incident correlation: Group related alerts to avoid noise
• Environmental awareness: Different handling for production vs. development

Integration with Incident Management

• Ticket creation: Automatically create support tickets from alerts
• Status updates: Send updates as incidents progress
• Resolution notifications: Confirm when issues are resolved
• Post-mortem triggers: Initiate review processes for major incidents

Tips for Effective Notification Management

• Start simple: Begin with email and add other methods gradually
• Test everything: Regularly test all notification handlers
• Document procedures: Keep records of notification setup and processes
• Review regularly: Periodically assess notification effectiveness
• Train your team: Ensure team members understand notification procedures
• Monitor the monitors: Watch for notification delivery issues
• Plan for scale: Ensure notification systems can handle growth
• Consider compliance: Meet any regulatory notification requirements
• Balance urgency: Match notification method to incident severity
• Stay updated: Keep up with changes to notification platforms and APIs