User Tools

Site Tools


discourse

Table of Contents

Discourse Installation on Fresh Ubuntu Server with Backup Restoration

Prerequisites

  • Fresh Ubuntu Server installation
  • Root or sudo access
  • Existing Discourse backup file (.tar.gz)
  • Domain name pointed to server IP

Installation Steps

1. Initial System Setup

sudo -s
apt-get install git

2. Clone Discourse Docker Repository

git clone https://github.com/discourse/discourse_docker.git /var/discourse
cd /var/discourse
chmod 700 containers

3. Edit app.yml for Reverse Proxy Configuration

Before running discourse-setup, edit ~/var/discourse/containers/app.yml:

Comment Out SSL Templates

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  ## SSL templates - COMMENTED OUT for reverse proxy setup
  # - "templates/web.ssl.template.yml"
  # - "templates/web.letsencrypt.ssl.template.yml"

Change Port Mapping

expose:
  - "8080:80"   # expose container port 80 on host port 8080
  # - "443:443" # https - commented out

Rationale: Reverse proxy (nginx) will handle TLS termination and forward HTTP traffic to port 8080. Configuring this before setup avoids needing to rebuild the container later.

4. Run Discourse Setup

./discourse-setup --skip-connection-test

Note: This process takes time. Good opportunity for a coffee break.

Configuration Parameters:

Provide these values during setup:

  • Hostname: example.com
  • Developer Email: admin@example.com
  • SMTP Server: Your SMTP server address
  • SMTP Port: Typically 587 for TLS
  • SMTP Username: Your SMTP username
  • SMTP Password: Your SMTP password
  • Notification Email: Email address for outgoing notifications
  • Let's Encrypt Email: OFF (critical - disables Let's Encrypt since TLS will be handled by reverse proxy)

5. Copy Backup File to Server

From your local machine (or source location), copy the backup file:

scp -P <ssh-port> backup-file.tar.gz user@example.com:/var/discourse/shared/standalone/backups/default/

Notes:

  • Replace <ssh-port> with your SSH port (default is 22, use -P 22 or omit if default)
  • Replace backup-file.tar.gz with your actual backup filename
  • Adjust username and hostname as needed for your setup

6. Restore Backup from Admin Panel

  1. Access Discourse at http://example.com (or server IP if DNS not yet configured)
  2. Navigate to Admin → Backups
  3. Locate the uploaded backup file in the list
  4. Click Restore on the backup file
  5. Confirm the restoration

Note: The restoration process will restart Discourse and may take several minutes depending on backup size.

7. Install and Configure Nginx

#This nginx setup in this guide is but one valid configuration.

apt-get install nginx

8. Create Nginx Configuration (Example)

Note: This configuration represents one working approach for proxying Discourse through nginx. Alternative configurations may work equally well depending on your specific requirements, existing infrastructure, or preferred nginx patterns.

Create /etc/nginx/sites-available/example.com:

server {
    listen 80;
    server_name example.com;
 
    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
 
        # Required for Discourse
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
 
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
 
        # Buffer settings
        proxy_buffering off;
        proxy_redirect off;
    }
 
    client_max_body_size 10m;
}

Key Discourse Requirements:

  • X-Forwarded-For and X-Forwarded-Host headers
  • WebSocket support (proxy_http_version 1.1, Upgrade, Connection headers)
  • Buffering disabled (proxy_buffering off)
  • File upload size limit (client_max_body_size 10m)

9. Enable Nginx Configuration

# Remove default site (prevents conflicts)
rm /etc/nginx/sites-enabled/default
 
# Create symlink to enable site
ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/
 
# Test configuration
nginx -t
 
# Reload nginx
systemctl reload nginx

10. Install Certbot

snap install --classic certbot

11. Generate SSL Certificate

certbot

Certbot will run interactively and:

  • Detect the nginx configuration for your domain
  • Prompt for domain selection
  • Automatically generate Let's Encrypt certificates
  • Modify nginx configuration to enable HTTPS
  • Configure automatic certificate renewal

12. Reload Nginx

systemctl reload nginx

Your Discourse instance should now be accessible at https://example.com

Important Notes

YAML Sensitivity

  • Be extremely careful with whitespace and alignment when editing app.yml
  • Validate syntax at http://www.yamllint.com/ if needed

Undocumented Discourse Behavior

  • The 8080:80 port mapping for reverse proxy setups is not well-documented in official Discourse guides
  • Entering OFF for Let's Encrypt email is not clearly documented
  • Commenting out SSL templates is required but not obvious for reverse proxy scenarios
  • Discourse's Docker setup is optimized for their standard configuration, making reverse proxy setups unnecessarily opaque

Security Considerations

  • Use strong, unique passwords for SMTP and admin accounts
  • Secure transfer of backup files containing sensitive data
  • Review and update SMTP credentials appropriately for your environment
  • Consider firewall rules to restrict access to port 8080

Troubleshooting

Nginx Shows Default Welcome Page

  • Verify default site removed: ls /etc/nginx/sites-enabled/
  • Confirm symlink exists: ls -la /etc/nginx/sites-enabled/example.com
  • Test config: nginx -t
  • Reload: systemctl reload nginx

SSL Certificate Permission Errors

  • Run nginx test with sudo: sudo nginx -t
  • Check for other configs referencing non-existent certificates
  • Ensure only HTTP (port 80) configuration exists before running certbot

Backup Not Visible in Admin Panel

  • Verify file copied to correct path: /var/discourse/shared/standalone/backups/default/
  • Check file permissions: ls -la /var/discourse/shared/standalone/backups/default/
  • Ensure filename ends in .tar.gz

==== SCP Syntax Issues ====https://wiki.scorpi.us/doku.php?id=discourse&do=

  • Port flag is capital -P not lowercase -p
  • Syntax: scp -P <port> <source> <user>@<host>:<destination>
  • Example: scp -P 22 backup.tar.gz user@host:/path/

```

Discourse API Configuration

Webhook Configuration

Available Events

Topic Events

  • Topic is created
  • Topic is revised
  • Topic is updated
  • Topic is deleted
  • Topic is recovered

Post Events

  • Post is created
  • Post is updated
  • Post is deleted
  • Post is recovered

User Events

  • User logged in
  • User logged out
  • User confirmed e-mail
  • User is created
  • User is approved
  • User is updated
  • User is deleted
  • User is suspended
  • User is unsuspended
  • User is anonymized

Group Events

  • Group is created
  • Group is updated
  • Group is deleted

Category Events

  • Category is created
  • Category is updated
  • Category is deleted

Tag Events

  • Tag is created
  • Tag is updated
  • Tag is deleted

Chat Events (Critical for sync)

  • Message is created
  • Message is edited
  • Message is trashed
  • Message is restored

Other Events

  • Reviewable Events
  • Notification Events
  • Solved Events
  • Badge Events
  • Group User Events
  • Like Events
  • User Promoted Events
  • Topic Voting Events

Webhook Settings

Content Type: application/json (recommended for easy parsing)

Secret: Optional string for generating HMAC signatures (implement signature verification to prevent spoofed webhooks)

Filtering Options:

  • Triggered Categories - only fire webhooks for specific categories
  • Triggered Tags - only fire webhooks for specific tags
  • Triggered Groups - only fire webhooks for specific groups
  • Leave blank to trigger for all

TLS Certificate Check: Enable unless using self-signed certs in development

Configured Webhook Endpoints

hobbiesync webhook:

general webhook:

Path structure: /chat/hooks/[uuid] suggests custom webhook handler service

Chat API Endpoints

Base path: /chat/api/channels

Channel Operations

List channels

GET /chat/api/channels

Get specific channel

GET /chat/api/channels/{channel_id}

List messages

GET /chat/api/channels/{channel_id}/messages

Send message

POST /chat/api/channels/{channel_id}/messages

Authentication

Discourse API requires headers:

Api-Key: {your_api_key}
Api-Username: {bot_username}

API Credentials:

  • API Key: [REDACTED]
  • Username: syncbot

Security Notes:

  • Store credentials in environment variables or secrets management (Vault)
  • API key has full access as the bot user
  • Revoke/rotate if compromised
  • Ensure bot user has appropriate permissions (chat access, posting rights to target channels)

Architecture Considerations

Critical Events for Chat Sync

Primary focus:

  • Chat events (message created/edited/trashed/restored) - core sync functionality
  • Post events - if bridging forum posts to Discord channels
  • User events (login/logout/created) - for presence sync and user mapping

Bidirectional Sync Challenges

Discourse → Discord:

  • Webhooks handle this direction
  • Parse incoming webhook payloads
  • Map to Discord API calls

Discord → Discourse:

  • Discord bot with message event listeners
  • POST to Discourse API /chat/api/channels/{channel_id}/messages
  • Requires Discord bot token and event subscriptions

Loop Prevention:

  • Track message IDs/sources to avoid infinite echo
  • Store mapping of Discourse message ID ↔ Discord message ID
  • Ignore messages from own bot user

State Management

Required mappings to store:

  • Message ID mapping (Discourse chat message ID ↔ Discord message ID)
  • User mapping (Discourse username ↔ Discord user/webhook representation)
  • Channel mapping (Discourse channel ID ↔ Discord channel ID)
  • Edit/delete operations require retrieving these mappings

Event Filtering Strategy

Options:

  • “Send me everything” - receives all events (high bandwidth/processing)
  • Selective events - choose specific event types
  • Category/tag/group filters - sync only specific Discourse areas to specific Discord channels

Recommendation: Use selective events and filtering to reduce noise and processing load

Implementation Flow

Webhook Receiver (Discourse → Discord):

  1. Receive webhook POST request
  2. Verify HMAC signature if secret configured
  3. Parse JSON payload
  4. Extract message/event data
  5. Check for existing mapping (to detect edits/deletes)
  6. Call Discord API
  7. Store message ID mapping

Discord Bot (Discord → Discourse):

  1. Listen for Discord message events
  2. Check if message is from bot (skip to prevent loop)
  3. Format message for Discourse
  4. POST to Discourse API with credentials
  5. Store message ID mapping

Message History/Backfill:

  • Use GET /chat/api/channels/{channel_id}/messages for initial sync
  • Implement pagination if needed
  • Consider rate limiting

Profile Image/Avatar Synchronization

How Avatar Syncing Works

The bridge synchronizes profile images/avatars bidirectionally between Discord and Discourse using webhook-based user impersonation.

Discourse → Discord Avatar Flow

  1. Bridge polls Discourse Chat API every 1.5 seconds for new messages
  2. Each message includes user info with an avatar_template field
  3. Avatar template format: /user_avatar/site/username/{size}/123_2.png
  4. Bridge calls build_avatar_url() to construct full URL:
    • Replaces {size} placeholder with pixel size (default: 128)
    • Prepends Discourse base URL if path is relative
  5. Full avatar URL passed to Discord webhook avatar_url parameter
  6. Discord displays the message with the Discourse user's actual avatar
# Avatar URL construction (bridge.py lines 160-186)
def build_avatar_url(self, avatar_template: str, size: int = 128) -> Optional[str]:
    if not avatar_template:
        return None
    avatar_path = avatar_template.replace('{size}', str(size))
    if not avatar_path.startswith('/'):
        avatar_path = '/' + avatar_path
    return f"{self.base_url}{avatar_path}"

Discord → Discourse Avatar Flow

Discord user avatars are not synced to Discourse. Messages from Discord appear with the bot's configured username in Discourse, prefixed with [Discord] username: to identify the source user.

User Impersonation

The bridge uses Discord webhooks to display messages from Discourse users with their actual:

  • Username - Discourse display name shown in Discord
  • Avatar - Discourse profile image displayed in Discord

This creates a seamless experience where Discourse users appear as distinct users in Discord channels.


Image Upload/Download Synchronization

Discord → Discourse Images

  1. Discord attachments detected via message.attachments
  2. Images downloaded to temporary files using download_image()
  3. Uploaded to Discourse via /uploads.json?type=image endpoint
  4. Upload URL appended to message as markdown: ![]({image_url})
  5. Temporary files cleaned up after processing

Discourse → Discord Images

  1. Images extracted from uploads field in Discourse messages
  2. Relative URLs converted to absolute URLs (prepends base_url)
  3. URLs appended to message content
  4. Discord auto-embeds images from URLs

GIF and Video Handling

Giphy/Tenor URL Detection

The bridge detects and converts Giphy/Tenor share URLs to direct media URLs for proper embedding:

MP4 to GIF Conversion

When Giphy/Tenor returns MP4 URLs (common for larger animations), the bridge converts them to GIF using FFmpeg:

  1. Downloads MP4 to temporary file
  2. Generates optimized color palette for compression
  3. Converts to GIF with configurable settings:
    • Frame rate (default: 15 FPS)
    • Width scaling (default: 480px)
    • Duration limit (default: 10 seconds)
    • CPU thread limit (default: 3 threads)
  4. Uploads converted GIF to Discourse
  5. Cleans up temporary files

FFmpeg configuration options in bridge_config.json:

{
  "bridge": {
    "ffmpeg_threads": 3,
    "ffmpeg_duration_limit": 10,
    "ffmpeg_fps": 15,
    "ffmpeg_scale_width": 480
  }
}

State Management

Message Tracking

The bridge tracks processed message IDs to prevent duplicates and enable resumption after restarts.

State file structure (bridge_state.json):

{
  "discord_channels": {
    "channel_id": {
      "last_message_id": 1234567890,
      "last_updated": "2024-01-15T12:34:56Z"
    }
  },
  "discourse_channels": {
    "channel_id": {
      "last_message_id": 456,
      "last_updated": "2024-01-15T12:34:56Z"
    }
  }
}

State Initialization

On first run, the bridge initializes state with the latest message IDs to prevent flooding channels with historical messages.


Loop Prevention

The bridge implements multiple safeguards to prevent infinite message loops:

Discord → Discourse

  • Messages prefixed with [Discord] tag
  • Webhook messages (non-user) are detected and skipped
  • Bot's own messages filtered by user ID

Discourse → Discord

  • Messages containing [Discord] prefix are ignored
  • Prevents echoing messages that originated from Discord

Configuration Reference

Full Configuration Structure

{
  "discord": {
    "token": "bot_token_here",
    "channels": [
      {
        "name": "channel_name",
        "discord_channel_id": 123456789012345678,
        "discourse_channel_id": 2,
        "discord_webhook_url": null,
        "discourse_webhook_url": "https://discourse.example.com/chat/hooks/UUID",
        "enabled": true
      }
    ]
  },
  "discourse": {
    "base_url": "https://discourse.example.com",
    "api_key": "api_key_here",
    "username": "syncbot"
  },
  "bridge": {
    "polling_interval": 1.5,
    "state_file": "bridge_state.json",
    "log_level": "INFO",
    "rate_limit_delay": 1.0,
    "use_discord_webhooks": true,
    "ffmpeg_threads": 3,
    "ffmpeg_duration_limit": 10,
    "ffmpeg_fps": 15,
    "ffmpeg_scale_width": 480
  }
}

Configuration Options

Option Default Description
polling_interval 1.5 Seconds between Discourse API polls
state_file bridge_state.json Path to state persistence file
log_level INFO Logging verbosity (DEBUG, INFO, WARNING, ERROR)
rate_limit_delay 1.0 Minimum seconds between API calls
use_discord_webhooks true Enable user impersonation via webhooks
ffmpeg_threads 3 CPU threads for MP4 conversion
ffmpeg_duration_limit 10 Max seconds of video to convert
ffmpeg_fps 15 Frame rate for converted GIFs
ffmpeg_scale_width 480 Width in pixels for converted GIFs

Environment Variable Overrides

  • DISCORD_TOKEN - Discord bot token
  • DISCOURSE_API_KEY - Discourse API key
  • DISCOURSE_USERNAME - Discourse bot username
  • DISCOURSE_BASE_URL - Discourse instance URL
  • LOG_LEVEL - Logging level
  • CONFIG_FILE - Path to config file
  • STATE_FILE - Path to state file

Channel Mapping

Multi-Channel Support

The bridge supports multiple independent channel pairs. Each pair has:

  • Discord channel ID
  • Discourse channel ID
  • Separate webhook URLs
  • Independent enable/disable flag
  • Separate state tracking

Adding a New Channel Pair

  1. Add entry to discord.channels array in config
  2. Set discord_channel_id and discourse_channel_id
  3. Create Discourse webhook and add URL to discourse_webhook_url
  4. Set enabled: true
  5. Restart bridge - Discord webhook created automatically

Error Handling and Retry Logic

Automatic Retry

  • Failed Discord webhook sends do not update state
  • Message remains in queue and retries on next poll cycle
  • Prevents message loss on temporary failures

Graceful Degradation

  • API errors caught and logged, processing continues
  • Image download failures don't block message delivery
  • FFmpeg conversion failures fall back to posting MP4 URL

Timeouts

  • API requests: 30-60 second timeout
  • FFmpeg conversion: 30-60 second timeout
  • Prevents indefinite hangs

Docker Deployment

Container Configuration

services:
  syncbot:
    build: .
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    volumes:
      - ./bridge-config.json:/app/bridge-config.json:ro
      - ./bridge_state.json:/app/bridge_state.json

Security

  • Runs as non-root user inside container
  • Config mounted read-only
  • State file writable for persistence
  • Resource limits prevent runaway processes

Message Flow Diagrams

Discord → Discourse

Discord User posts message
         ↓
on_message() event triggered
         ↓
Check: Is webhook message? → Skip
Check: Is bot message? → Skip
         ↓
Find channel config by Discord channel ID
         ↓
Download image attachments (if any)
         ↓
Upload images to Discourse
         ↓
Extract/convert Giphy/Tenor URLs
         ↓
Convert MP4 to GIF (if needed)
         ↓
Format: "[Discord] username: message"
         ↓
Append image URLs as markdown
         ↓
Send via Discourse webhook
         ↓
Update Discord channel state

Discourse → Discord

poll_discourse() runs every 1.5 seconds
         ↓
For each enabled channel:
         ↓
GET messages after last_message_id
         ↓
For each new message:
         ↓
Check: Has [Discord] prefix? → Skip
         ↓
Extract username from user info
Extract avatar_template from user info
         ↓
Build avatar URL (replace {size}, add base_url)
         ↓
Extract image uploads, make URLs absolute
         ↓
Truncate content to 2000 chars if needed
         ↓
Send via Discord webhook:
  - username: Discourse username
  - avatar_url: Discourse avatar
  - content: message text
         ↓
Success? → Update Discourse channel state
Failed? → Retry on next poll

Dependencies

Python Packages

  • discord.py>=2.0.0 - Discord bot framework
  • requests>=2.28.0 - HTTP requests
  • aiohttp>=3.8.0 - Async HTTP
  • python-dotenv>=0.19.0 - Environment variables

System Dependencies

  • Python 3.11+
  • FFmpeg (for MP4 to GIF conversion)

Known Limitations

Not Currently Implemented

  • Message Edit Sync - Edits are not synchronized
  • Message Delete Sync - Deletions are not synchronized
  • Reaction Sync - Emoji reactions not bridged
  • PluralKit Support - Removed in current version (was in earlier versions per git history)
  • Discord Avatar → Discourse - Only Discourse avatars sync to Discord, not vice versa

Troubleshooting

Messages Not Syncing

  • Check enabled: true in channel config
  • Verify Discord bot has webhook permissions in channel
  • Check Discourse webhook URL is correct
  • Review logs for API errors

Duplicate Messages

  • Check state file permissions (must be writable)
  • Verify state file path matches config

Images Not Appearing

  • Verify Discourse API key has upload permissions
  • Check temporary directory is writable
  • Review logs for upload errors

GIFs Not Converting

  • Ensure FFmpeg is installed: ffmpeg -version
  • Check FFmpeg is in PATH
  • Review logs for conversion errors
  • Try reducing ffmpeg_duration_limit or ffmpeg_scale_width
discourse.txt · Last modified: by 208.92.105.160