====== Discourse Installation on Fresh Ubuntu Server with Backup Restoration ======

===== Prerequisites =====
  * Fresh Ubuntu Server installation
  * Root or sudo access
  * Existing Discourse backup file (.tar.gz)
  * Domain name pointed to server IP

===== Installation Steps =====

==== 1. Initial System Setup ====
<code bash>
sudo -s
apt-get install git
</code>

==== 2. Clone Discourse Docker Repository ====
<code bash>
git clone https://github.com/discourse/discourse_docker.git /var/discourse
cd /var/discourse
chmod 700 containers
</code>

==== 3. Edit app.yml for Reverse Proxy Configuration ====

**Before running discourse-setup**, edit ''~/var/discourse/containers/app.yml'':

=== Comment Out SSL Templates ===
<code yaml>
templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  ## SSL templates - COMMENTED OUT for reverse proxy setup
  # - "templates/web.ssl.template.yml"
  # - "templates/web.letsencrypt.ssl.template.yml"
</code>

=== Change Port Mapping ===
<code yaml>
expose:
  - "8080:80"   # expose container port 80 on host port 8080
  # - "443:443" # https - commented out
</code>

**Rationale:** Reverse proxy (nginx) will handle TLS termination and forward HTTP traffic to port 8080. Configuring this before setup avoids needing to rebuild the container later.

==== 4. Run Discourse Setup ====
<code bash>
./discourse-setup --skip-connection-test
</code>

**Note:** This process takes time. Good opportunity for a coffee break.

**Configuration Parameters:**

Provide these values during setup:

  * **Hostname:** ''example.com''
  * **Developer Email:** ''admin@example.com''
  * **SMTP Server:** Your SMTP server address
  * **SMTP Port:** Typically ''587'' for TLS
  * **SMTP Username:** Your SMTP username
  * **SMTP Password:** Your SMTP password
  * **Notification Email:** Email address for outgoing notifications
  * **Let's Encrypt Email:** ''OFF'' (critical - disables Let's Encrypt since TLS will be handled by reverse proxy)

==== 5. Copy Backup File to Server ====

From your local machine (or source location), copy the backup file:
<code bash>
scp -P <ssh-port> backup-file.tar.gz user@example.com:/var/discourse/shared/standalone/backups/default/
</code>

**Notes:**
  * Replace ''<ssh-port>'' with your SSH port (default is 22, use ''-P 22'' or omit if default)
  * Replace ''backup-file.tar.gz'' with your actual backup filename
  * Adjust username and hostname as needed for your setup

==== 6. Restore Backup from Admin Panel ====

  - Access Discourse at ''http://example.com'' (or server IP if DNS not yet configured)
  - Navigate to **Admin → Backups**
  - Locate the uploaded backup file in the list
  - Click **Restore** on the backup file
  - Confirm the restoration

**Note:** The restoration process will restart Discourse and may take several minutes depending on backup size.

==== 7. Install and Configure Nginx ====

#This nginx setup in this guide is but one valid configuration.
<code bash>
apt-get install nginx
</code>

==== 8. Create Nginx Configuration (Example) ====

**Note:** This configuration represents one working approach for proxying Discourse through nginx. Alternative configurations may work equally well depending on your specific requirements, existing infrastructure, or preferred nginx patterns.

Create ''/etc/nginx/sites-available/example.com'':
<code nginx>
server {
    listen 80;
    server_name example.com;
    
    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        
        # Required for Discourse
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
        
        # Buffer settings
        proxy_buffering off;
        proxy_redirect off;
    }
    
    client_max_body_size 10m;
}
</code>

**Key Discourse Requirements:**
  * ''X-Forwarded-For'' and ''X-Forwarded-Host'' headers
  * WebSocket support (''proxy_http_version 1.1'', ''Upgrade'', ''Connection'' headers)
  * Buffering disabled (''proxy_buffering off'')
  * File upload size limit (''client_max_body_size 10m'')

==== 9. Enable Nginx Configuration ====
<code bash>
# Remove default site (prevents conflicts)
rm /etc/nginx/sites-enabled/default

# Create symlink to enable site
ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

# Test configuration
nginx -t

# Reload nginx
systemctl reload nginx
</code>

==== 10. Install Certbot ====
<code bash>
snap install --classic certbot
</code>

==== 11. Generate SSL Certificate ====
<code bash>
certbot
</code>

Certbot will run interactively and:
  * Detect the nginx configuration for your domain
  * Prompt for domain selection
  * Automatically generate Let's Encrypt certificates
  * Modify nginx configuration to enable HTTPS
  * Configure automatic certificate renewal

==== 12. Reload Nginx ====
<code bash>
systemctl reload nginx
</code>

Your Discourse instance should now be accessible at ''https://example.com''

===== Important Notes =====

==== YAML Sensitivity ====
  * Be extremely careful with whitespace and alignment when editing ''app.yml''
  * Validate syntax at http://www.yamllint.com/ if needed

==== Undocumented Discourse Behavior ====
  * The ''8080:80'' port mapping for reverse proxy setups is not well-documented in official Discourse guides
  * Entering ''OFF'' for Let's Encrypt email is not clearly documented
  * Commenting out SSL templates is required but not obvious for reverse proxy scenarios
  * Discourse's Docker setup is optimized for their standard configuration, making reverse proxy setups unnecessarily opaque

==== Security Considerations ====
  * Use strong, unique passwords for SMTP and admin accounts
  * Secure transfer of backup files containing sensitive data
  * Review and update SMTP credentials appropriately for your environment
  * Consider firewall rules to restrict access to port 8080

===== Troubleshooting =====

==== Nginx Shows Default Welcome Page ====
  * Verify default site removed: ''ls /etc/nginx/sites-enabled/''
  * Confirm symlink exists: ''ls -la /etc/nginx/sites-enabled/example.com''
  * Test config: ''nginx -t''
  * Reload: ''systemctl reload nginx''

==== SSL Certificate Permission Errors ====
  * Run nginx test with sudo: ''sudo nginx -t''
  * Check for other configs referencing non-existent certificates
  * Ensure only HTTP (port 80) configuration exists before running certbot

==== Backup Not Visible in Admin Panel ====
  * Verify file copied to correct path: ''/var/discourse/shared/standalone/backups/default/''
  * Check file permissions: ''ls -la /var/discourse/shared/standalone/backups/default/''
  * Ensure filename ends in ''.tar.gz''

==== SCP Syntax Issues ====https://wiki.scorpi.us/doku.php?id=discourse&do=
  * Port flag is capital ''-P'' not lowercase ''-p''
  * Syntax: ''scp -P <port> <source> <user>@<host>:<destination>''
  * Example: ''scp -P 22 backup.tar.gz user@host:/path/''

```
===== Discourse API Configuration =====

==== Webhook Configuration ====

=== Available Events ===

**Topic Events**
  * Topic is created
  * Topic is revised
  * Topic is updated
  * Topic is deleted
  * Topic is recovered

**Post Events**
  * Post is created
  * Post is updated
  * Post is deleted
  * Post is recovered

**User Events**
  * User logged in
  * User logged out
  * User confirmed e-mail
  * User is created
  * User is approved
  * User is updated
  * User is deleted
  * User is suspended
  * User is unsuspended
  * User is anonymized

**Group Events**
  * Group is created
  * Group is updated
  * Group is deleted

**Category Events**
  * Category is created
  * Category is updated
  * Category is deleted

**Tag Events**
  * Tag is created
  * Tag is updated
  * Tag is deleted

**Chat Events** (Critical for sync)
  * Message is created
  * Message is edited
  * Message is trashed
  * Message is restored

**Other Events**
  * Reviewable Events
  * Notification Events
  * Solved Events
  * Badge Events
  * Group User Events
  * Like Events
  * User Promoted Events
  * Topic Voting Events

=== Webhook Settings ===

**Content Type:** ''application/json'' (recommended for easy parsing)

**Secret:** Optional string for generating HMAC signatures (implement signature verification to prevent spoofed webhooks)

**Filtering Options:**
  * Triggered Categories - only fire webhooks for specific categories
  * Triggered Tags - only fire webhooks for specific tags
  * Triggered Groups - only fire webhooks for specific groups
  * Leave blank to trigger for all

**TLS Certificate Check:** Enable unless using self-signed certs in development

=== Configured Webhook Endpoints ===

**hobbiesync webhook:**
  * URL: ''https://example.com/chat/hooks/WEBHOOK_ID_1''

**general webhook:**
  * URL: ''https://example.com/chat/hooks/WEBHOOK_ID_2''

Path structure: ''/chat/hooks/[uuid]'' suggests custom webhook handler service

==== Chat API Endpoints ====

Base path: ''/chat/api/channels''

=== Channel Operations ===

**List channels**
<code>
GET /chat/api/channels
</code>

**Get specific channel**
<code>
GET /chat/api/channels/{channel_id}
</code>

**List messages**
<code>
GET /chat/api/channels/{channel_id}/messages
</code>

**Send message**
<code>
POST /chat/api/channels/{channel_id}/messages
</code>

=== Authentication ===

Discourse API requires headers:
<code>
Api-Key: {your_api_key}
Api-Username: {bot_username}
</code>

**API Credentials:**
  * API Key: ''[REDACTED]''
  * Username: ''syncbot''

**Security Notes:**
  * Store credentials in environment variables or secrets management (Vault)
  * API key has full access as the bot user
  * Revoke/rotate if compromised
  * Ensure bot user has appropriate permissions (chat access, posting rights to target channels)

==== Architecture Considerations ====

=== Critical Events for Chat Sync ===

**Primary focus:**
  * Chat events (message created/edited/trashed/restored) - core sync functionality
  * Post events - if bridging forum posts to Discord channels
  * User events (login/logout/created) - for presence sync and user mapping

=== Bidirectional Sync Challenges ===

**Discourse → Discord:**
  * Webhooks handle this direction
  * Parse incoming webhook payloads
  * Map to Discord API calls

**Discord → Discourse:**
  * Discord bot with message event listeners
  * POST to Discourse API ''/chat/api/channels/{channel_id}/messages''
  * Requires Discord bot token and event subscriptions

**Loop Prevention:**
  * Track message IDs/sources to avoid infinite echo
  * Store mapping of Discourse message ID ↔ Discord message ID
  * Ignore messages from own bot user

=== State Management ===

Required mappings to store:
  * Message ID mapping (Discourse chat message ID ↔ Discord message ID)
  * User mapping (Discourse username ↔ Discord user/webhook representation)
  * Channel mapping (Discourse channel ID ↔ Discord channel ID)
  * Edit/delete operations require retrieving these mappings

=== Event Filtering Strategy ===

**Options:**
  * "Send me everything" - receives all events (high bandwidth/processing)
  * Selective events - choose specific event types
  * Category/tag/group filters - sync only specific Discourse areas to specific Discord channels

**Recommendation:** Use selective events and filtering to reduce noise and processing load

=== Implementation Flow ===

**Webhook Receiver (Discourse → Discord):**
  - Receive webhook POST request
  - Verify HMAC signature if secret configured
  - Parse JSON payload
  - Extract message/event data
  - Check for existing mapping (to detect edits/deletes)
  - Call Discord API
  - Store message ID mapping

**Discord Bot (Discord → Discourse):**
  - Listen for Discord message events
  - Check if message is from bot (skip to prevent loop)
  - Format message for Discourse
  - POST to Discourse API with credentials
  - Store message ID mapping

**Message History/Backfill:**
  * Use ''GET /chat/api/channels/{channel_id}/messages'' for initial sync
  * Implement pagination if needed
  * Consider rate limiting


===== Profile Image/Avatar Synchronization =====

==== How Avatar Syncing Works ====

The bridge synchronizes profile images/avatars bidirectionally between Discord and Discourse using webhook-based user impersonation.

=== Discourse → Discord Avatar Flow ===

  - Bridge polls Discourse Chat API every 1.5 seconds for new messages
  - Each message includes user info with an ''avatar_template'' field
  - Avatar template format: ''/user_avatar/site/username/{size}/123_2.png''
  - Bridge calls ''build_avatar_url()'' to construct full URL:
    * Replaces ''{size}'' placeholder with pixel size (default: 128)
    * Prepends Discourse base URL if path is relative
  - Full avatar URL passed to Discord webhook ''avatar_url'' parameter
  - Discord displays the message with the Discourse user's actual avatar

<code python>
# Avatar URL construction (bridge.py lines 160-186)
def build_avatar_url(self, avatar_template: str, size: int = 128) -> Optional[str]:
    if not avatar_template:
        return None
    avatar_path = avatar_template.replace('{size}', str(size))
    if not avatar_path.startswith('/'):
        avatar_path = '/' + avatar_path
    return f"{self.base_url}{avatar_path}"
</code>

=== Discord → Discourse Avatar Flow ===

Discord user avatars are **not** synced to Discourse. Messages from Discord appear with the bot's configured username in Discourse, prefixed with ''[Discord] username:'' to identify the source user.

=== User Impersonation ===

The bridge uses Discord webhooks to display messages from Discourse users with their actual:
  * **Username** - Discourse display name shown in Discord
  * **Avatar** - Discourse profile image displayed in Discord

This creates a seamless experience where Discourse users appear as distinct users in Discord channels.

----

===== Image Upload/Download Synchronization =====

==== Discord → Discourse Images ====

  - Discord attachments detected via ''message.attachments''
  - Images downloaded to temporary files using ''download_image()''
  - Uploaded to Discourse via ''/uploads.json?type=image'' endpoint
  - Upload URL appended to message as markdown: ''![]({image_url})''
  - Temporary files cleaned up after processing

==== Discourse → Discord Images ====

  - Images extracted from ''uploads'' field in Discourse messages
  - Relative URLs converted to absolute URLs (prepends base_url)
  - URLs appended to message content
  - Discord auto-embeds images from URLs

----

===== GIF and Video Handling =====

==== Giphy/Tenor URL Detection ====

The bridge detects and converts Giphy/Tenor share URLs to direct media URLs for proper embedding:

  * **Giphy**: ''https://giphy.com/gifs/...'' → ''https://media.giphy.com/media/{ID}/giphy.gif''
  * **Tenor**: ''https://tenor.com/view/...'' → ''https://media.tenor.com/{ID}/tenor.gif''

==== MP4 to GIF Conversion ====

When Giphy/Tenor returns MP4 URLs (common for larger animations), the bridge converts them to GIF using FFmpeg:

  - Downloads MP4 to temporary file
  - Generates optimized color palette for compression
  - Converts to GIF with configurable settings:
    * Frame rate (default: 15 FPS)
    * Width scaling (default: 480px)
    * Duration limit (default: 10 seconds)
    * CPU thread limit (default: 3 threads)
  - Uploads converted GIF to Discourse
  - Cleans up temporary files

FFmpeg configuration options in ''bridge_config.json'':
<code json>
{
  "bridge": {
    "ffmpeg_threads": 3,
    "ffmpeg_duration_limit": 10,
    "ffmpeg_fps": 15,
    "ffmpeg_scale_width": 480
  }
}
</code>

----

===== State Management =====

==== Message Tracking ====

The bridge tracks processed message IDs to prevent duplicates and enable resumption after restarts.

State file structure (''bridge_state.json''):
<code json>
{
  "discord_channels": {
    "channel_id": {
      "last_message_id": 1234567890,
      "last_updated": "2024-01-15T12:34:56Z"
    }
  },
  "discourse_channels": {
    "channel_id": {
      "last_message_id": 456,
      "last_updated": "2024-01-15T12:34:56Z"
    }
  }
}
</code>

==== State Initialization ====

On first run, the bridge initializes state with the **latest** message IDs to prevent flooding channels with historical messages.

----

===== Loop Prevention =====

The bridge implements multiple safeguards to prevent infinite message loops:

==== Discord → Discourse ====
  * Messages prefixed with ''[Discord]'' tag
  * Webhook messages (non-user) are detected and skipped
  * Bot's own messages filtered by user ID

==== Discourse → Discord ====
  * Messages containing ''[Discord]'' prefix are ignored
  * Prevents echoing messages that originated from Discord

----

===== Configuration Reference =====

==== Full Configuration Structure ====

<code json>
{
  "discord": {
    "token": "bot_token_here",
    "channels": [
      {
        "name": "channel_name",
        "discord_channel_id": 123456789012345678,
        "discourse_channel_id": 2,
        "discord_webhook_url": null,
        "discourse_webhook_url": "https://discourse.example.com/chat/hooks/UUID",
        "enabled": true
      }
    ]
  },
  "discourse": {
    "base_url": "https://discourse.example.com",
    "api_key": "api_key_here",
    "username": "syncbot"
  },
  "bridge": {
    "polling_interval": 1.5,
    "state_file": "bridge_state.json",
    "log_level": "INFO",
    "rate_limit_delay": 1.0,
    "use_discord_webhooks": true,
    "ffmpeg_threads": 3,
    "ffmpeg_duration_limit": 10,
    "ffmpeg_fps": 15,
    "ffmpeg_scale_width": 480
  }
}
</code>

==== Configuration Options ====

^ Option ^ Default ^ Description ^
| ''polling_interval'' | 1.5 | Seconds between Discourse API polls |
| ''state_file'' | bridge_state.json | Path to state persistence file |
| ''log_level'' | INFO | Logging verbosity (DEBUG, INFO, WARNING, ERROR) |
| ''rate_limit_delay'' | 1.0 | Minimum seconds between API calls |
| ''use_discord_webhooks'' | true | Enable user impersonation via webhooks |
| ''ffmpeg_threads'' | 3 | CPU threads for MP4 conversion |
| ''ffmpeg_duration_limit'' | 10 | Max seconds of video to convert |
| ''ffmpeg_fps'' | 15 | Frame rate for converted GIFs |
| ''ffmpeg_scale_width'' | 480 | Width in pixels for converted GIFs |

==== Environment Variable Overrides ====

  * ''DISCORD_TOKEN'' - Discord bot token
  * ''DISCOURSE_API_KEY'' - Discourse API key
  * ''DISCOURSE_USERNAME'' - Discourse bot username
  * ''DISCOURSE_BASE_URL'' - Discourse instance URL
  * ''LOG_LEVEL'' - Logging level
  * ''CONFIG_FILE'' - Path to config file
  * ''STATE_FILE'' - Path to state file

----

===== Channel Mapping =====

==== Multi-Channel Support ====

The bridge supports multiple independent channel pairs. Each pair has:
  * Discord channel ID
  * Discourse channel ID
  * Separate webhook URLs
  * Independent enable/disable flag
  * Separate state tracking

==== Adding a New Channel Pair ====

  - Add entry to ''discord.channels'' array in config
  - Set ''discord_channel_id'' and ''discourse_channel_id''
  - Create Discourse webhook and add URL to ''discourse_webhook_url''
  - Set ''enabled: true''
  - Restart bridge - Discord webhook created automatically

----

===== Error Handling and Retry Logic =====

==== Automatic Retry ====

  * Failed Discord webhook sends **do not** update state
  * Message remains in queue and retries on next poll cycle
  * Prevents message loss on temporary failures

==== Graceful Degradation ====

  * API errors caught and logged, processing continues
  * Image download failures don't block message delivery
  * FFmpeg conversion failures fall back to posting MP4 URL

==== Timeouts ====

  * API requests: 30-60 second timeout
  * FFmpeg conversion: 30-60 second timeout
  * Prevents indefinite hangs

----

===== Docker Deployment =====

==== Container Configuration ====

<code yaml>
services:
  syncbot:
    build: .
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    volumes:
      - ./bridge-config.json:/app/bridge-config.json:ro
      - ./bridge_state.json:/app/bridge_state.json
</code>

==== Security ====

  * Runs as non-root user inside container
  * Config mounted read-only
  * State file writable for persistence
  * Resource limits prevent runaway processes

----

===== Message Flow Diagrams =====

==== Discord → Discourse ====

<code>
Discord User posts message
         ↓
on_message() event triggered
         ↓
Check: Is webhook message? → Skip
Check: Is bot message? → Skip
         ↓
Find channel config by Discord channel ID
         ↓
Download image attachments (if any)
         ↓
Upload images to Discourse
         ↓
Extract/convert Giphy/Tenor URLs
         ↓
Convert MP4 to GIF (if needed)
         ↓
Format: "[Discord] username: message"
         ↓
Append image URLs as markdown
         ↓
Send via Discourse webhook
         ↓
Update Discord channel state
</code>

==== Discourse → Discord ====

<code>
poll_discourse() runs every 1.5 seconds
         ↓
For each enabled channel:
         ↓
GET messages after last_message_id
         ↓
For each new message:
         ↓
Check: Has [Discord] prefix? → Skip
         ↓
Extract username from user info
Extract avatar_template from user info
         ↓
Build avatar URL (replace {size}, add base_url)
         ↓
Extract image uploads, make URLs absolute
         ↓
Truncate content to 2000 chars if needed
         ↓
Send via Discord webhook:
  - username: Discourse username
  - avatar_url: Discourse avatar
  - content: message text
         ↓
Success? → Update Discourse channel state
Failed? → Retry on next poll
</code>

----

===== Dependencies =====

==== Python Packages ====

  * ''discord.py>=2.0.0'' - Discord bot framework
  * ''requests>=2.28.0'' - HTTP requests
  * ''aiohttp>=3.8.0'' - Async HTTP
  * ''python-dotenv>=0.19.0'' - Environment variables

==== System Dependencies ====

  * Python 3.11+
  * FFmpeg (for MP4 to GIF conversion)

----

===== Known Limitations =====

==== Not Currently Implemented ====

  * **Message Edit Sync** - Edits are not synchronized
  * **Message Delete Sync** - Deletions are not synchronized
  * **Reaction Sync** - Emoji reactions not bridged
  * **PluralKit Support** - Removed in current version (was in earlier versions per git history)
  * **Discord Avatar → Discourse** - Only Discourse avatars sync to Discord, not vice versa

----

===== Troubleshooting =====

==== Messages Not Syncing ====

  * Check ''enabled: true'' in channel config
  * Verify Discord bot has webhook permissions in channel
  * Check Discourse webhook URL is correct
  * Review logs for API errors

==== Duplicate Messages ====

  * Check state file permissions (must be writable)
  * Verify state file path matches config

==== Images Not Appearing ====

  * Verify Discourse API key has upload permissions
  * Check temporary directory is writable
  * Review logs for upload errors

==== GIFs Not Converting ====

  * Ensure FFmpeg is installed: ''ffmpeg -version''
  * Check FFmpeg is in PATH
  * Review logs for conversion errors
  * Try reducing ''ffmpeg_duration_limit'' or ''ffmpeg_scale_width''