Scorpi Wiki

User Tools

Site Tools

Trace:
discourse

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
discourse [2026/01/23 06:05] – 208.92.105.181discourse [2026/01/26 21:07] (current) – 208.92.105.160
Line 1: Line 1:
-You're right, I apologize. DokuWiki uses its own wiki syntax, not Markdown. Here's the proper DokuWiki format: 
- 
-``` 
 ====== Discourse Installation on Fresh Ubuntu Server with Backup Restoration ====== ====== Discourse Installation on Fresh Ubuntu Server with Backup Restoration ======
  
Line 93: Line 90:
  
 ==== 7. Install and Configure Nginx ==== ==== 7. Install and Configure Nginx ====
 +
 +#This nginx setup in this guide is but one valid configuration.
 <code bash> <code bash>
 apt-get install nginx apt-get install nginx
Line 215: Line 214:
   * Ensure filename ends in ''.tar.gz''   * Ensure filename ends in ''.tar.gz''
  
-==== SCP Syntax Issues ====+==== SCP Syntax Issues ====https://wiki.scorpi.us/doku.php?id=discourse&do=
   * Port flag is capital ''-P'' not lowercase ''-p''   * Port flag is capital ''-P'' not lowercase ''-p''
   * Syntax: ''scp -P <port> <source> <user>@<host>:<destination>''   * Syntax: ''scp -P <port> <source> <user>@<host>:<destination>''
   * Example: ''scp -P 22 backup.tar.gz user@host:/path/''   * Example: ''scp -P 22 backup.tar.gz user@host:/path/''
 +
 ``` ```
 +===== Discourse API Configuration =====
 +
 +==== Webhook Configuration ====
 +
 +=== Available Events ===
 +
 +**Topic Events**
 +  * Topic is created
 +  * Topic is revised
 +  * Topic is updated
 +  * Topic is deleted
 +  * Topic is recovered
 +
 +**Post Events**
 +  * Post is created
 +  * Post is updated
 +  * Post is deleted
 +  * Post is recovered
 +
 +**User Events**
 +  * User logged in
 +  * User logged out
 +  * User confirmed e-mail
 +  * User is created
 +  * User is approved
 +  * User is updated
 +  * User is deleted
 +  * User is suspended
 +  * User is unsuspended
 +  * User is anonymized
 +
 +**Group Events**
 +  * Group is created
 +  * Group is updated
 +  * Group is deleted
 +
 +**Category Events**
 +  * Category is created
 +  * Category is updated
 +  * Category is deleted
 +
 +**Tag Events**
 +  * Tag is created
 +  * Tag is updated
 +  * Tag is deleted
 +
 +**Chat Events** (Critical for sync)
 +  * Message is created
 +  * Message is edited
 +  * Message is trashed
 +  * Message is restored
 +
 +**Other Events**
 +  * Reviewable Events
 +  * Notification Events
 +  * Solved Events
 +  * Badge Events
 +  * Group User Events
 +  * Like Events
 +  * User Promoted Events
 +  * Topic Voting Events
 +
 +=== Webhook Settings ===
 +
 +**Content Type:** ''application/json'' (recommended for easy parsing)
 +
 +**Secret:** Optional string for generating HMAC signatures (implement signature verification to prevent spoofed webhooks)
 +
 +**Filtering Options:**
 +  * Triggered Categories - only fire webhooks for specific categories
 +  * Triggered Tags - only fire webhooks for specific tags
 +  * Triggered Groups - only fire webhooks for specific groups
 +  * Leave blank to trigger for all
 +
 +**TLS Certificate Check:** Enable unless using self-signed certs in development
 +
 +=== Configured Webhook Endpoints ===
 +
 +**hobbiesync webhook:**
 +  * URL: ''https://example.com/chat/hooks/WEBHOOK_ID_1''
 +
 +**general webhook:**
 +  * URL: ''https://example.com/chat/hooks/WEBHOOK_ID_2''
 +
 +Path structure: ''/chat/hooks/[uuid]'' suggests custom webhook handler service
 +
 +==== Chat API Endpoints ====
 +
 +Base path: ''/chat/api/channels''
 +
 +=== Channel Operations ===
 +
 +**List channels**
 +<code>
 +GET /chat/api/channels
 +</code>
 +
 +**Get specific channel**
 +<code>
 +GET /chat/api/channels/{channel_id}
 +</code>
 +
 +**List messages**
 +<code>
 +GET /chat/api/channels/{channel_id}/messages
 +</code>
 +
 +**Send message**
 +<code>
 +POST /chat/api/channels/{channel_id}/messages
 +</code>
 +
 +=== Authentication ===
 +
 +Discourse API requires headers:
 +<code>
 +Api-Key: {your_api_key}
 +Api-Username: {bot_username}
 +</code>
 +
 +**API Credentials:**
 +  * API Key: ''[REDACTED]''
 +  * Username: ''syncbot''
 +
 +**Security Notes:**
 +  * Store credentials in environment variables or secrets management (Vault)
 +  * API key has full access as the bot user
 +  * Revoke/rotate if compromised
 +  * Ensure bot user has appropriate permissions (chat access, posting rights to target channels)
 +
 +==== Architecture Considerations ====
 +
 +=== Critical Events for Chat Sync ===
 +
 +**Primary focus:**
 +  * Chat events (message created/edited/trashed/restored) - core sync functionality
 +  * Post events - if bridging forum posts to Discord channels
 +  * User events (login/logout/created) - for presence sync and user mapping
 +
 +=== Bidirectional Sync Challenges ===
 +
 +**Discourse → Discord:**
 +  * Webhooks handle this direction
 +  * Parse incoming webhook payloads
 +  * Map to Discord API calls
 +
 +**Discord → Discourse:**
 +  * Discord bot with message event listeners
 +  * POST to Discourse API ''/chat/api/channels/{channel_id}/messages''
 +  * Requires Discord bot token and event subscriptions
 +
 +**Loop Prevention:**
 +  * Track message IDs/sources to avoid infinite echo
 +  * Store mapping of Discourse message ID ↔ Discord message ID
 +  * Ignore messages from own bot user
 +
 +=== State Management ===
 +
 +Required mappings to store:
 +  * Message ID mapping (Discourse chat message ID ↔ Discord message ID)
 +  * User mapping (Discourse username ↔ Discord user/webhook representation)
 +  * Channel mapping (Discourse channel ID ↔ Discord channel ID)
 +  * Edit/delete operations require retrieving these mappings
 +
 +=== Event Filtering Strategy ===
 +
 +**Options:**
 +  * "Send me everything" - receives all events (high bandwidth/processing)
 +  * Selective events - choose specific event types
 +  * Category/tag/group filters - sync only specific Discourse areas to specific Discord channels
 +
 +**Recommendation:** Use selective events and filtering to reduce noise and processing load
 +
 +=== Implementation Flow ===
 +
 +**Webhook Receiver (Discourse → Discord):**
 +  - Receive webhook POST request
 +  - Verify HMAC signature if secret configured
 +  - Parse JSON payload
 +  - Extract message/event data
 +  - Check for existing mapping (to detect edits/deletes)
 +  - Call Discord API
 +  - Store message ID mapping
 +
 +**Discord Bot (Discord → Discourse):**
 +  - Listen for Discord message events
 +  - Check if message is from bot (skip to prevent loop)
 +  - Format message for Discourse
 +  - POST to Discourse API with credentials
 +  - Store message ID mapping
 +
 +**Message History/Backfill:**
 +  * Use ''GET /chat/api/channels/{channel_id}/messages'' for initial sync
 +  * Implement pagination if needed
 +  * Consider rate limiting
 +
 +
 +===== Profile Image/Avatar Synchronization =====
 +
 +==== How Avatar Syncing Works ====
 +
 +The bridge synchronizes profile images/avatars bidirectionally between Discord and Discourse using webhook-based user impersonation.
 +
 +=== Discourse → Discord Avatar Flow ===
 +
 +  - Bridge polls Discourse Chat API every 1.5 seconds for new messages
 +  - Each message includes user info with an ''avatar_template'' field
 +  - Avatar template format: ''/user_avatar/site/username/{size}/123_2.png''
 +  - Bridge calls ''build_avatar_url()'' to construct full URL:
 +    * Replaces ''{size}'' placeholder with pixel size (default: 128)
 +    * Prepends Discourse base URL if path is relative
 +  - Full avatar URL passed to Discord webhook ''avatar_url'' parameter
 +  - Discord displays the message with the Discourse user's actual avatar
 +
 +<code python>
 +# Avatar URL construction (bridge.py lines 160-186)
 +def build_avatar_url(self, avatar_template: str, size: int = 128) -> Optional[str]:
 +    if not avatar_template:
 +        return None
 +    avatar_path = avatar_template.replace('{size}', str(size))
 +    if not avatar_path.startswith('/'):
 +        avatar_path = '/' + avatar_path
 +    return f"{self.base_url}{avatar_path}"
 +</code>
 +
 +=== Discord → Discourse Avatar Flow ===
 +
 +Discord user avatars are **not** synced to Discourse. Messages from Discord appear with the bot's configured username in Discourse, prefixed with ''[Discord] username:'' to identify the source user.
 +
 +=== User Impersonation ===
 +
 +The bridge uses Discord webhooks to display messages from Discourse users with their actual:
 +  * **Username** - Discourse display name shown in Discord
 +  * **Avatar** - Discourse profile image displayed in Discord
 +
 +This creates a seamless experience where Discourse users appear as distinct users in Discord channels.
 +
 +----
 +
 +===== Image Upload/Download Synchronization =====
 +
 +==== Discord → Discourse Images ====
 +
 +  - Discord attachments detected via ''message.attachments''
 +  - Images downloaded to temporary files using ''download_image()''
 +  - Uploaded to Discourse via ''/uploads.json?type=image'' endpoint
 +  - Upload URL appended to message as markdown: ''![]({image_url})''
 +  - Temporary files cleaned up after processing
 +
 +==== Discourse → Discord Images ====
 +
 +  - Images extracted from ''uploads'' field in Discourse messages
 +  - Relative URLs converted to absolute URLs (prepends base_url)
 +  - URLs appended to message content
 +  - Discord auto-embeds images from URLs
 +
 +----
 +
 +===== GIF and Video Handling =====
 +
 +==== Giphy/Tenor URL Detection ====
 +
 +The bridge detects and converts Giphy/Tenor share URLs to direct media URLs for proper embedding:
 +
 +  * **Giphy**: ''https://giphy.com/gifs/...'' → ''https://media.giphy.com/media/{ID}/giphy.gif''
 +  * **Tenor**: ''https://tenor.com/view/...'' → ''https://media.tenor.com/{ID}/tenor.gif''
 +
 +==== MP4 to GIF Conversion ====
 +
 +When Giphy/Tenor returns MP4 URLs (common for larger animations), the bridge converts them to GIF using FFmpeg:
 +
 +  - Downloads MP4 to temporary file
 +  - Generates optimized color palette for compression
 +  - Converts to GIF with configurable settings:
 +    * Frame rate (default: 15 FPS)
 +    * Width scaling (default: 480px)
 +    * Duration limit (default: 10 seconds)
 +    * CPU thread limit (default: 3 threads)
 +  - Uploads converted GIF to Discourse
 +  - Cleans up temporary files
 +
 +FFmpeg configuration options in ''bridge_config.json'':
 +<code json>
 +{
 +  "bridge": {
 +    "ffmpeg_threads": 3,
 +    "ffmpeg_duration_limit": 10,
 +    "ffmpeg_fps": 15,
 +    "ffmpeg_scale_width": 480
 +  }
 +}
 +</code>
 +
 +----
 +
 +===== State Management =====
 +
 +==== Message Tracking ====
 +
 +The bridge tracks processed message IDs to prevent duplicates and enable resumption after restarts.
 +
 +State file structure (''bridge_state.json''):
 +<code json>
 +{
 +  "discord_channels": {
 +    "channel_id": {
 +      "last_message_id": 1234567890,
 +      "last_updated": "2024-01-15T12:34:56Z"
 +    }
 +  },
 +  "discourse_channels": {
 +    "channel_id": {
 +      "last_message_id": 456,
 +      "last_updated": "2024-01-15T12:34:56Z"
 +    }
 +  }
 +}
 +</code>
 +
 +==== State Initialization ====
 +
 +On first run, the bridge initializes state with the **latest** message IDs to prevent flooding channels with historical messages.
 +
 +----
 +
 +===== Loop Prevention =====
 +
 +The bridge implements multiple safeguards to prevent infinite message loops:
 +
 +==== Discord → Discourse ====
 +  * Messages prefixed with ''[Discord]'' tag
 +  * Webhook messages (non-user) are detected and skipped
 +  * Bot's own messages filtered by user ID
 +
 +==== Discourse → Discord ====
 +  * Messages containing ''[Discord]'' prefix are ignored
 +  * Prevents echoing messages that originated from Discord
 +
 +----
 +
 +===== Configuration Reference =====
 +
 +==== Full Configuration Structure ====
 +
 +<code json>
 +{
 +  "discord": {
 +    "token": "bot_token_here",
 +    "channels": [
 +      {
 +        "name": "channel_name",
 +        "discord_channel_id": 123456789012345678,
 +        "discourse_channel_id": 2,
 +        "discord_webhook_url": null,
 +        "discourse_webhook_url": "https://discourse.example.com/chat/hooks/UUID",
 +        "enabled": true
 +      }
 +    ]
 +  },
 +  "discourse": {
 +    "base_url": "https://discourse.example.com",
 +    "api_key": "api_key_here",
 +    "username": "syncbot"
 +  },
 +  "bridge": {
 +    "polling_interval": 1.5,
 +    "state_file": "bridge_state.json",
 +    "log_level": "INFO",
 +    "rate_limit_delay": 1.0,
 +    "use_discord_webhooks": true,
 +    "ffmpeg_threads": 3,
 +    "ffmpeg_duration_limit": 10,
 +    "ffmpeg_fps": 15,
 +    "ffmpeg_scale_width": 480
 +  }
 +}
 +</code>
 +
 +==== Configuration Options ====
 +
 +^ Option ^ Default ^ Description ^
 +| ''polling_interval'' | 1.5 | Seconds between Discourse API polls |
 +| ''state_file'' | bridge_state.json | Path to state persistence file |
 +| ''log_level'' | INFO | Logging verbosity (DEBUG, INFO, WARNING, ERROR) |
 +| ''rate_limit_delay'' | 1.0 | Minimum seconds between API calls |
 +| ''use_discord_webhooks'' | true | Enable user impersonation via webhooks |
 +| ''ffmpeg_threads'' | 3 | CPU threads for MP4 conversion |
 +| ''ffmpeg_duration_limit'' | 10 | Max seconds of video to convert |
 +| ''ffmpeg_fps'' | 15 | Frame rate for converted GIFs |
 +| ''ffmpeg_scale_width'' | 480 | Width in pixels for converted GIFs |
 +
 +==== Environment Variable Overrides ====
 +
 +  * ''DISCORD_TOKEN'' - Discord bot token
 +  * ''DISCOURSE_API_KEY'' - Discourse API key
 +  * ''DISCOURSE_USERNAME'' - Discourse bot username
 +  * ''DISCOURSE_BASE_URL'' - Discourse instance URL
 +  * ''LOG_LEVEL'' - Logging level
 +  * ''CONFIG_FILE'' - Path to config file
 +  * ''STATE_FILE'' - Path to state file
 +
 +----
 +
 +===== Channel Mapping =====
 +
 +==== Multi-Channel Support ====
 +
 +The bridge supports multiple independent channel pairs. Each pair has:
 +  * Discord channel ID
 +  * Discourse channel ID
 +  * Separate webhook URLs
 +  * Independent enable/disable flag
 +  * Separate state tracking
 +
 +==== Adding a New Channel Pair ====
 +
 +  - Add entry to ''discord.channels'' array in config
 +  - Set ''discord_channel_id'' and ''discourse_channel_id''
 +  - Create Discourse webhook and add URL to ''discourse_webhook_url''
 +  - Set ''enabled: true''
 +  - Restart bridge - Discord webhook created automatically
 +
 +----
 +
 +===== Error Handling and Retry Logic =====
 +
 +==== Automatic Retry ====
 +
 +  * Failed Discord webhook sends **do not** update state
 +  * Message remains in queue and retries on next poll cycle
 +  * Prevents message loss on temporary failures
 +
 +==== Graceful Degradation ====
 +
 +  * API errors caught and logged, processing continues
 +  * Image download failures don't block message delivery
 +  * FFmpeg conversion failures fall back to posting MP4 URL
 +
 +==== Timeouts ====
 +
 +  * API requests: 30-60 second timeout
 +  * FFmpeg conversion: 30-60 second timeout
 +  * Prevents indefinite hangs
 +
 +----
 +
 +===== Docker Deployment =====
 +
 +==== Container Configuration ====
 +
 +<code yaml>
 +services:
 +  syncbot:
 +    build: .
 +    restart: unless-stopped
 +    deploy:
 +      resources:
 +        limits:
 +          cpus: '1'
 +          memory: 512M
 +    logging:
 +      driver: "json-file"
 +      options:
 +        max-size: "10m"
 +        max-file: "3"
 +    volumes:
 +      - ./bridge-config.json:/app/bridge-config.json:ro
 +      - ./bridge_state.json:/app/bridge_state.json
 +</code>
 +
 +==== Security ====
 +
 +  * Runs as non-root user inside container
 +  * Config mounted read-only
 +  * State file writable for persistence
 +  * Resource limits prevent runaway processes
 +
 +----
 +
 +===== Message Flow Diagrams =====
 +
 +==== Discord → Discourse ====
 +
 +<code>
 +Discord User posts message
 +         ↓
 +on_message() event triggered
 +         ↓
 +Check: Is webhook message? → Skip
 +Check: Is bot message? → Skip
 +         ↓
 +Find channel config by Discord channel ID
 +         ↓
 +Download image attachments (if any)
 +         ↓
 +Upload images to Discourse
 +         ↓
 +Extract/convert Giphy/Tenor URLs
 +         ↓
 +Convert MP4 to GIF (if needed)
 +         ↓
 +Format: "[Discord] username: message"
 +         ↓
 +Append image URLs as markdown
 +         ↓
 +Send via Discourse webhook
 +         ↓
 +Update Discord channel state
 +</code>
 +
 +==== Discourse → Discord ====
 +
 +<code>
 +poll_discourse() runs every 1.5 seconds
 +         ↓
 +For each enabled channel:
 +         ↓
 +GET messages after last_message_id
 +         ↓
 +For each new message:
 +         ↓
 +Check: Has [Discord] prefix? → Skip
 +         ↓
 +Extract username from user info
 +Extract avatar_template from user info
 +         ↓
 +Build avatar URL (replace {size}, add base_url)
 +         ↓
 +Extract image uploads, make URLs absolute
 +         ↓
 +Truncate content to 2000 chars if needed
 +         ↓
 +Send via Discord webhook:
 +  - username: Discourse username
 +  - avatar_url: Discourse avatar
 +  - content: message text
 +         ↓
 +Success? → Update Discourse channel state
 +Failed? → Retry on next poll
 +</code>
 +
 +----
 +
 +===== Dependencies =====
 +
 +==== Python Packages ====
 +
 +  * ''discord.py>=2.0.0'' - Discord bot framework
 +  * ''requests>=2.28.0'' - HTTP requests
 +  * ''aiohttp>=3.8.0'' - Async HTTP
 +  * ''python-dotenv>=0.19.0'' - Environment variables
 +
 +==== System Dependencies ====
 +
 +  * Python 3.11+
 +  * FFmpeg (for MP4 to GIF conversion)
 +
 +----
 +
 +===== Known Limitations =====
 +
 +==== Not Currently Implemented ====
 +
 +  * **Message Edit Sync** - Edits are not synchronized
 +  * **Message Delete Sync** - Deletions are not synchronized
 +  * **Reaction Sync** - Emoji reactions not bridged
 +  * **PluralKit Support** - Removed in current version (was in earlier versions per git history)
 +  * **Discord Avatar → Discourse** - Only Discourse avatars sync to Discord, not vice versa
 +
 +----
 +
 +===== Troubleshooting =====
 +
 +==== Messages Not Syncing ====
 +
 +  * Check ''enabled: true'' in channel config
 +  * Verify Discord bot has webhook permissions in channel
 +  * Check Discourse webhook URL is correct
 +  * Review logs for API errors
 +
 +==== Duplicate Messages ====
 +
 +  * Check state file permissions (must be writable)
 +  * Verify state file path matches config
 +
 +==== Images Not Appearing ====
 +
 +  * Verify Discourse API key has upload permissions
 +  * Check temporary directory is writable
 +  * Review logs for upload errors
 +
 +==== GIFs Not Converting ====
 +
 +  * Ensure FFmpeg is installed: ''ffmpeg -version''
 +  * Check FFmpeg is in PATH
 +  * Review logs for conversion errors
 +  * Try reducing ''ffmpeg_duration_limit'' or ''ffmpeg_scale_width''
 +
discourse.1769148318.txt.gz · Last modified: by 208.92.105.181