Ingest Slack channels data into Port via Meltano, S3 and webhook

This guide will demonstrate how to ingest Slack channels and channel membership data into Port using Meltano, S3 and a webhook integration.

Disclaimer

S3 integrations lack some of the features (such as reconciliation) found in Ocean or other Port integration solutions.

As a result, if a record ingested during the initial sync is later deleted in the data source, there’s no automatic mechanism to remove it from Port. The record simply won’t appear in future syncs, but it will remain in Port indefinitely.

If the data includes a flag for deleted records (e.g., is_deleted: "true"), you can configure a webhook delete operation in your webhook’s mapping configuration to remove these records from Port automatically.

Prerequisites

Ensure you have a Port account and have completed the onboarding process.
This feature is part of Port's limited-access offering. To obtain the required S3 bucket, please contact our team directly via chat, Slack, or e-mail, and we will create and manage the bucket on your behalf.
Access to an available Meltano app - for reference, follow the quick start guide, or follow the following steps:

shell

Install python3
```
brew install python3
```

Create a python virtual env:

python -m venv .venv
source .venv/bin/activate

Install meltano & follow installation instructions
```
pip install meltano
```
Change to meltano project
```
cd <name_of_project>
```

Setup a Slack Meltano exporter app - follow Meltano's guide for slack connector.

Include email data
If you wish to include email data, in addition to the permissions listed in the guide above, you will need to include user.email:read in the app's permissions.

Data model setup

Add Blueprints

Add the Slack Channel Membership blueprint:

Go to the Builder page of your portal.
Click on "+ Blueprint".
Click on the {...} button in the top right corner, and choose "Edit JSON".
Paste the following JSON schema into the editor:

Slack Channel Membership (Click to expand)

{
  "identifier": "slack_channel_membership",
  "description": "Slack Channel Membership",
  "title": "Slack Channel Membership",
  "icon": "Slack",
  "schema": {
    "properties": {
      "member_id": {
        "type": "string",
        "description": "ID of the user who is a member of the channel."
      },
      "channel_id": {
        "type": "string",
        "description": "ID of the channel the user belongs to."
      }
    },
    "required": [
      "member_id",
      "channel_id"
    ]
  },
  "mirrorProperties": {},
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {}
}

Add the Slack Channel blueprint in the same way:

Slack Channel (Click to expand)

{
  "identifier": "slack_channel",
  "description": "Slack Channel",
  "title": "Slack Channel",
  "icon": "Slack",
  "schema": {
    "properties": {
      "is_private": {
        "type": "boolean",
        "description": "Indicates if the channel is private."
      },
      "context_team_id": {
        "type": "string",
        "description": "ID of the team the channel belongs to."
      },
      "is_channel": {
        "type": "boolean",
        "description": "Indicates if this is a channel (true) or a direct message (false)."
      },
      "is_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is shared across teams."
      },
      "previous_names": {
        "type": "array",
        "description": "List of previous names of the channel."
      },
      "creator": {
        "type": "string",
        "description": "ID of the user who created the channel."
      },
      "createdAt": {
        "type": "number",
        "description": "Timestamp of when the channel was created."
      },
      "is_ext_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is externally shared."
      },
      "is_group": {
        "type": "boolean",
        "description": "Indicates if this is a group DM."
      },
      "is_archived": {
        "type": "boolean",
        "description": "Indicates if the channel is archived."
      },
      "shared_team_ids": {
        "type": "array",
        "description": "List of teams the channel is shared with."
      },
      "is_org_shared": {
        "type": "boolean",
        "description": "Indicates if the channel is shared across the entire organization."
      },
      "num_members": {
        "type": "number",
        "title": "num_members"
      },
      "purpose": {
        "type": "string",
        "description": "Information about the channel's purpose."
      },
      "topic": {
        "type": "string",
        "description": "Information about the channel's topic."
      }
    },
    "required": []
  },
  "mirrorProperties": {
    "member_id": {
      "title": "member_id",
      "path": "users.member_id"
    }
  },
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {
    "users": {
      "title": "Users",
      "target": "slack_channel_membership",
      "required": false,
      "many": true
    }
  }
}

Add the Slack User blueprint in the same way:

Slack User (Click to expand)

{
  "identifier": "slack_user",
  "description": "Slack User",
  "title": "Slack User",
  "icon": "Slack",
  "schema": {
    "properties": {
      "tz": {
        "type": "string",
        "description": "The user's time zone."
      },
      "is_restricted": {
        "type": "boolean",
        "description": "Indicates if the user is restricted."
      },
      "is_primary_owner": {
        "type": "boolean",
        "description": "Indicates if the user is the primary owner."
      },
      "real_name": {
        "type": "string",
        "description": "The user's real name."
      },
      "team_id": {
        "type": "string",
        "description": "The user's team ID."
      },
      "is_admin": {
        "type": "boolean",
        "description": "Indicates if the user is an admin."
      },
      "is_app_user": {
        "type": "boolean",
        "description": "Indicates if the user is an app user."
      },
      "deleted": {
        "type": "boolean",
        "description": "Indicates if the user is deleted."
      },
      "is_bot": {
        "type": "boolean",
        "description": "Indicates if the user is a bot."
      },
      "email": {
        "type": "string",
        "title": "email"
      }
    },
    "required": []
  },
  "mirrorProperties": {
    "channel_id": {
      "title": "channel_id",
      "path": "membership.channel_id"
    }
  },
  "calculationProperties": {},
  "aggregationProperties": {},
  "relations": {
    "user": {
      "title": "User",
      "target": "_user",
      "required": false,
      "many": false
    },
    "membership": {
      "title": "membership",
      "target": "slack_channel_membership",
      "required": false,
      "many": true
    }
  }
}

Create Webhook Integration

Create a webhook integration to ingest the data into Port:

Go to the Data sources page of your portal.
Click on "+ Data source".
In the top selection bar, click on Webhook, then select Custom Integration.
Enter a name for your Integration (for example: "Slack Integration"), enter a description (optional), then click on Next.
Copy the Webhook URL that was generated and include set up the Meltano connection (see Below).
Scroll down to the section titled "Map the data from the external system into Port" and paste the following mapping:

Slack Webhook Mapping (Click to expand)

[
  {
    "blueprint": "slack_channel",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and (.body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[2] | IN(\"channels\"))",
    "entity": {
      "identifier": ".body.id | tostring",
      "title": ".body.name_normalized | tostring",
      "properties": {
        "is_private": ".body.is_private",
        "purpose": ".body.purpose.value",
        "context_team_id": ".body.context_team_id",
        "is_shared": ".body.is_shared",
        "previous_names": ".body.previous_names",
        "creator": ".body.creator",
        "createdAt": ".body.created",
        "is_ext_shared": ".body.is_ext_shared",
        "is_group": ".body.is_group",
        "is_archived": ".body.is_archived",
        "num_members": ".body.num_members | tonumber? // .",
        "topic": ".body.topic.value",
        "shared_team_ids": ".body.shared_team_ids",
        "is_org_shared": ".body.is_org_shared"
      },
      "relations": {
        "users": {
          "combinator": "'and'",
          "rules": [
            {
              "property": "'channel_id'",
              "operator": "'='",
              "value": ".body.id | tostring"
            }
          ]
        }
      }
    }
  },
  {
    "blueprint": "slack_user",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and (.body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[2] | IN(\"users\"))",
    "entity": {
      "identifier": ".body.id | tostring",
      "title": ".body.name | tostring",
      "properties": {
        "tz": ".body.tz",
        "is_restricted": ".body.is_restricted",
        "is_primary_owner": ".body.is_primary_owner",
        "real_name": ".body.real_name",
        "team_id": ".body.team_id",
        "is_admin": ".body.is_admin",
        "is_app_user": ".body.is_app_user",
        "deleted": ".body.deleted",
        "is_bot": ".body.is_bot",
        "email": ".body.profile.email"
      },
      "relations": {
        "user": ".body.profile.email"
      }
    }
  },
  {
    "blueprint": "slack_channel_membership",
    "operation": "create",
    "filter": "(.body | has(\"_PORT_SOURCE_OBJECT_KEY\")) and (.body._PORT_SOURCE_OBJECT_KEY | split(\"/\") | .[2] | IN(\"channel_members\"))",
    "entity": {
      "identifier": ".body.channel_id + \"_\" + .body.member_id | tostring",
      "title": ".body.channel_id + \"_\" + .body.member_id | tostring",
      "properties": {
        "member_id": ".body.member_id",
        "channel_id": ".body.channel_id"
      }
    }
  }
]

Meltano Setup

Recommended

Refer to this GitHub Repository to view examples and prepared code sample for this integration.

Set up S3 Destination

If you haven't already set up S3 Destination for Port S3, follow these steps:

Meltano provides detailed documentation on how to generate/receive the appropriate credentials to set the s3-target loader. Once the appropriate credentials are prepared, you may set up the meltano extractor:

shell

Navigate to your meltano environment:
```
cd path/to/your/meltano/project/
```
Install the source plugin you wish to extract data from:
```
meltano add loader target-s3
```

Configure the plugin using the interactive CLI prompt:

meltano config target-s3 set --interactive

Or set the configuration parameters individually using the CLI:

# required
meltano config target-s3 set cloud_provider.aws.aws_access_key_id $AWS_ACCESS_KEY_ID 
meltano config target-s3 set cloud_provider.aws.aws_secret_access_key $AWS_SECRET_ACCESS_KEY 
meltano config target-s3 set cloud_provider.aws.aws_bucket $AWS_BUCKET 
meltano config target-s3 set cloud_provider.aws.aws_region $AWS_REGION
# recommended
meltano config target-s3 set append_date_to_filename_grain microsecond
meltano config target-s3 set partition_name_enabled true
meltano config target-s3 set prefix 'data/'

Set up Slack Connection

Install and configure a Slack extractor, for more information go to: Slack extractor.

Private Channels
Meltano will not read information from private channels by default. If you wish to include private channels: tick the "include private channels" option, and manually add the Slack-export App to your desired private channels.

Add the tap-slack extractor to your project using meltano add :
```
meltano add extractor tap-slack
```
Configure the tap-slack settings using meltano config :
```
meltano config tap-slack set --interactive
```
Test that extractor settings are valid using meltano config :
```
meltano config tap-slack test
```

Create a specific target-s3 loader for the webhook you created, and enter the Webhook URL you have copied when setting up the webhook as the part of the prefix configuration field, for example: "data/wSLvwtI1LFwQzXXX".

meltano add loader target-s3--slackintegration --inherit-from target-s3
meltano config target-s3--slackintegration set prefix data/<WEBHOOK_URL>
meltano config target-s3--slackintegration set format format_type jsonl

Run the connection:

meltano el tap-slack target-s3--slackintegration

Important

If for any reason you have entered different values than the ones specified in this guide, inform us so we can assist to ensure the integration will run smoothly.

Prerequisites​

Data model setup​

Add Blueprints​

Create Webhook Integration​

Meltano Setup​

Set up S3 Destination​

Set up Slack Connection​

Additional relevant guides​