Tableau
Important Capabilities
Capability | Status | Notes |
---|---|---|
Column-level Lineage | ✅ | Enabled by default, configure using extract_column_level_lineage |
Dataset Usage | ✅ | Dashboard/Chart view counts, enabled using extract_usage_stats config |
Descriptions | ✅ | Enabled by default |
Detect Deleted Entities | ✅ | Enabled by default when stateful ingestion is turned on. |
Domains | ❌ | Requires transformer |
Extract Ownership | ✅ | Requires recipe configuration |
Extract Tags | ✅ | Requires recipe configuration |
Platform Instance | ✅ | Enabled by default |
Table-Level Lineage | ✅ | Enabled by default |
Prerequisites
In order to ingest metadata from Tableau, you will need:
- Tableau Server Version 2021.1.10 and above. It may also work for older versions.
- Enable the Tableau Metadata API for Tableau Server, if its not already enabled.
- Tableau Credentials (Username/Password or Personal Access Token)
- The user or token must have Site Administrator Explorer permissions.
Ingestion through UI
The following video shows you how to get started with ingesting Tableau metadata through the UI.
Integration Details
This plugin extracts Sheets, Dashboards, Embedded and Published Data sources metadata within Workbooks in a given project
on a Tableau site. Tableau's GraphQL interface is used to extract metadata information. Queries used to extract metadata are located
in metadata-ingestion/src/datahub/ingestion/source/tableau_common.py
Concept Mapping
This ingestion source maps the following Source System Concepts to DataHub Concepts:
Source Concept | DataHub Concept | Notes |
---|---|---|
"Tableau" | Data Platform | |
Project | Container | SubType "Project" |
Embedded DataSource | Dataset | SubType "Embedded Data Source" |
Published DataSource | Dataset | SubType "Published Data Source" |
Custom SQL Table | Dataset | SubTypes "View" , "Custom SQL" |
Embedded or External Tables | Dataset | |
Sheet | Chart | |
Dashboard | Dashboard | |
User | User (a.k.a CorpUser) | Optionally Extracted |
Workbook | Container | SubType "Workbook" |
Tag | Tag | Optionally Extracted |
Lineage
Lineage is emitted as received from Tableau's metadata API for
- Sheets contained within a Dashboard
- Embedded or Published Data Sources depended on by a Sheet
- Published Data Sources upstream to Embedded datasource
- Tables upstream to Embedded or Published Data Source
- Custom SQL datasources upstream to Embedded or Published Data Source
- Tables upstream to Custom SQL Data Source
Caveats
- Tableau metadata API might return incorrect schema name for tables for some databases, leading to incorrect metadata in DataHub. This source attempts to extract correct schema from databaseTable's fully qualified name, wherever possible. Read Using the databaseTable object in query for caveats in using schema attribute.
Troubleshooting
Why are only some workbooks/custom SQLs/published datasources ingested from the specified project?
This may happen when the Tableau API returns NODE_LIMIT_EXCEEDED error in response to metadata query and returns partial results with message "Showing partial results. , The request exceeded the ‘n’ node limit. Use pagination, additional filtering, or both in the query to adjust results." To resolve this, consider
- reducing the page size using the
page_size
config param in datahub recipe (Defaults to 10). - increasing tableau configuration metadata query node limit to higher value.
PERMISSIONS_MODE_SWITCHED
error in ingestion report
This error occurs if the Tableau site is using external assets. For more detail, refer to the Tableau documentation Manage Permissions for External Assets.
Follow the below steps to enable the derived permissions:
- Sign in to Tableau Cloud or Tableau Server as an admin.
- From the left navigation pane, click Settings.
- On the General tab, under Automatic Access to Metadata about Databases and Tables, select the
Automatically grant authorized users access to metadata about databases and tables
check box.
CLI based Ingestion
Install the Plugin
pip install 'acryl-datahub[tableau]'
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: tableau
config:
# Coordinates
connect_uri: https://prod-ca-a.online.tableau.com
site: acryl
platform_instance: acryl_instance
project_pattern: ["^default$", "^Project 2$", "^/Project A/Nested Project B$"]
# Credentials
username: "${TABLEAU_USER}"
password: "${TABLEAU_PASSWORD}"
# Options
ingest_tags: True
ingest_owner: True
default_schema_map:
mydatabase: public
anotherdatabase: anotherschema
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
connect_uri ✅ string | Tableau host URL. |
add_site_container boolean | When enabled, sites are added as containers and therefore visible in the folder structure within Datahub. Default: False |
database_hostname_to_platform_instance_map map(str,string) | |
default_schema_map map(str,string) | |
extract_column_level_lineage boolean | When enabled, extracts column-level lineage from Tableau Datasources Default: True |
extract_lineage_from_unsupported_custom_sql_queries boolean | [Experimental] Whether to extract lineage from unsupported custom sql queries using SQL parsing Default: False |
extract_project_hierarchy boolean | Whether to extract entire project hierarchy for nested projects. Default: True |
extract_usage_stats boolean | [experimental] Extract usage statistics for dashboards and charts. Default: False |
force_extraction_of_lineage_from_custom_sql_queries boolean | [Experimental] Force extraction of lineage from custom sql queries using SQL parsing, ignoring Tableau metadata Default: False |
ingest_embed_url boolean | Ingest a URL to render an embedded Preview of assets within Tableau. Default: False |
ingest_external_links_for_charts boolean | Ingest a URL to link out to from charts. Default: True |
ingest_external_links_for_dashboards boolean | Ingest a URL to link out to from dashboards. Default: True |
ingest_multiple_sites boolean | When enabled, ingests multiple sites the user has access to. If the user doesn't have access to the default site, specify an initial site to query in the site property. By default all sites the user has access to will be ingested. You can filter sites with the site_name_pattern property. This flag is currently only supported for Tableau Server. Tableau Cloud is not supported. Default: False |
ingest_owner boolean | Ingest Owner from source. This will override Owner info entered from UI Default: False |
ingest_tables_external boolean | Ingest details for tables external to (not embedded in) tableau as entities. Default: False |
ingest_tags boolean | Ingest Tags from source. This will override Tags entered from UI Default: False |
max_retries integer | Number of retries for failed requests. Default: 3 |
page_size integer | [advanced] Number of metadata objects (e.g. CustomSQLTable, PublishedDatasource, etc) to query at a time using the Tableau API. Default: 10 |
password string | Tableau password, must be set if authenticating using username/password. |
platform_instance string | The instance of the platform that all assets produced by this recipe belong to |
platform_instance_map map(str,string) | |
project_path_separator string | The separator used for the project_pattern field between project names. By default, we use a slash. You can change this if your Tableau projects contain slashes in their names, and you'd like to filter by project. Default: / |
site string | Tableau Site. Always required for Tableau Online. Use emptystring to connect with Default site on Tableau Server. Default: |
sql_parsing_disable_schema_awareness boolean | [Experimental] Ignore pre ingested tables schemas during parsing of SQL queries (allows to workaround ingestion errors when pre ingested schema and queries are out of sync) Default: False |
ssl_verify One of boolean, string | Whether to verify SSL certificates. If using self-signed certificates, set to false or provide the path to the .pem certificate bundle. Default: True |
token_name string | Tableau token name, must be set if authenticating using a personal access token. |
token_value string | Tableau token value, must be set if authenticating using a personal access token. |
username string | Tableau username, must be set if authenticating using username/password. |
workbook_page_size integer | [advanced] Number of workbooks to query at a time using the Tableau API. Default: 1 |
env string | Environment to use in namespace when constructing URNs. Default: PROD |
lineage_overrides TableauLineageOverrides | Mappings to change generated dataset urns. Use only if you really know what you are doing. |
lineage_overrides.database_override_map map(str,string) | |
lineage_overrides.platform_override_map map(str,string) | |
project_pattern AllowDenyPattern | Filter for specific Tableau projects. For example, use 'My Project' to ingest a root-level Project with name 'My Project', or 'My Project/Nested Project' to ingest a nested Project with name 'Nested Project'. By default, all Projects nested inside a matching Project will be included in ingestion. You can both allow and deny projects based on their name using their name, or a Regex pattern. Deny patterns always take precedence over allow patterns. By default, all projects will be ingested. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
project_pattern.ignoreCase boolean | Whether to ignore case sensitivity during pattern matching. Default: True |
project_pattern.allow array | List of regex patterns to include in ingestion Default: ['.*'] |
project_pattern.allow.string string | |
project_pattern.deny array | List of regex patterns to exclude from ingestion. Default: [] |
project_pattern.deny.string string | |
projects array | [deprecated] Use project_pattern instead. List of tableau projects Default: ['default'] |
projects.string string | |
site_name_pattern AllowDenyPattern | Filter for specific Tableau sites. By default, all sites will be included in the ingestion. You can both allow and deny sites based on their name using their name, or a Regex pattern. Deny patterns always take precedence over allow patterns. This property is currently only supported for Tableau Server. Tableau Cloud is not supported. Default: {'allow': ['.*'], 'deny': [], 'ignoreCase': True} |
site_name_pattern.ignoreCase boolean | Whether to ignore case sensitivity during pattern matching. Default: True |
site_name_pattern.allow array | List of regex patterns to include in ingestion Default: ['.*'] |
site_name_pattern.allow.string string | |
site_name_pattern.deny array | List of regex patterns to exclude from ingestion. Default: [] |
site_name_pattern.deny.string string | |
stateful_ingestion StatefulStaleMetadataRemovalConfig | Base specialized config for Stateful Ingestion with stale metadata removal capability. |
stateful_ingestion.enabled boolean | Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or datahub_api is specified, otherwise False Default: False |
stateful_ingestion.remove_stale_metadata boolean | Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled. Default: True |
The JSONSchema for this configuration is inlined below.
{
"title": "TableauConfig",
"description": "Any non-Dataset source that produces lineage to Datasets should inherit this class.\ne.g. Orchestrators, Pipelines, BI Tools etc.",
"type": "object",
"properties": {
"connect_uri": {
"title": "Connect Uri",
"description": "Tableau host URL.",
"type": "string"
},
"username": {
"title": "Username",
"description": "Tableau username, must be set if authenticating using username/password.",
"type": "string"
},
"password": {
"title": "Password",
"description": "Tableau password, must be set if authenticating using username/password.",
"type": "string"
},
"token_name": {
"title": "Token Name",
"description": "Tableau token name, must be set if authenticating using a personal access token.",
"type": "string"
},
"token_value": {
"title": "Token Value",
"description": "Tableau token value, must be set if authenticating using a personal access token.",
"type": "string"
},
"site": {
"title": "Site",
"description": "Tableau Site. Always required for Tableau Online. Use emptystring to connect with Default site on Tableau Server.",
"default": "",
"type": "string"
},
"platform_instance": {
"title": "Platform Instance",
"description": "The instance of the platform that all assets produced by this recipe belong to",
"type": "string"
},
"max_retries": {
"title": "Max Retries",
"description": "Number of retries for failed requests.",
"default": 3,
"type": "integer"
},
"ssl_verify": {
"title": "Ssl Verify",
"description": "Whether to verify SSL certificates. If using self-signed certificates, set to false or provide the path to the .pem certificate bundle.",
"default": true,
"anyOf": [
{
"type": "boolean"
},
{
"type": "string"
}
]
},
"extract_column_level_lineage": {
"title": "Extract Column Level Lineage",
"description": "When enabled, extracts column-level lineage from Tableau Datasources",
"default": true,
"type": "boolean"
},
"env": {
"title": "Env",
"description": "Environment to use in namespace when constructing URNs.",
"default": "PROD",
"type": "string"
},
"stateful_ingestion": {
"$ref": "#/definitions/StatefulStaleMetadataRemovalConfig"
},
"platform_instance_map": {
"title": "Platform Instance Map",
"description": "A holder for platform -> platform_instance mappings to generate correct dataset urns",
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"projects": {
"title": "Projects",
"description": "[deprecated] Use project_pattern instead. List of tableau projects ",
"default": [
"default"
],
"type": "array",
"items": {
"type": "string"
}
},
"project_pattern": {
"title": "Project Pattern",
"description": "Filter for specific Tableau projects. For example, use 'My Project' to ingest a root-level Project with name 'My Project', or 'My Project/Nested Project' to ingest a nested Project with name 'Nested Project'. By default, all Projects nested inside a matching Project will be included in ingestion. You can both allow and deny projects based on their name using their name, or a Regex pattern. Deny patterns always take precedence over allow patterns. By default, all projects will be ingested.",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"project_path_separator": {
"title": "Project Path Separator",
"description": "The separator used for the project_pattern field between project names. By default, we use a slash. You can change this if your Tableau projects contain slashes in their names, and you'd like to filter by project.",
"default": "/",
"type": "string"
},
"default_schema_map": {
"title": "Default Schema Map",
"description": "Default schema to use when schema is not found.",
"default": {},
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"ingest_tags": {
"title": "Ingest Tags",
"description": "Ingest Tags from source. This will override Tags entered from UI",
"default": false,
"type": "boolean"
},
"ingest_owner": {
"title": "Ingest Owner",
"description": "Ingest Owner from source. This will override Owner info entered from UI",
"default": false,
"type": "boolean"
},
"ingest_tables_external": {
"title": "Ingest Tables External",
"description": "Ingest details for tables external to (not embedded in) tableau as entities.",
"default": false,
"type": "boolean"
},
"page_size": {
"title": "Page Size",
"description": "[advanced] Number of metadata objects (e.g. CustomSQLTable, PublishedDatasource, etc) to query at a time using the Tableau API.",
"default": 10,
"type": "integer"
},
"workbook_page_size": {
"title": "Workbook Page Size",
"description": "[advanced] Number of workbooks to query at a time using the Tableau API.",
"default": 1,
"type": "integer"
},
"lineage_overrides": {
"title": "Lineage Overrides",
"description": "Mappings to change generated dataset urns. Use only if you really know what you are doing.",
"allOf": [
{
"$ref": "#/definitions/TableauLineageOverrides"
}
]
},
"database_hostname_to_platform_instance_map": {
"title": "Database Hostname To Platform Instance Map",
"description": "Mappings to change platform instance in generated dataset urns based on database. Use only if you really know what you are doing.",
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"extract_usage_stats": {
"title": "Extract Usage Stats",
"description": "[experimental] Extract usage statistics for dashboards and charts.",
"default": false,
"type": "boolean"
},
"ingest_embed_url": {
"title": "Ingest Embed Url",
"description": "Ingest a URL to render an embedded Preview of assets within Tableau.",
"default": false,
"type": "boolean"
},
"ingest_external_links_for_dashboards": {
"title": "Ingest External Links For Dashboards",
"description": "Ingest a URL to link out to from dashboards.",
"default": true,
"type": "boolean"
},
"ingest_external_links_for_charts": {
"title": "Ingest External Links For Charts",
"description": "Ingest a URL to link out to from charts.",
"default": true,
"type": "boolean"
},
"extract_project_hierarchy": {
"title": "Extract Project Hierarchy",
"description": "Whether to extract entire project hierarchy for nested projects.",
"default": true,
"type": "boolean"
},
"extract_lineage_from_unsupported_custom_sql_queries": {
"title": "Extract Lineage From Unsupported Custom Sql Queries",
"description": "[Experimental] Whether to extract lineage from unsupported custom sql queries using SQL parsing",
"default": false,
"type": "boolean"
},
"force_extraction_of_lineage_from_custom_sql_queries": {
"title": "Force Extraction Of Lineage From Custom Sql Queries",
"description": "[Experimental] Force extraction of lineage from custom sql queries using SQL parsing, ignoring Tableau metadata",
"default": false,
"type": "boolean"
},
"sql_parsing_disable_schema_awareness": {
"title": "Sql Parsing Disable Schema Awareness",
"description": "[Experimental] Ignore pre ingested tables schemas during parsing of SQL queries (allows to workaround ingestion errors when pre ingested schema and queries are out of sync)",
"default": false,
"type": "boolean"
},
"ingest_multiple_sites": {
"title": "Ingest Multiple Sites",
"description": "When enabled, ingests multiple sites the user has access to. If the user doesn't have access to the default site, specify an initial site to query in the site property. By default all sites the user has access to will be ingested. You can filter sites with the site_name_pattern property. This flag is currently only supported for Tableau Server. Tableau Cloud is not supported.",
"default": false,
"type": "boolean"
},
"site_name_pattern": {
"title": "Site Name Pattern",
"description": "Filter for specific Tableau sites. By default, all sites will be included in the ingestion. You can both allow and deny sites based on their name using their name, or a Regex pattern. Deny patterns always take precedence over allow patterns. This property is currently only supported for Tableau Server. Tableau Cloud is not supported. ",
"default": {
"allow": [
".*"
],
"deny": [],
"ignoreCase": true
},
"allOf": [
{
"$ref": "#/definitions/AllowDenyPattern"
}
]
},
"add_site_container": {
"title": "Add Site Container",
"description": "When enabled, sites are added as containers and therefore visible in the folder structure within Datahub.",
"default": false,
"type": "boolean"
}
},
"required": [
"connect_uri"
],
"additionalProperties": false,
"definitions": {
"DynamicTypedStateProviderConfig": {
"title": "DynamicTypedStateProviderConfig",
"type": "object",
"properties": {
"type": {
"title": "Type",
"description": "The type of the state provider to use. For DataHub use `datahub`",
"type": "string"
},
"config": {
"title": "Config",
"description": "The configuration required for initializing the state provider. Default: The datahub_api config if set at pipeline level. Otherwise, the default DatahubClientConfig. See the defaults (https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/graph/client.py#L19).",
"default": {},
"type": "object"
}
},
"required": [
"type"
],
"additionalProperties": false
},
"StatefulStaleMetadataRemovalConfig": {
"title": "StatefulStaleMetadataRemovalConfig",
"description": "Base specialized config for Stateful Ingestion with stale metadata removal capability.",
"type": "object",
"properties": {
"enabled": {
"title": "Enabled",
"description": "Whether or not to enable stateful ingest. Default: True if a pipeline_name is set and either a datahub-rest sink or `datahub_api` is specified, otherwise False",
"default": false,
"type": "boolean"
},
"remove_stale_metadata": {
"title": "Remove Stale Metadata",
"description": "Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
},
"AllowDenyPattern": {
"title": "AllowDenyPattern",
"description": "A class to store allow deny regexes",
"type": "object",
"properties": {
"allow": {
"title": "Allow",
"description": "List of regex patterns to include in ingestion",
"default": [
".*"
],
"type": "array",
"items": {
"type": "string"
}
},
"deny": {
"title": "Deny",
"description": "List of regex patterns to exclude from ingestion.",
"default": [],
"type": "array",
"items": {
"type": "string"
}
},
"ignoreCase": {
"title": "Ignorecase",
"description": "Whether to ignore case sensitivity during pattern matching.",
"default": true,
"type": "boolean"
}
},
"additionalProperties": false
},
"TableauLineageOverrides": {
"title": "TableauLineageOverrides",
"type": "object",
"properties": {
"platform_override_map": {
"title": "Platform Override Map",
"description": "A holder for platform -> platform mappings to generate correct dataset urns",
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"database_override_map": {
"title": "Database Override Map",
"description": "A holder for database -> database mappings to generate correct dataset urns",
"type": "object",
"additionalProperties": {
"type": "string"
}
}
},
"additionalProperties": false
}
}
}
Code Coordinates
- Class Name:
datahub.ingestion.source.tableau.TableauSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for Tableau, feel free to ping us on our Slack.