Microsoft Azure Databricks lineage configuration

Connection mode

You can connect to Microsoft Azure Databricks by using one of the following connection modes:

• Direct connection
• Remote connection with a Manta agent. When an agent is configured, select it from the list. For more information, see Configuring agents for lineage metadata import.

Include and exclude lists

You can include or exclude assets up to the schema level. Provide catalogs and schemas in the format catalog/schema. Each part is evaluated as a regular expression. Assets which are added later in the data source will also be included or excluded if they match the conditions specified in the lists. Example values:

• myCatalog/: all schemas in myCatalog,
• myCatalog/.*: all schemas in myCatalog,
• myCatalog3/mySchema1: mySchema1 from myCatalog3,
• myCatalog4/mySchema[1-5]: any schema in my myCatalog4 with a name that starts with mySchema and ends with a digit between 1 and 5

External inputs

If you use external Microsoft Azure Databricks dll archives, you can add them in a .zip file as an external input. You can organize the structure of the .zip file as the dll folder with subfolders or archives that represent the workspace structure. The .zip file can have the following structure:

<dll>
    <catalog_name_folder>
      <schema_name_folder>
        <tables>
          <table_name.sql>
        <views>
          <view_name.sql>

Advanced import options

Display table lineage

Generate edges between tables for which the column-level lineage information was not found.