Create a data stream to start the flow of data from your Databricks source to Data 360 .
| User Permissions Needed | |
|---|---|
| To create a data stream: | Data Cloud Architect |
Before you begin:
- Make sure that the Databricks File Federation connection is set up.
- Review IP addresses to make sure that the Databricks connection has the necessary access.
-
In Data Cloud, on the Data Streams tab, click New.
-
Select Databricks File Federation as the source of the data stream, and click Next.
-
From the menus, select the connection and database to use.
-
Under Schemas, select the schema to use.
-
From the Available Objects, select the objects that you want to include.
note: You can't choose objects listed under In Use Objects.
-
Click Next.
-
Enter the object name and object API name. See Data Lake Object Naming Standards.
-
Select the category to specify the type of data to ingest. See Category.
-
Select a primary key to uniquely identify the incoming records. See Primary Key.
-
Select the source fields you want to ingest and click Next.
Fields with compatible data types are listed under Supported Fields. It's recommended to select all supported fields.
note: Unsupported Fields lists the fields with data types that aren't supported in Data 360.
-
Select a data space. If you don't have multiple data spaces provisioned, select Default. For more information, see Data Spaces.
-
Click Deploy.
Data Type Compatibility
This table lists the mapping of Databricks data types with Data 360 data types.
| Databricks Data Type | Is Auto-mapped to Data 360 Data Type |
|---|---|
| INT, SMALLINT, TINYINT, BIGINT | Number |
| FLOAT, DOUBLE. DECIMAL(p,s) | Number |
| STRING | Text |
| TIMESTAMP with Timezone | DateTime |
| DATE | Date |
| BOOLEAN | Boolean |
| VARIANT, OBJECT, ARRAY, GEOGRAPHY, GEOMETRY | Not Suported |
| BINARY, VARBINARY | Not Supported |
| STRUCT | Not Supported |