Create an Unstructured Data Connection from Azure Storage

Connect unstructured data from Azure Storage for use in your Agentforce, RAG, automation, and analytics workflows. First create an unstructured data lake object (UDLO) in Data 360 to reference the unstructured data. Then create a file notification pipeline to keep your UDLO up to date.

See the Unstructured Data File Formats and Connectors for a list of supported file formats for unstructured data.

important: To ensure your unstructured data is properly connected to Data 360, first perform the steps to connect unstructured data from your external blob store, then set up file notifications, and finally put data in your external blob store.

Create a UDLO for Unstructured Data from Azure Storage

Create a UDLO in Data 360 to reference unstructured data from Azure Storage.

note: Assign the Data Cloud Architect permission set to the user creating the UDLO. Don't also assign Data Cloud User to the same user. Assigning both removes Data Cloud from the App Launcher, which prevents you from opening the Data Cloud app to complete and verify these steps.

Before you begin:

Make sure you've set up a connection to Azure Storage.

1. From App Launcher, select Data Cloud.
2. Click Data Lake Objects and then click New.
3. From the New Data Lake Object menu, select From External Files, and click Next.
4. Choose the Azure Storage connector, and click Next.
5. From the Select Connection dropdown list, select a connection. Data Cloud auto-populates the source based on the connection that you select.
6. In the Directory field, point to a specific folder or an entire directory in your blob store. All folders and subfolders in a directory are included. Optionally, use wildcard characters to specify a file name pattern for multiple files.
   • A directory value is required in two places: the connection's Container Path and this UDLO Directory field. Neither can be saved empty. Data Cloud resolves the lookup path by concatenating the two: <Container Path>/<Directory>. Split your blob prefix across the two fields so that the concatenation reconstructs the real path exactly once, and don't repeat the same folder in both. For example, for blobs at inbound/eml/, set Container Path to inbound/ and Directory to eml/ so the resolved path is inbound/eml/, not inbound/inbound/. Open the connection and check its Container Path before you set this field.
   • Wildcard file name patterns are case-sensitive. *.eml and *.EML are different patterns. Use the casing your files actually use.
7. To add more directories, click More Files. You can include up to 5 directories.
8. Click Next.
9. Add an Object Name and an Object API Name for the UDLO. See Data Lake Object Naming Standards.
   • Data Cloud appends __dll to the API name of the underlying Data Lake Object. For example, an Object API Name of Email_Archive_264_UDLO is queryable as Email_Archive_264_UDLO__dll. Account for this suffix in any downstream queries, tooling, or code references.
10. From the Data Space dropdown list, select a data space in which to create the new UDMO or a data space from which to select an existing UDMO.
11. Map the UDLO to a UDMO.
    • To create a new UDMO, click New.
    • To use an existing UDMO, click Existing, and select a UDMO from the list.
12. Optionally, leave the checkbox selected to create a search index configuration for the UDMO using system defaults that automatically selects text fields and a chunking strategy for each field. You can deselect the checkbox and create a search index configuration later if you choose not to do so now.
13. Click Next, or if you created a search index configuration, review the details, and save your work.
14. After establishing the connection from your external blob store, set up a file notification pipeline to notify Data Cloud whenever files are added, updated, or deleted from your external blob store.

Send File Notifications from Azure Storage to Data 360

Create a file notification pipeline for Azure Storage to notify a Salesforce connected app whenever unstructured data files are added, updated, or deleted from a bucket.

Required User Permissions

Create a Private-Public Key Pair and Certificate

To configure OAuth for the connected app you will use in your file notification pipeline, create a private/public RSA key pair and a digital x509 certificate.

note: If you already have a connected app that you want to use, ensure that you have the private/public RSA key pair you used to create the x509 certificate for that app, as you need them to enable OAuth in a subsequent step.

1. From your terminal, change directories to any folder.

2. Create the private/public key pair.

   openssl genrsa -out keypair.pem 2048
   

3. Create a digital certificate from the key pair.

   openssl req -new -x509 -nodes -sha256 -days 365 -key keypair.pem -out certificate.crt
   

4. Complete the questions as prompted.

5. Create a pkcs8 private key from the key pair.

   openssl pkcs8 -topk8 -nocrypt -in keypair.pem -out private.key
   

6. Keep the private/public key pair and the digital x509 certificate, as you need them in the following task.

Download the Azure Storage File Notification Installer Script and Function

1. Download the Azure Storage file notification installer script.
2. Download and unzip azure_function.zip.

Set Up A Connected App and OAuth Settings

Set up a connected app to use in the file notification pipeline and apply necessary OAuth settings.

1. Create a basic connected app. If you already have a connected app, continue to Enable OAuth settings for the API integration.
2. Enable OAuth settings for the API integration.
   1. To use JWT OAuth Flow, select Use Digital Signatures.
   2. Enter the callback URL (endpoint) that Salesforce calls back to your application during OAuth. It’s the same as the OAuth redirect URI.
   3. Click Choose File, and select the certificate (.crt file) you created in Create a Private/Public Key Pair and Certificate.
   4. Set these OAuth scopes: Manage user data via APIs (api), Perform requests at any time (refresh_token, offline_access), and Manage Data Cloud Ingestion API data (cdp_ingest_api).
   5. Make sure Issue JSON Web Token (JWT)-based access tokens for named users is unchecked. If this option is checked, the org issues JWT-format access tokens that the Data Cloud token-exchange endpoint (/services/a360/token) rejects, and the file notification pipeline can't acquire a Data Cloud token.
   6. Note the Consumer Key and Callback URL.
3. Click Save.
4. In Setup, search for OAuth and OpenID Connect Settings.
5. Turn on Allow OAuth Username-Password Flows.
6. Using the Consumer Key and Callback URL created in Enable OAuth settings for the API integration. Paste the following URL into your browser using the noted values.

   <YOUR_ORG_URL>/services/oauth2/authorize?response_type=code&client_id=<YOUR_CONSUMER_KEY>&scope=api refresh_token cdp_ingest_api&redirect_uri=<YOUR_CALLBACK_URL>&code_challenge=SHA256
   

7. When prompted to provide permission for each of the scopes you requested, click Allow.
8. (Optional): If you want to verify that the app is correctly authorized, use the Quick Find search for “Connected Apps OAuth Usage”. Your connected app should be listed.

note: Depending on your downstream Data 360 usage (for example, querying or profiling ingested data), you may also need the cdp_query_api and cdp_profile_api scopes on the connected app.

Run the Install Script

Separate installer scripts are provided for macOS/Linux and for Windows.

macOS and Linux

1. In the directory where you extracted the file notification installer script, go to the installers/azure folder.
2. Open input_parameters_azure.conf and replace the environment variable values with your values.
3. Run the installer script. This command makes the file notification script executable.

   chmod +x setup_azure_file_notification.sh ./setup_azure_file_notification.sh input_parameters_azure.conf
   

• macOS users: When you run the installer script, you are transferred to the UI to authenticate. After authentication, return to the terminal. The script continues to run in the terminal.
• Linux users: When you run the installer script, an authentication link and code is displayed in the terminal. Copy the link and code to authenticate from your browser.

Windows

The Windows installer uses a PowerShell script and a .txt configuration file instead of the .sh script and .conf file used on macOS and Linux.

1. In the directory where you extracted the file notification installer script, go to the windows_installers/azure folder.
2. Open input_parameters_azure_windows.txt and replace the environment variable values with your values. This file uses the same variables as input_parameters_azure.conf; see the Azure Storage Variable Reference.
3. In Windows PowerShell, run the installer script, passing the configuration file with the -ConfigFile parameter.

   .\setup_azure_file_notification_windows.ps1 -ConfigFile input_parameters_azure_windows.txt
   

• When you run the installer script, a browser window opens and redirects you to the Azure portal to authenticate. Log in and close the window. The script continues to run in PowerShell to completion.

Azure Storage Variable Reference

Refer to these variables when creating a file notification pipeline from an Azure blob store to Data 360.

Troubleshooting File Notifications

These common issues can occur when setting up a connected app or Azure Storage file notification in Data 360. If you don’t see data connected in your org after loading unstructured data into Azure and creating a file notification pipeline, review these requirements.

Connected App

Ensure that you have:

• Used a valid consumer key and RSA private key. If you're using a consumer key from a previously configured connected app, check that the connected app and Salesforce org still exists.
• Set all three required scopes in the connected app, as documented.
• Ensured that Issue JSON Web Token (JWT)-based access tokens for named users is unchecked. When this option is checked, the pipeline receives a JWT-format access token that Data Cloud's token-exchange endpoint rejects.
• Authenticated your connected app. In the Quick Find, search for "Connected Apps OAuth Usage". You should see your connected app and a user count of 1.
• Uploaded a .crt file in the connected app, not a .pem or .key file.
• Used a .pem file when creating the RSA private key.
• Used a single set of keys (.crt, .pem, and .key) for both the cloud function and the connected app.
• Used the correct pattern for key names, as specified in the variable reference.

Unstructured Data Files and UDLO Creation

Ensure that you have:

• Aligned the directory structure of the external blob store source directory and the UDLO directory. Data 360 looks for files at <Container Path>/<UDLO Directory>. For example, if the connection's Container Path is data/ and you configure the UDLO Directory field as files, Data 360 looks for files at <Cloud Provider Bucket>/data/files. Don't repeat the same folder in both fields, as that produces a doubled path that matches no files.
• Correctly set the file type extensions and have matched the case used in the file names for the unstructured data files when creating the UDLO.

Installer Script and Cloud Function

Ensure that you have:

• Properly set all input parameters.
• Provided a valid Salesforce Org login url and username.
• Uploaded the correct, latest cloud function zip for your notification pipeline. For example, don't use the GCS cloud function zip for an Azure notification pipeline.
• Set up access to the Azure portal from your Salesforce account.

Additional Troubleshooting

See Also

• Salesforce Help: Unstructured Data Workflow
• Salesforce Help: Use Data 360 Search for AI, Automation, and Analytics
• Salesforce Help: Create a Vector Search Index
• Salesforce Help: Create a Hybrid Search Index
• Salesforce Help: Using Retrieval Augmented Generation
• Salesforce Help: Data Stream Settings and Refresh Modes