--- schema_version: "1" id: da3672558985e4bea3a8202c50bc3b844da057efc3527a03429d9e2553deffd6 title: Write and Validate a Custom Chunking Function collection: developer version: current locale: en-us portal_url: https://docs.salesforce.com/developer/current/en-us/data-cloud/data-cloud-code-ext/guides/function/write-custom-chunking-function.md canonical_url: https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/write-custom-chunking-function.html source_site: developer.salesforce.com content_type: documentation content_hash: d47db0f527fe2fbea126eecdcbfe9cc44bc7133fd50c039eb0f002fb93a978ee products: - Data 360 Code Extension Guide - Data 360 Developer Documentation --- # Write and Validate a Custom Chunking Function In code extension, function runs are specific to search index chunking, where your code controls how content is split before vectorization and indexing. When you select your custom function in the search index builder, Data 360 passes a list of document elements to your function, and your function must return a dictionary with generated chunks. This topic focuses on writing that logic and validating it before deployment. | Edition Table | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Available in: **Developer**, **Enterprise**, **Performance**, and **Unlimited** Editions. See [Data 360 edition availability](https://help.salesforce.com/s/articleView?id=sf.c360_a_dc_editions.htm&type=5). | | Permission Sets Needed | | | ------------------------------------------------- | ------------------------------------------------------ | | To write and validate a custom chunking function: | Permission set: | ## Authoring ### Required and Optional Fields at a Glance - Required input: request object with an `input` list. - Optional input metadata: fields such as `page_number`, `transcript_fields`, `text_as_html`, `source_dmo_fields`, `prepend`, and image metadata fields (`image_base64`, `image_mime_type`, `image_type`), when available. - Required output: top-level `output` list. - Required chunk fields: `text`, `seq_no`, `chunk_type`. - Optional output fields: `citations`. Before you implement `payload/entrypoint.py`, define the logic your function applies to each DocElement list: 1. Identify source patterns: List the content patterns you expect, for example, titles, section text, tables, lists, or mixed HTML blocks from your Data 360 source data. 2. Define chunk boundaries: Decide exactly where one chunk starts and ends, for example, per heading, per section, or by token/length limits. 3. Define context rules: Specify what must stay together in a chunk, for example, product name, specs, and link, to preserve retrieval quality. 4. Define citation mapping: Decide which metadata fields to carry into `citations`, for example, `FilePath__c`, `page_number`, or source type, for traceability. 5. Define validation scenarios: List the sample and org-derived payloads you use to verify output shape and chunk quality. ### Function Signature Encapsulate your custom chunking logic in a top-level callable in `payload/entrypoint.py`. In the initiated function package scaffold, the callable is: ```py def function(request: SearchIndexChunkingV1Request, runtime: Runtime) -> SearchIndexChunkingV1Response: ``` - **request (`SearchIndexChunkingV1Request`)**: The typed request object containing input documents to chunk. - **runtime (`Runtime`)**: Runtime context provided by the platform (required by the callable signature). - **returns (`SearchIndexChunkingV1Response`)**: The typed response object containing generated chunks under `output`. The runtime reads the entry point file from `payload/config.json` and executes your callable from that file. In the initiated scaffold, keep the callable signature as just shown. ### Input Specification The callable receives `request` and `runtime`. The `request` object uses the `SearchIndexChunkingV1Request` schema, which contains one top-level key, `input`. | Request Key | Data Type | Required | Description | | ----------- | -------------------- | -------- | ----------------------------------- | | `input` | List of `DocElement` | Yes | List of document elements to chunk. | ```json { "input": [ { ... DocElement 1 ... }, { ... DocElement 2 ... } ] } ``` ### DocElement Object Each `DocElement` in the list is a dictionary representing a piece of content from the source document. | Key | Description | | ------------ | -------------------------------------------------------- | | **text** | The text content of the element. | | **metadata** | Information about the element (type, origin, and so on). | Example: ```json { "text": "The actual text content of the element.", "metadata": { ... } } ``` The fields inside `metadata` vary by element type. ### DocElement Types The element type is specified by the `type` key inside `metadata`. Types are grouped into these categories. | Type | Description | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `title` | Structural titles and section headers. | | `text` | Standard blocks of text. | | `list_item` | Individual items within a list. | | `table` | Structured, tabular data where `text` contains extracted table text. When Code Extension parsing and preprocessing is enabled, metadata can also include table HTML. | | `code_snippet` | Pre-formatted source code or command snippets. | | `image` | Visual elements (charts, diagrams, pictures). Metadata includes `image_base64`. | | `page_metadata` | Peripheral content such as headers, footers, and page numbers | For simple file formats, for example, .html, .txt, .log, and .aspx, you can pass the entire file content as a single DocElement. In that case, `text` holds the full content and the metadata `type` is `text`. ### Metadata Fields Use this as the complete metadata field reference. | Metadata Key | Data Type | Required | Description | | ------------------- | -------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------- | | `type` | Enum (`text`, `title`, `table`, `image`, `list_item`, `code_snippet`, `page_metadata`) | No | Element type. Defaults to `text` when omitted. | | `page_number` | Integer | No | Page number in the source document. | | `transcript_fields` | Object | No | Transcript metadata for audio/video content. | | `text_as_html` | String | No | HTML representation of content (for example, table HTML). | | `source_dmo_fields` | Dictionary (`string -> string/int/float`) | No | Source DMO field values as key-value pairs. | | `prepend` | List of objects | No | Fields whose values are prepended before indexing. | | `image_base64` | String | No | Base64 image data for image-type elements. | | `image_mime_type` | String | No | Image MIME type (for example, `image/png`). | | `image_type` | String | No | Semantic image category (for example, `diagram`). | Nested object fields: - `transcript_fields.speaker` (String, optional) - `transcript_fields.start_timestamp` (Float, optional) - `transcript_fields.end_timestamp` (Float, optional) - `prepend[].dmo_name` (String, optional) - `prepend[].field_name` (String, optional) - `prepend[].value` (String, optional) ### Standard Metadata Fields for UDMOs and DMOs - Unstructured Data Model Objects (UDMOs): Input metadata sent to your function can include fields such as `Col1__c` through `Col5__c`, `ContentType__c`, `DataSourceObject__c`, `DataSource__c`, `ETag__c`, `FilePath__c`, `InternalOrganization__c`, `KQ_FilePath__c`, `LastModified__c`, `Metadata__c`, `ParentId__c`, `ResolvedFilePath__c`, and `Size__c`, as applicable. - Structured DMOs: Input metadata sent to your function can include fields such as `datasourceobject__c` and `datasource__c` when chunking from a specific field. These are input fields provided in the request metadata when available. You don't create these metadata fields in your function. Use available fields in your `citations` to trace chunks back to their source. ### Output Specification Your function generates the output. It must return a dictionary with an `output` key. The value of `output` must be a list of `Chunk` dictionaries, in sequential order. ```json { "output": [ { ... Chunk 1 ... }, { ... Chunk 2 ... } ] } ``` ### Top-Level Response Use this table to decide which keys are mandatory. | Key | Required | Description | | -------- | -------- | ----------------------------------------------------- | | `output` | Yes | List of chunk dictionaries returned by your function. | ### Chunk Object Each chunk in the list must be a dictionary with these keys. | Key | Data Type | Description | | -------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | **text** | String (required) | The text content of the chunk. | | **seq_no** | Integer | The sequence number of the chunk, starting from 1. Must be a continuous, ordered sequence. | | **chunk_type** | String | Chunk type. Use `text`. | | **citations** | Dictionary of string key-value pairs | Optional metadata from source DocElements (for example, `FilePath__c`, `page_number`) to identify the origin of the chunk. | The initiated project sample can include additional keys that are ignored by the typed model. Keep your function output aligned with the response schema by always returning a valid `output` list with correctly ordered chunk objects. ### Minimum Valid Function Use this template as a minimal baseline when you start authoring. ```py from datacustomcode.function import Runtime from datacustomcode.function.feature_types.chunking import ( ChunkType, SearchIndexChunkingV1Output, SearchIndexChunkingV1Request, SearchIndexChunkingV1Response, ) def function( request: SearchIndexChunkingV1Request, runtime: Runtime ) -> SearchIndexChunkingV1Response: chunks = [] for i, element in enumerate(request.input, start=1): chunks.append( SearchIndexChunkingV1Output( text=element.text, seq_no=i, chunk_type=ChunkType.TEXT, ) ) return SearchIndexChunkingV1Response(output=chunks) ``` ## Example Function The following example creates one chunk per DocElement. ### Example Input ```json { "input": [ { "text": "Document Title", "metadata": { "type": "text", "page_number": 1, "source_dmo_fields": { "FilePath__c": "relative_file_path_from_UDMO" } } }, { "text": "This is the first paragraph of the document, discussing important topics.", "metadata": { "type": "text", "page_number": 1, "source_dmo_fields": { "FilePath__c": "docs/sample.pdf" } } } ] } ``` ### Example Implementation ```py from datacustomcode.function import Runtime from datacustomcode.function.feature_types.chunking import ( ChunkType, SearchIndexChunkingV1Output, SearchIndexChunkingV1Request, SearchIndexChunkingV1Response, ) def function( request: SearchIndexChunkingV1Request, runtime: Runtime ) -> SearchIndexChunkingV1Response: """ A simple chunking strategy that creates one chunk per DocElement. """ chunks = [] for i, element in enumerate(request.input): source_fields = element.metadata.source_dmo_fields if element.metadata else {} citation_data = { "source_type": element.metadata.type.value if element.metadata and element.metadata.type else "", "page_number": str(element.metadata.page_number) if element.metadata and element.metadata.page_number is not None else "", "file_path": str(source_fields.get("FilePath__c")) if source_fields and source_fields.get("FilePath__c") else "", } chunk = SearchIndexChunkingV1Output( text=element.text, seq_no=i + 1, chunk_type=ChunkType.TEXT, citations={k: v for k, v in citation_data.items() if v}, ) chunks.append(chunk) return SearchIndexChunkingV1Response(output=chunks) ``` ### Example Output ```json { "output": [ { "chunk_type": "text", "text": "Document Title", "seq_no": 1, "citations": { "source_type": "text", "page_number": "1", "file_path": "relative_file_path_from_UDMO" } }, { "chunk_type": "text", "text": "This is the first paragraph of the document, discussing important topics.", "seq_no": 2, "citations": { "source_type": "text", "page_number": "1", "file_path": "docs/sample.pdf" } } ] } ``` ## Validating 1. Use the auto-generated sample payload `payload/tests/test.json` to run an initial validation of your custom logic in `payload/entrypoint.py`. 2. Validate the package by running this command. ```bash sf data-code-extension function run --entrypoint --test-with [--target-org ] [--config-file ] [--dependencies ] ``` **Example:** ```bash sf data-code-extension function run --entrypoint ./payload/entrypoint.py --test-with ./payload/tests/test.json ``` - `--entrypoint ./payload/entrypoint.py` (required): Path to your function entry point. - `--test-with ./payload/tests/test.json` (required): Path to the JSON file that provides test input for the function run. - `--target-org ` (optional): The alias or username of the Salesforce org where you want to run the code extension package. Required when your function testing includes callouts to generative or predictive models registered in your org. The org must have Data 360 enabled with the appropriate permissions. - `--config-file ` (optional): JSON file that supplies input for the run. If omitted, the command uses the default `payload/config.json`. - `--dependencies ` (optional): Comma-separated dependency overrides for local runs. 3. If you need additional scenarios, create more test payloads in `payload/tests/` by following the same request structure as `test.json`, and run the command again with the new file path in `--test-with`. 4. Verify that the returned dictionary has the correct `output` shape, such as a list of chunks with `text`, `seq_no`, `chunk_type`, and optional `citations`, and that chunking logic behaves as expected. 5. Refresh package metadata before packaging or deployment. ```bash sf data-code-extension function scan [--entrypoint ] [--config-file ] [--dry-run] [--no-requirements] ``` **Example:** ```bash sf data-code-extension function scan --entrypoint ./payload/entrypoint.py ``` - `--entrypoint ./payload/entrypoint.py`: (optional if you run the command from the package root): Path to your function entry point. - `--dry-run` (optional): Preview changes without modifying files. - `--no-requirements` (optional): Update `config.json` without changing `requirements.txt`. For best practices and limitations (for example, deterministic output, edge-case handling, and execution model constraints), see [Considerations When Writing Custom Chunking Functions](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/considerations-when-writing-custom-function.md). If you want to implement prompt-driven chunking logic by calling the LLM gateway from your function, see [Use LLM Gateway Calls in Custom Chunking Functions](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/use-llm-gateway-in-custom-function.md). ## Next Step - [Deploy Custom Function to Data 360](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/deploy-custom-function-overview.md) ## See Also - [Considerations When Writing Custom Chunking Functions](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/considerations-when-writing-custom-function.md) - [Use LLM Gateway Calls in Custom Chunking Functions](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/use-llm-gateway-in-custom-function.md) - [Use Custom Functions in Data 360](https://developer.salesforce.com/docs/data/data-cloud-code-ext/guide/use-custom-function.md) - [Execute the Code Extension Workflow by Using Agentforce Vibes](https://developer.salesforce.com/docs/data/data-cloud-code-ext/use-agentforce-vibes-ide-code-extension.md) - [Execute the Code Extension Workflow by Using Agentic Tools and Salesforce Skills](https://developer.salesforce.com/docs/data/data-cloud-code-ext/use-agentic-tools-code-extension.md)