Microsoft Dynamics (via Collector method) - v3.0.0

About Collectors

This collector has been tested with Microsoft Dataverse Web API v9.2

https://learn.microsoft.com/en-us/power-apps/developer/data-platform/webapi/web-api-versions

Step 1: Registering A Kada App in Azure AD

This step is performed by the Azure AD Admin.

Skip this step if you have already configured a Kada app in Azure AD for Entra SSO or Power BI integration

• Log in to your company's Azure Portal and open the Azure Active Directory page

• Select App Registration in the side panel and click New registration

• Complete the registration form

  • Name: Enter a name for the integration e.g. KADA Dynamics API Integration

  • Supported account types: Select Accounts in this organisation directory only

  • Redirect URL: Add Web / https://www.kada.ai

• Click Register to complete the registration

• Click on the newly created KADA Dynamics API Integration App

• Save the Application (client) ID and Directory (tenant) ID for use in a later step

• Click on Endpoints and save the URL for OpenID Connect metadata document for use in a later step

• Select Certificates & secrets in the side panel and click New client secret

• Complete the new secret form and save the Secret Value for use in a later step

Make sure you send the following information to the K Admin so that they can complete the next process.

• Application (client) ID

• Directory (tenant) ID

• Secret Value

Step 2: Kada App access to Dynamics API

This step is performed by a Dynamics admin

• Log in to your company's Dynamics / Power Apps

  • https://admin.powerplatform.microsoft.com/home

• Select Environments and click on the Environment that you want to load into K.

• Click on Settings

• Select on Users + permissions and click on Application Users

• Click New App User. In the slide out:

  • Click Add an App. Search for the App that was created in Step 1 and add it

  • Select an appropriate org

  • Add the Service Reader role

• Click Create

Step 3: Obtain the Web Resource API

This step is performed by a Dynamics admin

• Log in to your company's Power App Maker

  • https://make.powerapps.com/

• Select Settings. Click on Developer resources

• Save the Web API endpoint for use in a later step

Step 4: Review the list of entities to be loaded into K (Optional)

This step is performed by a Dynamics admin

The Dynamics integration will extract all entities (and fields). This may include some entities that you do not want to load into K. K supports extracting a list of entities (and associated fields) as well.

To review the list of entities you want to extract:

• Log in to your company's Power App Maker - https://make.powerapps.com/

• Select Tables

• Take down the list of table names that you want to include in the extract

Step 5: Create the Source in K

This and the following step is performed by a K admin

Create a source in K

• Go to Settings, Select Sources and click Add Source

• Select a Microsoft Dynamics as the Source type

• Select "Load from File" option

• Give the source a Name - e.g. Microsoft Dynamics Production

• Add the Host name for the Microsoft Dynamics Instance - e.g. https://<org-id>.api.crm6.dynamics.com

• Click Next & Finish Setup

Step 6: Getting Access to the Source Landing Directory

Step 7: Install the Collector

You can download the Latest Core Library and Microsoft Dynamics whl via Platform Settings → Sources → Download Collectors

Run the following command to install the collector

pip install kada_collectors_extractors_microsoft_dynamics-3.0.0-py3-none-any.whl

You will also need to install the corresponding common library kada_collectors_lib-x.x.x for this collector to function properly.

pip install kada_collectors_lib-x.x.x-py3-none-any.whl

Step 8: Configure the Collector

kada_microsoft_dynamics_extractor_config.json

{
    "client": "",
    "secret": "",
    "tenant": "",
    "resource": "",
    "entity_filter": [],
    "output_path": "",
    "mask": false,
    "compress": false,
    "timeout": 30,
    "meta_only": true
}

Step 9: Run the Collector

This code sample uses the kada_microsoft_dynamics_extractor.py for handling the configuration details

import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.microsoft_dynamics import Extractor

get_generic_logger('root')

_type = 'microsoft_dynamics'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA Dynamics API Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename)
parser.add_argument('--name', '-n', dest='name', default=_type)
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(args.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

publish_hwm(args.name, end_hwm)

Step 10: Check the Collector Outputs

K Extracts

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

High Water Mark File

A high water mark file is created called microsoft_dynamics_hwm.txt.

Step 11: Push the Extracts to K

Once the files have been validated, you can push the files to the K landing directory.

Example: Using Airflow to orchestrate the Extract and Push to K