Loading a full manifest.json for dbt (Cloud & Core)

Currently a full manifest.json (generated after a dbt compile) is required to ensure all dbt models are loaded into K. We recommend pushing a new full manifest.json as part of your dbt CI/CD process so that every time you add/update/delete a dbt model, you update K as well.

We are working to remove this step for dbt cloud users using the new dbt APIs that have been recently been made available.

Step 1) Generate the manifest.json

• If you are using dbt Cloud, log into your dbt cloud account and go a project (you will need to repeat this for every project)

• Create a new job to run dbt compile.

  • You do not need to run any other dbt command or schedule this job.

• Run the dbt compile job

• After the job has been completed, download the manifest.json from the artifacts tab

• Open the manifest.json file and find the "DBT_CLOUD_PROJECT_ID" for dbt Cloud

  • Use "project_id" for dbt Core

• Append the ID to the start of the manifest.json file name

  • e.g. 11231_manifest.json

Step 2) Upload the manifest.json to the dbt landing folder

• Follow the steps in How to upload a file to the K landing directory to upload the manifest.json in Step 1