Currently a full manifest.json (generated after a dbt compile) is required to ensure all dbt models are loaded into K. We recommend pushing a new full manifest.json as part of your dbt CI/CD process so that every time you add/update/delete a dbt model, you update K as well.
We are working to remove this step for dbt cloud users using the new dbt APIs that have been recently been made available.
Step 1) Generate the manifest.json
-
If you are using dbt Cloud, log into your dbt cloud account and go a project (you will need to repeat this for every project)
-
Create a new job to run dbt compile.
-
You do not need to run any other dbt command or schedule this job.
-
-
Run the dbt compile job
-
After the job has been completed, download the manifest.json from the artifacts tab
-
Open the manifest.json file and find the
"DBT_CLOUD_PROJECT_ID"for dbt Cloud-
Use
"project_id"for dbt Core
-
-
Append the ID to the start of the manifest.json file name
-
e.g. 11231_manifest.json
-
Step 2) Upload the manifest.json to the dbt landing folder
-
Follow the steps in How to upload a file to the K landing directory to upload the manifest.json in Step 1