Skip to main content

Creating SCD2 tables in dbt

I don't want this blog to become the dbt blog so I have taken my time to post about this but do fully intend to do some more posts on dbt and the cool built in functionality. Equally I am quite happy with where my model is at at the moment so until I find some new tool to use or a new data source I am going to look to expand the section on the free training available. 

dbt has the ability to cater for creating scd(2) style tables called snapshots, details of it are included on the advanced materialization training. I set up my first snapshot model by creating an scd2 table for the activity type dim, so that if I get a new exercise type added it will create a new rows, equally if I delete or modify one of the old columns it will end date the old row and insert the new row. The preference is to do this off a date column for change time however I don't have this so do the merge against all columns. 

Snapshots sit in their own folder and have a fairly simple modelling structure shown in my example below. 



Labels:

Comments

Post a Comment

Popular posts from this blog

Keboola Flows

Really finding Keboola was the thing that kickstarted this project otherwise I would be trying to build custom code on a python cloud server and building everything from scratch.  In Keboola you build you data sources and destinations using connection details which is fairly simple and something I will likely cover in another post, same goes for transformations etc. Here though I am going to discuss Flows, this is where you bring everything together. On my free account there are some limitations.  My easiest flow is very basic:  Pull parkrun results e-mail from Gmail to Google Sheets (actually done by Zap not Keboola).  Keboola will, as often as I like, in this case once a week, pull the data from the sheet into its storage.  It will then transfer this to the target database. Currently I have this setup to be MySQL database but I can and might expand that to the Snowflake instance within Keboola.  I then, outside of Keboola, connect to the MySQL database f...
Post a Comment
Read more

Zoho Analytics

Have I finally found my BI Tool, one that lets me import data from Snowflake and share it for free? I know, no sooner have I posted about how hard it was to find a tool that could do anything from Snowflake than I come across Zoho. You can check out my dashboard on the following page . Below is a diagram that outlines the processes I have used to obtain this data. In summary my parkrun e-mail is pushed to Google Sheets every week by Zapier and Forms I submit every day are used to track the strength training I do. Keboola is then used to ingest this data into MySQL and or Snowflake where I then use views or the built in transformation processes in Keboola to shift the data into a format for reporting. Google Data Studio then connects to MySQL and Zoho to Snowflake to visualise the data. 
Post a Comment
Read more

Generative AI News - 03/03/2024

Here are some of the latest news highlights related to Generative AI from this week: Venus Williams and Brain.ai : Tennis legend Venus Williams has ventured into interior design with Brain.ai , a new generative AI-powered platform 1 . It’s fascinating to see how generative AI is being integrated into various domains beyond smartphones. Google’s Gemini AI Controversy : Google’s chatbot, Gemini , faced controversy due to its image generator mistakenly placing people of colour in Nazi-era uniforms. This incident highlights the importance of responsible AI development 2 . Apple’s Focus on GenAI : Apple CEO Tim Cook announced that the company will “break new ground” in generative AI this year 2 . Keep an eye out for exciting developments from Apple. Morph Studio for Filmmaking : Morph Studio introduces an AI filmmaking platform that allows weaving Stability AI-generated video clips into movies 2 . A creative application of generative AI! Adobe’s Project Music : Adobe unveiled Project Music...
Post a Comment
Read more