Skip to main content

Creating SCD2 tables in dbt

I don't want this blog to become the dbt blog so I have taken my time to post about this but do fully intend to do some more posts on dbt and the cool built in functionality. Equally I am quite happy with where my model is at at the moment so until I find some new tool to use or a new data source I am going to look to expand the section on the free training available. 

dbt has the ability to cater for creating scd(2) style tables called snapshots, details of it are included on the advanced materialization training. I set up my first snapshot model by creating an scd2 table for the activity type dim, so that if I get a new exercise type added it will create a new rows, equally if I delete or modify one of the old columns it will end date the old row and insert the new row. The preference is to do this off a date column for change time however I don't have this so do the merge against all columns. 

Snapshots sit in their own folder and have a fairly simple modelling structure shown in my example below. 



Labels:

Comments

Post a Comment

Popular posts from this blog

Gen AI News - 01/04/2024

According to Gemini here is the latest news:  Here's a rundown on some recent Generative AI news you might find interesting: Nvidia's New Architecture: Nvidia announced their next-gen "Blackwell" architecture for GPUs [1]. This promises a significant leap in performance for running large language models and other generative AI tasks. AI for Mental Health: An article highlighted how chatbots powered by generative AI are being used to help Gen Z with mental health struggles [2]. Amazon Invests in Anthropic: Amazon is making a further investment in Anthropic, an AI startup focused on developing safe and beneficial AI [2]. This suggests big players are keen on the potential of generative AI. Google and Reddit Partner Up: There have been talks about Google and Reddit forming a new partnership [4]. This could involve using generative AI for content creation or recommendation on Reddit. Overall, the field of Generative AI is seeing continued investment and de...
Post a Comment
Read more

Gemini Gen AI News Today

Generative AI: This Week's Trends Shaping the Future The world of Generative AI (GenAI) is a whirlwind of innovation. This week has been no different, with exciting advancements and discussions shaping the future of this powerful technology. Here's a quick dive into the top trends that caught our eye: 1. The Rise of AI Memory: OpenAI made waves by introducing memory capabilities to a limited group of ChatGPT users. This allows the AI to remember details from past conversations, fostering more personalized and relevant interactions. This technology could revolutionize customer service chatbots and virtual assistants. 2. Generative AI for Healthcare Gets Specific: Research continues on how GenAI can benefit specific areas within healthcare. This week, a study emerged showcasing the potential for AI to streamline drug development by analyzing massive datasets of molecules and proteins. 3. The Power of Video and 3D Models: The creation of video and 3D models using generative ...
Post a Comment
Read more

Creating a date dimension in dbt and Snowflake

A lot of the data that I am working with doesn't lend itself to creating complex star schemas (more on data modelling in a later post) however I want to try and at least want to go to some effort. The one thing that pretty much all my tables have in common is date, so I want to build a date dimension. My strength training happens on a day, my running and walking, parkrun and hopefully the weather data I intend to integrate will have a date on it. This means if I have a date dimension and I aggregate my various facts at date level I can use them together in a single visualisation.  There are various methods I could have used to create the date dimension and I went with using the dbt spine, this was really simple using the utilities:  Used the date functions in Snowflake to extract relevant fields such as year, month, day in words and year month etc. 
Post a Comment
Read more