Skip to main content

dbt - pivoting on list of values

Say you have a massive table but for ease of use for reporting you need to split numbers out. For example say you are a service that provides numerous different services, each person gets a single form with a mark for each service provided and the transactional system stores this a a row per form per service. For reporting you may well want to pivot these out and have a separate columns for each service with a 1 / 0 and a count for each. 

In traditional databases and tools you have to create all your views in advanced and you cannot pivot on a list of values. The joy I am finding with dbt and jinja coding is that you can create the views dynamically. And whilst that may pose some risk if means that in this scenario a new service code is catered for automatically as you can dynamically generate the column by looping through a list of values. 

I have applied this logic my strength exercise data, pivoting it from rows to columns. Below is the jinja code and the resulting SQL, 

The other thing I used here was from the utils was slugify utility to replace spaces with underscores in my strings so I can use them in column names.

Comments

Popular posts from this blog

Generative AI News - 03/03/2024

Here are some of the latest news highlights related to Generative AI from this week: Venus Williams and Brain.ai : Tennis legend Venus Williams has ventured into interior design with Brain.ai , a new generative AI-powered platform 1 . It’s fascinating to see how generative AI is being integrated into various domains beyond smartphones. Google’s Gemini AI Controversy : Google’s chatbot, Gemini , faced controversy due to its image generator mistakenly placing people of colour in Nazi-era uniforms. This incident highlights the importance of responsible AI development 2 . Apple’s Focus on GenAI : Apple CEO Tim Cook announced that the company will “break new ground” in generative AI this year 2 . Keep an eye out for exciting developments from Apple. Morph Studio for Filmmaking : Morph Studio introduces an AI filmmaking platform that allows weaving Stability AI-generated video clips into movies 2 . A creative application of generative AI! Adobe’s Project Music : Adobe unveiled Project Music...

Creating a date dimension in dbt and Snowflake

A lot of the data that I am working with doesn't lend itself to creating complex star schemas (more on data modelling in a later post) however I want to try and at least want to go to some effort. The one thing that pretty much all my tables have in common is date, so I want to build a date dimension. My strength training happens on a day, my running and walking, parkrun and hopefully the weather data I intend to integrate will have a date on it. This means if I have a date dimension and I aggregate my various facts at date level I can use them together in a single visualisation.  There are various methods I could have used to create the date dimension and I went with using the dbt spine, this was really simple using the utilities:  Used the date functions in Snowflake to extract relevant fields such as year, month, day in words and year month etc. 

Gen AI News - 01/04/2024

According to Gemini here is the latest news:  Here's a rundown on some recent Generative AI news you might find interesting: Nvidia's New Architecture: Nvidia announced their next-gen "Blackwell" architecture for GPUs [1]. This promises a significant leap in performance for running large language models and other generative AI tasks. AI for Mental Health: An article highlighted how chatbots powered by generative AI are being used to help Gen Z with mental health struggles [2]. Amazon Invests in Anthropic: Amazon is making a further investment in Anthropic, an AI startup focused on developing safe and beneficial AI [2]. This suggests big players are keen on the potential of generative AI. Google and Reddit Partner Up: There have been talks about Google and Reddit forming a new partnership [4]. This could involve using generative AI for content creation or recommendation on Reddit. Overall, the field of Generative AI is seeing continued investment and de...