Skip to main content

dbt - pivoting on list of values

Say you have a massive table but for ease of use for reporting you need to split numbers out. For example say you are a service that provides numerous different services, each person gets a single form with a mark for each service provided and the transactional system stores this a a row per form per service. For reporting you may well want to pivot these out and have a separate columns for each service with a 1 / 0 and a count for each. 

In traditional databases and tools you have to create all your views in advanced and you cannot pivot on a list of values. The joy I am finding with dbt and jinja coding is that you can create the views dynamically. And whilst that may pose some risk if means that in this scenario a new service code is catered for automatically as you can dynamically generate the column by looping through a list of values. 

I have applied this logic my strength exercise data, pivoting it from rows to columns. Below is the jinja code and the resulting SQL, 

The other thing I used here was from the utils was slugify utility to replace spaces with underscores in my strings so I can use them in column names.
{%- set activities = dbt_utils.get_column_values(
table=ref('stg_exercise_star'),
column= 'description'
) -%}
SELECT
{{ dbt_utils.current_timestamp() }} last_refresh,
date_done,
{% for activity in activities -%}
sum(case when description = '{{ activity }}' then act_count else 0 end)
as {{ dbt_utils.slugify(activity) }}_count
{%- if not loop.last -%}
,
{% endif -%}
{%- endfor %}
from {{ ref('stg_exercise_star') }}
group by 2
SELECT
current_timestamp::
timestamp_ntz
last_refresh,
date_done,
sum(case when description = 'Pull Up' then act_count else 0 end)
as pull_up_count,
sum(case when description = 'Crunch' then act_count else 0 end)
as crunch_count,
sum(case when description = 'Push Up' then act_count else 0 end)
as push_up_count,
sum(case when description = 'Dumbbells' then act_count else 0 end)
as dumbbells_count,
sum(case when description = 'Box Jump' then act_count else 0 end)
as box_jump_count,
sum(case when description = 'Plank Time' then act_count else 0 end)
as plank_time_count
from KEBOOLA_7127.WORKSPACE_15661914.stg_exercise_star
group by 2
view raw pivoted_gen.sql hosted with ❤ by GitHub

Comments

Popular posts from this blog

AWS training cloud academy free course

One of the things I like about this course are the instructors are really clear but also that it provides free labs that allow you to actually sign into AWS and perform some actions to actually create and do things without worrying that you are going to incur a cost.  Today I complete one of the hands on labs.  This was to create a lambda function, in this case it was a very basic python script that was searching a website for a keyword. I then placed this into a schedule and used cloudwatch to create a dashboard that monitored the running of this function. Overall it was a very simple use case but it was also a very simple process to setup.  I don't have much to add to this other than it is well worth signing up to cloud academy for the free training if nothing else, I am tempted, once i have done some more training, to give the paid for option a go to get the full sandboxes. 

AI News

Here’s a concise roundup of the latest AI news from the past couple of days: AI Technology: Friend or Foe? Researchers and experts continue to debate the impact of artificial intelligence. Is it a boon or a threat? The discussion ranges from AI ethics to its potential in various fields. Read more here . 5 Ways Artificial Intelligence Will Change the World by 2050 Futurists predict that AI will revolutionize our lives in the coming decades. From healthcare to transportation, AI is set to transform industries. Explore the possibilities here . How AI Will Transform Businesses in 2023 Business leaders are embracing AI to enhance efficiency, decision-making, and customer experiences. Stay updated on the latest AI trends in the corporate world here . China’s High-Level AI Modules China is pushing the boundaries of AI with modular next-generation systems. These high-level AI technologies promise breakthroughs in fields like robotics, healthcare, and smart cities. Learn more here . The Future ...

Data Cleansing View in Snowflake

For part of one of my free ETLs I am using Zapps to transfer e-mails from Google Sheets and then Keboola to transfer the sheets into my Snowflake database. I am familiar with string searches and cleansing in Oracle and using python but have not had the chance to do this in Snowflake. I wanted to give it a go as a proof of concept if nothing else. There were some difference in functions between Oracle and Snowflake, no INSTR and using POSITION instead and some difference in working with dates / timestamps but overall it was very similar.  The code below is what I ended up using:  I think want to use this to create some overview graphics to allow me to track the success or failure of my ETLs. Assuming the aspects of Retool remain free you can see how much ETL is going on this link .  In case things aren't working, here is a table of the output I am producing.