Skip to main content

Keboola Flows

Really finding Keboola was the thing that kickstarted this project otherwise I would be trying to build custom code on a python cloud server and building everything from scratch. 

In Keboola you build you data sources and destinations using connection details which is fairly simple and something I will likely cover in another post, same goes for transformations etc. Here though I am going to discuss Flows, this is where you bring everything together. On my free account there are some limitations. 

My easiest flow is very basic: 

  • Pull parkrun results e-mail from Gmail to Google Sheets (actually done by Zap not Keboola). 
  • Keboola will, as often as I like, in this case once a week, pull the data from the sheet into its storage. 
  • It will then transfer this to the target database. Currently I have this setup to be MySQL database but I can and might expand that to the Snowflake instance within Keboola. 
  • I then, outside of Keboola, connect to the MySQL database from Google Data Studio and make some visualisations. 
Within Keboola flows you have several tabs. The builder tab where you configure your flow. The All Runs tab to look at the logs of Flow Runs, Notifications where you can configure e-mail notifications for various results of the Flow (Success, Error etc.) and the versions tab where you can look at the history of the flow. 

You can see many of these steps and the basic config in the below gif. 


The end result of this flow is the e-mail date, subject and body being passed into the MySQL database. I then do some data cleansing on this (post to come) and then visualise (currently poorly) in Google Data Studio. I intend to do another post on comparing this very simple data set by visualising in GDS, Power BI and Retool. 


Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

Gen AI news 29-04-2024

Here are some recent updates and insights related to Generative AI (gen AI) : Enterprise Hits and Misses - Robotics and Gen AI Converge : This article discusses the convergence of robotics and generative AI. It explores breakthroughs needed in the field, the FTC’s policy change regarding non-competes, and the impact on AI model sizes for enterprises 1 . Read more All You Need To Know About The Upcoming AI-Powered OLED iPad Pro : This piece provides a summary of rumors surrounding the next-gen AI-fused OLED iPad Pro, powered by the new Apple M4 chip 2 . Read more Delivering on the Promise of Gen AI : New Electronics reflects on NVIDIA GTC and key announcements that contribute to delivering on the promises made for generative AI 3 . Read more The Future of Generative AI - An Early View in 15 Charts (McKinsey): Since the release of ChatGPT in November 2022, generative AI has been making headlines. McKinsey research estimates that gen AI features could add up to $4.4 trillion to the globa...
Post a Comment
Read more

Snowflake Scripting - SQL Cursors

Snowflake scripting in SQL seems to be in preview and I have decided to have a play with it. Given how new it is there is limited documentation so I am using a combination of what I can find on the Snowflake site and the odd blog that has been written about it. There appear to be a few quirks, at least when compared to Oracle PL/SQL (though that has been round for years). How many of these are intentional and how many are things to be ironed out I don't know. You can see the procedure I have created it:  Accepts an id as a parameter  Creates a result set selecting from a table, using the parameter as a filter Loads the results set into a cursor.  Loops through the cursor loading the id in the cursor into variable Calls procedure passing in the variable as the parameter.  Then as a proof of concept I tried the Snowflake feature of allowing declaration of variables within the main start and end block. 
Post a Comment
Read more