Skip to main content

Keboola Flows

Really finding Keboola was the thing that kickstarted this project otherwise I would be trying to build custom code on a python cloud server and building everything from scratch. 

In Keboola you build you data sources and destinations using connection details which is fairly simple and something I will likely cover in another post, same goes for transformations etc. Here though I am going to discuss Flows, this is where you bring everything together. On my free account there are some limitations. 

My easiest flow is very basic: 

  • Pull parkrun results e-mail from Gmail to Google Sheets (actually done by Zap not Keboola). 
  • Keboola will, as often as I like, in this case once a week, pull the data from the sheet into its storage. 
  • It will then transfer this to the target database. Currently I have this setup to be MySQL database but I can and might expand that to the Snowflake instance within Keboola. 
  • I then, outside of Keboola, connect to the MySQL database from Google Data Studio and make some visualisations. 
Within Keboola flows you have several tabs. The builder tab where you configure your flow. The All Runs tab to look at the logs of Flow Runs, Notifications where you can configure e-mail notifications for various results of the Flow (Success, Error etc.) and the versions tab where you can look at the history of the flow. 

You can see many of these steps and the basic config in the below gif. 


The end result of this flow is the e-mail date, subject and body being passed into the MySQL database. I then do some data cleansing on this (post to come) and then visualise (currently poorly) in Google Data Studio. I intend to do another post on comparing this very simple data set by visualising in GDS, Power BI and Retool. 


Labels:

Comments

Post a Comment

Popular posts from this blog

AI News

Here’s a concise roundup of the latest AI news from the past couple of days: AI Technology: Friend or Foe? Researchers and experts continue to debate the impact of artificial intelligence. Is it a boon or a threat? The discussion ranges from AI ethics to its potential in various fields. Read more here . 5 Ways Artificial Intelligence Will Change the World by 2050 Futurists predict that AI will revolutionize our lives in the coming decades. From healthcare to transportation, AI is set to transform industries. Explore the possibilities here . How AI Will Transform Businesses in 2023 Business leaders are embracing AI to enhance efficiency, decision-making, and customer experiences. Stay updated on the latest AI trends in the corporate world here . China’s High-Level AI Modules China is pushing the boundaries of AI with modular next-generation systems. These high-level AI technologies promise breakthroughs in fields like robotics, healthcare, and smart cities. Learn more here . The Future ...
Post a Comment
Read more

MySQL - Free

 So I was looking at trying to get a cloud based database that was always on. I wanted to build some visuals over whatever data I ended up building and have the DB accessible from a cloud server seemed like the easy way. I wanted to keep it free because I hate spending when I don't need to, so that others could use it for free and because I was sure there must be options out there. In the end my life was made much easier by spending £10 but you can go with the same free option on this site. https://www.freemysqlhosting.net/ Although not super fast or super sized it gives you a free and easily accessible database. So far I have easily connected using phpAdmin, BeeKeeper Studio, Python, Google Data Studio and Keboola. I have had no issues at all unlike several other solutions I have tried including Heroku.  To setup the DB you just set your location and hit start, you will then be e-mailed the connection details and then use your favourite MySQL IDE and you are in.  Above i...
Post a Comment
Read more

Gen AI News - 05-04-2024

Here are some recent developments in the field of General AI (Gen AI) : China’s Use of AI to Sow Division in the US According to a report from Microsoft’s threat centre, China is increasingly using AI-generated content and fake social media accounts to inflame division in the United States and other regions. These tactics have raised concerns about the influence of AI in shaping public discourse 1 . UN Chief Concerned About Israel’s AI Target Identification in Gaza UN Secretary-General Antonio Guterres expressed serious concern over reports that Israel is using artificial intelligence to identify targets in Gaza. The use of AI in military operations has resulted in civilian casualties, prompting global scrutiny 2 . The AI Deepfake Apocalypse Deepfake technology continues to evolve, raising ethical and security concerns. Researchers are exploring ways to combat deepfakes, but the challenge remains. Adobe’s Dana Rao emphasizes the need for combined methods to control AI-generated images ...
Post a Comment
Read more