Not only is markdown useful for bloggers and writers, it’s also an easy to use language which can improve your software development processes and practices. History According to wikipedia, “Markdown is a lightweight markup language for creating formatted text using a plain-text editor.” Don’t confuse markdown with markup languages, with the…

How many times have you had to write the same query, on a dataset or dataframe, over and over again. Functions and/or procedures can provide many benefits such as code reusability and abstraction, even in big data projects such as PySpark. Let’s begin by getting some sample CSV data using…

You’ve got some big data which you have filtered and transformed into something “manageable” and would like to display it on a web page or mobile phone. So how can you get the data into a format which can then be consumed via a web service or another application? The…

Google Colab provides us with a notebook to start developing ETL pipelines with Apache Beam, an open-source programming model used with Google’s DataFlow cloud product. Practical Introduction Firstly install the Apache Beam open-source, programming model via the following pip install in your Google Colab notebook: !pip install apache-beam Then get some CSV…

Google produced a whitepaper way back in 2010, describing its experiences with “massively parallel computing done using shared clusters of commodity machines.” This white paper introduced us to the Dremel system, built on Google’s experiences with web search and parallel database management systems (DBMS). However, what really interested me, as…

While Jupyter style notebooks are excellent for data analysts and developers, there are numerous business cases for providing a web-based front-end to big data projects. This article describes how to build a web front-end for a PySpark project. Business Use Cases What business benefit is gained from moving a big…

Like so many of us, I started using big data tools such as PySpark after having spent many, many years working with traditional SQL. Therefore, the easy way in is to write big data queries using the same “traditional” SQL approach of SELECT statements, which is the typical approach I…

Streaming is an extension of the PySpark core API that allows us to process data from streaming or static data sources. Whilst there are myriad articles online about how to use Streaming for PySpark with databricks and other configurations, I found that attempting to set this up using Google Colab…

When multiple Scrum teams depend upon each other, this creates dependencies. How can we identify and manage the dependencies between teams to reduce risk and improve our development efficiency? In this article, I will provide an overview of a number of techniques which can be used to manage dependencies. According…