A simple use case of Automating administrative tasks in EKS using Kubernetes Cron job — Kubernetes is one of the most widely used open source container orchestration platforms. The easy, quick and handy way of automating tasks in Unix for most DevOps Engineers is to schedule a script with Unix crontab. Similarly, Kubernetes provides a native way of scheduling tasks at a fixed schedule. This…

Spark remote job submission allows client to submit Spark jobs to Yarn cluster from anywhere, decoupling the client from the Yarn cluster. — Spark remote job submission allows clients to submit Spark jobs to the Yarn cluster from anywhere. This can also be leveraged to submit Spark jobs to different Yarn clusters running different versions of Hadoop and Spark. Problem Statement We are using AWS EMR cluster with Yarn as a resource manager to run…

A simple approach to compare Pyspark DataFrames based on grain and to generate reports with data samples — Comparing two datasets and generating accurate meaningful insights is a common and important task in the BigData world. By running parallel jobs in Pyspark we can efficiently compare huge datasets based on grain and generate efficient reports to pinpoint the difference at each column level. Requirement:

Simple way of implementing S3 logging in Python3 — One of the Key aspects of a software framework is to log necessary information and persist them. Using boto3 and native python logger its easy to persist logs into AWS S3 within a python program. Logs would be helpful to debug, monitor, audit and understand the behaviour of the framework…