Blog | Pythian | Data Engineering

Oracle

Oracle’s AI Data Platform Workbench: “Hello World”

Oracle has recently released a new service within OCI called AI Data Platform Workbench, or AIDPW for short.

Data Engineering

The Art of the Deployment Pipeline

Navigating the complexities of data engineering releases can cause endless sleepless nights for even the most seasoned ...

A close-up photograph of a modern developer's workspace monitor.

Analytics

Tip Tuesday | Use Python For Querying JSON Files

Don't waste time importing simple JSON files into a database just to run a quick query. With Python's list ...

Database

Updating Elasticsearch indexes with Spark

With the extensive adoption of Elasticsearch as a search and analytics engine, more often we build data pipelines that ...

Data Engineering

How to schedule weekdays only on Airflow

Consider the following situation: You have a data ingestion pipeline where the data comes in real-time on weekdays and ...

Google Cloud

Spark UDF memoization

Memoization is a powerful technique that allows you to improve performance of repeatable computations. Although it ...

Databricks

Where in a Modern Data Platform Should You Process Your Data?

Data warehouse or data lake? We break down the pros and cons of each In the book Designing Cloud Data Platforms, ...

Database

An Introduction to Chaos Engineering: Trying to Break Stuff

Some time ago there was a car ad with the slogan "Don't use it, abuse it", meaning that no matter what you do to the ...

Analytics

How the England and Wales Cricket Board scored with big data

The England and Wales Cricket Board governs every aspect of the sport in those two countries, and it holds massive ...

Google Cloud

The data platform in the age of serverless architecture

“The transformation of the superstructure, which takes place far more slowly than that of the substructure, has taken ...

BigQuery

Data modeling for cloud DW

Intro In this blog post, I would like to share some options that you can consider to model your cloud DW for better ...

Oracle

An Oracle-based approach to the "taxi fare" prediction problem - episode 1

Although I haven't produced a lot of posts, those I have produced were always strictly related to the Oracle RDBMS as a ...

Cloud

Data Streaming with Kafka and Flink on AWS – Part 1

Apache Kafka and Apache Flink are popular data streaming applications platforms. However, provisioning and managing ...

Analytics

Approaching Master Data Management

Our previous post focused on ‘lightweight governance’ – enabling engineering and product teams across an organization ...

BigQuery

The five key Requirements for a successful SAP data management strategy

Introduction In our last blog article on data management and the DART methodology, we discussed the importance of ...

Google Cloud

Building an ETL Pipeline with Multiple External Data Sources in Cloud Data Fusion

In this post, I’ll share a quick start guide on Google Cloud Platform’s (GCP) Cloud Fusion. We’ll first take a look at ...

Analytics

Metadata-Driven Data Governance: the How and Why

In our previous discussion, we explored the role of data stewards and their vital function for data governance ...

Google Cloud

Data preparation with dbt and BigQuery

Raw incoming data needs to go through a series of data preparation steps before it can be used for analysis. These ...

Data Engineering

Python: Using Dataclasses to Model Your Data

Here at Pythian, we love our data. Our code is no exception (pun sort of intended), so I’ll be covering dataclasses in ...

Google Cloud

Apache Beam: the Future of Data Processing?

Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines. It’s a ...

Snowflake

Snowflake System Function Error: Argument 0 to Function SYSTEM$PIPE_STATUS Needs to Be Constant

I recently encountered the above issue which prompted me to write this blog post so I can easily reference the solution ...

Cloud

Data Streaming with Kafka and Flink on AWS – Part 2

Apache Kafka and Apache Flink are popular platforms for data streaming applications. However, provisioning and managing ...

Google Cloud

Near Real-Time Data Processing for BigQuery: Part Two

This post is part two of describing (near) real-time data processing for BigQuery. In this post, I will use Dataform to ...

Google Cloud

Migrate RDB to Cloud SQL Using Google’s Dataflow

Most corporations have huge amounts of data in RDBMS (relational database management system). When considering a RDBMS ...

Google Cloud

Consuming Tweets Using Apache Beam on Dataflow

Apache Beam is an SDK (software development kit) available for Java, Python, and Go that allows for a streamlined ETL ...

Microsoft SQL Server

Monitoring Azure Data Factory, Self Hosted IR and Pipelines

Architecture diagram For effective monitoring of ADF pipelines, we are going to use Log Analytics, Azure Monitor and ...

Google Cloud

Secure Data Ingestion and its Impact on Cloud Modernization

Secure data ingestion is an essential component of cloud modernization—and an essential next step in the broader effort ...

Analytics

Data Governance in Legacy Environments

Bridging Analytical Models and Modern Governance In our previous post we discussed the governance requirements for ...

MySQL

Replicating MySQL to Snowflake with Kafka and Debezium—Part One: Data Extraction

What, Snowflake? Yes, Snowflake. While my core skills are based on the Oracle database, lately I’ve been working more ...

Data Engineering

DAG Monitoring with Airflow

What is Apache Airflow? Airflow is a platform to programmatically author (designing pipelines, creating workflows), ...

AI Related Latest Blogs

Oracle

Oracle AI Data Platform (AIDP): No-Nonsense Platform Overview

Oracle’s AI Data Platform (AIDP) is essentially an attempt to fix the biggest annoyance ...

Google Workspace

AI in the Workplace: Employee Productivity & Sentiment Survey

2025 employee survey on the use of AI In November 2025, Pythian, a professional services ...

Google Workspace

Is Agentspace good for non-Google shops?

So you’re not a Google shop. Does that mean you can’t take advantage of the AI ...

View All →

Analytics related blog posts

Analytics

Using Multi-Agent Reasoning for NLP-to-SQL

As data-driven decision-making becomes essential, access to that data shouldn’t require deep technical ...

Analytics

Becky's BI Apps Corner: OBIA 11.1.1.10.2

Upgrade jitters? If you’re running older versions of OBIA, the new 11.1.1.10.2 release brings game-changing features like automatic error handling and ...

Analytics

Oracle Data Integrator (aka Sunopsis) Now On OTN

Oracle Data Integrator (formerly Sunopsis) has officially landed on the Oracle Technology Network, bringing

View All →

Build a scalable data foundation to accelerate with AI.

Get started today →

Oracle’s AI Data Platform Workbench: “Hello World”

The Art of the Deployment Pipeline

Tip Tuesday | Use Python For Querying JSON Files

Updating Elasticsearch indexes with Spark

How to schedule weekdays only on Airflow

Spark UDF memoization

Where in a Modern Data Platform Should You Process Your Data?

An Introduction to Chaos Engineering: Trying to Break Stuff

How the England and Wales Cricket Board scored with big data

The data platform in the age of serverless architecture

Data modeling for cloud DW

An Oracle-based approach to the "taxi fare" prediction problem - episode 1

Data Streaming with Kafka and Flink on AWS – Part 1

Approaching Master Data Management

The five key Requirements for a successful SAP data management strategy

Building an ETL Pipeline with Multiple External Data Sources in Cloud Data Fusion

Metadata-Driven Data Governance: the How and Why

Data preparation with dbt and BigQuery

Python: Using Dataclasses to Model Your Data

Apache Beam: the Future of Data Processing?

Snowflake System Function Error: Argument 0 to Function SYSTEM$PIPE_STATUS Needs to Be Constant

Data Streaming with Kafka and Flink on AWS – Part 2

Near Real-Time Data Processing for BigQuery: Part Two

Migrate RDB to Cloud SQL Using Google’s Dataflow

Consuming Tweets Using Apache Beam on Dataflow

Monitoring Azure Data Factory, Self Hosted IR and Pipelines

Secure Data Ingestion and its Impact on Cloud Modernization

Data Governance in Legacy Environments

Replicating MySQL to Snowflake with Kafka and Debezium—Part One: Data Extraction

DAG Monitoring with Airflow

Recent news

Pythian Recognized as a Top AI & Data Consulting Firm for 2026

Pythian Recognized as Silver Winner in UKOUG Data Integration Partner of the Year Awards 2026

Pythian Leaders Recognized on CRN’s 2026 Women of the Channel List

Pythian and Three of Its Leaders Named Finalists for the 2025 CRN Women of the Year Awards

Pythian India Earns “Great Place to Work” Certification for Fourth Consecutive Year

Pythian’s Vanessa Simmons and Donna Williston Again Named to CRN’s 2025 Women of the Channel List

Pythian Positioned as Oracle Database @ Google Cloud Leader with Acquisition of Rittman Mead

Pythian Doubles Down on AI-Readiness, Expanding AI Practice with Valuable Services and Expertise

Pythian Launches Agentspace QuickStart

Pythian Wins 2025 Google Cloud Databases Partner of the Year for North America

Pythian and GigaOm Announce AI Partnership

Pythian Named Top Employer in Canada's National Capital Region for 2025

Pythian Welcomes Shawn Gandhi as Senior Vice President, Customer Engineering, and Country Managing Director for Canada

Pythian Announces New Oracle Database@Google Cloud Migration Services

Pythian India Earns “Great Place to Work” Certification for Third Consecutive Year

Pythian Recognized on CRN’s 2024 Solution Provider 500 List and DBTA’s 100 Companies That Matter Most in Data for 2024

Pythian Welcomes Brooks Borcherding as Chief Executive Officer

Pythian’s Vanessa Simmons and Donna Williston Honored by CRN on 2024 Women of the Channel Lists

Pythian Recognized as a Top Employer in Canada’s National Capital Region in 2024

Pythian Announces Generative AI Strategy and Offerings to Accelerate Enterprise Innovation

CRN Names Pythian’s Vanessa Simmons a 2023 Women of the Channel Power 80 Solution Provider

Award-Winning Marketing Veteran Christina O’Reilly Joins Pythian Services as Senior Vice President, Marketing

Pythian Makes Database Trends and Applications’ List of Trendsetting Solution Providers for 2023

Pythian Celebrates 25 Years of Enterprise Data, Analytics, and Cloud Success

Pythian India Achieves “Great Place to Work” Certification After Building High-Trust, High-Performance Culture

Pythian Named One of ‘The Companies That Matter Most in Data’ by Database Trends and Applications Magazine

Vanessa Simmons from Pythian Named on CRN’s 2022 Women of the Channel Power 70 Solution Providers List

Pythian Named Partner in Launch of Google Cloud Cortex Framework

Pythian’s SAP Services Practice Grows by 300 Percent in 2021

Pythian Renews Google Cloud Infrastructure Specialization in the Google Cloud Partner Advantage Program

Pythian Services Recognized on CRN’s 2022 MSP 500 List

Pythian Renews Machine Learning Partner Specialization in the Google Cloud Partner Advantage Program

Pythian Sets Company Record with Over 100 New Customers in 2021

Pythian’s CTO Paul Lewis Featured Commentary Published by InformationWeek

How to Modernize Backup Without Breaking the Bank

Pythian Launches FinOps Managed Services

Pythian Acquires SAP MSP Partner ManageServe

Pythian Taps Seasoned Cloud and Data Expert Paul Lewis as CTO

Pythian Achieves Managed Services Provider Status In the Google Cloud Partner Advantage Program

Keith Angell: Accelerating Enterprise Growth With Data And Analytics Through Remarkable Leadership

Pythian Achieves the Data Management Partner Specialization in the Google Cloud Partner Advantage Program

Pythian Adds Snowflake Data Cloud Services Suite

New CEO Keith Angell: Pythian Services A ‘Golden Nugget’

Pythian Appoints Keith Angell As CEO

Pythian Services Acquires Google Cloud Premier Partner Agosto

Pythian Announces Support For AWS Outposts As A Launch Partner

Pythian Wins 2018 Google Cloud Data Analytics Partner of the Year Award

Pythian achieves Machine Learning Partner Specialization in the Google Cloud Partner Specialization Program

Pythian achieves Cloud Migration Partner Specialization in the Google Cloud Partner Specialization Program