Big Data: What’s New? What’s Next? Part Two

In this post, we will look at recent developments in the world of data and technology and extrapolate what comes next. While there is a bit of guesswork in predicting the future, we can look around and detect patterns that might develop into one possible future. Two themes of 2022 in particular lend themselves to a bit of prognosticating.

Article Highlight

Automate the mundane. The only constant is change.

First, I have read several articles that predict the death of ETL and ELT as we know it. Personally, I am still skeptical. Running reports against transactional source systems is a recipe for a performance nightmare. Also, I’ve found it impossible to imagine a data warehouse without facts and dimensions. Reading a little deeper into these articles, the authors usually propose that ETL/ELT will not die but will be automated if you purchase their wares. But, what if…? Even if it is automation of the process, I know more than a few people who would like to have ready-to-use data without all the ingesting, transforming and normalizing – not to mention the hours of troubleshooting edge cases and source system changes.

Second, have you heard of Chat GPT by OpenAI? Apparently, a machine could write this article. Along with the promise of easy content generation for your blogs (source: Medium) and directions for removing a sandwich from a VCR written in King James English (source: Twitter), Chat GPT offers term papers written beyond a student’s given abilities (source: New York Times).

So, what if we assume a future without ETL/ELT and content generated by a bot? With a bit of training, a bot could learn to read an ERP database and determine which tables contain inventory, orders, customers and shipping. The bot could then build a data lake consisting of the facts and dimensions required for answering business questions. Another well-trained bot could then create reports based on metadata created by the first bot. One day a CFO will ask a bot about how year-to-date sales compare to the last 10 years and receive a chart and a narrative about how sales are performing. The concept is not too far fetched. We already use bots for anomaly detection, loan approvals and writing blog content.

What becomes of the ETL developers and report writers? ETL developers will be well-suited to building and training models. Report writers understand the value of combining data sources. These shifts will echo the shifts from SQL to Python and on-prem to cloud. In fact, the future, like the past, will be full of change.