Automating data extraction with dynamic transformation to enable ETL for enhanced data catalogue and dashboard insights

The session introduced an innovative approach to automate the generation of Data Frames, CSV and Parquet files from multiple Excel datasets (data resides in various columns, rows and worksheet), enabling seamless integration with minimal ETL coding for utilization in data catalogues and dashboards. This streamlined process enhances faster data accessibility.

Key Takeaways

• Dynamic Extraction and Transformation support in configuration file
• Automating data extraction from Excel to Data Frames, CSV and Parquet formats, hence streamlines data processing workflows.
• No ETL tool is required. It requires a tremendous effort to read an Excel sheet with different rows, columns and repeating rows efficiently. 
• Ability to read Excel datasets URL from sharepoint

————————————————————————————————————————————————————

Bio

Dr. Seah Boon Keong- Senior Software Engineer, Monetary Policy Department | Bank Negara Malaysia (BNM)| Malaysia

Seah is a Data Scientist and Data Engineer with over 20 years’ experience in technology, Banking, FMCG, telco, industrial predictive maintenance, and research. Seah’s areas of expertise include Data Analytics, Machine Learning, Data Modeling, Data Warehouse and Solution Architect. Seah has extensive experience in building large platforms which have delivered significant business results. He has also successfully developed AI related services on document Q&A, AI chart generation. Presently he helps enabled the Financial Data Catalogue, Dashboarding and data related pipelines for faster data insights and decision making. Aside from financial services, Seah previously Head the Data Science Platforms and Analytics in BAT, hence enabling business growth in new category products and marketing in BAT. Seah was the pioneer in establishing the Data Science Platform and Analytics in BAT.