fbpx
supertype logo dark (wide)
supertype logo dark (wide)
idx logo

Automated Market Insights: Revolutionizing analytics creation through automatic agents

Featuring: Supertype Summary

This project features Supertype Summary, a programmatic report generation service that produces analytical reports from raw data in seconds, not days.

IDX (Indonesia Stock Exchange) adopts analytics automation in their stock market reporting, greatly increasing statistical accuracy while eliminating manual laborious report creation

Challenges At a Glance

  • Manual creation of stock market reports consumes significant time and is prone to human error.
  • Data is scattered across 100+ Excel spreadsheet files manually collected, cleansed, and organized
  • Lack of a coherent data architecture and overarching schematic strategy results in frequent breaking changes and manual PowerPoint tweaks to accommodate these changes
  • Absence of a data validation process during data processing compromises the reliability of the daily stock market statistical report
  • The statistics reporting content is outdated and overloaded with excessive information
  • Data are stored in spreadsheets, with no persistent database; Risk of files deleting without backup and loss of data

Why: Transitioning from a Excel Spreadsheet PowerPoint workflow to a fully automatic report generation workflow with Supertype Summary

Challenges

In service of the Indonesian economy, the Indonesia Stock Exchange (IDX) strives to ensure equitable information access for all stakeholders in the capital market. Towards this goal, IDX generates in-depth stock market statistical reports at varying intervals (i.e. daily, weekly, monthly, quarterly, and yearly periods). These reports are labor-intensive work, necessitating repetitive queries against the data source and design input that are adjusted to suit each reporting format and informational needs.

Beyond the substantial time investment, a key issue that arise from this workflow revolves around data validation and accuracy. Supertype identifies 4 key Challenges in the current analytics reporting process within IDX’s Data Services division.

  • Scattered and manually gathered data

    Data is scattered across multiple sources, with a reliance on spreadsheet files that are manually gathered and compiled

  • Inaccuracies and Delays

    Data is manually processed each day, allowing for human-dependent delays and erroneous data entry that are hard to reconcile or troubleshoot

  • Breaking Changes & Inconsistencies

    These challenges are further compounded by a lack of a consistent data architecture, resulting in frequent breaking changes and moving targets in the generation of these statistical reports

  • Data Security & Governance Risk

    Data are primarily stored in spreadsheet file formats, with a notable absence of dedicated databases or data warehousing solutions. This poses a considerable data security and governance risk, as files may be accidentally deleted without backup, potentially resulting in the loss of valuable historical data.

Approach

1. Create a Data Architecture

Employing the TOGAF (The Open Group Architectural Framework) framework, we design a data architecture that eliminates spreadsheet files and manual data processing from the existing structure.

2. Data Migration to Database

Supertype introduces a process for migrating historical data stored in spreadsheet files into the database. Our analytics engineers then create the necessary Python scripts to prepare the data for downstream i.e the stock market statistical report.

3. Data Validation Program

Supertype’s engineers develop a bespoke data validation pipeline that re-validates each row of data as it gets added into the database.

4. Automated Report Generation

The adoption of Supertype Summary brings end-to-end automation to the reporting creation process while guaranteeing consistency in each iteration of its report.

Technologies

Python Programming Language

Python is often referred to as a general-purpose computer programming language and is especially suited for IDX's goal in data engineering, statistical visualization and analytics automation.

PostgreSQL Database

PostgreSQL is a powerful, open source object-relational database management system widely used among enterprises and startups alike.

Data Visualization Libraries

Matplotlib is a comprehensive plotting library for creating rich data visualizations in Python.

Apache Airflow

Apache Airflow is an open-source workflow management platform for data engineering pipelines. Data Engineers at Supertype used it to manage and organize ETL (Extract-transform-load) pipelines through Directed Acyclic Graphs (DAGs).

Solutions

End to End Analytics Automation: A pragmatic approach to programmatic report creation

1. Create a Data Architecture

Supertype and the IDX (Indonesia Stock Exchange)’s Data Services division collaboratively outline the primary objectives and hurdles in generating stock market statistical reports. This involves identifying the key stakeholders and establishing the business overview of the report. Employing the TOGAF framework, we design a data architecture that eliminates spreadsheet files and manual data processing from the existing structure. The replacement involves implementing a database as the data warehouse and leveraging Python and Apache Airflow for automated, on-the-fly data processing.

2. Data Migration & Database Creation

Building upon the established data architecture, Supertype introduces a process for migrating historical data stored in spreadsheet files into the database. Analytic Engineers from Supertype team then create the necessary Python scripts to prepare the data for downstream i.e the stock market statistical report. These scripts are configured to run automatically every working day, alleviating the manual processing requirements for the IDX team. The migration to a database not only facilitates easier data management for the IDX team but also ensures data integrity and safeguards against potential losses. The Analytics Engineer at Supertype create a data dictionary document as part of this delivery.

3. Data Validation Program

Data reliability was identified as a significant challenge early into this project. While automating data processing and report generation through Supertype Summary — a programmatic report creation progam by Supertype — greatly improves data reliability, there remains a possibility of inaccuracies due to external factors. To address this, Supertype’s engineers developed a bespoke data validation pipeline that re-validates the as it flows into the database.

This pipeline conducts two types of evaluations: (1) data presence and (2) completeness. With (1), it verifies the successful daily storage of data in the database while guaranteeing the absence of null values in new data. With (2), it intelligently compare similar data from different tables to deduce correctness and validity, qualifying each row as fit for use by stakeholders in the financial market.

4. Automated Report Generation

A key deliverable of this project is a fully automated analytics system that takes advantage of Supertype Summary to generate 30-page PowerPoint slides in a matter of seconds. The transition to Summary brings end-to-end automation to the reporting creation process, but also increases consistency in each iteration of its report, It features the use of modern visualization and statistical graphs, replacing the limited graphs created in Microsoft Excel by its predecessor.

Supertype brings new perspective from their work on Supertype Sectors, incorporating elements that enhance the report while greatly streamline the report creation process. It also features the use of Apache Airflow, leading to an automatic scheduler that generates detailed analytical reports on predetermined intervals.

arsyil_idx_bursa_efek

Collaborating with Supertype for our integrated reporting project has been a transformative experience.

Their deep understanding of data science has revolutionized our reporting processes, making them more efficient and insightful.

Their team's commitment to delivering solutions that meet our requirements has been truly impressive. Thanks to their innovative solutions, we now have integrated reporting that saves us time and resources."

Arsyil Hendra Saputra

Indonesia Stock Exchange (Bursa Efek)

Key Deliverables

Well-Designed Analytics Architecture

A proper, well thought-out data architecture that uses the TOGAF (The Open Group Architecture Framework) framework. It consolidates:

  • The business process and business objectives of this reporting project.
  • The challenges faced in the previous analytics production process.
  • A specific recommendation on the technology stack and architectural design for this project.

Automatic, on-the-fly Data Processing

Automatic data cleansing and data processing pipelines executed everyday as new data flows in, modeled after Supertype Sectors.

Single Source of Truth

A database to store all of the 120+ tables that act as a unified source of truth for the statistical reports, its programmatic report process and an accompanying data dictionary document.

Automated Report Generation

Incorporation of Supertype Summary: a fully automated report creation program for 8 different types of statistical reports. Each of the report are executed on its respective schedule, and are respectively 30+ pages in length with plenty of statistical analyses, tabular presentations, and modern data visualization

Integrated Data Validators

Bespoke data validation programs to make sure that the data are qualified independently for validity and correctness; thus upholding the integrity of reported figures in the final reports

Key Benefits

Analytics Automation: 10 days to 1 hour Reporting

The ETL process automatically retrieves data from local storage and transforms it into ready-to-use data for various analyses. Supertype Summary is a key piece of automation that significantly reduce the overhead required in analytics reporting. Client’s estimates put the time reduction at 95%, from 10-20 days to just 1-2 hours, where the analytical process is fully automatic and programmatically-generated.

The manpower required to operate the analytical process is reduced by 67%, shaving 450 hours of labor hours each month by eliminating data entry, spreadsheet compilations, manual data preparation, Excel charting, presentation design and editing work.

Improved Reporting Accuracy and Validity

By eliminating human labor, the accuracy of IDX’s reporting has increased from 85% to 99.5% in internal benchmarks. The introduction of a centralized database replaces the use of static spreadsheet files scattered across different locations, improving query efficiency considerably.

Incorporating better informational design, the new reporting system conveys an equal amount of insights in just 30 pages, de-bloating from over 200 pages in the predecessor design. The resulting report is more accessible, has greater semantic consistency, and more purposeful in its analytics presentation.

Analytics Engineering and Automation powered by Supertype's core Services

Supertype Summary

From data to Complete Presentation PDFs, in seconds.

IDX (Indonesia Stock Exchange) Data Services plays a critical role in fulfilling the information needs of Indonesia's capital and financial markets. In service of these goals, it engages Supertype to architect a fully automatic analytics creation process.

Supertype's PDF report generation as a service enables IDX to create IDX-branded, visually-rice statistical reports and analyses in matters of seconds.

Supertype Summary
supertype summary
supertype summary
supertype summary
supertype summary pdf generation
Previous
Next

Supertype Sectors

https://sectors.app/enterprise

Indonesia's financial data layer

Supertype Sectors is an always-learning financial data layer that powers our suite of enterprise-grade market intelligence products.

Plug-n-Play Fintech Enablement

Build your financial services, market analytics, in-house research, and other data-driven applications on top of our data warehouses and API services.

Onboarding & Customization

Dedicated on-ramp training, white-labeling, custom domain and hosting, and dedicated account management services to make Sectors truly yours.

Contact Us

supertype_idx_bursa_efek_case_study

Want more case study?

This case study, along with others in our archive are available in the PDF format. To schedule a call with us and learn more about our past projects, leave your contact method below.

Your work email, so we can reach you. We respect your privacy and do not engage in any spamming activities.

Inquiries

s@supertype.ai

Back to homepage

More content from Supertype