Jong Ha Shin
Jong Ha Shin/
Macrogen: Data-driven platform using AI
Search
Macrogen: Data-driven platform using AI

Macrogen: Data-driven platform using AI

Tag
LLM
RAG
Tech Stack
OpenAI, Python, Streamlit, FastAPI, MySQL, AWS
🏢Example of AX from the Leadership PerspectiveExecutive SummaryKey ContributionsAchievementsIntroduction, Problem, and GoalIntroductionProblemGoalTechnical OverviewProblem-Solving in Action: Insights from Overcoming Project Hurdles1. Solving Cross-System Fragmentation2. Enhancing Insight Generation3. Ensuring Stability During Scale-Up

🏢Example of AX from the Leadership Perspective

Executive Summary

In genomic research operations, critical data, including sample and analysis information from LIMS and revenue or operational data from SAP, were siloed across separate systems, forcing teams to manually gather and reconcile information. To address this, I led the development of the AI-based Genome Dashboard platform, which consolidates LIMS and SAP data for real-time tracking, visualization, and as a data-driven decision-making tool leveraging AI. The platform now significantly improves accessibility and efficiency by unifying siloed data, empowering business teams to make data-driven strategies and decisions.

Key Contributions

  • A. Product & System Architecture Planning: Defined system objectives and workflows to unify fragmented LIMS and SAP data.
  • B. Real-Time Data Pipeline & Integration: Built APIs connecting LIMS and SAP systems to a centralized database.
  • C. Dashboard & UI Development: Built a Streamlit-based interactive dashboard for data visualization and AI-based inquiry.
  • D. Cloud Deployment & Operation: Designed scalable deployment using AWS infrastructure (AMI, ASG, ALB, CloudWatch).

Achievements

  • Successfully validated performance for real-time tracking of clinical samples and revenue data, used by internal researchers and business teams.
  • Teams now utilize integrated data to generate trend-based insights and predictive reports.
  • Improved operational efficiency and reduced manual data processing.

Introduction, Problem, and Goal

Introduction

With the rapid advancement of genomics and sequencing technologies, we at Macrogen generate vast amounts of both scientific and business data. Sample and analysis data are managed in LIMS, while revenue and operational data reside in SAP. However, the lack of integration between these systems created significant barriers: our researchers and business teams struggled to obtain a comprehensive, real-time view of projects, leading to duplicated effort, slow workflows, and missed opportunities for insight. As data volume and complexity increased, the need for a unified, accessible platform became critical to support both scientific discovery and business agility. The platform now significantly improves accessibility and efficiency by unifying siloed data, empowering business teams to make data-driven strategies and decisions.

Problem

  • Scientific (LIMS) and business (SAP) data were siloed across separate infrastructures, requiring manual cross-checking and reconciliation.
  • High-volume sequencing and business operations made manual tracking slow, error-prone, and inefficient.
  • Difficulty accessing comprehensive, integrated data limited analytical insights, slowed downstream research, and delayed business reporting.

Goal

  • Build a unified platform enabling real-time tracking and interpretation of both sequencing/clinical and revenue/operational data.
  • Improve accessibility through a visual dashboard and AI-driven decision support for both research and business users.
  • Reduce time spent collecting, validating, and reconciling data manually across LIMS and SAP systems.

Technical Overview

  • LLM & AI Integration
    • OpenAI
  • Real-Time Data Pipeline
    • FastAPI (data ingestion and integration between systems)
  • Database
    • MySQL
  • Web Interface
    • Streamlit (for interactive dashboard and AI interface)
  • Cloud Infrastructure
    • AWS AMI (Amazon Machine Image, for easy updates)
    • AWS ASG (Auto Scaling Group, for scalability)
    • AWS ALB (Application Load Balancer, for scalability and routing)
    • AWS CloudWatch (for logging and monitoring)

Problem-Solving in Action: Insights from Overcoming Project Hurdles

1. Solving Cross-System Fragmentation

Problem:
Researchers and business teams were forced to access multiple, disconnected systems—such as LIMS for scientific data and SAP for business data—to collect the information needed for project tracking and reporting. This manual, repetitive process not only wasted valuable time but also increased the risk of errors and inconsistencies, ultimately reducing productivity and slowing down both research and business operations.
Solution:
Designed a real-time integration pipeline using FastAPI and a unified MySQL database structure, consolidating previously separated datasets into a single interface.

2. Enhancing Insight Generation

Problem:
Even after data collection, researchers struggled to generate insights due to complex manual analysis.
Solution:
Developed a Streamlit-based dashboard providing structured visualization and LLM-assisted queries for rapid interpretation.

3. Ensuring Stability During Scale-Up

Problem:
Increasing sample volume created performance pressure on internal systems. using caching and improve the performance of data cleaning and visualization logic to use the lowest possible instnace the cost optimiaation of the system
Solution:
Deployed a scalable AWS architecture using ASG and ALB, enabling dynamic resource expansion and stable operation.