EPIC Database Documentation Index

Quick Navigation

📖 For First-Time Users

1. Start here: QUICK_REFERENCE.md — 5-minute overview
2. Then read: DATABASE_DOCUMENTATION.md — Sections 1–3 (input, processing, output)
3. Try this: Run python scripts/usage_examples.py to see working code

🏗️ For System Design

• ARCHITECTURE.md — Data flow diagrams, tensor construction, edge building logic

💻 For Implementation

• Database_DOCUMENTATION.md — Code recipes and patterns
• scripts/usage_examples.py — 7 runnable examples
• src/epic_preprocess.py — Core preprocessing functions

🔍 For Debugging & Validation

• DATABASE_DOCUMENTATION.md — QC checks and error handling
• QUICK_REFERENCE.md — Common errors & fixes

📊 For Data Analysis

• DATABASE_DOCUMENTATION.md — Manifest structure, batch loading
• scripts/usage_examples.py — Example 6 (batch statistics)

File Manifest

docs/
├── README.md (this file)
├── DATABASE_DOCUMENTATION.md      (Comprehensive: 11 sections, ~1000 lines)
├── QUICK_REFERENCE.md             (Quick-lookup: 10 sections, ~300 lines)
├── ARCHITECTURE.md                (Visual/technical: data flow, queries)
│
scripts/
├── preprocess_dataset.py           (Main preprocessing script)
├── usage_examples.py               (7 runnable examples)
├── make_figures.py                 (Visualization utils)
│
src/
├── epic_preprocess.py              (Core preprocessing functions)
│   ├── build_local_index()
│   ├── preprocess_epic_file_sparse()
│   └── helper functions
│
dataset/
├── raw/                            (260 *.csv files — raw EPIC input)
│   ├── CD011505_end1red_bright.csv
│   ├── CD011605_5a_bright.csv
│   ├── ... (258 more)
│
└── processed/by_embryo/            (260 *.npz files — processed output)
    ├── CD011505_end1red_bright.npz
    ├── CD011605_5a_bright.npz
    ├── ... (258 more)
    └── manifest.txt

Document Overview

📄 DATABASE_DOCUMENTATION.md (Comprehensive Reference)

Length: ~1200 lines | Audience: Everyone (but especially developers)

📋 QUICK_REFERENCE.md (Quick Lookup)

Length: ~300 lines | Audience: Experienced users needing fast reference

🏛️ ARCHITECTURE.md (System Design & Data Flow)

Length: ~400 lines |Audience: System designers, advanced users

Usage Patterns by Role

🧪 Data Scientist / ML Researcher

Goal: Train models on cell dynamics

Read:

1. QUICK_REFERENCE.md (Section “Minimal Working Example”)
2. DATABASE_DOCUMENTATION.md (Sections 3, 6, 9)
3. scripts/usage_examples.py (Examples 1, 2, 6)

Code pattern:

# Load batch of embryos
from pathlib import Path
import numpy as np

for npz_path in sorted(Path("dataset/processed/by_embryo").glob("*.npz"))[:10]:
    X = np.load(npz_path)["X"]  # (N, 5, T)
    # Train your model...

🔬 Biologist / Domain Expert

Goal: Understand cell behavior, verify data quality

Read:

1. DATABASE_DOCUMENTATION.md (Sections 1, 4, 8)
2. QUICK_REFERENCE.md (Sections “Feature Definitions”, “Cell Name ↔ Index”)
3. scripts/usage_examples.py (Examples 3, 5)

Code pattern:

# Track a cell's developmental history
cell_name = "ABal"
trajectory = [X[cell_idx, :, t] for t in range(T) if alive_mask[cell_idx, t]]
# Analyze migration, division timing, etc.

🛠️ Software Engineer / DevOps

Goal: Process data, manage pipelines, ensure data quality

Read:

1. DATABASE_DOCUMENTATION.md (Sections 2, 5, 7, 8, 11)
2. ARCHITECTURE.md (Sections “Data Pipeline”, “Validation”)
3. scripts/preprocess_dataset.py (runnable script)

Code pattern:

# Run full preprocessing pipeline
python scripts/preprocess_dataset.py \
    --raw_dir dataset/raw \
    --out dataset/processed/by_embryo \
    --distance_threshold 20

# Verify output
python scripts/usage_examples.py  # Run all QC checks

📚 Student / New Contributor

Goal: Understand the system from scratch

Path:

1. Week 1: QUICK_REFERENCE.md (all sections)
2. Week 1: Run python scripts/usage_examples.py
3. Week 2: Read DATABASE_DOCUMENTATION.md (sections 1–6)
4. Week 2: Study ARCHITECTURE.md diagrams
5. Week 3: Dive into src/epic_preprocess.py code

FAQ: Where Do I Find…?

Key Concepts Glossary

Core Terms

Biological Terms

Troubleshooting Guide

Problem: “FileNotFoundError: No such file or directory”

Likely cause: You haven’t run preprocessing yet

Solution:

cd d:\Github\Spatio-Temporal-Evolution
python scripts/preprocess_dataset.py --raw_dir dataset/raw --out dataset/processed/by_embryo

See DB_DOC §2.3

Problem: “KeyError: ‘X’”

Likely cause: Wrong file format; trying to load non-NPZ file

Solution:

import numpy as np
# Correct: load from processed/by_embryo/
npz = np.load("dataset/processed/by_embryo/CD011605_5a_bright.npz", allow_pickle=True)
X = npz["X"]  # Now works

Problem: Shape mismatch or “index out of range”

Likely cause: Different embryos have different N and T

Solution:

# Always check shape per embryo
for npz_path in paths:
    npz = np.load(npz_path)
    N, d, T = npz["X"].shape
    print(f"{npz_path.name}: N={N}, T={T}")

See Example 6

Problem: Memory error loading all 260 embryos

Likely cause: Not enough RAM for all data simultaneously

Solution:

# Load batch-wise
BATCH_SIZE = 10
for i in range(0, 260, BATCH_SIZE):
    batch = load_embryos(i, min(i+BATCH_SIZE, 260))
    # Process batch
    del batch  # Free memory

See DB_DOC §5.2

Contact & Support

• Code issues: Check DB_DOC §11
• Data quality: See DB_DOC §8 for validation checklist
• Biological questions: Refer to DB_DOC §4

Citation

If you use this preprocessed database, please cite:

@article{sulston1983lineage,
  title={The embryonic cell lineage of the nematode {C}aenorhabditis elegans},
  author={Sulston, JE and Schierenberg, E and White, JG and Thomson, JN},
  journal={Developmental Biology},
  volume={100},
  number={1},
  pages={64--119},
  year={1983}
}

Documentation Index Version: 1.0
Last Updated: April 2026
Total Docs: 4 files (~2000 lines + code examples)