Workflow Guide
Every data task in MAGIC follows a five-phase pipeline that moves from raw inputs to finished outputs. The magic-data-lifecycle skill orchestrates this pipeline, routing work to the right specialist skill at each phase.
You don't invoke phases manually. Describe your goal in plain language and the lifecycle skill determines which phase to activate and when to transition.
The Pipeline at a Glance
| Phase | Skill | What Happens |
|---|---|---|
| Discover | magic-data-loading | Format detection, schema inference, initial load |
| Plan | magic-data-lifecycle | Quality gating, skill routing, checkpoint strategy |
| Execute | Domain skills | Cleaning, transformation, analysis, synthesis |
| Validate | magic-data-validation | Schema enforcement, constraint checking, quality gates |
| Deliver | magic-report-generation, magic-data-visualization | Reports, charts, exports |
Phase Walkthrough
Discover
Skill: magic-data-loading
The agent inspects the input source — a file path, database connection string, or API endpoint — and determines how to read it. Format detection handles CSV, JSON, Parquet, Excel, and more. For databases the agent infers the connection driver and previews the schema.
At the end of Discover the raw data is checkpointed as ckpt_01_raw_data.\{ext\} so that no subsequent phase needs to re-fetch the source.
Key decisions made here:
- Encoding and delimiter (for delimited files)
- Row count and memory footprint
- Whether a profiling pass is warranted before proceeding
Plan
Skill: magic-data-lifecycle (orchestrator)
The lifecycle skill reviews the loaded data and the user's stated goal, then builds a skill route — an ordered list of specialist skills to activate. It also decides:
- Which quality gates to enforce (e.g. minimum completeness score before transformation)
- How many checkpoint steps to create
- Whether the task needs
magic-data-profilingbefore cleaning or can skip straight to transformation
The plan is surfaced as a brief summary in the agent's response. You can ask the agent to explain or adjust the plan before it begins executing.
Execute
Skills: magic-data-profiling, magic-data-cleaning, magic-data-exploration, magic-statistical-analysis, magic-data-transformation, magic-data-synthesis
This is where the actual data work happens. The lifecycle skill activates each specialist in sequence, passing checkpointed outputs between them. Each specialist:
- Reads the previous checkpoint
- Performs its operation (imputation, normalization, reshape, etc.)
- Writes a new checkpoint with a descriptive name
If a step produces unexpected results — for example, a cleaning pass drops more rows than expected — the agent pauses, explains what happened, and asks whether to continue or adjust parameters.
Validate
Skill: magic-data-validation
After execution, the agent runs a validation pass against the final dataset. Validation checks:
- Schema conformance — column names, data types, and cardinality match the expected schema
- Constraint checks — nullability, range bounds, referential integrity
- Quality score — the profiling score must meet the configured threshold (default 85/100)
If validation fails, the agent does not proceed to Deliver. It surfaces the failing checks and recommends targeted fixes. You can ask it to apply fixes automatically or resolve them manually.
Deliver
Skills: magic-report-generation, magic-data-visualization
The final phase produces human-readable outputs from the validated data. Depending on your goal the agent may generate:
- A Markdown quality report with findings and recommendations
- Static charts (PNG/SVG via matplotlib) or interactive charts (HTML via Plotly)
- A cleaned export file in the requested format
All outputs are written under workspace/ so they are easy to find and share.
Skill-to-Phase Mapping
magic-data-loading
Discover phase — multi-format detection and ingestion
magic-data-profiling
Execute phase — quality scoring and distribution analysis
magic-data-cleaning
Execute phase — imputation, normalization, deduplication
magic-data-validation
Validate phase — schema enforcement and constraint checking
magic-data-transformation
Execute phase — reshape, pivot, merge, derived columns
magic-report-generation
Deliver phase — structured Markdown reports
Skipping Phases
Not every task requires all five phases. The lifecycle skill skips phases that are not relevant:
- No validation needed — if you are exploring raw data without a target schema, the Validate phase is omitted
- No delivery needed — if your goal is a cleaned file rather than a report, the Deliver phase writes the export and stops
- No planning needed — single-skill tasks (e.g. "profile this CSV") bypass the lifecycle skill entirely and invoke the specialist directly
Single-skill invocations still save checkpoints and update the workspace state file so the session remains resumable.
Last updated on