Built on WIX Headless

The Data Migration Lifecycle

From a committed price to a result proven against the source, the lifecycle runs in nine steps. The first two fix the scope and the price. The next five turn the actual source data into approved rules and generated code, business decisions before technical ones. The last two prove the result and keep every decision traceable.

  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Phase 1 · Steps 1–2

Fix the Scope and the Price

Two steps that turn numbers you already have into a scope you can count and a price that is fixed before you are locked in.

  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Count & Commit

The price starts from counts you already have. You fill in a short questionnaire: your systems, the source and target files, and their field counts.

Committed Scope · Predefined Price

From the questionnaire we commit to a scope, a price, and a schedule. Small differences in the counts leave the price unchanged. Larger ones are settled at agreed unit rates. Work that turns out harder than we expected does not change the price. Logic that is a piece of the migration in its own right, such as building one target record from several source records held in different files, is priced on its own.

The same holds in a tender or an integrator’s bid, where the price is due before the data can be seen.

What drives the price

  • Target fieldsThe largest driver. Each target field is discussed, defined, coded, tested, and verified.
  • Target modulesEach target module (load unit) is a running unit with its own mapping sessions, test cycles, and migration process.
  • Target tables / record typesEach table is a separate unit of migration logic producing records for it, including its relationships to other tables.
  • Source files and fieldsEach source file adds extraction and format handling, and each of its fields has to be analysed and understood.
  • Complex fieldsFields whose logic needs other source files, history, or non-trivial derivation. The questionnaire gives an estimated number, and discovery confirms it. Logic that is a piece of the migration in its own right, such as building one target record from several source records held in different files, is priced on its own.

Stated in the scope

  • Source extracts: you provide the agreed source files as flat extracts (fixed-width or CSV), with the record layout needed to read them, for discovery and for the migration, unless a direct read is agreed.
  • Premises: the migration runs in an environment you provide, so the source files stay on your premises. Analysis results (counts, distributions, and example keys, not the source files) may be used by the project team away from that environment, unless you require otherwise.
  • Source data quality: cleansing of source data is outside the fixed price or priced separately.
  • Decision turnaround: customer decisions within an agreed number of working days.
  • History depth: how far back source data is migrated.
  • Environments: the number of target environments to load, and who provides them.
  • Data volume: affects runtime and rehearsal duration, not the fixed price.
  • Production runs: rehearsals, delta runs, and the cutover are scoped separately when the migration itself is defined (see step 7).
  • Validation: the items compared and the depth of comparison are agreed during mapping; target extracts are provided by the customer at the agreed granularity and format (see step 8).
  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Discover

The project opens with a short discovery. It is the first part of the analysis in step 3, run on an agreed set of source files, so you extract only those files at the start and nothing is done twice. You provide those files as extracts, with the record layout needed to read them, and we profile them in your environment. Discovery confirms the counts in the questionnaire. The gaps themselves surface later, in mapping (step 4), and the full analysis of every source file follows in step 3.

If discovery finds the data far from the questionnaire, the agreement has an exit point there, and what is paid up to it is set in advance.

Phase 2 · Steps 3–7

Build It from the Data

Five steps that start from the actual source data: understand it, map it to the target with business decisions before technical ones, resolve the gaps, generate the migration code, then test, rehearse, and migrate.

  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Analyze & Understand

Before migration execution, understand what actually exists:

  • Source structures and metadata
  • Data profiling
  • Populations and distributions
  • Anomalies and exceptions
  • Relationships
  • Historical patterns
  • Business semantics

The analysis results are the evidence the rest of the project works from. Documentation says what a field was meant to hold. The analysis shows the values it actually holds and how often each occurs. Mapping, gap decisions, test data, and validation all refer back to it.

Principle 1: Data Is the Source of Truth
  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Define & Map

Mapping is more than matching fields. For each part of the target, the mapping sessions define:

What goes where

  • Source-to-target mapping
  • Semantic interpretation
  • Transformation requirements
  • Exception handling

What does not fit

  • Data gaps
  • Migration gaps
  • Target / functional gaps
  • Cleansing requirements

How it will be checked

  • Balancing requirements
  • Test and verification requirements
  • Acceptance criteria

Mapping is done in two passes, business first. The business pass maps the meaning of the source to the target model and brings the business decisions forward. The technical pass (fields, keys, formats, transformation rules) starts only once those decisions are settled, so it is quicker, issues surface earlier, and technical work is not redone.

Every mapping is checked against the data. When a source field is proposed for a target field, we open its analysis results and ask:

  • Do the values, and how often each occurs, fit the meaning of the target field?
  • Does the target accept the values that actually exist, not only the documented ones?
  • Are there odd values, or values that occur unexpectedly rarely? These become cleansing requirements now, not surprises in testing.

Mapping is also an early test of the target solution.

Mapping your real data to the target shows where the planned solution does not yet cover what the business does today. These are business gaps, not migration gaps. They typically surface within the first weeks of mapping, while they are still design decisions, not months later in user acceptance testing. The whole project benefits, not only the migration.

  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Resolve Gaps and Decisions

  1. Evidence
  2. Gap
  3. Resolution
  4. Approved outcome

The resolution is yours. Your business and target-system owners decide. We bring the evidence and record the outcome.

Not every gap ends in a transformation rule. A decision may add a target component or a target field, add target logic without changing any field, change a migration rule, or change the migration process itself without touching the rules. Whatever the outcome, it is recorded with the evidence that raised the gap, the decision, and who made it.

Migration discoveries should not remain disconnected spreadsheet issues or meeting notes. They feed governed migration knowledge.

Gap Governance
  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Implement

Business logic should not disappear into code.

  1. Approved mapping
  2. Predefined rules
  3. Generated program
  4. Execution

Typically the migration is carried by predefined rules captured in the mapping. A code snippet is the exception, used only where a predefined rule cannot express the case. We do not hand-write the migration.

The requirement and its implementation sit in the same place. Each mapping element, for example one source field mapped to one target field, holds both what the business asked for and the rule that implements it. Where a predefined rule cannot express the case, a short code snippet is added at that same element, and AutoCode places it at the right point in the generated program.

Because each element is small and self-contained, there is little room for the specification and the code to drift apart, and a reviewer sees the requirement and its logic side by side.

That is why changes are always made in the mapping, never in the generated code. Code can therefore be regenerated whenever required, and wherever the mapping has not changed, the regenerated code is exactly the same.

AutoCode in MCC & Tools
  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Test, Rehearse & Migrate

  1. Test
  2. Rehearse
  3. Full migration
  4. Deltas
  5. Cutover

Three activities, each followed by validation (step 8), each run on the generated code from step 6.

Test

Testing starts from the analysis. For every source field, the analysis records each distinct value and the first occurrences (keys) that carry it. Once mapping is done, we know exactly which source fields each unit uses, so the Sample Extractor builds a test source that contains only, and all, the records needed to cover every distinct value of those fields. The result is a small file with no redundant records, in which every value can be checked individually. Where needed, fields are combined so the sample covers every combination of values that exists in the source.

Rehearse

Rehearsals run the migration on far more data than the test sample: the covering sample plus random records, and when the plan calls for it, the full source. Findings go back into the mapping, never into the code.

Migrate

The migration itself starts with a full run, batched where volumes require it, followed by delta runs that carry new and changed source data forward, so the target reflects the source as it is, not as it was. The final delta is the cutover.

Production rehearsals, delta runs, and the cutover are scoped with you when the migration itself is defined. How many are needed, how deep and wide they go, and how long each run takes depend on your organisation and on how fast the target system loads data in bulk, which ranges from hours to months. None of this is known at commitment, so it is agreed on its own terms rather than guessed into the fixed price.

Sample Extractor in MCC & Tools

Phase 3 · Steps 8–9

Prove & Govern

Two steps that compare what arrived with where it came from, and keep every piece of evidence, every decision, and every version traceable from the first analysis to acceptance.

  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Validate & Prove

Compare what arrived with where it came from.

  1. Load / technical validation
  2. Source-target reconciliation
  3. Validation against approved transformation rules
  4. Discrepancy identification
  5. Investigation and resolution
  6. Acceptance evidence

Not everything is compared. Comparing every value costs more than it proves. You define the items that are critical to verify, and for each we agree whether counts per value are sufficient or a full record-level comparison is required. You provide the extracted target data at the agreed granularity and format; the comparison logic may aggregate values by key on both sides before comparing them, and every discrepancy is explained.

Migration is not complete merely because the load completed successfully.

Validator: encounter count 4,820 matched; ICU visits 320 on the source and 298 on the target, marked failed
Encounter count 4,820 matched. ICU visits are 320 on the source and 298 on the target, marked failed.
  1. Fix the Scope and the Price
    1. Count & Commit
    2. Discover
  2. Build It from the Data
    1. Analyze
    2. Map
    3. Resolve Gaps
    4. Implement
    5. Test, Rehearse & Migrate
  3. Prove & Govern
    1. Prove
    2. Govern

Govern Throughout

Governance is not a closing step; it runs under all the others. Every piece of evidence, every gap, every decision and who made it, every rule version, and every generated unit is recorded in MCC as the work proceeds, not reconstructed afterwards.

  • Evidence
  • Traceability
  • Actions
  • Decisions
  • Versions
  • Reproducibility
  • Validation

The result is a migration that can be explained and repeated: any value in the target can be traced back to the source data, the rule that produced it, and the decision behind the rule; any unit can be regenerated from its mapping; and validation evidence is kept with the version it proves.

Migration Principles

Explore MCC & Tools

When to Call Us

Talk to Ofakim when your migration involves one or more of these.

  • a major system replacement
  • complex or poorly documented legacy data
  • several source systems into one target
  • regulated, clinical, or financially sensitive information
  • delta migration and a constrained cutover window
  • source-to-target reconciliation as the acceptance criterion
  • business rules that live in the old programs or in how people work, and that have to be written down or they will be applied wrongly or lost
  • a merger, acquisition, or consolidation of data

If none of these apply, a conventional ETL project may be all you need, and we will say so.

Planning a Complex Data Migration?

Let's examine the source environment, target objectives, and migration challenge.