Migration Process
The migration from 2.2.1.1 to 3.0 is a two-phase migration. The first phase is done by the Codebeamer version 2.2.1.1 migration module in the background.
During this migration Codebeamer can be used normally. The second phase of the migration is done during the first deployment of Codebeamer 3.0. This step finalizes and completes the migration.
Use the following steps to migrate your tasks:
1. Upgrade to the provided Codebeamer version 2.2.1.1, that contains the migration module.
 |
This step is mandatory. You must first upgrade to version 2.2.1.1 to prepare the 3.0 schema.
|
2. Start the migration manually using the migration module UI as described in this topic. The migration process operates asynchronously. A background job is scheduled, and the migration is considered as started, but the job may not start immediately. Pre-migration checks are part of this job. If the job detects check failures such as the check for abandoned branches, the migration immediately stops in the background without any notification.
3. Check the migration process regularly based on the information in the migration module. The migration will run in the background, and Codebeamer can be used as usual during the migration process. Some operations are unavailable during the migration.
The migration process is as follows:
a. The migration process checks for stuck in-progress items that could be a result of an application crash, or other severe error. The state of such items is changed to MODIFIED, to process them again. For more information, see Item Status Values.
b. The migration collects all the data that needs to be migrated. If the migration process has been run earlier, only the changes since the last run are collected.
c. The collected data is migrated into the new tables or marked as faulty if the migration fails for some reason. More details are in subsequent sections in this topic.
d. The successfully migrated data is validated. The data in the new tables is compared with the data in the earlier tables. If a difference is detected, the data is marked as inconsistent. More details are in subsequent sections in this topic.
e. When all the data is migrated, the process sleeps for ten minutes.
f. The process automatically restarts at this step and begins the cycle again to check if new data needs to be migrated.
4. When most of the data is migrated, put Codebeamer in maintenance mode and restart Codebeamer. For more information, see Maintenance mode. This will stop all integration services that could modify the data. For instance, Git, JIRA, and so on.
The migration will stop automatically when all the following conditions are met:
◦ Codebeamer is in maintenance mode.
◦ All items are in one of the terminal states.
◦ There are no recent changes.
When these conditions are met, the migration module will enter the FINAL_VALIDATION_PHASE and do a final validation of the migrated data. After this phase, the migration will stop and go to the FINISHED state regardless of the validation outcome, whether validation errors were detected or not. For more information, see Migration Module.
|
|
The migration process will never stop automatically, because new data could emerge if the application is still used during the migration. The migration should not be stopped if any item is not in one of the terminal states as described in subsequent sections in this topic.
|
The steps must be executed in the order specified in the following tables.
Upgrade preparation steps:
|
Step Number
|
Description
|
|---|---|
|
1
|
Ensure you are not trying to deploy Codebeamer 3.0.0.0.
|
|
2
|
Delete 3.0.0.0 schema parts if necessary.
|
|
3
|
Back up the database.
|
|
4
|
Stop Codebeamer.
|
|
5
|
Ensure the earlier steps have been completed. For more information, refer to Preparing for
the Schema Upgrade.
|
|
6
|
Deploy Codebeamer 2.2.1.1.
|
Migration steps:
|
Step Number
|
Description
|
|---|---|
|
1
|
Start the migration. The migration can take from a few days to a week or more, to complete.
|
|
2
|
Monitor migration and wait until most of the data is migrated. Wait until most of the items are in one of the following terminal states: FINISHED, PARTIALLY_FINISHED, UNRECOVERABLE_ERROR, or VALIDATION_ERROR.
|
|
3
|
Export the migration results for analysis.
If there are issues, you can use analyze the exported data to decide if you should go ahead with the migration. You can reach out to PTC Technical Support if you encounter issues or need guidance.
|
If you decide to proceed with the migration:
|
Step Number
|
Description
|
|---|---|
|
1
|
Put Codebeamer in maintenance mode.
|
|
2
|
Restart Codebeamer.
Optionally, you can stop the migration before restarting Codebeamer and resume the migration after restart. This is not mandatory. You can restart Codebeamer when migration is in progress.
|
|
3
|
Check if there are changes that are not migrated and wait until all the data is migrated.
|
|
4
|
Wait until the migration goes to the FINISHED state.
|
|
5
|
Exclude all items with a status of UNRECOVERABLE_ERROR. For more information see Migration Module.
|
|
6
|
Stop Codebeamer.
|
|
7
|
Deploy Codebeamer 3.0.0.0 with the updated Working-Sets feature.
|
|
8
|
Run a quick smoke test.
|
If you are applying a new version:
|
Step Number
|
Description
|
|---|---|
|
1
|
Stop the migration.
|
|
2
|
Stop Codebeamer.
|
|
3
|
Deploye the new version of Codebeamer 2.2.1.1.
|
|
4
|
Again migrate and re-validate all items. Continue again from this point.
|
Additional Information About the Migration Process
• The migration is fully asynchronous, so no errors will be shown on the migration module page.
• There are two types of errors/warnings/messages as follows:
◦ Process-level: These pertain to the migration process itself. For example, if there are orphaned branches, this is a process-level error and the migration will immediately stop.
◦ Item: These pertain to the tracker item being migrated. For example, if a task cannot be migrated, the error is an item-level error and pertains to the item.
• The only errors that stop the migration are the errors described in Preparing for
the Schema Upgrade. In every other case the migration will keep trying to migrate the items. Temporary database or network outages, or other errors won't stop the migration process.
• With a single exception mentioned in the next bullet, the migration module only reads the existing tables and creates new tables that are not being used by Codebeamer 2.2.0.0. It also creates a set of temporary migration tables to keep track of the migration process and to store any errors or warnings messages.
• If tasks contain a Modified at field value different from the Submitted at field value, or a Modified by field value different from the Submitted by field value, the data is corrupted because the values must be the same for the first revision. The new database schema is not able to store different values for these fields in the first revision. To address this issue, the inconsistent entries for submittedBy and submittedAt are deleted before migration.
Throttling, Retries, and Revision Skip
• Throttling: The migration supports throttling. If enabled, the process keeps track of the average number free database connections, and the average CPU load. If either of them exceeds a predefined threshold, the migration is scaled down. For instance, the threshold could be as follows:
◦ Number of free database connections are less than 20% of the pool on average.
◦ CPU load is more than 80% on average.
On the other hand, throttling is scaled up until the threshold values are not exceeded. This ensures a balance between migration speed and application database load.
• Automatic retry: If an item migration fails, migration is automatically retried thrice. This is to ensure that there is no item lost because of temporary outages. However, if the item is not migrated despite reaching the maximum number of retries, it is possible to reset the migration or re-validate the failed tasks. For more information, see Migration Module. This is useful when the outage was longer than expected.
• Revision skip: If a revision cannot be migrated due to a programmatic error, the migration module tries to create an empty revision which is a dummy revision in which there are no field value changes, and proceeds to the next revision. Revision information is retained to avoid broken references in the data. If there are one or more skipped revisions, but not all revisions are skipped, the final item status is PARTIALLY_FINISHED. If all revisions are skipped, the final item status is UNRECOVERABLE_ERROR.
Process Status Values
The following are the possible process status values based on the order in which the migration module executes them:
|
Status
|
Description
|
State
|
|---|---|---|
|
NEW
|
The upgrade has not yet started.
|
|
|
MIGRATION_ITERATION_SCHEDULED
|
The upgrade has started, and a new iteration is already scheduled. (The migration is currently in the ten-minute sleeping period.)
|
|
|
PREPARE_MIGRATION_PHASE_IN_PROGRESS
|
The pre-upgrade step is currently in progress. The migration module is bulk loading the tables where possible and preparing the migration tables for later phases.
|
|
|
TASK_UPGRADE_PHASE_IN_PROGRESS
|
The tracker item upgrade is currently in progress.
|
|
|
VALIDATION_PHASE_IN_PROGRESS
|
The tracker item validation phase is currently in progress. Each revision of each tracker item is checked to ensure that the content loaded from the new tables is the same as the content loaded from the old (earlier) tables.
|
|
|
MERGE_INFO_MIGRATION_IN_PROGRESS
|
The merge information migration is currently in progress.
|
|
|
POST_MIGRATION_PHASE_IN_PROGRESS
|
A post processing check to see if if FINAL_VALIDATION_PHASE can be entered, or a new iteration must be started.
|
|
|
FINAL_VALIDATION_PHASE_IN_PROGRESS
|
The process only enters this phase when the conditions described in the migration process here are met. The is a bulk validation to check if the data does not violate the database constraints that will be enforced later.
|
|
|
ERROR
|
The migration failed with an error. This does not stop the migration but only the current iteration. Migration will be retried in the next iteration.
|
|
|
STOPPING
|
The migration is being stopped.
|
|
|
STOPPED
|
The migration is stopped manually or due to an unrecoverable error.
|
Terminal state
|
|
UNDER_RESET
|
The migration tables are under HARD RESET. The migration module truncates all target tables and resets the migration state as if it was never run previously.
|
|
|
FINISHED
|
The migration is now complete.
|
Terminal state
|
Item Status Values
The following are the possible item status values:
|
Status
|
Description
|
State
|
||
|---|---|---|---|---|
|
NEW
|
Migration of the item has not yet started.
|
|||
|
MIGRATION_IN_PROGRESS
|
Migration is currently in progress.
|
|||
|
ERROR
|
Migration failed due to an unexpected error. For example, a network outage. Will be retried if the number of retry attempts are not yet reached.
|
|||
|
UNRECOVERABLE_ERROR
|
The maximum number of retries has been reached and the migration failed with an unrecoverable error. For example, mandatory data is missing in the task. For such a status, migration will not be retried.
|
Terminal state
|
||
|
VALIDATION_ERROR
|
Migration failed with a validation error. The generated hashes of the source and target objects are different. Applies only to Task items.
|
Terminal state
|
||
|
READY_FOR_VALIDATION
|
The item is successfully migrated and ready for validation. Applies only to Task items.
|
|||
|
VALIDATION_IN_PROGRESS
|
Validation is currently in progress.
|
|||
|
FINISHED
|
Migration and validation have successfully completed.
|
Terminal state
|
||
|
PARTIALLY_FINISHED
|
The migration finished with one or more revision errors (skipped revisions).
|
Terminal state
|
||
|
MODIFIED
|
The item has been modified since the last successful migration. For example, a new revision is created, and the modifications need to be migrated. This status is similar to NEW but indicates that previous work or migration attempts are already done on this item.
|
|||
|
EXCLUDED
|
The task is manually excluded from migration.
|
The migration flowchart is as follows:
Task Migration Flowchart
Status values with a regular border indicate healthy states. Status values with a dashed border indicate intermediate states. Status values with a double-dashed border indicate an error or exclusion, as displayed in the flowchart.
Field Conversion Errors
The following value conversion or value transfer failures are not considered as an error but logged in the TEMP_FIELD_MIGRATION_ERROR table. Such failures can be fixed or ignored:
• FieldConversion: The old schema stores field values as a string, while the new schema stores them based on data type. This means that during the migration the values need to be converted to the appropriate type. For example the string “123” is migrated as a number, 123. As the old schema allowed the storing of invalid values for a particular type, the conversion may fail. For example, the string “two” in a number field may result in a failure. For field conversion errors, the value is carried over as a string as a fallback mechanism.
• BrokenReference: A reference field points to an item which is missing from the database. Such fields are not migrated.
• InvalidReference: The reference is not in a valid format and cannot be parsed.
• MissingCellIdForTableReference: The cell ID of the field value cannot be determined. The field is skipped as it cannot be migrated.
• Duplicate: If there are multiple references with the same properties except for the ordinal, only the first field is migrated, and all other fields are skipped.
• LegacyReferenceWarning: The reference is in a legacy format, and the migration module cannot ascertain if it is valid.
• AmbigousCreatorUsers: The Submitted by and Modified by field values are different for the first revision which is not a valid case. The Submitted by field is used during migration to fix the error.
• MissingLayout: The field is missing from the tracker's basic layout. As there is no layout where type information can be found, the field is migrated as a string.
• TableLayoutMismatch: Fields were found that don’t align with the table layout. For instance, cells outside existing columns.
• DuplicateChoice: The choice field contains multiple occurrences of the same choice.
• ReferenceTargetNotMigrated: The target of the reference is in an unrecoverable error state. So the reference is skipped from validation.
• ReferenceInChoiceOptionField: A reference value found in a choice field.
The following temporary tables are used by the migration module:
• temp_baseline_item_status
• temp_common_id_mapper
• temp_field_migration_error
• temp_merge_info
• temp_upgrade_item_status
• temp_upgrade_process_status
• temp_validation_result
• temp_ws_only_item_id_mapper
Migration Module
The migration module can be used to migrate your legacy Working-Sets to use them in Codebeamer 3.0.
Accessing the Migration Module
To access the migration module, in Codebeamer version 2.2.1.1, go to System Administration and click Migration module.
The migration module page is displayed. The following is a sample of a migration that is in progress:
Before beginning the migration, click Show branchable flag mappings. The following screen appears:
The Action column displays one of the following messages:
• Nothing can be done! “Shared” config is unalterable.
As the text suggests, no action can be taken.
• Remove the tracker's branches!
In this case, you must remove the branches of the tracker.
• Change the Shared configuration on the tracker config page!
In this case, you must change the tracker Shared attribute configuration on the tracker configuration page. For more information, see Customize a Tracker.
On the main Migration module page, the checkboxes with item status names appear if there is at least one item that has been migrated containing that status.
Starting and Stopping the Migration
Click Start migration to start the migration. If the migration is running, the button changes to Stop migration. To stop the migration, click it.
When migrating the database, keep the following in mind:
• Both start and stop functions are asynchronous.
• Start migration will schedule the migration job, but it may not start immediately.
• Stop migration will notify the migration module to stop, but the migration will not be immediately halted. Depending on the operation in progress when the migration is being stopped, it could take from a few seconds to several minutes. For instance, in the post migration phase, there are large-scale queries running for ten to twenty minutes. The queries are not aborted but between each query, there are checks to determine if the job should abort and exit. The status appears as STOPPING until the job exits.
The migration module will not process any new items but will continue processing the ongoing items. This might take some time based on the number of items currently being processed.
Migration Parameters
Before you start the migration, the following Migration parameters dialog box appears:
The migration parameters are as follows:
Thread pool size: The number of threads used for migration. Higher values improve performance, but also significantly increase CPU usage and database load. The value must be less than 70% of the maximum size of the connection pool. For example, if the maximum number of connections are 35, the thread pool count must not exceed 24. Increasing the thread pool count above the threshold will degrade the performance instead of improving it. To allow for higher thread pool values, the size of the connection pool size must be correspondingly increased. The default thread pool size is 20.
Throttling: If this checkbox is selected, the thread pool size is dynamically adjusted. In this case the set size acts as the maximum value. If the migration module detects that the average CPU usage is very high, or the average number of idle connections is too low, it will decrease the pool size to lower the load. Conversely, if the average CPU load is too low, or the average number of idle connections is very high, throttling will increase the pool size up to the specified maximum value, to increase the load. This checkbox is useful to automatically scale down the migration when Codebeamer is used during the day, and to scale up during the nights.
Detailed DB log: If this checkbox is selected, a very large amount of log information will be added to the migration module temp tables. Use this checkbox only for development purposes. Do not use it on a large dataset, because it will significantly decrease migration performance.
Hard Reset Migration
Click Hard reset migration to reset the migration job. It will be considered as it was never run before. Clicking this button will truncate all migration related tables except for temp_upgrade_process_status. The migration can be restarted from the beginning. Click Hard reset migration only when the migration stops.
Migration Process Details
Refer to the summary and the table on this page to know the migration status. For more information, see Process Status Values and Item Status Values.
The Messages window shows migration-related information.
Exporting the Report
To export the report, click Export report. This button is visible only when the migration is not running.
You can export the report as a ZIP file.
If you encounter errors or special cases that are not handled by the migration module, reach out to PTC Technical Support for help.
The exported ZIP file contains the following CSV and TXT files:
• process_messages.txt
This file contains all the process-level messages and errors. It does not contain item-level messages. Post validation, database constraint warnings which cannot be tied to a task are also reported in this file.
• status_summary.csv
The table as visible on the screen, is exported as a CSV file.
• field_migration_errors.csv
This file contains all the field-related errors encountered during the migration. Checkboxes have no effect on the content of this file. For details on these errors, see Field Conversion
Errors.
A sample is as follows:
globalUniqueItemId,revision,fieldId,targetReference,type,message
1294,1,13001,,MissingLayout,"The field is missing from the revision's layout, migrating it as string"
1294,1,13001,,MissingLayout,"The field is missing from the revision's layout, migrating it as string"
• tasks.csv
This file contains the tasks with the selected status values. Status values are included if there is at least one item with that status.
| | If you select FINISHED and there are many items with that status, the file size of the CSV file can be large. The same applies if you select VALIDATION_ERROR. |
A sample is as follows:
itemId,status,latestKnownRevisionId,highestProcessedRevisionId,remainingRetryAttempts,messages
23927,EXCLUDED,2,0,0,"
2024-11-11 09:14:58: Task item successfully collected
Error during trying to upgrade task: org.springframework.dao.DataIntegrityViolationException:
### Error updating database. Cause: java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into (""CBTEST"".""BRANCH_ITEM_HISTORY_REVISION"".""ITEM_ID"")"
23927,EXCLUDED,2,0,0,"
2024-11-11 09:14:58: Task item successfully collected
Error during trying to upgrade task: org.springframework.dao.DataIntegrityViolationException:
### Error updating database. Cause: java.sql.SQLIntegrityConstraintViolationException: ORA-01400: cannot insert NULL into (""CBTEST"".""BRANCH_ITEM_HISTORY_REVISION"".""ITEM_ID"")"
• validation_errors.csv
This file contains all the tasks with validation errors.
A sample is as follows:
globalUniqueItemId,revisionId,status,oldHash,oldObject,newObject
1270,1,FAILED,39bdf82f543569b1ba813a36f34589debb85fafcb679b707786d48a76f6f2775,"[{""assocId"":null,""biDirect"":false,""biDirectIncomingSuspected"":false,...[TRUNCATED]...}]","[{""assocId"":null,""biDirect"":false,""biDirectIncomingSuspected"":false,...[TRUNCATED]...}]"
1270,1,FAILED,39bdf82f543569b1ba813a36f34589debb85fafcb679b707786d48a76f6f2775,"[{""assocId"":null,""biDirect"":false,""biDirectIncomingSuspected"":false,...[TRUNCATED]...}]","[{""assocId"":null,""biDirect"":false,""biDirectIncomingSuspected"":false,...[TRUNCATED]...}]"
Get New Changes Since Last Iteration
While Codebeamer can be used in parallel during the migration, you must create a maintenance window to ensure data integrity. When Maintenance mode is turned on, and the users are no longer allowed to use Codebeamer, there is a small window when the very last changes may not have been migrated.
Click Get new changes since last iteration, to verify that all the data is processed, or if there still are pending tasks or task revisions that need to be migrated. If some data is pending, you need to wait for a final migration iteration to migrate the remaining data. When all numbers appear as 0, the upgrade process can advance to the next step.
Again Migrate Failed Tasks
The Re-migrate failed tasks button is enabled if there is at least one task item with a status of UNRECOVERABLE_ERROR or PARTIALLY_FINISHED. When you click this button, all migrated data is removed and the item status is changed to MODIFIED, so it can be migrated again.
Use this button only if there was a temporary error such as a connection pool exhaust that lasted a few minutes and caused the migration of some items to fail.
Do not use this button in other cases because it will result in the same output. If you continue to see same errors, contact PTC Technical Support.
Re-Validate Tasks
The Re-validate failed tasks button is enabled if there is at least one task item with a status of VALIDATION_ERROR. When you click this button, the item status is changed to READY_FOR_VALIDATION.
Use this button only if there was a temporary error such as a connection pool exhaust that lasted a few minutes and caused the migration of some items to fail.
Do not use this button in other cases because it will result in the same output. If you continue to see same errors, contact PTC Technical Support.
| | Do not use the Re-validate failed tasks button if there are items with a status of PARTIALLY_FINISHED, because skipped revisions are not validated. Attempting to validate such items will always result in a status of VALIDATION_ERROR. |
Exclude Unrecoverable Errors
The Exclude unrecoverable errors button is enabled if there is at least one task with a status of UNRECOVERABLE_ERROR.
Despite all attempts to migrate the data, there might be items that cannot be migrated because of data corruption or other issues. Un-migrated or half-migrated items would prevent applying database restrictions such as foreign keys or not-null constraints.
To ensure every constraint can be applied later you may decide to skip such items from the migration. In such a case, the problematic data needs to be deleted from the old and the new database tables.
 | Exercise caution when clicking the Exclude unrecoverable errors button. Excluded items are fully deleted from all tables, except the temporary migration tables to keep a record of such an action. This action cannot be undone, even with a hard reset. |
Changes Post Migration
This section discusses the changes that occur after the migration is complete.
Handling of Shared Trackers and Working-Sets
On the migration module page, the Working-Set and tracker combinations section shows the shared trackers that are no longer shared after the upgrade, because they were previously branched before the schema upgrade. If a tracker is branched in any Working-Set (including the Default Working-Set), it is set as branched in the configuration and removed from all the Working-Sets where it was shared. This is based on changes introduced in Codebeamer version 3.0.
Migration of Merge Information
On upgrade, all cleared item badge information is lost.
Items are compared manually between the current and parent Working-Set. If they are different, a merge badge is created for the item on the current Working-Set. This scenario can be a bit challenging if the user had previously merged an item to another Working-Set using the Mark as merged checkbox (without actually merging the content). In this case, the item badge is cleared although the content in the current Working-Set is different than the content in the other Working-Set. When such an item is migrated, the migration module will find a difference between the item in the current Working-Set and the other Working-Set and create a merge badge for this item, disregarding the Mark as merged checkbox that was selected previously. Newly added badges have to be manually removed by the user by again attempting the merge and selecting the Mark as merged checkbox post data migration. This can be challenging if there are a large number of items with newly added merge badges after data migration. There is no way to provide a report to identify such items with merge badges.