Migration Planning

This section provides basic guidelines on how to prepare and plan for your migrations, and tasks that will need to be completed after migration to restore your data and configurations. These are basic guidelines that we have devised based on what we know about VA’s GitHub usage and our analysis of the resources and features used by VA repositories. VA teams should use these as a starting point but teams will be responsible for developing their own migration plans by understanding how the migration process and limitations will impact your use of GitHub, how much downtime may be necessary to restore missing DevOps workflows and functionality, and how to restore whatever integrations may be present with internal or third-party applications. The GitHub team will be available to provide support at every step, so please reach out to us if you need help planning your migrations.

Pre-Migration

General Repository Metadata

The VA is using the migration as an opportunity to capture required information about all repositories. Ensure you have the following information about your repositories before you begin migrations:

1. eMASS and VASI IDs of the VA projects that the repositories are related to.
2. Security maintainers
3. Product Line/organizational structure: Portfolio, Product Line, and System or Application Name.

All of the above information except System or Application Name is required in the repository migration form. However we understand that some repositories are not related to production projects and may not have eMASS or VASI IDs, or perhaps those projects do not yet have those IDs. The migration form supports these scenarios by allowing users to enter pending or N/A in the eMASS and VASI ID fields. The migration instructions in the GHEC-US handbook will have more details.

Planning Downtime

Migrations will incur necessary downtime for the repositories being migrated. When a repository is being migrated it will also be locked so that no one can submit any changes (to prevent data being lost). Migrating a repository takes a bit of time, though we believe that most VA repositories will migrate in under 30 minutes. Teams will also need to rebuild missing configuration and restore integrations and other processes after migrations are complete.

Based on our analysis and internal migration testing, we are confident that most VA repositories will migrate without error. As we help VA teams overcome migration errors that they encounter, we will publish updated guidance and solutions to help other VA teams that may run into the same problems. VA teams will be able to file support requests to request assistance with any migration issues.

Network Details

VA GHEC-US will have its own unique domain name: va.ghe.com. Any other subdomains that you would typically use in github.com should now be based on va.ghe.com, for example instead of api.github.com use api.va.ghe.com. See our GHE Network Details for specific IP ranges and other network information for the US infrastructure.

Service Accounts and PATs

Service Accounts are supported in GHEC-US, however these must be va.gov Entra ID accounts since Entra ID is the authentication mechanism for GHEC-US. The process for creating GHEC-US service accounts is the same as what we have currently documented for the GHEC-EMU environment. While service accounts are supported we strongly encourage teams to use migrations as an opportunity to switch from using service accounts and PATs to GitHub Apps, where possible.

There are numerous benefits to using GitHub App tokens instead of service account or user PATs:

1. They provide enhanced security because the tokens only last 1 hour. Furthermore, GHEC-US enforces a lifetime limit of 90 days on user PATs, so they must be rotated at least every 90 days. GitHub App tokens are acquired on the fly so they never need rotating.
2. GitHub Apps also use fine-grained permissions which allow you to scope an App’s permissions to the bare minimum that is necessary, further enhancing security.
3. They provide more scalability because each GitHub App has its own rate limits, compared to shared rate limits for user PATs since all PATs from a single user share the rate limit. The rate limits for a user and a GitHub App are generally similar, however it’s easier to scale out by creating multiple GitHub Apps which would each receive their own rate limits. Workflows that need extremely high rate limits can also achieve greater scalability by using different GitHub Apps (with more limited permission sets) for various parts of the workflow, or even cycling through multiple instances of a similar GitHub App in a round robin fashion to spread the rate limit load among multiple GitHub Apps.
4. GitHub Apps are not tied to a user, so they remain functional even if users leave the organization or if problems were to arise with the va.gov service account configuration.
5. GitHub Apps will also be much easier for VA GHEC-US users to create and maintain, because you will not need to go through the process to create a va.gov account and maintain it.

Basic steps to switch from using PATs to GitHub App Tokens:

1. Analyse your current usage: Identify the API calls and permissions your PATs are using, and document the repositories and resources accessed by the PATs.
2. Register a GitHub App: Create the app in your user namespace. Configure it with the necessary permissions. Note that you will need to add private keys to an App in order to generate installation authentication tokens.
3. Transfer ownership to the GHEC-US software organization: The app must be owned by the organization before we will approve installation requests. This is necessary so that we can inspect the configuration and ensure that the app is not sending webhooks to unauthorized destinations.
4. Update your workflows to authenticate as the App installation instead of using PATs. That documentation provides instructions on using the GitHub APIs to generate App installation tokens, however there are also several publicly available utilities that you can use to simplify this process. If you are using tokens in an Actions workflow you can use the create-github-app-token custom action to generate a token. There is also the gh-token CLI extension that can be used to easily create an installation token.

Be aware that there may still be some scenarios and GitHub APIs that will require the use of a user PAT instead of an App token, for example APIs that require full user-level access rather than fine-grained permissions. It is beyond the scope of this documentation to outline every scenario for which GitHub Apps may not work, however our knowledge of the VA’s GitHub infrastructure leads us to believe that GitHub Apps can replace the vast majority of service account use. Create a support request if you would like consultation on your specific scenarios.

CodeQL Security Scanning

The default CodeQL scanning configuration will be the same as the default configuration in GHEC. Our recommendation is for teams to use the default configuration. If you have needs that aren’t met by the default configuration then you can create your own advanced CodeQL configuration in your repository (or a reusable workflow that can be used across multiple repositories). Migrated repositories will be automatically opted into the default scanning configuration.

Teams should note that two previous code scanning workflows will be deprecated and no longer be supported by the GitHub team. These workflows are contained in the GHEC codeql-tools repository.

Reusable Actions Workflows

The codeql-tools repository contains the reusable action codeql-analysis that implemented CodeQL scanning with requirements from OIS. These OIS-driven requirements are deprecated and no longer necessary, and by extension, the use of the codeql-analysis workflow is no longer necessary or supported. Teams using a code scanning workflow based on this action should transition to the default CodeQL configuration.

Jenkins Shared Libraries

The codeql-tools repository also contains a Jenkins Shared Library for running CodeQL in Jenkins pipelines. This Jenkins Shared Library primarily served two purposes:

1. Perform compile-time build tracing: At the time, CodeQL required being able to hook into your build process to achieve full fidelity.
2. Facilitate CodeQL access to private dependencies: CodeQL also required access to the application’s dependencies, both for scanning and to build the application.

CodeQL now supports a buildless mode for C/C++, C#, and Java, where it doesn’t need access to your build process or all dependencies to provide quality results. Our recommendation is that teams should use the default configuration and assess whether it meets your security scanning needs.

If teams wish to continue using CodeQL from Jenkins workflows they will still have that option, but the codeql-tools repository will be removed in the near future so teams using the Jenkins Shared Library will need to implement their own Shared Libraries leveraging the codeql CLI directly. To use the Jenkins shared libraries in GHEC-US two specific changes must be made to the scripts:

1. A github.com authorization token will need to be used for the API requests that check for the latest version of the codeql-action custom action. GHEC-US tokens will not work when making API requests to api.github.com.
2. The code that uploads the database must be removed. GHEC-US does not support the endpoint that allows uploading the database.

The above links are to the linux versions of these Jenkins groovy scripts, but the windows file has similar code so the instructions apply there as well.

Git submodules

A search of VA GHEC shows there is some use of Git submodules. There is no automated process for updating submodules after migration, but users can follow these steps to update their submodule references.

1. Ideally you would migrate the repositories that are used as submodules first, then migrate the “parent” repositories that use those submodules. It’s OK if there are some circular references here that prevent this order from being accomplished cleanly. The GitHub GEI migration process does not recognize submodules, so there is no harm if the parent is migrated first. It only matters when you are attempting to clone the repositories locally and fix the submodule references to point to the newly migrated repositories. At this time, the parent and all submodule repositories must already be migrated to GHEC-US.
2. Clone the parent repositories and update the submodule references. Update the submodule references by opening the .gitmodules files in the root of your repositories, and updating the URLs to the submodule repositories with the new URLs to your migrated repositories. E.g. change https://github.com/department-of-veterans-affairs/my-submodule-repo.git to https://va.ghe.com/software/my-submodule-repo.git.
3. Sync the updated URLs by running git submodule sync.
4. Update the submodule references with: git submodule update --init --recursive
5. Commit the changes to the .gitmodules file
6. Verify the status with git submodule status

For most customers we would also recommend rewriting the history of the repository to update the submodule references in all past commits, using the Git filter-branch command:

git filter-branch --tree-filter "find . -name '.gitmodules' -exec sed -i -e 's/<OLD_SUBMODULE_URL>/<NEW_SUBMODULE_URL>/g' {} \;"

However we understand the VA may have policies against rewriting the history of repositories in this fashion. If teams are able to rewrite the repository history and wish to do so, make note of the following:

1. In both the old and new submodule URLs, forward slashes must be escaped with backslashes e.g. \/.
2. On macOS machines there needs to be empty single quotes after the sed -i flag e.g. sed -i ''.

Planning for Reconfiguration and Limitations

After repositories have been successfully migrated, VA teams will need to restore missing configuration. These activities will also impact downtime, so VA teams should analyze their repositories’ use of these resources and develop a plan for how to restore them with minimal disruption. We believe the following resources will be of most concern for VA teams:

1. Users and teams: The user performing the migration will be a repository admin on the migrated repository. That repository admin will then need to reconfigure repository access for other VA users and teams. Teams will probably want to ensure that all necessary users are onboarded to GHEC-US and establish GitHub teams and membership before performing migrations, so that users and teams can be granted access as soon as the repository is migrated and open for business.
2. GitHub Apps: Since Marketplace is not available for GHEC-US, all apps will need to be created as custom apps. The process will be the same as with the current GHEC organization - users can create custom apps in their account and request to transfer ownership to the organization. Once ownership has been transferred to the organization, VA teams can submit support requests in the github-admin/support repository to install the apps in their desired repositories. The primary pre-migration activities for VA teams will be:
   1. Inventory the apps your team owns or manages
   2. For third-party apps, research whether the app is supported in GitHub’s Enterprise Cloud with data residency product and if so, how to recreate it as a custom app. You may need to contact the app vendor to determine this. Also note that VA must be more strict with approving third party apps in GHEC-US since it is (or will be) a fully FedRAMP Moderate system. Third party apps represent interconnects to external systems and therefore the standard VA compliance process for approving and documenting these interconnects must be followed. We are still working with VA to establish the process users can follow to request approval for third party apps and will provide updated guidance when that process has been established. At bare minimum it will mean that the external system must also have a VA ATO and an MOU must be filed to document the interconnect between GitHub and the external system.
   3. Create the app, transfer it to the GHEC-US software organization, and create a support request asking for the app to be installed into your desired repositories. If there is code behind your custom app, it will need to be updated to support the new API endpoint for VA GHEC-US: https://api.va.ghe.com.
3. Packages: Packages are migrated separately from repositories and Maven packages will not be supported in GHEC-US.
   1. With the VA Nexus being the potential replacement for Maven packages, VA teams need to consider how they can transition to using Nexus. Teams will most likely want to reconfigure maven poms/profiles and Actions workflows to use Nexus ahead of migrations, so they can transition on their own schedule.
   2. For other types of GitHub Packages, teams should consider whether they need to migrate current package versions to the new environment, or simply reconfigure publishing workflows to publish new packages to GHEC-US. The hostname for the package registry will be different in GHEC-US, it will be https://<registry-type>.va.ghe.com where <registry-type> is one of: containers, npm, rubygems, or nuget.
   3. The GitHub team has developed a separate workflow and issue template for migrating packages. This workflow can be used to migrate packages after the repository that the packages are associated with has been migrated. There are several caveats to the package migration workflow, see the migration process section for details on using the package migration workflow.
4. Other dependency management tools: There may be other dependency management tools that need to be configured to use the new va.ghe.com hostname if they are pulling private VA dependencies directly from GitHub, for example Golang. VA teams consuming private VA dependencies from other teams/repositories will need to coordinate their migrations with those other teams to ensure that dependencies are migrated before the consumers are updated for the new environment.
5. MacOS Actions runners: macOS actions runners are not yet available in GHEC-US, however they are scheduled to be available at a future date. We are working closely with the va-mobile-app team which is the main user of macOS runners in GHEC. There are some other repositories with light usage of macOS runners but our analysis of those workflows leads us to believe they could all be either disabled or transitioned to Linux runners. If you are currently using macOS runners and would like to consult on other options then create a support request and we will be in touch.
6. Self-hosted Actions runners: self-hosted runners are not migrated and will need to be recreated after migration.
   1. For repository runners the process should be relatively straightforward. Once a repository has been exported in the first stage of migration it will be locked therefore no Actions workflows will be invoked from the source repository. Once the migration is complete teams can go through the documented process of adding the existing self-hosted runner system to the new repository in GHEC-US. If teams wish to experiment with adding self-hosted runners to a migrated repository in GHEC-US that used the Dry Run migration option (meaning the source repository remains unlocked and could run Actions workflows), then teams can either provision new systems for the self-hosted runners in GHEC-US or configure a second instance of the runner application on the same system. This is possible as long as the runner application is installed into an entirely separate directory and none of the configuration is shared between the two instances of the runner application.
   2. For organization runners the GitHub team has duplicated the same runner groups from GHEC in GHEC-US. We are available to support requests to add and configure self-hosted runners for the VA. There are only a handful of these registered in the current GHEC enterprise and we expect the teams that own these will reach out to us about migrating them to GHEC-US.
7. Secrets: Secrets of all types will need to be restored after migrations, including Dependabot, Actions, and webhooks. Teams will need to ensure that these secret values are saved in a safe place that can be accessed after migrations, to restore the values in the migrated repositories.
8. Actions variables and environments: These are not migrated by GEI but the GitHub team plans to add additional support for these resources. That additional support may not be available in the first iteration of the migration tooling so VA teams can choose to proceed with migrations and repopulate these resources themselves, or wait for the additional support in later versions of the migration tooling.
9. Projects: Projects are not migrated by GEI but the GitHub team has implemented custom support for migrating some facets of Projects. There is a separate Projects migration workflow that can be invoked after all of the related repositories have been migrated. Even with this support there are still important aspects of your projects that will need to be rebuilt manually, including views, the order of items displayed in your views, workflows, and iteration custom fields. You should still have access to your source project to view your current configuration when rebuilding it in GHEC-US, so the primary pre-migration task for VA teams with respect to Projects is to understand these limitations and plan time for running the Projects migration workflow after migrating your repositories, and rebuilding the missing aspects of your Projects after they have been partially migrated. See the migration process section for details on the Projects migration workflow.
10. Issues: Issue types and relationships are not supported by GEI, but the GitHub team is planning to add custom support to migrate both. We will update with more guidance once our issue relationship support is available.
11. Repository rulesets: Rulesets are supported through custom migration support, but teams will need to make note of the bypass settings for each of their custom rulesets. The actors identified in the bypass settings will not exist in the destination organization, therefore they cannot be migrated.
12. Actions workflows and any other code or integrations that make GitHub API requests will need to be updated to use the new API endpoint for VA GHEC-US: https://api.va.ghe.com.
13. Code security scanning: Teams will need to migrate to the default code security scanning configuration as outlined above, or account for the absence of the codeql-tools repository.

Post-Migration

The post-migration requirements for VA teams are mostly aligned with the pre-migration requirements, to rebuild resources and configuration that did not completely migrate, and to ensure the connectivity and function of any GitHub integrations such as GitHub Apps and self-hosted runners.

1. Users, teams, and repository permissions: VA teams will need to re-establish GitHub access and repository permissions for their users and GitHub teams. GitHub access is facilitated through the GitHub app in Entra ID, which new users can onboard to through a self-serve process. GitHub teams can be tied to an Entra ID group so that team membership is controlled by membership in the Entra ID group. GitHub teams can also be managed as they were in GHEC - entirely in GitHub without Entra ID integration.
2. GitHub resources that reference users and teams will also need to be updated, for example CODEOWNERS files and ruleset bypass rules.
3. GitHub Apps: VA teams will need to create and transfer custom GitHub apps if they did not do so prior to migrating their repositories. After migration VA teams will need to create support requests to add the newly migrated repositories to the GitHub apps. As mentioned in the pre-migration section, GitHub App code that makes API requests will need to be updated to use the new VA GHEC-US API endpoint https://api.va.ghe.com.
4. Packages:
   1. VA teams will need to ensure that the transition to using VA Nexus for maven packages is working as expected.
   2. For other packages, VA teams can use our Packages migration workflow to migrate some or all of a repository’s package versions to GHEC-US, after the repository has been migrated.
   3. Configure local or repository package managers that pull private VA packages directly from github.com to now use va.ghe.com, when those dependencies have also been migrated to GHEC-US.
5. Self-hosted Actions runners: Teams should ensure that their new self-hosted Actions runners are working as expected (or stand up and register them, if that was not done pre-migration).
6. Secrets: Teams will need to recreate all secrets in the new repositories.
7. Actions variables and environments: Similar to secrets, if you migrated your repositories after the GitHub team’s additional support for these is in place, then the Actions variables and environments should have already been migrated for you and teams should examine these to ensure that they were migrated correctly. Otherwise, teams will need to recreate these resources.
8. Projects: Even with the GitHub team’s custom support for migrating Projects, teams will still need to restore the following aspects of their projects: views, item display order in views, workflows, and custom iteration fields.
9. Issue relationships: Our custom support for issue relationships should migrate some issue relationships for you, but there may be edge cases where all relationships are not covered and those would need to be manually re-established after migration.
10. Repository rulesets: Rulesets will be migrated by our custom migration support, but teams will need to re-apply the bypass settings for each ruleset to reference the new users and teams in GHEC-US.

Updating Local Configuration

VA GitHub users will need to configure their local environments to use https://va.ghe.com for standard GitHub access, and https://api.va.ghe.com for API requests. We cannot predict all configurations that may need to change on VA users’ systems, but we will provide some examples for reference.

1. Note the new authentication page for GHEC-US: https://va.ghe.com/enterprises/va/sso.
2. Visual Studio Code: Update settings to provide the new hostname at Settings > Extensions > GHE.com & GitHub Enterprise Server Authentication, https://va.ghe.com. You may need to log out of the GitHub extensions and log back in using the Enterprise settings.
3. Reauthenticate the GitHub CLI:
   1. First ensure you have an active SSO session to va.ghe.com.
   2. Log out of the GitHub CLI using gh auth logout
   3. Authenticate against va.ghe.com using gh auth login. When prompted, choose Other for where you use GitHub, and enter the hostname va.ghe.com. Follow the remaining prompts.
   4. Optionally set the GH_HOST environment variable to va.ghe.com if you want the GitHub CLI to default to GHEC-US for all commands.
   5. Verify the authentication status with gh auth status. It should display the currently authenticated account and host.
   6. If you need to use both the VA GitHub.com organization and GHEC-US, you can authenticate to both and switch between them using gh auth login, then use the --hostname flag to specify the target host for commands.
4. Update your local git repository remotes to use the migrated repositories on va.ghe.com instead of github.com. For example: git remote set-url {remote name} https://va.ghe.com/software/{repository name}.git. Optionally, you can just create new local clones of all of your repositories.
5. You will also need to reauthenticate git but how to do this will depend on your system configuration and which Git credential manager you are using. Typically if you just attempt a Git operation after updating your remotes, then Git will prompt you to authenticate against the remote.

These are the general guidelines that we can provide in advance of VA team migrations, but we expect this documentation to grow with more specific guidance as we help VA teams migrate and discover solutions to whatever problems teams encounter. We will work closely with VA teams as you migrate your repositories and integrations, especially the early adopters, and we will continuously update this documentation to provide those solutions to all VA teams.