Skip to main content

Release issues

The automated release process works well a vast majority of the time but occasionally there are issues that may require some manual intervention. These issues can be caused by either upstream issues with the app stores or due to issues with our build. This page is meant to document some of the issues we may run into with our releases and how to resolve them.

Re-releasing a failed build or resubmitting​

Occasionally, a build may fail for one reason or another, or we may determine that we want to resubmit a build. In order to do this, we need to re-trigger the Release Build workflow. This workflow is triggered by tagging the release branch with a vX.X.X tag. Since this tag exists already, follow the steps below to re-tag a branch and retrigger the build.

The releases for both iOS and Android are tied together in the same workflow. Following these steps will re-run the release build for both platforms, even if only one platform failed. This means a previously successful build for the other platform will be rebuilt, which wastes CI time and introduces risk of a new failure on the previously passing platform. If only one platform failed, consider whether the issue can be resolved without a full rebuild (e.g., see App Store screenshot failure for iOS screenshot issues). :::
  1. Determine the version number of the build you want to rebuild (vX.X.X)
  2. If possible, remove any pending versions for review
    1. App Store Connect
    2. Google Play Consolei. Note: Archiving pending Android builds may cause issues with the build number. The Android Fastlane review lane compares version codes across all Play Store tracks (including archived builds), so archiving could cause the next build to get a conflicting version code. We recommend waiting until the issue is resolved before archiving any unwanted builds.
  3. Make any changes necessary to the corresponding release/vX.X.X branch or have an FE engineer do so
  4. Since the vX.X.X tag exists already from the previous build attempt, we need to delete the tag, both locally and remotely
    1. Delete the local tag: git tag --delete vX.X.X
    2. Delete the remote tag: git push --delete origin vX.X.X
    3. Checkout the release/vX.X.X branch
    4. Run git pull to ensure you have the latest changes
    5. Tag the branch with the version we're trying to release: git tag -a vX.X.X -m vX.X.X
    6. Push the tag git push origin vX.X.X
  5. Confirm that the Release Build has been retriggered
  6. Upon successful build, check the corresponding App Store console to confirm that the build is present and has been submitted for review

App Store screenshot failure​

Occasionally the uploading of screenshots to the iOS App Store will fail due to timeout. A recent fastlane version update included some optimizations to fix this but it's possible it can occur again. Screenshots are uploaded after a successful build has already been uploaded to App Store Connect. Sometimes we can fix the screenshots manually in App Store connect. To do so:

  1. Go to the pending release in App Store Connect
  2. On the Distribution tab, you should see the "iOS Previews and Screenshots" section
  3. For each screen size and device:
    1. Remove any failed screenshots. They usually appear as blank with no image preview
    2. Check if there is a button in the top right that allows you to re-submit the app for review.
      1. If not, you will likely need to re-release the entire build by following the steps for "Re-releasing a failed build" instead
      2. If so, continue with the next steps
    3. Comparing it to the screenshots in the release/vX.X.X branch, check the order and upload any missing screenshots
  4. Re-submit the app for review

Need to add code into a release after it has been created (but not pushed to the stores)​

Sometimes, after a release branch has been cut, we need to get an important update included before approval. When this happens, we need to recreate the release candidate build. If QA hasn't happened yet, there's no extra effort, but if QA has completed testing on the release candidate, it will need to be retested. Updating the release needs to be approved by the QA team.

Because the release is tied into a commit and not the current state of the branch, follow the steps below to "re-cut" the release. The RC tag format is RC-vX.X.X-MMDDYY-HHMM (e.g., RC-v2.71.0-032623-0636):

  1. Add updated code into the release branch release/vX.X.X and push to GitHub
  2. Find the existing RC tag name. You can list tags with: git tag -l 'RC-v*'
  3. Delete the local release candidate tag: git tag --delete RC-vX.X.X-MMDDYY-HHMM
  4. Delete the remote tag: git push --delete origin RC-vX.X.X-MMDDYY-HHMM
  5. Checkout the release/vX.X.X branch
  6. Run git pull to ensure you have the latest changes
  7. Create a new RC tag with the current date/time: git tag -a RC-vX.X.X-MMDDYY-HHMM -m "Release Candidate for vX.X.X. tagged on MM/DD/YY at HH:MM"
  8. Push the tag: git push origin RC-vX.X.X-MMDDYY-HHMM
Like the release build, the RC build workflow builds both iOS and Android. There is currently no way to rebuild an RC for a single platform. :::

Release Ticket Failure​

Outlined below are the steps to follow when the automation to create a release ticket and Slack coordination thread fails due to issues like incorrect GitHub username assignment.

  1. Update the release_ticket template with the correct GitHub username and push to GitHub
  2. Determine the version number of the build (vX.X.X) and find the RC tag name (format: RC-vX.X.X-MMDDYY-HHMM)
  3. Clean up artifacts from the failed attempt:
    1. Delete the local release candidate tag: git tag --delete RC-vX.X.X-MMDDYY-HHMM
    2. Delete the remote tag: git push --delete origin RC-vX.X.X-MMDDYY-HHMM
    3. Delete the local release branch: git branch -d release/vX.X.X
    4. Delete the remote release branch: git push origin --delete release/vX.X.X
    5. Close any partially-created release ticket in GitHub (if one was created before the failure)
    6. Close any TestRail milestone that was created for this version (if applicable)
  4. Manually run the new_release_branch workflow using workflow_dispatch targeting the develop branch.
  5. Confirm that:
    1. New RC tag has been created
    2. New release branch exists
    3. Release ticket has been created in GitHub
    4. Slack coordination thread has been posted to the va-mobile-app channel

Hotfix / Off-Cadence Release​

There are two types of out-of-band releases depending on what code you need to ship:

New release from develop (includes all merged work since last release)​

Use this when you need to ship everything currently in develop as an off-cadence release:

  1. Manually run the new_release_branch workflow using workflow_dispatch targeting the develop branch.
  2. Confirm that:
    1. New RC tag has been created
    2. New release branch exists
    3. Release ticket has been created in GitHub
    4. Slack coordination thread has been posted to the va-mobile-app channel

True hotfix (only the fix, based on current production code)​

Use this when you need to ship a targeted fix without pulling in all of develop. This approach branches from the last release tag so only the hotfix code is included:

  1. Determine the current production version tag (e.g., v2.71.0)

  2. Create a hotfix branch from that tag:

    git checkout v2.71.0
    git checkout -b release/v2.71.1
    git push -u origin release/v2.71.1

  3. Cherry-pick or commit the fix onto the hotfix branch and push to GitHub

  4. Tag the branch to trigger a release candidate build:

    git tag -a RC-v2.71.1-MMDDYY-HHMM -m "Release Candidate for v2.71.1"
    git push origin RC-v2.71.1-MMDDYY-HHMM

  5. After QA validation, follow the standard release approval process. The release branch creation will trigger the release ticket automation.

  6. After the hotfix is released, ensure the fix is also merged back into develop to prevent regression.

The true hotfix approach uses a patch version bump (e.g., v2.71.0 to v2.71.1) to distinguish it from the regular release cadence. This requires manually creating the release branch and tag rather than using the new_release_branch workflow, since that workflow always branches from develop. :::

Single Platform Build Failure​

When a release build succeeds for one platform but fails for the other (e.g., iOS fails due to a screenshot timeout while Android succeeds), the options are:

Option 1: Fix without rebuilding (preferred when possible)​

For certain iOS failures (like screenshot upload timeouts), you may be able to fix the issue directly in App Store Connect without triggering a new build. See App Store screenshot failure for details.

Option 2: Re-trigger the full release build​

If the issue requires a code change or a full rebuild, follow the steps in Re-releasing a failed build. Be aware this rebuilds both platforms.

Option 3: Manually trigger a single-platform build (if available)​

If the build workflows have been updated to support workflow_dispatch with a platform input, you can manually trigger a rebuild for just the failed platform from the Release Build workflow page. Select the release branch, choose the failed platform, and run.