Skip to content

Repo sync #27950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/get-started/using-git/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,5 @@ children:
- /using-git-rebase-on-the-command-line
- /resolving-merge-conflicts-after-a-git-rebase
- /dealing-with-special-characters-in-branch-and-tag-names
- /troubleshooting-the-2-gb-push-limit
---

Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,5 @@ For more information on working with forks, see "[AUTOTITLE](/pull-requests/coll
- [`git remote` main page](https://git-scm.com/docs/git-remote.html)
- "[AUTOTITLE](/get-started/quickstart/git-cheatsheet)"
- "[AUTOTITLE](/get-started/getting-started-with-git/git-workflows)"
- "[Git Handbook](https://guides.github.com/introduction/git-handbook/)"
- "[Git Handbook](https://guides.github.com/introduction/git-handbook/)"{% ifversion fpt or ghec %}
- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"{% endif %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
title: Troubleshooting the 2 GB push limit
intro: 'Learn how to work around the 2 GB push limit.'
versions:
fpt: '*'
ghec: '*'
shortTitle: Maximum push limit
---

## About the push limit

{% data variables.product.prodname_dotcom %} has a maximum 2 GB limit for a single push. You might hit this limit when trying to upload very large repositories for the first time, importing large repositories from other platforms, or when trying to rewrite the history of large existing repositories.

If you hit this limit, you may see one of the following error messages:

- `fatal: the remote end hung up unexpectedly`
- `remote: fatal: pack exceeds maximum allowed size`

You can either split up your push into smaller parts, or delete the Git history and start from scratch. If you have made a single commit that's larger than 2 GB and you can't delete the Git history and start from scratch, then you will need to perform an interactive rebase to split the large commit into multiple smaller ones.

## Splitting up a large push

You can avoid hitting the limit by breaking your push into smaller parts, each of which should be under 2 GB in size. If a branch is within this size limit, you can push it all at once. However, if a branch is larger than 2 GB, you'll need to split the push into even smaller portions and push only a few commits at a time.

1. If you haven't configured the remote yet, add the repository as a new remote. For more information, see "[AUTOTITLE](/get-started/getting-started-with-git/managing-remote-repositories#adding-a-remote-repository)."
1. To find suitable commits spread out along the history of the main branch in your local repository, run the following command:

```shell
git log --oneline --reverse refs/heads/BRANCH-NAME | awk 'NR % 1000 == 0'
```

This command reveals every 1000th commit. You can increase or decrease the number to adjust the step size.

1. Push each of these commits one at a time to your {% data variables.product.prodname_dotcom %} hosted repository.

```shell
git push REMOTE-NAME <YOUR_COMMIT_SHA_NUMBER>:refs/heads/BRANCH-NAME
```

If you see the message `remote: fatal: pack exceeds maximum allowed size`, reduce the step size in step 2 and try again.

1. Go through the same process for every commit you identified in the history from step 2.
1. If this is the first time this repository is being pushed to {% data variables.product.prodname_dotcom %}, perform a final mirror push to ensure any remaining refs are pushed up.

```shell
git push REMOTE-NAME --mirror
```

If this is still too large, you'll need to push up other branches in stages using the same steps.

Once you're familiar with the procedure, you can automate steps 2 to 4 to simplify the process. For example:

```shell
step_commits=$(git log --oneline --reverse refs/heads/BRANCH-NAME | awk 'NR % 1000 == 0')
echo "$step_commits" | while read commit message; do git push REMOTE-NAME $commit:refs/heads/BRANCH-NAME; done
```

## Starting from scratch

If the repository does not have any history, or your initial commit was over 2 GB on its own and you don't mind resetting the Git history, you can also start from scratch.

1. On your local copy, delete the hidden `.git` folder to remove all the previous Git history and convert it back into a normal folder full of files.
1. Create a new empty folder.
1. Run `git init` and `git lfs install` on the new folder, and add the new empty {% data variables.product.prodname_dotcom %} repository as a remote.
1. If you already use {% data variables.large_files.product_name_long %} and have all of the {% data variables.large_files.product_name_short %} tracking rules you intend to use already listed in the `.gitattributes` file in the old folder, that should be the first file you copy across to the new folder. You should ensure the tracking rules are in place before you add any other files, so that there's no chance things intended for {% data variables.large_files.product_name_short %} will be committed to regular Git storage.

If you do not already use {% data variables.large_files.product_name_short %}, you can skip this step, or you can set up the tracking rules you intend to use in the `.gitattributes` file in the new folder before you copy any other files across. For more information, see "[AUTOTITLE](/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage)."

1. Move batches of files that are smaller than 2 GB from the old folder to the new folder. After each batch is moved, create a commit and push it before moving the next batch. You can take a cautious approach and stick to around 2 GB. Alternatively, if you have a folder with files meant for {% data variables.large_files.product_name_short %}, you can ignore those files when considering the 2 GB limit per batch.

Once the old folder is empty, the {% data variables.product.prodname_dotcom %} repository should contain everything. If you are using {% data variables.large_files.product_name_short %}, all files meant for {% data variables.large_files.product_name_short %} should be pushed to {% data variables.large_files.product_name_short %} storage.
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,9 @@ You can use the command line to import source code and, if the code has been tra
{% endif %}

All of these tools import source code and revision history, only. If you also want to import your settings and your collaboration history, such as issues and pull requests, you'll need to use more advanced tools. To determine the best tool to use for your migration, see "[AUTOTITLE](/migrations/overview/planning-your-migration-to-github)."

{% ifversion fpt or ghec %}
## Further reading

- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ After you've initialized a Git repository, you can push the repository to {% dat

{% data reusables.migrations.create-empty-repo %}
1. At the top of your repository on {% ifversion ghae %}{% data variables.product.product_name %}{% else %}{% data variables.location.product_location %}{% endif %}'s Quick Setup page, click {% octicon "copy" aria-label="Copy to clipboard" %} to copy the remote repository URL.

![Screenshot of the "Quick Setup" header in a repository. Next to the remote URL, an icon of two overlapping squares is highlighted with an orange outline.](/assets/images/help/repository/copy-remote-repository-url-quick-setup.png)
{% data reusables.command_line.open_the_multi_os_terminal %}
1. Change the current working directory to your local project.
Expand Down Expand Up @@ -167,4 +167,5 @@ After you've initialized a Git repository, you can push the repository to {% dat

## Further reading

- "[AUTOTITLE](/repositories/working-with-files/managing-files/adding-a-file-to-a-repository#adding-a-file-to-a-repository-using-the-command-line)"
- "[AUTOTITLE](/repositories/working-with-files/managing-files/adding-a-file-to-a-repository#adding-a-file-to-a-repository-using-the-command-line)"{% ifversion fpt or ghec %}
- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,9 @@ To follow these steps, you must use a macOS or Linux system and have the followi
1. After the import finishes, to check out your newly-created Git repository, run `git checkout HEAD`.
{% data reusables.migrations.add-github-repo-as-remote %}
{% data reusables.migrations.push-to-github %}

{% ifversion fpt or ghec %}
## Further reading

- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,9 @@ To follow these steps, you must use a macOS or Linux system and have the followi
{% data reusables.migrations.move-into-git-repo-directory %}
{% data reusables.migrations.add-github-repo-as-remote %}
{% data reusables.migrations.push-to-github %}

{% ifversion fpt or ghec %}
## Further reading

- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,9 @@ To follow these steps, you must use Windows and have the following tools install
{% data reusables.migrations.move-into-git-repo-directory %}
{% data reusables.migrations.add-github-repo-as-remote %}
{% data reusables.migrations.push-to-github %}

{% ifversion fpt or ghec %}
## Further reading

- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,9 @@ For purposes of demonstration, we'll use:
```

If the repository you are importing contains large files, you may run into a warning or error. For more information on large files and how to manage them, see "[AUTOTITLE](/repositories/working-with-files/managing-large-files/about-large-files-on-github)."

{% ifversion fpt or ghec %}
## Further reading

- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"
{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -93,4 +93,5 @@ permissions: Site administrators can use the administrative shell to import data

## Further reading

- "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#import-and-export)"
- "[AUTOTITLE](/admin/configuration/configuring-your-enterprise/command-line-utilities#import-and-export)"{% ifversion fpt or ghec %}
- "[AUTOTITLE](/get-started/using-git/troubleshooting-the-2-gb-push-limit)"{% endif %}
Original file line number Diff line number Diff line change
Expand Up @@ -2624,7 +2624,9 @@
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [],
"additional-permissions": [
"contents"
],
"access": "write"
},
{
Expand Down Expand Up @@ -3708,6 +3710,17 @@
"additional-permissions": [],
"access": "write"
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [
"administration"
],
"access": "read"
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3271,7 +3271,9 @@
"access": "write",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": []
"additional-permissions": [
"contents"
]
},
{
"category": "interactions",
Expand Down Expand Up @@ -4578,6 +4580,19 @@
"server-to-server": true,
"additional-permissions": []
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"access": "read",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": [
"administration"
]
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
15 changes: 14 additions & 1 deletion src/github-apps/data/ghae/fine-grained-pat-permissions.json
Original file line number Diff line number Diff line change
Expand Up @@ -1708,7 +1708,9 @@
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [],
"additional-permissions": [
"contents"
],
"access": "write"
},
{
Expand Down Expand Up @@ -2263,6 +2265,17 @@
"additional-permissions": [],
"access": "write"
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [
"administration"
],
"access": "read"
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
17 changes: 16 additions & 1 deletion src/github-apps/data/ghae/server-to-server-permissions.json
Original file line number Diff line number Diff line change
Expand Up @@ -2065,7 +2065,9 @@
"access": "write",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": []
"additional-permissions": [
"contents"
]
},
{
"category": "collaborators",
Expand Down Expand Up @@ -2735,6 +2737,19 @@
"server-to-server": true,
"additional-permissions": []
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"access": "read",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": [
"administration"
]
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3282,7 +3282,9 @@
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [],
"additional-permissions": [
"contents"
],
"access": "write"
},
{
Expand Down Expand Up @@ -4366,6 +4368,17 @@
"additional-permissions": [],
"access": "write"
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [
"administration"
],
"access": "read"
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4082,7 +4082,9 @@
"access": "write",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": []
"additional-permissions": [
"contents"
]
},
{
"category": "interactions",
Expand Down Expand Up @@ -5389,6 +5391,19 @@
"server-to-server": true,
"additional-permissions": []
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"access": "read",
"user-to-server": true,
"server-to-server": true,
"additional-permissions": [
"administration"
]
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2681,7 +2681,9 @@
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [],
"additional-permissions": [
"contents"
],
"access": "write"
},
{
Expand Down Expand Up @@ -3349,6 +3351,17 @@
"additional-permissions": [],
"access": "write"
},
{
"category": "repos",
"slug": "create-a-fork",
"subcategory": "forks",
"verb": "post",
"requestPath": "/repos/{owner}/{repo}/forks",
"additional-permissions": [
"administration"
],
"access": "read"
},
{
"category": "git",
"slug": "create-a-blob",
Expand Down
Loading