* reuse recoverable error checks across mirror_pull
* add new cases for 'cannot lock ref/not our ref' (race condition in
fetch) and 'Unable to create/lock"
* move lfs sync right after commit graph write, and before other
maintenance which may fail
* try a prune for 'broken reference' as well as 'not our ref'
* always sync LFS right after commit graph write, and before other
maintenance which may fail
This handles a few cases where our very large and very active
repositories could serve mirrored git refs, but be missing lfs files:
## Case 1 (multiple variants): Race condition in git fetch
There was already a check for 'unable to resolve reference' on a failed
git fetch, after which a git prune and then subsequent fetch are
performed. This is to work around a race condition where the git remote
tells Gitea about a ref for some HEAD of a branch, then fails a few
seconds later because the remote branch was deleted, or the ref was
updated (force push).
There are two more variants to the error message you can get, but for
the same kind of race condition. These *may* be related to the git
binary version Gitea has access to (in my case, it was 2.48.1).
## Case 2: githttp.go can serve updated git refs before it's synced lfs
oids
There is probably a more aggressive refactor we could do here to have
the cat-file loop use FETCH_HEAD instead of relying on the commit graphs
to be committed locally (and thus serveable to clients of Gitea), but a
simple reduction in the occurrences of this for me was to move the lfs
sync block immediately after the commit-graph write and before any other
time-consuming (or potentially erroring/exiting) blocks.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This contains two backwards-compatible changes:
* in the lfs http_client, the number of lfs oids requested per batch is
loaded from lfs_client#BATCH_SIZE and defaulted to the previous value of
20
* in the lfs server/service, the max number of lfs oids allowed in a
batch api request is loaded from server#LFS_MAX_BATCH_SIZE and defaults
to 'nil' which equates to the previous behavior of 'infinite'
This fixes#32306
---------
Signed-off-by: Royce Remer <royceremer@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Support legacy _links LFS batch response.
Fixes#31512.
This is backwards-compatible change to the LFS client so that, upon
mirroring from an upstream which has a batch api, it can download
objects whether the responses contain the `_links` field or its
successor the `actions` field. When Gitea must fallback to the legacy
`_links` field a logline is emitted at INFO level which looks like this:
```
...s/lfs/http_client.go:188:performOperation() [I] <LFSPointer ee95d0a27ccdfc7c12516d4f80dcf144a5eaf10d0461d282a7206390635cdbee:160> is using a deprecated batch schema response!
```
I've only run `test-backend` with this code, but added a new test to
cover this case. Additionally I have a fork with this change deployed
which I've confirmed syncs LFS from Gitea<-Artifactory (which has legacy
`_links`) as well as from Gitea<-Gitea (which has the modern `actions`).
Signed-off-by: Royce Remer <royceremer@gmail.com>
