Backport #34355 by @lunny
The tags synchronization is very slow for a non-mirror repository with
many tags especially forking. This PR make all repositories' tags
synchronization use the same function and remove the low performance
synchronization function. The commit count of tag now will not be stored
into database when syncing. Since the commits count will always be read
from cache or git data, the `NumCommits` in the release table will be
updated for the first read from git data.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Fix#880
Design:
1. A global setting `security.TWO_FACTOR_AUTH`.
* To support org-level config, we need to introduce a better "owner
setting" system first (in the future)
2. A user without 2FA can login and may explore, but can NOT read or
write to any repositories via API/web.
3. Keep things as simple as possible.
* This option only aggressively suggest users to enable their 2FA at the
moment, it does NOT guarantee that users must have 2FA before all other
operations, it should be good enough for real world use cases.
* Some details and tests could be improved in the future since this
change only adds a check and seems won't affect too much.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This PR uniform all temporary directory usage so that it will be easier
to manage.
Relate to #31792
- [x] Added a new setting to allow users to configure the global
temporary directory.
- [x] Move all temporary files and directories to be placed under
os.Temp()/gitea.
- [x] `setting.Repository.Local.LocalCopyPath` now will be
`setting.TempPath/local-repo` and the customized path is removed.
```diff
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;[repository.local]
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;
-;; Path for local repository copy. Defaults to TEMP_PATH + `local-repo`, this is deprecated and cannot be changed
-;LOCAL_COPY_PATH = local-repo
```
- [x] `setting.Repository.Upload.TempPath` now will be
`settting.TempPath/uploads` and the customized path is removed.
```diff
;[repository.upload]
-;;
-;; Path for uploads. Defaults to TEMP_PATH + `uploads`
-;TEMP_PATH = uploads
```
- [x] `setting.Packages.ChunkedUploadPath` now will be
`settting.TempPath/package-upload` and the customized path is removed.
```diff
;[packages]
-;;
-;; Path for chunked uploads. Defaults it's `package-upload` under `TEMP_PATH` unless it's an absolute path.
-;CHUNKED_UPLOAD_PATH = package-upload
```
- [x] `setting.SSH.KeyTestPath` now will be
`settting.TempPath/ssh_key_test` and the customized path is removed.
```diff
[server]
-;;
-;; Directory to create temporary files in when testing public keys using ssh-keygen,
-;; default is the system temporary directory.
-;SSH_KEY_TEST_PATH =
```
TODO:
- [ ] setting.PprofDataPath haven't been changed because it may need to
be kept until somebody read it but temp path may be clean up any time.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Replace all contexts in tests with go1.24 t.Context()
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* download endpoint has to use 302 redirect
* fake blob download used if direct download not possible
* downloading v3 artifacts not possible
New repo apis based on GitHub Rest V3
- GET /runs/{run}/artifacts (Cannot use run index of url due to not
being unique)
- GET /artifacts
- GET + DELETE /artifacts/{artifact_id}
- GET /artifacts/{artifact_id}/zip
- (GET /artifacts/{artifact_id}/zip/raw this is a workaround for a http
302 assertion in actions/toolkit)
- api docs removed this is protected by a signed url like the internal
artifacts api and no longer usable with any token or swagger
- returns http 401 if the signature is invalid
- or change the artifact id
- or expired after 1 hour
Closes#33353Closes#32124
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Adds user facing messages to errors when submitting agit pull request
Tries to highlight the returned error more and fixes agit suggestion to
create PR on first submission.
Closes: https://github.com/go-gitea/gitea/issues/32965
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
1. Add a OpenTelemetry-like shim-layer to collect traces
2. Add a simple builtin trace collector and exporter, end users could
download the diagnosis report to get the traces.
This PR's design is quite lightweight, no hard-dependency, and it is
easy to improve or remove. We can try it on gitea.com first to see
whether it works well, and fine tune the details.
---------
Co-authored-by: silverwind <me@silverwind.io>
unset XDG_CONFIG_HOME early to enable gitea to manage git configuration.
simple error checking to satisfy the linting. Closes#33039
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Gitea instance keeps reporting a lot of errors like "LFS SSH transfer connection denied, pure SSH protocol is disabled". When starting debugging the problem, there are more problems found. Try to address most of them:
* avoid unnecessary server side error logs (change `fail()` to not log them)
* figure out the broken tests/user2/lfs.git (added comments)
* avoid `migratePushMirrors` failure when a repository doesn't exist (ignore them)
* avoid "Authorization" (internal&lfs) header conflicts, remove the tricky "swapAuth" and use "X-Gitea-Internal-Auth"
* make internal token comparing constant time (it wasn't a serous problem because in a real world it's nearly impossible to timing-attack the token, but good to fix and backport)
* avoid duplicate routers (introduce AddOwnerRepoGitLFSRoutes)
* avoid "internal (private)" routes using session/web context (they should use private context)
* fix incorrect "path" usages (use "filepath")
* fix incorrect mocked route point handling (need to check func nil correctly)
* split some tests from "git general tests" to "git misc tests" (to keep "git_general_test.go" simple)
Still no correct result for Git LFS SSH tests. So the code is kept there
(`tests/integration/git_lfs_ssh_test.go`) and a FIXME explains the details.
This will allow instance admins to view signup pattern patterns for
public instances. It is modelled after discourse, mastodon, and
MediaWiki's approaches.
Note: This has privacy implications, but as the above-stated open-source
projects take this approach, especially MediaWiki, which I have no doubt
looked into this thoroughly, it is likely okay for us, too. However, I
would be appreciative of any feedback on how this could be improved.
---------
Co-authored-by: Giteabot <teabot@gitea.io>
All refs under `refs/pull` should only be changed from Gitea inside but
not by pushing from outside of Gitea.
This PR will prevent the pull refs update but allow other refs to be
updated on the same pushing with `--mirror` operations.
The main changes are to add checks on `update` hook but not
`pre-receive` because `update` will be invoked by every ref but
`pre-receive` will revert all changes once one ref update fails.
Change the copy to use `ActionsArtifact.StoragePath` instead of the
`ArtifactPath`. Skip artifacts that are expired, and don't error if the
file to copy does not exist.
---
When trying to migrate actions artifact storage from local to MinIO, we
encountered errors that prevented the process from completing
successfully:
* The migration tries to copy the files using the per-run
`ArtifactPath`, instead of the unique `StoragePath`.
* Artifacts that have been marked expired and had their files deleted
would throw an error
* Artifacts that are pending, but don't have a file uploaded yet will
throw an error.
This PR addresses these cases, and allow the process to complete
successfully.
This PR implemented object storages(LFS/Packages/Attachments and etc.)
for Azure Blob Storage. It depends on azure official golang SDK and can
support both the azure blob storage cloud service and azurite mock
server.
Replace #25458Fix#22527
- [x] CI Tests
- [x] integration test, MSSQL integration tests will now based on
azureblob
- [x] unit test
- [x] CLI Migrate Storage
- [x] Documentation for configuration added
------
TODO (other PRs):
- [ ] Improve performance of `blob download`.
---------
Co-authored-by: yp05327 <576951401@qq.com>
Add a configuration item to enable S3 virtual-hosted style (V2) to solve
the problem caused by some S3 service providers not supporting path
style (V1).
Merging PR may fail because of various problems. The pull request may
have a dirty state because there is no transaction when merging a pull
request. ref
https://github.com/go-gitea/gitea/pull/25741#issuecomment-2074126393
This PR moves all database update operations to post-receive handler for
merging a pull request and having a database transaction. That means if
database operations fail, then the git merging will fail, the git client
will get a fail result.
There are already many tests for pull request merging, so we don't need
to add a new one.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Follow #30472:
When a user is created by command line `./gitea admin user create`:
Old behavior before #30472: the first user (admin or non-admin) doesn't
need to change password.
Revert to the old behavior before #30472
Noteable additions:
- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches
Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
Attempts to resolve#28720.
---
Note that I am not a Gitea administrator so I don't normally use the
gitea CLI. Just saw this issue and wanted an opportunity to understand
how this subcommand works and see if I can add this feature :^)
I tested both with `--skip-db` and without and it appears to not add any
database-specific files to the generated archive i.e. I don't see a
`gitea-db.sql` or `gitea.db` file:
```console
$ TAGS="bindata sqlite sqlite_unlock_notify" make backend
Running go generate...
bindata for migration already up-to-date
bindata for options already up-to-date
bindata for public already up-to-date
bindata for templates already up-to-date
$ ./gitea dump --skip-db
2024/04/20 01:16:11 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/04/20 01:16:11 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/attachments
2024/04/20 01:16:11 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/avatars
2024/04/20 01:16:11 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-avatars
2024/04/20 01:16:11 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/lfs
2024/04/20 01:16:11 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-archive
2024/04/20 01:16:11 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/packages
2024/04/20 01:16:11 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_log
2024/04/20 01:16:11 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_artifacts
2024/04/20 01:16:11 cmd/dump.go:172:runDump() [I] Dumping local repositories... /workspaces/gitea/data/gitea-repositories
2024/04/20 01:16:11 cmd/dump.go:195:runDump() [I] Skipping database
2024/04/20 01:16:11 cmd/dump.go:229:runDump() [I] Adding custom configuration file from /workspaces/gitea/custom/conf/app.ini
2024/04/20 01:16:11 cmd/dump.go:256:runDump() [I] Packing data directory.../workspaces/gitea/data
2024/04/20 01:16:11 cmd/dump.go:335:runDump() [I] Finish dumping in file /workspaces/gitea/gitea-dump-1713575771.zip
$ unzip /workspaces/gitea/gitea-dump-1713575771.zip -d example
Archive: /workspaces/gitea/gitea-dump-1713575771.zip
. . .
$ ls example/
app.ini custom data repos
$ ls example/data/
actions_artifacts actions_log avatars home indexers jwt queues repo-archive repo-avatars tmp
```
Co-authored-by: Giteabot <teabot@gitea.io>
Agit returned result should be from `ProcReceive` hook but not
`PostReceive` hook. Then for all non-agit pull requests, it will not
check the pull requests for every pushing `refs/pull/%d/head`.
Major changes:
* Move some functions like "addReader" / "isSubDir" /
"addRecursiveExclude" to a separate package, and add tests
* Clarify the filename&dump type logic and add tests
* Clarify the logger behavior and remove FIXME comments
Co-authored-by: Giteabot <teabot@gitea.io>
