`UpdateAllCols` is dangerous, the columns should be updated when
necessary.
This PR replaces some `updateRepository` invokes to reduce possible
problems and wrongly updated time. Some parts have been fixed in #34388,
but some are hidden in the function `updateRepository`. Alternatively,
using `UpdateRepositoryColsNoAutoTime` to update the changed columns.
Some `UpdateRepoSize` invokes are duplicated, so they will be removed
when extracting from `updateRepository`.
Noticed the SQL will be executed 4 times when visit the file render view
page. For a repository which have many pull requests, it maybe slow.
```SQL
2025/06/08 15:24:44 models/issues/pull_list.go:69:GetUnmergedPullRequestsByHeadInfo() [I] [SQL] SELECT * FROM `pull_request` INNER JOIN `issue` ON issue.id = pull_request.issue_id WHERE (head_repo_id = ? AND head_branch = ? AND has_merged = ? AND issue.is_closed = ? AND flow = ?) [393 main false false 0] - 2.004167ms
2025/06/08 15:24:44 models/issues/pull_list.go:69:GetUnmergedPullRequestsByHeadInfo() [I] [SQL] SELECT * FROM `pull_request` INNER JOIN `issue` ON issue.id = pull_request.issue_id WHERE (head_repo_id = ? AND head_branch = ? AND has_merged = ? AND issue.is_closed = ? AND flow = ?) [393 main false false 0] - 1.03975ms
2025/06/08 15:24:44 models/issues/pull_list.go:69:GetUnmergedPullRequestsByHeadInfo() [I] [SQL] SELECT * FROM `pull_request` INNER JOIN `issue` ON issue.id = pull_request.issue_id WHERE (head_repo_id = ? AND head_branch = ? AND has_merged = ? AND issue.is_closed = ? AND flow = ?) [393 main false false 0] - 881.583µs
2025/06/08 15:24:44 models/issues/pull_list.go:69:GetUnmergedPullRequestsByHeadInfo() [I] [SQL] SELECT * FROM `pull_request` INNER JOIN `issue` ON issue.id = pull_request.issue_id WHERE (head_repo_id = ? AND head_branch = ? AND has_merged = ? AND issue.is_closed = ? AND flow = ?) [393 main false false 0] - 935.084µs
```
This PR did a refactor to query it once only.
Extract from #34531
## Move Commit status state to a standalone package
Move the state from `structs` to `commitstatus` package. It also
introduce `CommitStatusStates` so that the combine function could be
used from UI and API logic.
## Combined commit status Changed
This PR will follow Github's combined commit status. Before this PR,
every commit status could be a combined one.
According to
https://docs.github.com/en/rest/commits/statuses?apiVersion=2022-11-28#get-the-combined-status-for-a-specific-reference
> Additionally, a combined state is returned. The state is one of:
> failure if any of the contexts report as error or failure
> pending if there are no statuses or a context is pending
> success if the latest status for all contexts is success
This PR will follow that rule and remove the `NoBetterThan` logic. This
also fixes the inconsistent between UI and API. In the API convert
package, it has implemented this which is different from the UI. It also
fixed the missing `URL` and `CommitURL` in the API.
## `CalcCommitStatus` return nil if there is no commit statuses
The behavior of `CalcCommitStatus` is changed. If the parameter commit
statuses is empty, it will return nil. The reference places should check
the returned value themselves.
If user leaves the page, the context will become cancelled, so that the
update process maybe terminal in an unexpected status. This PR haven't
resolve the problem totally. It uses a background context to not cancel
the update process even if the user leaved the pull request view page.
Fix#31779
`GetIssuesLastCommitStatus` will revoke `GetIssuesAllCommitStatus` but
it has been invoked. The `CommitStatus` template variable has never been
used in notification subscription page so that it could be removed.
The tags synchronization is very slow for a non-mirror repository with
many tags especially forking. This PR make all repositories' tags
synchronization use the same function and remove the low performance
synchronization function. The commit count of tag now will not be stored
into database when syncing. Since the commits count will always be read
from cache or git data, the `NumCommits` in the release table will be
updated for the first read from git data.
Fix#34224
The previous implementation in #33744 will get the pushed commits
changed files. But it's not always right when push a merged commit. This
PR reverted the logic in #33744 and will always get the PR's changed
files and get code owners.
This PR moved git attributes related code to `modules/git/attribute` sub
package and moved language stats related code to
`modules/git/languagestats` sub package to make it easier to maintain.
And it also introduced a performance improvement which use the `git
check-attr --source` which can be run in a bare git repository so that
we don't need to create a git index file. The new parameter need a git
version >= 2.40 . If git version less than 2.40, it will fall back to
previous implementation.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: yp05327 <576951401@qq.com>
When list commits, some of the commits authors are the same at many
situations. But current logic will always fetch the same GPG keys from
database. This PR will cache the GPG keys, emails and users for the
context so that reducing the database queries.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR uniform all temporary directory usage so that it will be easier
to manage.
Relate to #31792
- [x] Added a new setting to allow users to configure the global
temporary directory.
- [x] Move all temporary files and directories to be placed under
os.Temp()/gitea.
- [x] `setting.Repository.Local.LocalCopyPath` now will be
`setting.TempPath/local-repo` and the customized path is removed.
```diff
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;[repository.local]
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;;
-;; Path for local repository copy. Defaults to TEMP_PATH + `local-repo`, this is deprecated and cannot be changed
-;LOCAL_COPY_PATH = local-repo
```
- [x] `setting.Repository.Upload.TempPath` now will be
`settting.TempPath/uploads` and the customized path is removed.
```diff
;[repository.upload]
-;;
-;; Path for uploads. Defaults to TEMP_PATH + `uploads`
-;TEMP_PATH = uploads
```
- [x] `setting.Packages.ChunkedUploadPath` now will be
`settting.TempPath/package-upload` and the customized path is removed.
```diff
;[packages]
-;;
-;; Path for chunked uploads. Defaults it's `package-upload` under `TEMP_PATH` unless it's an absolute path.
-;CHUNKED_UPLOAD_PATH = package-upload
```
- [x] `setting.SSH.KeyTestPath` now will be
`settting.TempPath/ssh_key_test` and the customized path is removed.
```diff
[server]
-;;
-;; Directory to create temporary files in when testing public keys using ssh-keygen,
-;; default is the system temporary directory.
-;SSH_KEY_TEST_PATH =
```
TODO:
- [ ] setting.PprofDataPath haven't been changed because it may need to
be kept until somebody read it but temp path may be clean up any time.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fix#28144
To make the resources will be cleanup once failed. All repository
operations now follow a consistent pattern:
- 1. Create a database record for the repository with the status
being_migrated.
- 2. Register a deferred cleanup function to delete the repository and
its related data if the operation fails.
- 3. Perform the actual Git and database operations step by step.
- 4. Upon successful completion, update the repository’s status to
ready.
The adopt operation is a special case — if it fails, the repository on
disk should not be deleted.
Extract from #33934
In the same goroutine, we should reuse the exist cat file sub process
which exist in `git.Repository` to avoid creating a unnecessary
temporary subprocess.
This PR reuse the exist cate file writer and reader in
`getCommitFromBatchReader`.
It also move `prepareLatestCommitInfo` before creating dataRc which will
hold the writer so other git operation will create a temporary cat file
subprocess.
Fix#32886
Add `last_committer_date` and `last_author_date` in the content API
which is not implemented by Github API v3 at the moment.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The pull request list API is slow, for every pull request, it needs to
open a git repository. Assume it has 30 records, there will be 30 sub
processes back because every repository will open a git cat-file --batch
sub process. This PR use base git repository to get the head commit id
rather than read it from head repository to avoid open any head git
repository.
When a team have no code unit permission of a repository, the member of
the team should not view activity contributors, recent commits and code
frequrency.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The pagination on the user dashboard sounds unnecessary, this will
change it to a prev/next buttons. For instances with around `10 million`
records in the action table, this option affects how the user dashboard
is loaded on first visit.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Giteabot <teabot@gitea.io>
Two SQLs are very slow when `action` table have over 5M records.
```
database duration=1.8881s db.sql="SELECT created_unix DIV 900 * 900 AS timestamp, count(user_id) as contributions FROM `action` WHERE user_id=? AND act_user_id=? AND (created_unix > ?) GROUP BY timestamp ORDER BY timestamp"
database duration=1.5408s db.sql="SELECT count(*) FROM `action` WHERE (user_id = ?) AND (is_deleted = ?)"
```
This will cache the count for the first loading or when the activities
changed.
When there are over 5M records on `action` table, the heatmap on
dashboard is very slow as below SQL.
```
database duration=1.8881s db.sql="SELECT created_unix DIV 900 * 900 AS timestamp, count(user_id) as contributions FROM `action` WHERE user_id=? AND act_user_id=? AND (created_unix > ?) GROUP BY timestamp ORDER BY timestamp"
```
This PR add a new index for `action` table with columns `user_id`,
`act_user_id` and `created_unix` so that this query will become about 6
times faster than before.
Extract from #28966
This PR uses `gitrepo.IsRepositoryExist` instead of `util.IsExist` to
detect whether the repository exist in disk. This will move `RepoPath`
detail behind of package `gitrepo` to make it easier to do possible
changes where storing the repositories.
No code change
Fix#33490
It will only read the changed file on the pushed commits but not all the
files of this PR.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fix#33582Fix#31698
When a user login, the dashboard should load all feed belongs to him
with no any conditions. The complicated conditions should be applied
only for another user view this user's profile.
Gitea Actions generated target url doesn't contain host and port. So we
need to include them for external webhook visiting.
Fix#33603
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Noticed a SQL in gitea.com has a bigger load. It seems both `is_pull`
and `pin_order` are not indexed columns in the database.
```SQL
SELECT `id`, `repo_id`, `index`, `poster_id`, `original_author`, `original_author_id`, `name`, `content`, `content_version`, `milestone_id`, `priority`, `is_closed`, `is_pull`, `num_comments`, `ref`, `pin_order`, `deadline_unix`, `created_unix`, `updated_unix`, `closed_unix`, `is_locked`, `time_estimate` FROM `issue` WHERE (repo_id =?) AND (is_pull = 0) AND (pin_order > 0) ORDER BY pin_order
```
I came across a comment
https://github.com/go-gitea/gitea/pull/24406#issuecomment-1527747296
from @delvh , which presents a more reasonable approach. Based on this,
this PR will migrate all issue and pull request pin data from the
`issue` table to the `issue_pin` table. This change benefits larger
Gitea instances by improving scalability and performance.
---------
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Lunny XiaoandGitHub
