1
1
mirror of https://github.com/go-gitea/gitea synced 2024-10-31 23:34:25 +00:00
Commit Graph

174 Commits

Author SHA1 Message Date
Aravinth Manivannan
fa73cbf5a7
Store the foreign ID of issues during migration (#18446)
Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table.

At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists.

The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps:

1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes)
2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists
3. Optimize the Gitea uploader to speed up the updates, when possible.

The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it.

The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration.

The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works.

The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together.

The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo.

The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not.

The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose.

The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues.

---
[source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36)

Signed-off-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: Loïc Dachary <loic@dachary.org>
2022-03-17 18:08:35 +01:00
6543
bc0d2c8ada
RSS/Atom support for Repos (#19055)
* support for repos
* refactor
* advertise the feeds via meta tags
* allow feed suffix and feed header
* optimize performance
2022-03-13 17:40:47 +01:00
KN4CK3R
5b0cdd6155
Set is_empty in fixtures (#18869) 2022-02-26 01:15:25 -05:00
KN4CK3R
9d7a431b71
Set is_private in fixtures. (#18868)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-02-25 16:22:20 -05:00
singuliere
367894adc8
add test coverage for original author conversion during migrations (#18506)
* add test coverage for original author conversion during migrations

And create a function to factorize a code snippet that is repeated
five times and would otherwise be more difficult to test and maintain
consistently.

Signed-off-by: Loïc Dachary <loic@dachary.org>

* fix variable scope and int64 formatting

* add missing calls to remapExternalUser and fix misplaced %d

Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-02-01 13:20:28 -05:00
Lunny Xiao
35c3553870
Support webauthn (#17957)
Migrate from U2F to Webauthn

Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-01-14 16:03:31 +01:00
Aravinth Manivannan
69a28299e2
migrations: a deadline at January 1st, 1970 is valid (#18237)
* migrations: a deadline at January 1st, 1970 is valid

Do not change the deadline value if it is set to January 1st, 1970.

Setting the deadline to year 9999 when it is zero (which is equal to
January 1st, 1970) modifies a deadline set to January 1st, 1970 which
is a valid date. In addition, setting a date in year 9999 will be
converted to a null date in some cases.

Signed-off-by: Loïc Dachary <loic@dachary.org>

* tests: set milestone.deadline_unix in fixtures

The value of deadline_unix must be set to 253370764800 (i.e. 9999-01-01) in
fixtures, otherwise it will be inserted as null which leads to
unexpected errors. For instance, DumpRepository will store a null
deadline_unix as 0 (i.e. 1970-01-01) and RestoreRepository will change
it to 9999-01-01.

Signed-off-by: Loïc Dachary <loic@dachary.org>

Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-01-13 14:03:30 +08:00
Lunny Xiao
8760af752a
Team permission allow different unit has different permission (#17811)
* Team permission allow different unit has different permission

* Finish the interface and the logic

* Fix lint

* Fix translation

* align center for table cell content

* Fix fixture

* merge

* Fix test

* Add deprecated

* Improve code

* Add tooltip

* Fix swagger

* Fix newline

* Fix tests

* Fix tests

* Fix test

* Fix test

* Max permission of external wiki and issues should be read

* Move team units with limited max level below units table

* Update label and column names

* Some improvements

* Fix lint

* Some improvements

* Fix template variables

* Add permission docs

* improve doc

* Fix fixture

* Fix bug

* Fix some bug

* fix

* gofumpt

* Integration test for migration (#18124)

integrations: basic test for Gitea {dump,restore}-repo
This is a first step for integration testing of DumpRepository and
RestoreRepository. It:

runs a Gitea server,
dumps a repo via DumpRepository to the filesystem,
restores the repo via RestoreRepository from the filesystem,
dumps the restored repository to the filesystem,
compares the first and second dump and expects them to be identical

The verification is trivial and the goal is to add more tests for each
topic of the dump.

Signed-off-by: Loïc Dachary <loic@dachary.org>

* Team permission allow different unit has different permission

* Finish the interface and the logic

* Fix lint

* Fix translation

* align center for table cell content

* Fix fixture

* merge

* Fix test

* Add deprecated

* Improve code

* Add tooltip

* Fix swagger

* Fix newline

* Fix tests

* Fix tests

* Fix test

* Fix test

* Max permission of external wiki and issues should be read

* Move team units with limited max level below units table

* Update label and column names

* Some improvements

* Fix lint

* Some improvements

* Fix template variables

* Add permission docs

* improve doc

* Fix fixture

* Fix bug

* Fix some bug

* Fix bug

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Aravinth Manivannan <realaravinth@batsense.net>
2022-01-05 11:37:00 +08:00
Aravinth Manivannan
ade41f3f04
tests: add coverage for models migration helpers (#18162)
They were previously not covered at all, either by integration tests or unit tests.

This PR also fixes a bug where the `num_comments` field was incorrectly set to include all types of comments.

It sets num_closed_issues: 0 as default in milestone unit test fixtures. If they are not set, Incr("num_closed_issues") will be a noop because the field is null.
2022-01-03 23:35:01 +08:00
Kamil Domański
021df29623
Allow U2F 2FA without TOTP (#11573)
This change enables the usage of U2F without being forced to enroll an TOTP authenticator.
The `/user/auth/u2f` has been changed to hide the "use TOTP instead" bar if TOTP is not enrolled.

Fixes #5410
Fixes #17495
2021-11-08 23:47:19 +01:00
zeripath
0b4a8be26b
Ensure that restricted users can access repos for which they are members (#17460)
There is a small bug in the way that repo access is checked in
repoAssignment: Accessibility is checked by checking if the user has a
marked access to the repository instead of checking if the user has any
team granted access.

This PR changes this permissions check to use HasAccess() which does the
correct test. There is also a fix in the release api ListReleases where
it should return draft releases if the user is a member of a team with
write access to the releases.

The PR also adds a testcase.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-10-28 10:54:40 +08:00
wxiaoguang
7bcbdd0707
Add user status filter to admin user management page (#16770)
It makes Admin's life easier to filter users by various status.

* introduce window.config.PageData to pass template data to javascript module and small refactor

move legacy window.ActivityTopAuthors to window.config.PageData.ActivityTopAuthors
make HTML structure more IDE-friendly in footer.tmpl and head.tmpl
remove incorrect <style class="list-search-style"></style> in head.tmpl
use log.Error instead of log.Critical in admin user search

* use LEFT JOIN instead of SubQuery when admin filters users by 2fa. revert non-en locale.

* use OptionalBool instead of status map

* refactor SearchUserOptions.toConds to SearchUserOptions.toSearchQueryBase

* add unit test for user search

* only allow admin to use filters to search users
2021-10-12 20:11:35 +02:00
a1012112796
bb39359668
Add a simple way to rename branch like gh (#15870)
- Update default branch if needed
- Update protected branch if needed
- Update all not merged pull request base branch name
- Rename git branch
- Record this rename work and auto redirect for old branch on ui

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: delvh <dev.lh@web.de>
2021-10-08 19:03:04 +02:00
Lunny Xiao
d9c69596ff
Fix commit status index problem (#17061)
* Fix commit status index problem

* remove unused functions

* Add fixture and test for migration

* Fix lint

* Fix fixture

* Fix lint

* Fix test

* Fix bug

* Fix bug
2021-09-23 18:50:06 +08:00
Lunny Xiao
ddc709ff7f
Add repo_id for attachment (#16958)
When create a new issue or comment and paste/upload an attachment/image, it will not assign an issue id before submit. So if user give up the creating, the attachments will lost key feature and become dirty content. We don't know if we need to delete the attachment even if the repository deleted.

This PR add a repo_id in attachment table so that even if a new upload attachment with no issue_id or release_id but should have repo_id. When deleting a repository, they could also be deleted.

Co-authored-by: 6543 <6543@obermui.de>
2021-09-08 17:19:30 +02:00
KN4CK3R
4debb74eda
Fix wrong user in OpenID response (#16736)
* Fixed usage of wrong user.

* Added tests.
2021-08-19 12:11:30 -04:00
Jimmy Praet
a4962a9440
Add filter by owner and team to issue/pulls search endpoint (#16662)
* Filter by owner and team in API issue/pulls search

* Add integration test
2021-08-13 22:47:25 +02:00
Sergey Dryabzhinsky
22a0636544
Add Visible modes function from Organisation to Users too (#16069)
You can limit or hide organisations. This pull make it also posible for users

- new strings to translte
- add checkbox to user profile form
- add checkbox to admin user.edit form
- filter explore page user search
- filter api admin and public user searches
- allow admins view "hidden" users
- add app option DEFAULT_USER_VISIBILITY
- rewrite many files to use Visibility field
- check for teams intersection
- fix context output
- right fake 404 if not visible

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-06-26 20:53:14 +01:00
siddweiker
f573e93ed4
Fix heatmap activity (#15252)
* Group heatmap actions by 15 minute intervals

Signed-off-by: Sidd Weiker <siddweiker@gmail.com>

* Add multi-contribution test for user heatmap

Signed-off-by: Sidd Weiker <siddweiker@gmail.com>

* Add timezone aware summation for activity heatmap

Signed-off-by: Sidd Weiker <siddweiker@gmail.com>

* Fix api user heatmap test

Signed-off-by: Sidd Weiker <siddweiker@gmail.com>

* Update variable declaration style

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-25 12:59:25 -04:00
Lunny Xiao
b223d36195
Rework repository archive (#14723)
* Use storage to store archive files

* Fix backend lint

* Add archiver table on database

* Finish archive download

* Fix test

* Add database migrations

* Add status for archiver

* Fix lint

* Add queue

* Add doctor to check and delete old archives

* Improve archive queue

* Fix tests

* improve archive storage

* Delete repo archives

* Add missing fixture

* fix fixture

* Fix fixture

* Fix test

* Fix archiver cleaning

* Fix bug

* Add docs for repository archive storage

* remove repo-archive configuration

* Fix test

* Fix test

* Fix lint

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-23 17:12:38 -04:00
6543
6ad5d0a306
[API] ListReleases add filter for draft and pre-releases (#16175)
* invent ctx.QueryOptionalBool

* [API] ListReleases add draft and pre-release filter

* Add X-Total-Count header

* Add a release to fixtures

* Add TEST for API ListReleases
2021-06-17 10:58:10 +02:00
6543
0e081ff0ce
[API] ListIssues add more filters (#16174)
* [API] ListIssues add more filters:
optional filter repo issues by:
 - since
 - before
 - created_by
 - assigned_by
 - mentioned_by

* Add Tests

* Update routers/api/v1/repo/issue.go

Co-authored-by: Lanre Adelowo <adelowomailbox@gmail.com>

* Apply suggestions from code review

Co-authored-by: Lanre Adelowo <adelowomailbox@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-16 18:33:37 -04:00
Lunny Xiao
0393a57511
Add a new table issue_index to store the max issue index so that issue could be deleted with no duplicated index (#15599)
* Add a new table issue_index to store the max issue index so that issue could be deleted with no duplicated index

* Fix pull index

* Add tests for concurrent creating issues

* Fix lint

* Fix tests

* Fix postgres test

* Add test for migration v180

* Rename wrong test file name

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-14 10:22:55 +08:00
Lunny Xiao
b9d611e917
Always store primary email address into email_address table and also the state (#15956)
* Always store primary email address into email_address table and also the state

* Add lower_email to not convert email to lower as what's added

* Fix fixture

* Fix tests

* Use BeforeInsert to save lower email

* Fix v180 migration

* fix tests

* Fix test

* Remove wrong submited codes

* Fix test

* Fix test

* Fix test

* Add test for v181 migration

* remove change user's email to lower

* Revert change on user's email column

* Fix lower email

* Fix test

* Fix test
2021-06-08 11:52:51 +08:00
a1012112796
3088866531
fix some ui bug about draft release (#15137)
* fix some ui bug about draft release

- should not show draft release in tag list because
  it will't create real tag
- still show draft release without tag and commit message
  for draft release instead of 404 error
- remove tag load for attachement links because it's useless

Signed-off-by: a1012112796 <1012112796@qq.com>

* add test code

* fix test

That's because has added a new release in relaese test database.

* fix dropdown link for draft release
2021-05-06 05:12:50 +02:00
Norwin
658d1bfac8
API: fix set milestone on PR creation (#14981)
* API: fix set milestone on PR creation

pr creation via API failed with 404, because we searched
for milestoneID 0, due to uninitialized var usage D:

* add tests

* fix expected status codes

* fix tests

Co-authored-by: 6543 <6543@obermui.de>
2021-03-13 19:06:52 +01:00
6543
a4148c0f12
Repository transfer has to be confirmed, if user can not create repo for new owner (#14792)
* make repo as "pending transfer" if on transfer start doer has no right to create repo in new destination

* if new pending transfer ocured, create UI & Mail notifications
2021-03-01 01:47:30 +01:00
a1012112796
ac701637b4
Add dismiss review feature (#12674)
* Add dismiss review feature

refs:
    https://github.blog/2016-10-12-dismissing-reviews-on-pull-requests/
    https://developer.github.com/v3/pulls/reviews/#dismiss-a-review-for-a-pull-request

* change modal ui and error message

* Add unDismissReview api

Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-02-11 18:32:25 +01:00
Andrew Bezold
bc05ddc0eb
Redirect on changed user and org name (#11649)
* Add redirect for user

* Add redirect for orgs

* Add user redirect test

* Appease linter

* Add comment to DeleteUserRedirect function

* Fix locale changes

* Fix GetUserByParams

* Fix orgAssignment

* Remove debug logging

* Add redirect prompt

* Dont Export DeleteUserRedirect & only use it within a session

* Unexport newUserRedirect

* cleanup

* Fix & Dedub API code

* Format Template

* Add Migration & rm dublicat

* Refactor: unexport newRepoRedirect() & rm dedub del exec

* if this fails we'll need to re-rename the user directory

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-01-24 16:23:05 +01:00
Elena Neuschild
564030336d
Issues overview should not show issues from archived repos (#13220)
* Add lots of comments to user.Issues()

* Answered some questions from comments

* fix typo in comment

* Refac user.Issues(): add func repoIDs

* Refac user.Issues(): add func userRepoIDs

* Refac user.Issues(): add func issueIDsFromSearch

* Refac user.Issues(): improve error handling

* Refac user.Issues(): add inline documentation and move variable declarations closer to their usages

* Refac user.Issues(): add func repoIDMap

* Refac user.Issues(): cleanup

* Refac: Separate Issues from Pulls during routing

* fix typo in comment

* Adapt Unittests to Refactoring

* Issue13171: Issue and PR Overviews now ignore archived Repositories

* changed some verbatim SQL conditions to builder.Eq

* models/issue.go: use OptionalBool properly

Co-authored-by: 6543 <6543@obermui.de>

* Use IsArchived rather than ExcludeArchivedRepos

* fixed broken test after merge

* added nil check

* Added Unit Test securing Issue 13171 fix

* Improved IsArchived filtering in issue.GetUserIssueStats

* Removed unused func

* Added grouping to avoid returning duplicate repo IDs

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Gitea <gitea@fake.local>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-01-12 23:19:17 -05:00
Johnny Oskarsson
a07e67d9cc
Minimal OpenID Connect implementation (#14139)
This is "minimal" in the sense that only the Authorization Code Flow
from OpenID Connect Core is implemented.  No discovery, no configuration
endpoint, and no user scope management.

OpenID Connect is an extension to the (already implemented) OAuth 2.0
protocol, and essentially an `id_token` JWT is added to the access token
endpoint response when using the Authorization Code Flow.  I also added
support for the "nonce" field since it is required to be used in the
id_token if the client decides to include it in its initial request.

In order to enable this extension an OAuth 2.0 scope containing
"openid" is needed. Other OAuth 2.0 requests should not be impacted by
this change.

This minimal implementation is enough to enable single sign-on (SSO)
for other sites, e.g. by using something like `mod_auth_openidc` to
only allow access to a CI server if a user has logged into Gitea.

Fixes: #1310

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-02 00:33:27 +08:00
Norwin
f6bec85296
rework heatmap permissions (#14080)
* now uses the same permission model as for the activity feed:
  only include activities in repos, that the doer has access to.
  this might be somewhat slower.

* also improves handling of user.KeepActivityPrivate (still shows
  the heatmap to self & admins)

* extend tests

* adjust integration test to new behaviour

* add access to actions for admins

* extend heatmap unit tests
2020-12-22 03:53:37 +01:00
Kyle Evans
e461f0854f
[RFC] Make archival asynchronous (#11296)
* Make archival asynchronous

The prime benefit being sought here is for large archives to not
clog up the rendering process and cause unsightly proxy timeouts.
As a secondary benefit, archive-in-progress is moved out of the
way into a /tmp file so that new archival requests for the same
commit will not get fulfilled based on an archive that isn't yet
finished.

This asynchronous system is fairly primitive; request comes in, we'll
spawn off a new goroutine to handle it, then we'll mark it as done.
Status requests will see if the file exists in the final location,
and report the archival as done when it exists.

Fixes #11265

* Archive links: drop initial delay to three-quarters of a second

Some, or perhaps even most, archives will not take all that long to archive.
The archive process starts as soon as the download button is initially
clicked, so in theory they could be done quite quickly.  Drop the initial
delay down to three-quarters of a second to make it more responsive in the
common case of the archive being quickly created.

* archiver: restructure a little bit to facilitate testing

This introduces two sync.Cond pointers to the archiver package. If they're
non-nil when we go to process a request, we'll wait until signalled (at all)
to proceed. The tests will then create the sync.Cond so that it can signal
at-will and sanity-check the state of the queue at different phases.

The author believes that nil-checking these two sync.Cond pointers on every
archive processing will introduce minimal overhead with no impact on
maintainability.

* gofmt nit: no space around binary + operator

* services: archiver: appease golangci-lint, lock queueMutex

Locking/unlocking the queueMutex is allowed, but not required, for
Cond.Signal() and Cond.Broadcast().  The magic at play here is just a little
too much for golangci-lint, as we take the address of queueMutex and this is
mostly used in archiver.go; the variable still gets flagged as unused.

* archiver: tests: fix several timing nits

Once we've signaled a cond var, it may take some small amount of time for
the goroutines released to hit the spot we're wanting them to be at. Give
them an appropriate amount of time.

* archiver: tests: no underscore in var name, ungh

* archiver: tests: Test* is run in a separate context than TestMain

We must setup the mutex/cond variables at the beginning of any test that's
going to use it, or else these will be nil when the test is actually ran.

* archiver: tests: hopefully final tweak

Things got shuffled around such that we carefully build up and release
requests from the queue, so we can validate the state of the queue at each
step. Fix some assertions that no longer hold true as fallout.

* repo: Download: restore some semblance of previous behavior

When archival was made async, the GET endpoint was only useful if a previous
POST had initiated the download. This commit restores the previous behavior,
to an extent; we'll now submit the archive request there and return a
"202 Accepted" to indicate that it's processing if we didn't manage to
complete the request within ~2 seconds of submission.

This lets a client directly GET the archive, and gives them some indication
that they may attempt to GET it again at a later time.

* archiver: tests: simplify a bit further

We don't need to risk failure and use time.ParseDuration to get 2 *
time.Second.

else if isn't really necessary if the conditions are simple enough and lead
to the same result.

* archiver: tests: resolve potential source of flakiness

Increase all timeouts to 10 seconds; these aren't hard-coded sleeps, so
there's no guarantee we'll actually take that long. If we need longer to
not have a false-positive, then so be it.

While here, various assert.{Not,}Equal arguments are flipped around so that
the wording in error output reflects reality, where the expected argument is
second and actual third.

* archiver: setup infrastructure for notifying consumers of completion

This API will *not* allow consumers to subscribe to specific requests being
completed, just *any* request being completed. The caller is responsible for
determining if their request is satisfied and waiting again if needed.

* repo: archive: make GET endpoint synchronous again

If the request isn't complete, this endpoint will now submit the request and
wait for completion using the new API. This may still be susceptible to
timeouts for larger repos, but other endpoints now exist that the web
interface will use to negotiate its way through larger archive processes.

* archiver: tests: amend test to include WaitForCompletion()

This is a trivial one, so go ahead and include it.

* archiver: tests: fix test by calling NewContext()

The mutex is otherwise uninitialized, so we need to ensure that we're
actually initializing it if we plan to test it.

* archiver: tests: integrate new WaitForCompletion a little better

We can use this to wait for archives to come in, rather than spinning and
hoping with a timeout.

* archiver: tests: combine numQueued declaration with next-instruction assignment

* routers: repo: reap unused archiving flag from DownloadStatus()

This had some planned usage before, indicating whether this request
initiated the archival process or not. After several rounds of refactoring,
this use was deemed not necessary for much of anything and got boiled down
to !complete in all cases.

* services: archiver: restructure to use a channel

We now offer two forms of waiting for a request:
- WaitForCompletion: wait for completion with no timeout
- TimedWaitForCompletion: wait for completion with timeout

In both cases, we wait for the given request's cchan to close; in the latter
case, we do so with the caller-provided timeout. This completely removes the
need for busy-wait loops in Download/InitiateDownload, as it's fairly clean
to wait on a channel with timeout.

* services: archiver: use defer to unlock now that we can

This previously carried the lock into the goroutine, but an intermediate
step just added the request to archiveInProgress outside of the new
goroutine and removed the need for the goroutine to start out with it.

* Revert "archiver: tests: combine numQueued declaration with next-instruction assignment"

This reverts commit bcc5214023.

Revert "archiver: tests: integrate new WaitForCompletion a little better"

This reverts commit 9fc8bedb56.

Revert "archiver: tests: fix test by calling NewContext()"

This reverts commit 709c35685e.

Revert "archiver: tests: amend test to include WaitForCompletion()"

This reverts commit 75261f56bc.

* archiver: tests: first attempt at WaitForCompletion() tests

* archiver: tests: slight improvement, less busy-loop

Just wait for the requests to complete in order, instead of busy-waiting
with a timeout.  This is slightly less fragile.

While here, reverse the arguments of a nearby assert.Equal() so that
expected/actual are correct in any test output.

* archiver: address lint nits

* services: archiver: only close the channel once

* services: archiver: use a struct{} for the wait channel

This makes it obvious that the channel is only being used as a signal,
rather than anything useful being piped through it.

* archiver: tests: fix expectations

Move the close of the channel into doArchive() itself; notably, before these
goroutines move on to waiting on the Release cond.

The tests are adjusted to reflect that we can't WaitForCompletion() after
they've already completed, as WaitForCompletion() doesn't indicate that
they've been released from the queue yet.

* archiver: tests: set cchan to nil for comparison

* archiver: move ctx.Error's back into the route handlers

We shouldn't be setting this in a service, we should just be validating the
request that we were handed.

* services: archiver: use regex to match a hash

This makes sure we don't try and use refName as a hash when it's clearly not
one, e.g. heads/pull/foo.

* routers: repo: remove the weird /archive/status endpoint

We don't need to do this anymore, we can just continue POSTing to the
archive/* endpoint until we're told the download's complete. This avoids a
potential naming conflict, where a ref could start with "status/"

* archiver: tests: bump reasonable timeout to 15s

* archiver: tests: actually release timedReq

* archiver: tests: run through inFlight instead of manually checking

While we're here, add a test for manually re-processing an archive that's
already been complete. Re-open the channel and mark it incomplete, so that
doArchive can just mark it complete again.

* initArchiveLinks: prevent default behavior from clicking

* archiver: alias gitea's context, golang context import pending

* archiver: simplify logic, just reconstruct slices

While the previous logic was perhaps slightly more efficient, the
new variant's readability is much improved.

* archiver: don't block shutdown on waiting for archive

The technique established launches a goroutine to do the wait,
which will close a wait channel upon termination. For the timeout
case, we also send back a value indicating whether the timeout was
hit or not.

The timeouts are expected to be relatively small, but still a multi-
second delay to shutdown due to this could be unfortunate.

* archiver: simplify shutdown logic

We can just grab the shutdown channel from the graceful manager instead of
constructing a channel to halt the caller and/or pass a result back.

* Style issues

* Fix mis-merge

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-11-07 22:27:28 +02:00
John Olheiser
b5e974c8a5
Delete tag API (#13358)
* Delete tag API

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Wording

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add conflict response and fix API tests

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix other test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-10-30 20:56:34 -05:00
6543
48a80096bb
Update heatmap fixtures to restore tests (#13224)
* hotfix

* update ...
2020-10-20 17:39:25 -05:00
a1012112796
b9850375fc
Add review request api (#11355)
* Add review request api

* add : POST /repos/{owner}/{repo}/pulls/{index}/requested_reviewers
* Remove : DELET /repos/{owner}/{repo}/pulls/{index}/requested_reviewers
* fix some request review bug
* block delet request review by models/DeleteReview()

Signed-off-by: a1012112796 <1012112796@qq.com>

* make fmt

* fix bug

* fix test code

* fix typo

* Apply suggestion from code review @jonasfranz

* fix swagger ref

* fix typo

Co-authored-by: Lauris BH <lauris@nix.lv>

* fix comment

* Change response message

* chang response so some simplfy

* Add ErrIllLegalReviewRequest
fix some nits

* make fmt

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* * Add team support
* fix test
* fix an known bug

* fix nit

* fix test

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* update get api and add test

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: zeripath <art27@cantab.net>
2020-10-20 14:18:25 -04:00
赵智超
8be3e439c2
Add team support for review request (#12039)
Add team support for review request

Block #11355

Signed-off-by: a1012112796 <1012112796@qq.com>
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2020-10-12 20:55:13 +01:00
zeripath
5c0697ad1e
Use argon as default password hash algorithm (#12688)
* Restrict TLS connections to 1.2 minimum

* Set Argon2 as the default KDF

* Fix user.yml

* Remove TLS minversion changes

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add migration as per @techknowlogick

Signed-off-by: Andrew Thornton <art27@cantab.net>

* set the password algo in the fixtures

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Remove the v148 migration - it needs recreate table to change the defaults

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Nadim Kobeissi <nadim@symbolic.software>
2020-09-03 14:58:31 -04:00
Lanre Adelowo
4027c5dd7c
Kanban board (#8346)
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: jaqra <48099350+jaqra@users.noreply.github.com>
Co-authored-by: Kerry <flatline-studios@users.noreply.github.com>
Co-authored-by: Jaqra <jaqra@hotmail.com>
Co-authored-by: Kyle Evans <kevans91@users.noreply.github.com>
Co-authored-by: Tsakiridis Ilias <TsakiDev@users.noreply.github.com>
Co-authored-by: Ilias Tsakiridis <ilias.tsakiridis@outlook.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-08-16 23:07:38 -04:00
6543
c97494a4f4
API: Add pull review endpoints (#11224)
* API: Added pull review read only endpoints

* Update Structs, move Conversion, Refactor

* refactor

* lint & co

* fix lint + refactor

* add new Review state, rm unessesary, refacotr loadAttributes, convert patch to diff

* add DeletePullReview

* add paggination

* draft1: Create & submit review

* fix lint

* fix lint

* impruve test

* DONT use GhostUser for loadReviewer

* expose comments_count of a PullReview

* infent GetCodeCommentsCount()

* fixes

* fix+impruve

* some nits

* Handle Ghosts 👻

* add TEST for GET apis

* complete TESTS

* add HTMLURL to PullReview responce

* code format as per @lafriks

* update swagger definition

* Update routers/api/v1/repo/pull_review.go

Co-authored-by: David Svantesson <davidsvantesson@gmail.com>

* add comments

Co-authored-by: Thomas Berger <loki@lokis-chaos.de>
Co-authored-by: David Svantesson <davidsvantesson@gmail.com>
2020-05-02 03:20:51 +03:00
6543
bfda0f3864
[API] ListIssues add filter for milestones (#10148)
* Refactor Issue Filter Func

* ListIssues add filter for milestones

* as per @lafriks

* documentation ...
2020-04-30 01:15:39 -03:00
mrsdizzie
8d99ee2773
Add Organization Wide Labels (#10814)
* Add organization wide labels

Implement organization wide labels similar to organization wide
webhooks. This lets you create individual labels for organizations that can be used
for all repos under that organization (so being able to reuse the same
label across multiple repos).

This makes it possible for small organizations with many repos to use
labels effectively.

Fixes #7406

* Add migration

* remove comments

* fix tests

* Update options/locale/locale_en-US.ini

Removed unused translation string

* show org labels in issue search label filter

* Use more clear var name

* rename migration after merge from master

* comment typo

* update migration again after rebase with master

* check for orgID <=0 per guillep2k review

* fmt

* Apply suggestions from code review

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* remove unused code

* Make sure RepoID is 0 when searching orgID per code review

* more changes/code review requests

* More descriptive translation var per code review

* func description/delete comment when issue label deleted instead of hiding it

* remove comment

* only use issues in that repo when calculating number of open issues for org label on repo label page

* Add integration test for IssuesSearch API with labels

* remove unused function

* Update models/issue_label.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Use subquery in GetLabelIDsInReposByNames

* Fix tests to use correct orgID

* fix more tests

* IssuesSearch api now uses new BuildLabelNamesIssueIDsCondition. Add a few more tests as well

* update comment for clarity

* Revert previous code change now that we can use the new BuildLabelNamesIssueIDsCondition

* Don't sort repos by date in IssuesSearch API

After much debugging I've found a strange issue where in some cases MySQL will return a different result than other enigines if a query is sorted by a null collumn. For example with our integration test data where we don't set updated_unix in repository fixtures:

SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` ORDER BY updated_unix DESC LIMIT 15 OFFSET 45

Returns different results for MySQL than other engines. However, the similar query:

SELECT `id`, `owner_id`, `owner_name`, `lower_name`, `name`, `description`, `website`, `original_service_type`, `original_url`, `default_branch`, `num_watches`, `num_stars`, `num_forks`, `num_issues`, `num_closed_issues`, `num_pulls`, `num_closed_pulls`, `num_milestones`, `num_closed_milestones`, `is_private`, `is_empty`, `is_archived`, `is_mirror`, `status`, `is_fork`, `fork_id`, `is_template`, `template_id`, `size`, `is_fsck_enabled`, `close_issues_via_commit_in_any_branch`, `topics`, `avatar`, `created_unix`, `updated_unix` FROM `repository` ORDER BY updated_unix DESC LIMIT 15 OFFSET 30

Returns the same results.

This causes integration tests to fail on MySQL in certain cases but would never show up in a real installation. Since this API call always returns issues based on the optionally provided repo_priority_id or the issueID itself, there is no change to results by changing the repo sorting method used to get ids earlier in the function.

* linter is back!

* code review

* remove now unused option

* Fix newline at end of files

* more unused code

* update to master

* check for matching ids before query

* Update models/issue_label.go

Co-Authored-By: 6543 <6543@obermui.de>

* Update models/issue_label.go

* update comments

* Update routers/org/setting.go

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: 6543 <6543@obermui.de>
2020-04-01 01:14:46 -03:00
guillep2k
62a1322cf9
Fix "access" fixtures and tests (#10247)
* Add fixture gen tool and fix "access" test

* Close file before exiting

* Add missing repo_unit for repo id: 5

* Fix count on TestAPIOrgRepos

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-02-15 12:29:06 +08:00
Lauris BH
a1d796f521
Index code and stats only for non-empty repositories (#10251)
Fix test and switch to unique queue

Fix MySQL support when deleting old statistics
2020-02-14 13:42:30 +01:00
6543
36943e56d6 Add "Update Branch" button to Pull Requests (#9784)
* add Divergence

* add Update Button

* first working version

* re-use code

* split raw merge commands and db-change functions (notify, cache, ...)

* use rawMerge (remove redundant code)

* own function to get Diverging of PRs

* use FlashError

* correct Error Msg

* hook is triggerd ... so remove comment

* add "branch2" to "user2/repo1" because it unit-test "TestPullView_ReviewerMissed" use it but dont exist jet :/

* move GetPerm to IsUserAllowedToUpdate

* add Flash Success MSG

* imprufe code
- remove useless js chage

* fix-lint

* TEST: add PullRequest ID:5
Repo: user2/repo1
Base: branch1
Head: pr-to-update

* correct comments

* make PR5 outdated

* fix Tests

* WIP: add pull update test

* update revs

* update locales

* working TEST

* update UI

* misspell

* change style

* add 1s delay so rev exist

* move row up (before merge row)

* fix lint nit

* UI remove divider

* Update style

* nits

* do it right

* introduce IsSameRepo

* remove useless check

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-17 08:03:40 +02:00
Manush Dodunekov
1751d5fcf2 Restricted users (#6274)
* Restricted users (#4334): initial implementation

* Add User.IsRestricted & UI to edit it

* Pass user object instead of user id to places where IsRestricted flag matters

* Restricted users: maintain access rows for all referenced repos (incl public)

* Take logged in user & IsRestricted flag into account in org/repo listings, searches and accesses

* Add basic repo access tests for restricted users

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Mention restricted users in the faq

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Revert unnecessary change `.isUserPartOfOrg` -> `.IsUserPartOfOrg`

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Remove unnecessary `org.IsOrganization()` call

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* Revert to an `int64` keyed `accessMap`

* Add type `userAccess`
* Add convenience func updateUserAccess()
* Turn accessMap into a `map[int64]userAccess`

Signed-off-by: Manush Dodunekov <manush@stendahls.se>

* or even better: `map[int64]*userAccess`

* updateUserAccess(): use tighter syntax as suggested by lafriks

* even tighter

* Avoid extra loop

* Don't disclose limited orgs to unauthenticated users

* Don't assume block only applies to orgs

* Use an array of `VisibleType` for filtering

* fix yet another thinko

* Ok - no need for u

* Revert "Ok - no need for u"

This reverts commit 5c3e886aab.

Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-13 18:33:46 +01:00
6543
0b3aaa6196 [API] Add "before" query to ListIssueComments and ListRepoIssue… (#9685)
* add "before" query to ListIssueComments and ListRepoIssueComments

* Add TEST

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
2020-01-13 17:02:24 +01:00
Lunny Xiao
5765212c6d
Add owner_name column for table repository for maintaince reason (#9717)
* Add owner_name column for table repository for maintaince reason

* refactor

* Fix tests

* fix test

* fix bug when fork repository

Co-authored-by: zeripath <art27@cantab.net>
2020-01-12 17:36:21 +08:00
6543
6baa5d7588 [API] Add notification endpoint (#9488)
* [API] Add notification endpoints

 * add func GetNotifications(opts FindNotificationOptions)
 * add func (n *Notification) APIFormat()
 * add func (nl NotificationList) APIFormat()
 * add func (n *Notification) APIURL()
 * add func (nl NotificationList) APIFormat()
 * add LoadAttributes functions (loadRepo, loadIssue, loadComment, loadUser)
 * add func (c *Comment) APIURL()
 * add func (issue *Issue) GetLastComment()
 * add endpoint GET /notifications
 * add endpoint PUT /notifications
 * add endpoint GET /repos/{owner}/{repo}/notifications
 * add endpoint PUT /repos/{owner}/{repo}/notifications
 * add endpoint GET /notifications/threads/{id}
 * add endpoint PATCH /notifications/threads/{id}

* Add TEST

* code format

* code format
2020-01-09 11:56:32 +00:00
Antoine GIRARD
8b24073713 Only serve attachments when linked to issue/release and if accessible by user (#9340)
* test: add current attachement responses

* refactor: check if attachement is linked and accessible by user

* chore: clean TODO

* fix: typo attachement -> attachment

* revert un-needed go.sum change

* refactor: move models logic to models

* fix TestCreateIssueAttachment which was wrongly successful

* fix unit tests with unittype added

* fix unit tests with changes

* use a valid uuid format for pgsql int. test

* test: add unit test TestLinkedRepository

* refactor: allow uploader to access unlinked attachement

* add missing blank line

* refactor: move to a separate function repo.GetAttachment

* typo

* test: remove err test return

* refactor: use repo perm for access checking generally + 404 for all reject
2020-01-05 01:20:08 +02:00