Commit Graph
100 Commits
Author SHA1 Message Date
zeripathand6543 02de43236a Gitlab Migrator: dont ignore reactions of last request (#16903) (#16913)
Backport #16903

Fix bug related to early breaking when migrating reactions.

Co-authored-by: 6543 <6543@obermui.de>
2021-09-01 17:52:38 -04:00
zeripath f949f9e9c5 Correctly return the number of Repositories for Organizations (#16807) (#16911)
Backport #16807

Calculate and return the number of Repositories on the dashboard
Organization list.

This PR restores some of the logic that was removed in #14032 to
calculate the number of repos on the dashboard orgs list.

Fix #16648
Replaces #16799

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-09-01 19:20:35 +02:00
zeripath 58f4a4114e Simplify split diff view generation and remove JS dependency (#16775) (#16863)
Backport #16775

Gitea has relied on some slow JS code to match up added and deleted lines on the
diff pages. This can cause a considerable slow down on large diff pages.

This PR makes a small change meaning that the matching up can occur much more simply.

Partial fix #1351

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-30 16:43:06 +01:00
zeripath b31307c41c Ensure that the default visibility is set on the user create page (#16845) (#16862)
Backport #16845

Set the default visibility on the user create page.

Fix #16840

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-29 17:33:13 +02:00
zeripathandLunny Xiao 1790f01dd9 Upgrade xorm to v1.2.2 (#16663) & Add test to ensure that dumping of login sources remains correct (#16847) (#16848)
* Upgrade xorm to v1.2.2 (#16663)

Backport #16663

Fix #16683

* Add test to ensure that dumping of login sources remains correct (#16847)

#16831 has occurred because of a missed regression. This PR adds a simple test to
try to prevent this occuring again.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-28 13:16:19 +02:00
zeripath 695e8ae81d Add primary_key to issue_index (#16813) (#16820)
Backport #16813

Make the group_id a primary key in issue_index. This already has an
unique index and therefore is a good candidate for becoming a primary
key.

This PR also changes all other uses of this table to add the group_id as
the primary key.

The migration v192 from #16813 has not been backported but Xorm will
work fine with non-primary keyed tables. If a user on 1.15 wishes to
have the correct schema sooner than 1.16 - they can use gitea doctor
recreate-table issue_index and gitea will recreate the table with the
primary key.

Fix #16802

Signed-off-by: Andrew Thornton art27@cantab.net
2021-08-25 18:10:15 -04:00
zeripath 2148b27bfa Prevent NPE on empty commit (#16812) (#16819)
Backport #16812

Handle completely empty commit as the first commit to a repository.

Fix #16668

Signed-off-by: Andrew Thornton art27@cantab.net
2021-08-25 13:22:48 +01:00
zeripathandtechknowlogick 63178b5654 Ensure that template compilation panics are sent to the logs (#16788) (#16792)
Backport #16788

Although panics within the rendering pipeline are caught and dealt with,
panics that occur before that starts are unprotected and will kill Gitea
without being sent to the logs.

This PR adds a basic recovery handler to catch panics that occur after
the logger is initialised and ensure that they're sent to the logger.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-23 19:50:04 -04:00
zeripathandtechknowlogick 2d15126de6 Remove spurious / from issues.opened_by (#16793)
Fix #16713

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-23 18:24:30 -04:00
zeripathandtechknowlogick 31e4e8205a Add missing return to handleSettingRemoteAddrError (#16794) (#16795)
Backport #16794

There is a missing return in handleSettingRemoteAddrError which means
that the error page for repo settings is duplicately rendered.

Fix #16771

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-23 18:09:28 -04:00
zeripathandjolheiser 37e4cdbbe6 Update only certmagic (#16790)
Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: jolheiser <john.olheiser@gmail.com>
2021-08-23 18:07:44 -04:00
zeripathandtechknowlogick 5ceff8fda2 Recreate Tables should Recreate indexes on MySQL (#16718) (#16739)
The MySQL indexes are not being renamed at the same time as RENAME table despite the
CASCADE. Therefore it is probably better to just recreate the indexes instead.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-20 22:47:37 -04:00
zeripath 25437672bf Restore functionality for pushing non-standard refs (Partial #16705) (#16706)
Partial Backport #16705

There was an inadvertent breaking change in #15629 meaning that notes refs and other
git extension refs will be automatically rejected.

This PR removes this incorrect forced rejection of non-standard refs.

Fix #16688

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-17 10:24:47 +02:00
zeripath d89029ebac Ensure empty lines are copiable and final new line too (#16678) (#16692)
Backport #16678

When files are highlighted the newline character needs to be added in a whitespace
compliant mode. Also ensure the final empty newline is rendered.

Fix #16434

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-15 05:39:23 +02:00
zeripath 46d62ad896 Fix direct creation of external users on admin page (#16613)
From #16612 it was noticed that when creating new external users directly it was not
possible to set their username correctly. This PR restores this ability.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-09 14:18:40 +02:00
zeripathandLunny Xiao 428d58f8da Restore compatibility with SQLServer 2008 R2 in migrations (#16638)
This fixes two problems with MSSQL:

* `ALTER TABLE DROP ... IF EXISTS ...` is only supported in SQL Server >16.

The `IF EXISTS` here is a belt-and-braces and does not need to be present. Therefore
can be dropped. Also stop attempting to drop the indexes as constraints as they're indexes!

* System tables like: `sys.indexes` should be lowercase not uppercase because of collation issues.

Fix #13615

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-08 13:32:07 +03:00
1a2256bf44 Changelog for 1.15.0-rc3 (#16621)
* Changelog for 1.15.0-rc3

 ## [1.15.0-rc3](https://github.com/go-gitea/gitea/releases/tag/v1.15.0-rc3) - 2021-08-04

* BREAKING
  * Upgrade to the latest version of golang-jwt and increase minimum go to 1.15 (#16590) (#16606)
* SECURITY
  * Upgrade to the latest version of golang-jwt and increase minimum go to 1.15 (#16590) (#16606)
  * Switch to maintained jwt lib (#16532) (#16533)
  * Correctly create of git-daemon-export-ok files (#16508) (#16514)
  * Don't show private user's repo in explore view (#16550) (#16554)
* API
  * Swagger AccessToken fixes (#16574) (#16597)
  * Set AllowedHeaders on API CORS handler (#16524) (#16618)
* BUGFIXES
  * Restore Accessibility for Dropdown (#16576) (#16617)
  * Pass down SignedUserName down to AccessLogger context (#16605) (#16616)
  * Fix table alignment in markdown (#16596) (#16602)
  * Fix 500 on first wiki page (#16586) (#16598)
  * Lock goth/gothic and Re-attempt OAuth2 registration on login if registration failed at startup (#16564) (#16570)
  * Upgrade levelqueue to v0.4.0 (#16560) (#16561)
  * Handle too long PR titles correctly (#16517) (#16549)
  * Fix data race in bleve indexer (#16474) (#16509)
  * Restore CORS on git smart http protocol (#16496) (#16506)
  * Fix race in log (#16490) (#16505)
  * Fix prepareWikiFileName to respect existing unescaped files (#16487) (#16498)
  * Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479) (#16480)
  * Update notification table with only latest data (#16445) (#16469)
  * Revert to use alpine 3.13 (#16451) (#16452)
  * Fix crash following ldap authentication update (#16447) (#16448)
  * Fix direct creation of external users on admin page (partial #16612) (#16613)

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update CHANGELOG.md

Co-authored-by: techknowlogick <techknowlogick@gitea.io>

* Update CHANGELOG.md

Co-authored-by: zeripath <art27@cantab.net>

* Update CHANGELOG.md

* Update CHANGELOG.md

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-08-06 16:08:13 -04:00
zeripathandtechknowlogick 619e6d6400 Frontport changelog for v1.14.6 (#16633)
* Frontport changelog for v1.14.6

Frontport #16619

Changelog frontported to below v1.15.0-rc1.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update config.yaml

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-06 01:17:44 +03:00
zeripathandtechknowlogick f438b6f33b Update node tar dependency to 6.1.6 (#16622) (#16623)
Backport #16622

Forcibly update dev dependency on tar to 6.1.6

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-05 00:40:25 -04:00
zeripath c47065cc29 Set AllowedHeaders on API CORS handler (#16524) (#16618)
Backport #16524

Set AllowedHeaders on API CORS handler and add missing Access-Control-Expose-Headers
to pull API.

Fix #16100

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-04 21:57:59 +02:00
zeripathandLunny Xiao 719e2f26d5 Pass down SignedUserName down to AccessLogger context (#16605) (#16616)
Backport #16605

Unfortunately when the AccessLogger was moved back before the contexters the
SignedUserName reporting was lost. This is due to Request.WithContext leading to a
shallow copy of the Request and the modules/context/Context being within that request.

This PR adds a new context variable of a string pointer which is set and handled
in the contexters.

Fix #16600

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-08-04 13:26:35 -04:00
zeripath 40687a2160 Restore #10096/#8638 and re-fix #15172 (#16576) (#16617)
Backport #16576

This PR restores the vendored and patched dropdow from #8638. It
however, it partially abandons the call to `click()` using instead the
default dropdown click calls instead. This prevents the issue of the
dropdown grabbing focus permanently however, this may have negative
effects on the effect of focus on the dropdowns.

Of note, the behaviour of the template selector dropdown on the repo
creation page is slightly odd - I don't believe that this odd behaviour
is caused by this PR but rather by the feed source for this. I suspect
that the dropdown should be adding a delete button to its selection.

Fix #15172
References: #7057

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-04 13:31:10 +01:00
zeripath f9120092c1 Fix table alignment in markdown (#16596) (#16602)
Backport #16596

Set the TableOptions in markdown to allow alignment of the tables to work correctly

Fix #15959

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-04 00:16:00 +01:00
zeripath a17edf446f Fix 500 on first wiki page (#16586) (#16598)
Backport #16586

There is a mistake in #16319 and #16487 which means that the first time
a wiki page is created a 500 is reported because the `master` branch is
not in existence in that wiki yet.

This PR simply checks for this error and returns not found.

Fix #16584

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-03 23:46:08 +01:00
zeripath ff8fadd2be Upgrade to the latest version of golang-jwt (#16590) (#16606)
* Upgrade to the latest version of golang-jwt.

Backport #16590

* Forcibly update the vendored versions too
* Update our minimal go lang version to 1.15 (differs from 1.16 in #16590)

Signed-off-by: Andrew Thornton <art27@cantab.net>

 ### ⚠️ BREAKING ⚠️

This PR raises the minimal version of go supported to 1.15 which will mean the end of support of 32-bit Mac and Mac OS versions before Sierra.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* update minimal go required

Signed-off-by: Andrew Thornton <art27@cantab.net>

* update config.yaml

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-08-03 22:22:52 +01:00
zeripathandtechknowlogick 5fe7c0ed7b Swagger AccessToken fixes (#16574) (#16597)
Backport #16574

There is a subtle problem with the Swagger definition for AccessTokens which causes
autogeneration of APIs for these endpoints to fail.

This PR corrects these errors.

Ref: https://github.com/zeripath/java-gitea-api/issues/4
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-08-02 20:24:47 +02:00
zeripath 903bdefb58 Lock goth/gothic and Re-attempt OAuth2 registration on login if registration failed at startup (#16570)
Backport #16564

This PR has two parts:

* Add locking to goth and gothic calls with a RWMutex

The goth and gothic calls are currently unlocked and thus are a cause of multiple potential races

* Reattempt OAuth2 registration on login if registration failed

If OAuth2 registration fails at startup we currently disable the login_source however an alternative approach could be to reattempt registration on login attempt.
    
Fix #16096

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-29 18:52:38 +01:00
zeripath 693275455e Handle too long PR titles correctly (#16517) (#16549)
Backport #16517

The CompareAndPullRequestPost handler for POST to /compare
incorrectly handles returning errors to the user. For a start
it does not set the necessary markers to switch SimpleMDE
but it also does not immediately return to the form.

This PR fixes this by setting the appropriate values, fixing
the templates and preventing the suggestion of a too long
title.

Fix #16507

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-26 17:49:48 +02:00
zeripathandMatti R 67942ac1a9 Switch to maintained jwt lib (#16532) (#16533)
Backport #16532

Co-authored-by: Matti R <matti@mdranta.net>
2021-07-24 11:13:50 -04:00
zeripathand6543 0b06b2019f Add basic edit ldap auth test & actually fix #16252 (#16465) (#16494)
Backport #16465

One of the reasons why #16447 was needed and why #16268 was needed in
the first place was because it appears that editing ldap configuration
doesn't get tested.

This PR therefore adds a basic test that will run the edit pipeline.

In doing so it's now clear that #16447 and #16268 aren't actually
solving #16252. It turns out that what actually happens is that is that
the bytes are actually double encoded.

This PR now changes the json unmarshal wrapper to handle this double
encode.

Fix #16252

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-07-22 17:24:21 +03:00
zeripath 057205a4b7 Restore creation of git-daemon-export-ok files (#16508) (#16514)
Backport #16508

Somewhere along the line the creation of git-daemon-export-ok
files disappeared but the updating of these files when
repo visibility changes remained. The problem is that the
current state will create files even when the org or user
is private.

This PR restores creation correctly.

Fix #15521

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-22 15:52:02 +02:00
zeripathand6543 1b6c0c6bdc Changelog for 1.15.0-rc2 (#16511)
* Changelog for 1.15.0-rc2

Results of `~/go/bin/changelog -m 1.15.0 --after 16422 generate`

We need to release RC2 as there are mulitple problems with alpine 3.14 related to
the seccomp issues on Docker <20.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update CHANGELOG.md

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-07-22 15:47:38 +03:00
zeripath 1f5011dff7 Restore CORS on git smart http protocol (#16496) (#16506)
Backport #16496

Unfortunately the chi changes have resulted in the CORS headers for the
git smart http protocol going missing.

This is mostly because the OPTIONS method is not being handled by
httpBase anymore.

This PR adds a GetOptions, PostOptions and Options methods to web
handler to allow OPTIONS method requests to still reach the httpBase
function.

Fix #16350
Close #16491

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-21 15:03:02 +01:00
zeripath cf9aeca508 Fix race in log (#16490) (#16505)
Backport #16490

A race has been detected in #1441 relating to getting log levels.

This PR protects the GetLevel and GetStacktraceLevel calls with a RW mutex.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-21 20:19:36 +08:00
zeripath 0c3467ffb7 Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479) (#16480)
* Make cancel from CatFileBatch and CatFileBatchCheck wait for the command to end (#16479)

Fix #16427 (again!)

* handle sharing violation error code

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-20 23:52:19 +02:00
zeripath 7f6019e492 Update notification table with only latest data (#16445) (#16469)
When marking notifications read the results may be returned out of order
or be delayed.  This PR sends a sequence number to gitea so that the
browser can ensure that only the results of the latest notification
change are shown.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-17 19:05:59 +02:00
zeripath 0f11c5f592 Retry rename on lock induced failures (re-fix) (#16461) (#16462)
Backport #16461

Unfortunately #16435 asserts the wrong error and should use
os.LinkError not os.PathError.

Fix #16439

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-16 13:22:44 -04:00
zeripathandLauris BH bae0e1d773 Frontport v1.14.5 (#16453)
Frontport the changelog from v1.14.5

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-16 11:17:16 -04:00
zeripath e5fde7ef00 Fix crash following ldap authentication update (#16447) (#16448)
Backport #16447

Unfortunately #16268 contained a terrible error, whereby there was a double
indirection taken when unmarshalling the source data. This fatally breaks
authentication configuration reading.

Fix #16342

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-15 20:17:44 -04:00
zeripath 6243638c11 Update documentation to reflect #15219 (#16442) (#16444)
Backport #16442

The move to render custom/public as within /assets in #15219 missed updating
several documentation pages.

This PR updates this documentation.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-15 22:28:07 +02:00
zeripathandtechknowlogick 33a8eec33e Retry rename on lock induced failures (#16435)
* Retry rename on lock induced failures

Due to external locking on Windows it is possible for an
os.Rename to fail if the files or directories are being
used elsewhere.

This PR simply suggests retrying the rename again similar
to how we handle the os.Remove problems.

Fix #16427

Signed-off-by: Andrew Thornton <art27@cantab.net>

* resolve CI fail

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-07-15 11:46:07 -04:00
3dcb3e9073 Second attempt at preventing zombies (#16326)
* Second attempt at preventing zombies

* Ensure that the pipes are closed in ssh.go
* Ensure that a cancellable context is passed up in cmd/* http requests
* Make cmd.fail return properly so defers are obeyed
* Ensure that something is sent to stdout in case of blocks here

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint 2

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint 3

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fixup

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Apply suggestions from code review

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-07-14 10:43:13 -04:00
zeripath b82293270c Add option to provide signature for a token to verify key ownership (#14054)
* Add option to provide signed token to verify key ownership

Currently we will only allow a key to be matched to a user if it matches
an activated email address. This PR provides a different mechanism - if
the user provides a signature for automatically generated token (based
on the timestamp, user creation time, user ID, username and primary
email.

* Ensure verified keys can act for all active emails for the user

* Add code to mark keys as verified

* Slight UI adjustments

* Slight UI adjustments 2

* Simplify signature verification slightly

* fix postgres test

* add api routes

* handle swapped primary-keys

* Verify the no-reply address for verified keys

* Only add email addresses that are activated to keys

* Fix committer shortcut properly

* Restructure gpg_keys.go

* Use common Verification Token code

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-13 15:28:07 +02:00
zeripath 2f725cbc9e Add LRU mem cache implementation (#16226)
The current default memory cache implementation is unbounded in size and number of
objects cached. This is hardly ideal.

This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea.
The cache is limited by the number of objects stored in the cache (rather than size)
for simplicity. The default number of objects is 50000 - which is perhaps too small
as most of our objects cached are going to be much less than 1kB.

It may be worth considering using a different LRU implementation that actively limits
sizes or avoids GC - however, this is just a beginning implementation.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-10 23:54:15 +02:00
zeripath d06f9ce274 Redirect on bad CSRF instead of presenting bad page (#14937)
The current CSRF handler is a bit harsh with bad CSRF tokens on webpages
I think we can be a little kinder and redirect to base page with a flash error

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-08 15:57:24 +02:00
zeripathand6543 32fd11395b Fix relative links in postprocessed images (#16334)
If a pre-post-processed file contains relative img tags these need to be updated
and joined correctly with the prefix. Finally, the node attributes need to be updated.

Fix #16308

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-07-04 10:26:04 +01:00
zeripath 62c278e4ab Fix modified files list in webhooks when there is a space (#16288)
* Fix modified files list in webhooks when there is a space

There is an unfortunate bug with GetCommitFileStatus where files with
spaces are misparsed and split at the space.

There is a second bug because modern gits detect renames meaning that
this function no longer works correctly.

There is a third bug in that merge commits don't have their modified
files detected correctly.

Fix #15865


Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-07-02 21:23:37 +02:00
zeripath 9979983283 Update Go-Git to take advantage of LargeObjectThreshold (#16316)
Following the merging of https://github.com/go-git/go-git/pull/330 we
can now add a setting to avoid go-git reading and caching large objects.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:58:45 +02:00
zeripath 302e8b6d02 Prevent zombie processes (#16314)
Unfortunately go doesn't always ensure that execd processes are completely
waited for. On linux this means that zombie processes can occur.

This PR ensures that these are waited for by using signal notifier in serv and
passing a context elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 22:07:23 +02:00
zeripath 365c4e9316 Add button to delete undeleted repositories from failed migrations (#16197)
This PR adds a button to delete failed repositories if there has been a
failure during migration and for whatever reason the repository doesn't
get deleted automatically.

Fix #16154

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 21:14:53 +02:00
zeripath 71c7d0a993 Fix default push instructions on empty repos (#16302)
* Fix default push instructions on empty repos

Use script block like in `repo/clone_buttons.tmpl` to set default instructions
for pushing to empty repos.

Fix #16295

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-30 20:40:51 +02:00
add74fb368 Fix panic in recursive cache (#16298)
There is a bug with last commit cache recursive cache where the last
commit information that refers to the current tree itself will cause a
panic due to its path ("") not being included in the expected tree entry
paths.

This PR fixes this by skipping the missing entry.

Fix #16290

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-29 16:12:43 -04:00
zeripath 2f3d3ac20b Update fail2ban documentation (#16286)
Following the merge of #16278 we need to update the fail2ban
documentation to take account of the availability of the new
sshConnectionFailed failed authentication attempt log message.

Also add a deprecation notice regarding the previous publicKeyHandler
messages, as these may be a source of false positives.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-28 19:26:40 -04:00
zeripath f825f20d49 Upgrade Gliderlabs SSH to 0.3.3 and add FailedConnectionCallback (#16278)
* Upgrade Gliderlabs SSH to 0.3.3 and add FailedConnectionCallback

Following the merging of https://github.com/gliderlabs/ssh/pull/143 we
can now report connections to the ssh server that have failed before
public key exchange has completed using the standard fail2ban message.

This PR updates Gliderlabs SSH and adds a callback that will provide this
logging.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* move the callback to its own function to make the logging appear little nicer

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-28 13:05:27 -04:00
zeripath 9a0cd3af3b Handle misencoding of login_source cfg in mssql (#16268)
* Handle misencoding of login_source cfg in mssql

Unfortunately due a bug in xorm (see https://gitea.com/xorm/xorm/pulls/1957) updating
loginsources on MSSQL causes them to become corrupted. (#16252)

Whilst waiting for the referenced PR to be merged and to handle the corrupted
loginsources correctly we need to add a wrapper to the `FromDB()` methods to look
for and ignore the misplaced BOMs that have been added.

Fix #16252

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update models/login_source.go
2021-06-27 16:21:16 -04:00
zeripathandtechknowlogick 35f37a3625 Add --quiet and --verbose to gitea web to control initial logging (#16260)
One of the repeatedly reported issues has been that gitea produces too much console
logging during set up even if the console logger is turned off.

Fundamentally this is due to some otherwise very helpful logging that has to occur
before logging is set up. This has come to a head with the merging of #16243 where
otherwise potentially helpful Trace logging in the git module now appears on the
console.

This PR proposes three things:

1. Change the initial default logger to Info not Trace.
2. Change the logging for the AppPath things to Info in recompense.
3. Add two new command line options to gitea web: --quiet and --verbose

`gitea web -q` or `gitea web --quiet` will only log Fatal level initially.
`gitea web -verbose` will log at Trace.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-26 20:56:58 -04:00
zeripath 19ac575d57 Limit stdout tracelog to actual stdout (#16258)
Related #16243

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-26 13:47:56 +01:00
zeripathandtechknowlogick 62a4879e84 Improve efficiency in FindRenderizableReferenceNumeric and getReferences (#16251)
* Fuzzer finds an NPE due to incorrect URLPrefix

The Fuzzer is running on a non-repo urlprefix which is incorrect for RenderRaw


* Make FindRenderizableReferenceNumeric and getReferences more efficient

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-26 01:11:33 +02:00
zeripathand6543 1a1ce9b721 Fuzzer finds an NPE due to incorrect URLPrefix (#16249)
The Fuzzer is running on a non-repo urlprefix which is incorrect for RenderRaw

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-25 14:14:49 -04:00
zeripath d13a0e621b Do not show No match found for tribute (#16231)
Tribute.js will show an untranslated no match found if no emoji or mentions.

Further the mentions should really require a preceding space.

This PR fixes both of these.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-23 19:02:23 -04:00
zeripath 5f2ef17fdb Don't WARN log UserNotExist errors on ExternalUserLogin failure (#16238)
Instead log these at debug - with warn logging for other errors.

Fix #16235

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-23 17:41:39 -04:00
zeripath d55b5eb0d3 Use html.Parse rather than html.ParseFragment (#16223)
* Use html.Parse rather than html.ParseFragment
  There have been a few issues with html.ParseFragment - just use html.Parse instead.

* Skip document node

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-22 00:12:22 +02:00
zeripath 6a083a7234 Update documentation for Implicit TLS (#16220)
As per RFC 8314, it is now recommended to prefer TLS over STARTTLS.

Fix #16160

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-21 14:34:37 -04:00
zeripathand6543 681e81babd reqOrgMembership calls need to be preceded by reqToken (#16198)
ReqOrgMembership calls need to be preceded by reqToken

Fix #16192

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 14:01:44 +01:00
zeripathand6543 196593e2e9 More efficiently parse shas for shaPostProcessor (#16101)
* More efficiently parse shas for shaPostProcessor

The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.

This PR provides functions for both backends which are much faster to use.

Fix #16092

* Add ShaExistCache to RenderContext

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-06-21 00:39:12 +02:00
23358bc55d Use git log name-status in get last commit (#16059)
* Improve get last commit using git log --name-status

git log --name-status -c provides information about the diff between a
commit and its parents. Using this and adjusting the algorithm to use
the first change to a path allows for a much faster generation of commit
info.

There is a subtle change in the results generated but this will cause
the results to more closely match those from elsewhere.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-06-21 01:00:46 +03:00
zeripath 0db1048c3a Run processors on whole of text (#16155)
There is an inefficiency in the design of our processors which means that Emoji
and other processors run in order n^2 time.

This PR forces the processors to process the entirety of text node before passing
back up. The fundamental inefficiency remains but it should be significantly
ameliorated.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-17 11:35:05 +01:00
zeripath 1ec9e906dc Ensure settings for Service and Mailer are read on the install page (#15943)
* Ensure settings for Service and Mailer are read on the install page

NewContext does not set the mailer or service settings so add
a new function that will run this.

Fix #15894

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 19:32:57 -04:00
zeripathandtechknowlogick ffbf35b7e9 Clean-up the settings hierarchy for issue_indexer queue (#16001)
There are a couple of settings in `[indexer]` relating to the `issue_indexer` queue
which override settings in unpredictable ways. This PR adjusts this hierarchy and makes
explicit that these settings are deprecated.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-16 18:19:20 -04:00
zeripathand6543 5f4522cd1f Fix missing discard in repo_language_stats (#16030)
Set the missing discard(1) in repo_language_stats.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:16:47 -04:00
zeripathand6543 6d69df2804 Add Status Updates whilst Gitea migrations are occurring (#15076)
* Add migrating message

Signed-off-by: Andrew Thornton <art27@cantab.net>

* simplify messenger

Signed-off-by: Andrew Thornton <art27@cantab.net>

* make messenger an interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* rename

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prepare for merge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per tech

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-16 18:02:24 -04:00
zeripath f4d3bf7867 issue-keyword class is being incorrectly stripped off spans (#16163)
Bluemonday sanitizer regexp rules are not additive, so the addition of the icons,
emojis and chroma syntax policy has led to this being stripped.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 03:02:03 +02:00
zeripath b8e4ce754e Only check access tokens if they are likely to be tokens (#16164)
* Only check access tokens if they are likely to be tokens

Gitea will currently check every if every password is an access token even though
most passwords are not and cannot be access tokens.

By creation access tokens are 40 byte hexadecimal strings therefore only these should
be checked.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-16 00:29:25 +02:00
zeripathandtechknowlogick 95352e6bd5 Make command in authorized keys a template (#16003)
Fix #15595
Replaces #15978

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-14 21:15:12 -04:00
zeripathandtechknowlogick 15fbf23d13 v180 migration should be standalone (#16151)
Unfortunately the v180 migration picked up a few non-standalone dependencies. This PR
forcibly copies the important parts back into the migration.

Fix #16150

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-06-14 15:07:49 -04:00
zeripathand6543 daa5a23548 Set self-adjusting deadline for connection writing (#16068)
* Set self-adjusting deadline for connection writing

In #16055 it appears that the simple 5s deadline doesn't work for large
file writes. Now we can't - or at least shouldn't just set no deadline
as go will happily let these connections block indefinitely. However,
what seems reasonable is to set some minimum rate we expect for writing.

This PR suggests the following algorithm:

* Every write has a minimum timeout of 5s (adjustable at compile time.)
* If there has been a previous write - then consider its previous
deadline, add half of the minimum timeout + 2s per kb about to written.
* If that new deadline is after the minimum timeout use that.

Fix #16055

* Linearly increase timeout

* Make PerWriteTimeout, PerWritePerKbTimeouts configurable

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-11 00:25:25 +03:00
zeripath e03a91a48e Remove spurious AppSubUrl in serviceworker request. (#16047)
There is another spurious AppSubUrl placement in the serviceworker registration.
This PR removes it.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-08 18:46:13 +01:00
zeripathand6543 51775f65bc Make commit info cancelable (#16032)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

* pass context into get-last-commit

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Make commit_info cancellable

Signed-off-by: Andrew Thornton <art27@cantab.net>

* use context as context

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-06 19:44:58 -04:00
zeripath fe18a85f54 Fix panic (#16072)
There is an incorrect casting in the wrapped queue.

Fix #16071

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-05 15:23:22 +01:00
zeripath 9fdda90085 Fix case change in ownernames (#16045)
If you change the case of a username the change needs to be propagated to their
repositories.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-06-02 13:03:59 +01:00
zeripathand6543 8947422781 Fix bug due to missing MaxStartups and MaxSessions (#16046)
Unforunately #16009 makes these settings mandatory. This PR uses the same technique
as used for the certificates to make these settings non-mandatory.

Fix #16044

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-06-01 15:55:17 -04:00
zeripath cbf30830d2 Add missing SameSite settings for the i_like_gitea cookie (#16037)
The i_like_gitea cookie appears to be missing the SameSite settings. I think they
were present at some point but may have been removed in a merge.

This PR ensures that they are set.

Fix #15972

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-31 14:22:36 -04:00
zeripathandtechknowlogick 7a484c0788 Hide mirror passwords on repo settings page (#16022)
This PR simply hides mirror passwords from being displayed on the repo settings page.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-31 11:46:20 +01:00
256b1a3561 Fix bug in reverse proxy (#16026)
Unfortunately go panics you try to cast a nil interface{} as another primitive
therefore you need to check interfaces are not nil before casting.

Fix #16025

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-31 02:54:16 -04:00
zeripathandLunny Xiao 3183a465d7 Make modules/context.Context a context.Context (#16031)
* Make modules/context.Context a context.Context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Simplify context calls

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Set the base context for requests to the HammerContext

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-31 02:18:11 -04:00
zeripath 36dce0e457 Close the dataRC reader sooner (#16023)
Fix #15932

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-30 17:38:44 +01:00
zeripath 6d6a65cf5c Allow Token/Basic auth on raw paths (#15987)
It appears that people have been using token authentication to navigate to raw paths
and recent changes have broken this. Whilst ideally these paths would not be being used
like this - it was not the intention to be a breaking change.

This PR restores access to these paths.

Fix #13772

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-27 19:46:11 +01:00
zeripathandtechknowlogick 6d39053711 Fix setting of SameSite on cookies (#15989)
Fix #15972

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-05-26 21:01:07 -04:00
zeripathandJohn Olheiser c1a80b7d6a Use filepath.ToSlash and Join in indexer defaults and queues (#15971)
As revealed by #15964 there is inconsistent use of filepath Join and path Join
for these directories. The best thing to do is to use filepath.Join but then ToSlash
them for consistency.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2021-05-25 22:50:35 -04:00
zeripath 3aaf64885f Change default queue settings to be low go-routines (#15964)
This PR suggests a change to the default configuration for queues:

* Use a common DATADIR for the queues
* Set starting workers to 0 and make boost a single worker

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-24 02:23:55 +03:00
zeripathandsilverwind 17be645498 Encrypt LDAP bind password in db with SECRET_KEY (#15547)
* Encrypt LDAP bind password in db with SECRET_KEY

The LDAP source bind password are currently stored in plaintext in the db
This PR simply encrypts them with the setting.SECRET_KEY.

Fix #15460

Signed-off-by: Andrew Thornton <art27@cantab.net>

* remove ui warning regarding unencrypted password

Co-authored-by: silverwind <me@silverwind.io>
2021-05-20 09:29:57 +01:00
zeripathand6543 0e56e9c9d9 Restore token authentication for git http when 2FA active (#15915)
There was a small regression in #15303 whereby token auth
with 2FA active would be disallowed.

This PR fixes this.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-05-18 22:30:33 -04:00
zeripathandLunny Xiao 3a04d6f43f Systemd needs After as well as Require (#15881)
If the gitea service is stopped because of the db going down
it needs an `After=db.service` to ensure it is restarted in
addition to the `Requires=db.service` to ensure that the db
is started before gitea is started.

Fix #15866

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2021-05-16 16:41:37 +08:00
zeripath 58646cab22 Move sans-serif fallback font higher than emoji fonts (#15855)
The Tor browser does not use the system-ui font and no other fonts in the stack match
its default fonts. In fact it is possible that it will in future only
match generic fonts. This means that all rendering will first try the
emoji fonts before falling back to the sans-serif font for glyphs.

In this case has the emoji fall back fonts for Tor contains empty glyphs
for numbers - in order to protect privacy - and leads to numbers being
rendered as empty glyphs. This is clearly not ideal and whilst we could
use the Arimo font - as I state above I suspect that Tor will eventually
ban detecting this and we should instead move the sans-serif font higher
in the stack so that it matches before the emoji fonts.

Partial fix of #15844

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-16 02:12:55 +02:00
zeripath 8e32eeb5de Hold the event source when there are no listeners (#15725)
* Hold the event source when there are no listeners

The event source does not need to run when there are no listeners. Therefore
pause it when there are none.

* add some more logging

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 23:46:13 +02:00
zeripath f582ec4e53 Create a session on ReverseProxy and ensure that ReverseProxy users cannot change username (#15304)
* Create a session on ReverseProxy and ensure that ReverseProxy users cannot change username

ReverseProxy users should generate a session on reverse proxy username change.

Also prevent ReverseProxy users from changing their username.

Fix #2407

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 20:33:13 +02:00
zeripath 17c5c654a5 Prevent double-login for Git HTTP and LFS and simplify login (#15303)
* Prevent double-login for Git HTTP and LFS and simplify login

There are a number of inconsistencies with our current methods for
logging in for git and lfs. The first is that there is a double login
process. This is particularly evident in 1.13 where there are no less
than 4 hash checks for basic authentication due to the previous
IsPasswordSet behaviour.

This duplicated code had individual inconsistencies that were not
helpful and caused confusion.

This PR does the following:

* Remove the specific login code from the git and lfs handlers except
for the lfs special bearer token
* Simplify the meaning of DisableBasicAuthentication to allow Token and
Oauth2 sign-in.
* The removal of the specific code from git and lfs means that these
both now have the same login semantics and can - if not
DisableBasicAuthentication - login from external services. Further it
allows Oauth2 token authentication as per our standard mechanisms.
* The change in the recovery handler prevents the service from
re-attempting to login - primarily because this could easily cause a
further panic and it is wasteful.

* add test

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-05-15 17:32:09 +02:00
zeripath ba526ceffe Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow level queue, reduce goroutines etc (#15693)
* move shutdownfns, terminatefns and hammerfns out of separate goroutines

Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.

This may help reduce the background select/poll load in certain
configurations.

* The LevelDB queues can actually wait on empty instead of polling

Slight refactor to cause leveldb queues to wait on empty instead of polling.

* Shutdown the shadow level queue once it is empty

* Remove bytefifo additional goroutine for readToChan as it can just be run in run

* Remove additional removeWorkers goroutine for workers

* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher

* Add shutdown flusher to CUQ

* move persistable channel shutdown stuff to Shutdown Fn

* Ensure that UPCQ has the correct config

* handle shutdown during the flushing

* reduce risk of race between zeroBoost and addWorkers

* prevent double shutdown

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-15 16:22:26 +02:00
zeripath d0ea10b21f Close the gitrepo when deleting the repository (#15876)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-14 21:19:38 +01:00
zeripath 1a5659943e Add timeout to writing to responses (#15831)
In #15826 it has become apparent that there are a few occasions when a response can
hang during writing, and because there is no timeout go will happily just block
interminably. This PR adds a fixed 5 second timeout to all writes to a connection.

Fix #15826

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-05-14 13:26:03 +01:00