Partial Backport of #18415
Instead of using an asynchronous goroutine to push to disk on shutdown
just close the datachan and immediately push to the disk.
Prevents messages of incompletely flushed queues.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Restart zero worker if there is still work to do (#18658)
Backport #18658
It is possible for the zero worker to timeout before all the work is finished.
This may mean that work may take a long time to complete because a worker will only
be induced on repushing.
Also ensure that requested count is reset after pulls and push mirror sync requests and add some more trace logging to the queue push.
Fix#18607
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/queue/workerpool.go
* Only attempt to flush queue if the underlying worker pool is not finished (#18593)
Backport #18593
There is a possible race whereby a worker pool could be cancelled but yet the
underlying queue is not empty. This will lead to flush-all cycling because it
cannot empty the pool.
* On shutdown of Persistant Channel Queues close datachan and empty
Partial Backport #18415
Although we attempt to empty the datachan in queues - due to
races we are better off just closing the channel and forcibly emptying
it in shutdown.
Fix#18618
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move zero workers warning to debug
Fix#18617
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/queue/manager.go
Co-authored-by: Gusted <williamzijl7@hotmail.com>
* Update modules/queue/manager.go
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
* Prevent deadlock in TestPersistableChannelQueue
There is a potential deadlock in TestPersistableChannelQueue due to attempting to
shutdown the test queue before it is ready.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* prevent npe
Signed-off-by: Andrew Thornton <art27@cantab.net>
Convert the old mirror syncing queue to the more modern queue format.
Fix a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead.
The documentation is also updated to add information about the deprecated configuration values.
Signed-off-by: Andrew Thornton <art27@cantab.net>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Timeout on flush in testing
At the end of each test the queues are flushed. At present there is no limit on the
length of time a flush can take which can lead to long flushes.
However, if the CI task is cancelled we lose the log information as to where the long
flush was taking place.
This PR simply adds a default time limit of 2 minutes - at which point an error will
be produced. This should allow us to more easily find the culprit.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* return better error
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
A race has been detected in #1441 relating to getting log levels.
This PR protects the GetLevel and GetStacktraceLevel calls with a RW mutex.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent race in TestPersistableChannelQueue
A slight race has become apparent in the TestPersistableChannelQueue.
This PR simply adds locking to prevent the race.
* make print value of "$(GOTESTFLAGS)" on test-backend and unit-test-coverage
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* move shutdownfns, terminatefns and hammerfns out of separate goroutines
Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.
This may help reduce the background select/poll load in certain
configurations.
* The LevelDB queues can actually wait on empty instead of polling
Slight refactor to cause leveldb queues to wait on empty instead of polling.
* Shutdown the shadow level queue once it is empty
* Remove bytefifo additional goroutine for readToChan as it can just be run in run
* Remove additional removeWorkers goroutine for workers
* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher
* Add shutdown flusher to CUQ
* move persistable channel shutdown stuff to Shutdown Fn
* Ensure that UPCQ has the correct config
* handle shutdown during the flushing
* reduce risk of race between zeroBoost and addWorkers
* prevent double shutdown
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Queue manager FlushAll can loop rapidly - add delay
Add delay within FlushAll to prevent rapid loop when workers are busy
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per lunny
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
This PR is another in the vein of queue improvements. It suggests an
exponential backoff for bytefifo queues to reduce the load from queue
polling. This will mostly be useful for redis queues.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
There is a potential race in TestChannelQueue_Batch due to boost workers starting up
This PR simply removes the boosts from this test.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* The leveldb shadow queue of a persistable channel queue should always start with 0
workers and just use boost to add additional workers if necessary.
* create a zero boost so that if there are no workers in a pool - boost to start the workers
* actually set timeout appropriately on boosted workers
Signed-off-by: Andrew Thornton <art27@cantab.net>
The issue is that the TestPatch will reset the PR MergeBase - and it is possible for TestPatch to update the MergeBase whilst a merge is ongoing. The ensuing merge will then complete but it doesn't re-set the MergeBase it used to merge the PR.
Fixes the intermittent error in git test.
Signed-off-by: Andrew Thornton art27@cantab.net
* _ to unused func options
* rm useless brakets
* rm trifial non used models functions
* rm dead code
* rm dead global vars
* fix routers/api/v1/repo/issue.go
* dont overload import module
There is potentially a race with a slow starting internal
queue causing a NPE if Has is checked before the internal
queue has been setup.
This PR adds a lock on the Has() fn.
Fix#14311
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update go-redis to v8.4.0
* github.com/go-redis/redis/v8 v8.4.0 -> v8.5.0
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* TODO
* Use the Queue termination channel as the default context for pushes
Signed-off-by: Andrew Thornton <art27@cantab.net>
* missed one
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* Allow common redis and leveldb connections
Prevents multiple reopening of redis and leveldb connections to the same
place by sharing connections.
Further allows for more configurable redis connection type using the
redisURI and a leveldbURI scheme.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add unit-test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @lunny
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/cache/cache_redis.go
* Update modules/queue/queue_disk.go
* Update modules/cache/cache_redis.go
* Update modules/cache/cache_redis.go
* Update modules/queue/unique_queue_disk.go
* Update modules/queue/queue_disk.go
* Update modules/queue/unique_queue_disk.go
* Update modules/session/redis.go
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.
Fix#12339
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
* Remove spurious spacing between Maintenance Operations and its table on dashboard
* Prevent (EXTRA string) comments in Task headers
* Redirect tasks started from monitor page back to monitor
* Fix#12107 - redirects from process cancel should use AppSubUrl
* When wrapping queues set the name correctly
Signed-off-by: Andrew Thornton <art27@cantab.net>
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.
Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.
Unfortunately in `Workerpool` these were checked the wrong way round.
However, there were a few other places that were not being checked.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Upgrade levelqueue to version 0.2.0
This adds functionality for Unique Queues
* Add UniqueQueue interface and functions to create them
* Add UniqueQueue implementations
* Move TestPullRequests over to use UniqueQueue
* Reduce code duplication
* Add bytefifos
* Ensure invalid types are logged
* Fix close race in PersistableChannelQueue Shutdown
* go function contexting is not what you expect
* Apply suggestions from code review
Co-Authored-By: Lauris BH <lauris@nix.lv>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Make WorkerPools and Queues flushable
Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Change to for loop
* Add IsEmpty and begin just making the queues composed WorkerPools
* subsume workerpool into the queues and create a flushable interface
* Add manager command
* Move flushall to queue.Manager and add to testlogger
* As per @guillep2k
* as per @guillep2k
* Just make queues all implement flushable and clean up the wrapped queue flushes
* cope with no timeout
Co-authored-by: Lauris BH <lauris@nix.lv>
