1
1
mirror of https://github.com/go-gitea/gitea synced 2024-11-14 06:04:25 +00:00
gitea/modules/setting
Jason Song 33cc5837a6
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
2024-08-09 10:10:30 +08:00
..
config Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
actions_test.go Restrict [actions].DEFAULT_ACTIONS_URL to only github or self (#25581) 2023-06-30 07:26:36 +00:00
actions.go Support compression for Actions logs (#31761) 2024-08-09 10:10:30 +08:00
admin.go Allow disabling authentication related user features (#31535) 2024-07-09 17:36:31 +00:00
api.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
asset_dynamic.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
asset_static.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
attachment_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
attachment.go Update allowed attachment types (#29688) 2024-03-09 12:41:32 +00:00
cache.go Always enable caches (#28527) 2023-12-19 09:29:05 +00:00
camo.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
config_env_test.go Fix environment-to-ini inherited key bug (#27543) 2023-10-10 01:10:37 +08:00
config_env.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
config_provider_test.go Fix INI parsing for value with trailing slash (#26995) 2023-09-10 16:15:51 +00:00
config_provider.go Initial support for colorblindness-friendly themes (#30625) 2024-04-24 00:18:41 +08:00
config.go Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
cors.go Refactor CORS handler (#28587) 2023-12-25 20:13:18 +08:00
cron_test.go Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
cron.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
database_sqlite.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
database_test.go Fix incorrect PostgreSQL connection string for Unix sockets (#28865) 2024-01-20 16:04:47 +00:00
database.go Add slow SQL query warning (#27545) 2024-02-23 00:57:24 +00:00
federation.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
git_test.go Use [git.config] for reflog cleaning up (#24958) 2023-05-28 01:07:14 +00:00
git.go Use [git.config] for reflog cleaning up (#24958) 2023-05-28 01:07:14 +00:00
glob.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
global.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
highlight.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
i18n.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
incoming_email.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-04-29 08:47:56 +00:00
indexer_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
indexer.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
lfs_test.go Display deprecated warning in admin panel pages as well as in the log file (#26094) 2023-07-26 03:53:37 +00:00
lfs.go Use general token signing secret (#29205) 2024-02-18 17:39:04 +00:00
log_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
log.go Refactor "dump" sub-command (#30240) 2024-04-03 02:16:46 +00:00
mailer_test.go Remove unnecessary code (#24610) 2023-05-10 04:57:06 +00:00
mailer.go Add option to change mail from user display name (#31528) 2024-07-14 23:27:00 +02:00
markup.go Add .livemd as a markdown extension (#22730) 2023-04-26 11:22:54 -04:00
metrics.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
migrations.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mime_type_map.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mirror.go Avoid polluting the config (#25345) 2023-06-18 16:10:44 +00:00
oauth2_test.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
oauth2.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
other.go Add [other].SHOW_FOOTER_POWERED_BY setting to hide Powered by (#30253) 2024-04-03 16:01:50 +00:00
packages_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
packages.go Add signature support for the RPM module (#27069) 2024-08-06 09:03:33 -04:00
path_test.go Refactor path & config system (#25330) 2023-06-21 13:50:26 +08:00
path.go Do not allow different storage configurations to point to the same directory (#30169) 2024-03-31 03:03:24 +00:00
picture.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
project.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
proxy.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
queue.go Increase queue length (#27555) 2023-10-10 18:47:49 +08:00
repository_archive_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
repository_archive.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
repository.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
security.go Remember login for a month by default (#30150) 2024-03-28 03:13:42 +00:00
server.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
service_test.go Fix allowed user types setting problem (#26200) 2023-07-28 12:15:39 -04:00
service.go Add reverseproxy auth for API back with default disabled (#26703) 2023-09-07 08:31:46 +00:00
session.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
setting_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
setting.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
ssh.go Expanded minimum RSA Keylength to 3072 (#26604) 2023-08-28 00:53:16 +00:00
storage_test.go Azure blob storage support (#30995) 2024-05-30 07:33:50 +00:00
storage.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
task.go handle deprecated settings (#22992) 2023-02-20 16:18:26 -06:00
time.go Enable more revive linter rules (#30608) 2024-04-22 11:48:42 +00:00
ui.go Add MAX_ROWS option for CSV rendering (#30268) 2024-06-06 16:06:59 +08:00
webhook.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00