forgejo/modules/setting
Jason Song a627b885c7
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">

(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)

Conflicts:
	assets/go-licenses.json
	go.mod
	go.sum
  resolved with make tidy
2024-08-13 06:51:49 +02:00
..
config Customizable "Open with" applications for repository clone (#29320) 2024-03-23 11:58:54 +01:00
actions.go Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
actions_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
admin.go fix(services): set SendNotificationEmailOnNewUser 2024-05-25 16:27:44 +02:00
admin_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
api.go
asset_dynamic.go
asset_static.go
attachment.go feat: allow .webp attachments by default 2024-07-21 13:01:08 +05:00
attachment_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
badges.go
cache.go
camo.go
config.go Customizable "Open with" applications for repository clone (#29320) 2024-03-23 11:58:54 +01:00
config_env.go Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
config_env_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
config_provider.go
config_provider_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
cors.go
cron.go
cron_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
database.go Limit database max connections by default 2024-04-23 00:47:50 +02:00
database_sqlite.go
database_test.go
f3.go fix(f3): do not run the F3 CLI if F3 is disabled 2024-06-16 13:33:33 +02:00
federation.go activitypub: Sign the Host header too 2024-08-04 23:57:48 +02:00
forgejo_storage_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
git.go
git_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
highlight.go
i18n.go i18n: restore Malayalam and Serbian files, remove ml-IN from the language selector (#4576) 2024-07-22 14:08:15 +00:00
incoming_email.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-05-05 08:38:16 +01:00
incoming_email_test.go setting: Infer [email.incoming].PORT from .USE_TLS 2024-04-22 16:30:35 +02:00
indexer.go [FEAT] Support Include/Exclude Filters for Grep (#3058) 2024-04-06 13:25:39 +00:00
indexer_test.go
lfs.go Port "Use general token signing secret" 2024-02-19 20:49:37 +01:00
lfs_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
log.go log: journald integration (#2869) 2024-08-09 07:49:13 +00:00
log_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
mailer.go Add option to change mail from user display name (#31528) 2024-07-22 15:44:13 +02:00
mailer_test.go setting: Allow aliases for some email settings 2024-04-22 16:09:01 +02:00
markup.go Add setting to restrict count of lines being displayed & only highlight those lines 2024-03-25 16:05:01 +01:00
metrics.go
migrations.go
mime_type_map.go
mirror.go
oauth2.go [FEAT] Enable INVALIDATE_REFRESH_TOKENS 2024-07-22 20:45:13 +02:00
oauth2_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
other.go Add [other].SHOW_FOOTER_POWERED_BY setting to hide Powered by (#30253) 2024-04-07 15:40:31 +02:00
packages.go Arch packages implementation (#4785) 2024-08-04 06:16:29 +00:00
packages_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
path.go
path_test.go
picture.go
project.go
proxy.go
queue.go
quota.go feat(quota): Humble beginnings of a quota engine 2024-08-02 11:10:34 +02:00
repository.go Change the default SSH clone url to the ssh:// style 2024-04-17 11:04:48 +02:00
repository_archive.go
repository_archive_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
security.go Remember login for a month by default (#30150) 2024-03-30 07:17:31 +01:00
server.go Add slogan config (#3752) 2024-06-07 17:12:48 +00:00
server_test.go Add slogan config (#3752) 2024-06-07 17:12:48 +00:00
service.go
service_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
session.go Fix incorrect cookie path for AppSubURL (#29534) 2024-03-06 12:10:46 +08:00
setting.go log: journald integration (#2869) 2024-08-09 07:49:13 +00:00
setting_test.go
ssh.go
storage.go Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
storage_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
task.go
time.go Enable more revive linter rules (#30608) 2024-04-28 15:39:00 +02:00
ui.go Add MAX_ROWS option for CSV rendering (#30268) 2024-06-09 16:04:57 +02:00
webhook.go [REFACTOR] simplify checkHookType 2024-03-27 22:26:55 +01:00