forgejo/modules
Jason Song a627b885c7
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">

(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)

Conflicts:
	assets/go-licenses.json
	go.mod
	go.sum
  resolved with make tidy
2024-08-13 06:51:49 +02:00
..
actions Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
activitypub feat: access ActivityPub client through interfaces to facilitate mocking in unit tests (#4853) 2024-08-07 05:45:24 +00:00
analyze Rename code_langauge.go to code_language.go (#26377) 2023-08-07 15:00:53 -04:00
assetfs Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
auth Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
avatar Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
base cherry-pick OIDC changes from gitea (#4724) 2024-08-08 06:32:14 +00:00
cache Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
charset Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
container Add container.FilterSlice function (gitea#30339) 2024-04-16 11:49:44 +02:00
csv Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
emoji Update emoji set to Unicode 15 (#25595) 2023-06-29 16:29:48 +00:00
eventsource Final round of db.DefaultContext refactor (#27587) 2023-10-14 08:37:24 +00:00
forgefed Adjust codespell config + make it fix few typos which sneaked in since addition of codespell support (#4857) 2024-08-08 16:07:35 +00:00
generate Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
git [CHORE] Drop go-git support 2024-08-12 19:11:09 +02:00
gitgraph Add codespell support and fix a good number of typos with its help (#3270) 2024-05-09 13:49:37 +00:00
gitrepo [CHORE] Drop go-git support 2024-08-12 19:11:09 +02:00
graceful Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
hcaptcha Consume hcaptcha and pwn deps (#22610) 2023-01-29 09:49:51 -06:00
highlight Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
hostmatcher Support allowed hosts for webhook to work with proxy (#27655) 2023-10-18 09:44:36 +00:00
html Refactor backend SVG package and add tests (#26335) 2023-08-05 04:34:59 +00:00
httpcache [BRANDING] add X-Forgejo-* headers 2024-02-05 16:02:14 +01:00
httplib Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
indexer Adjust codespell config + make it fix few typos which sneaked in since addition of codespell support (#4857) 2024-08-08 16:07:35 +00:00
issue/template Issue Templates: add option to have dropdown printed list (#31577) 2024-07-22 15:44:13 +02:00
json Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
label Make label templates have consistent behavior and priority (#23749) 2023-04-10 16:44:02 +08:00
lfs [CHORE] Drop go-git support 2024-08-12 19:11:09 +02:00
log [CHORE] Fix darwin compatibility 2024-08-09 17:44:41 +02:00
markup disallow javascript: URI in the repository description 2024-08-09 07:04:01 +02:00
mcaptcha Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
metrics Rename project board -> column to make the UI less confusing (#30170) 2024-06-02 09:42:39 +02:00
migration Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
nosql s/Gitea/Forgejo in various log messages and comments 2024-04-21 21:26:15 +05:00
optional Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
options Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
packages Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
paginator Use more specific test methods (#24265) 2023-04-22 17:56:27 -04:00
pprof Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
private fix(hook): ignore unknown push options instead of failing 2024-07-02 21:39:01 +02:00
process Add codespell support and fix a good number of typos with its help (#3270) 2024-05-09 13:49:37 +00:00
proxy Use proxy for pull mirror (#22771) 2023-02-11 08:39:50 +08:00
proxyprotocol Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
public Refactor CORS handler (#28587) 2023-12-25 20:13:18 +08:00
queue Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
recaptcha Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
references Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
regexplru Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
repository Distinguish LFS object errors to ignore missing objects during migration (#31702) 2024-08-04 18:24:10 +02:00
secret Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
session Avoid importing modules/web/middleware in modules/session (#30584) 2024-04-21 16:28:16 +02:00
setting Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
sitemap Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
ssh Remove SSH workaround (#27893) 2023-11-03 15:21:05 +00:00
storage Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
structs Fix null requested_reviewer from API (#31773) 2024-08-13 06:51:49 +02:00
svg Refactor backend SVG package and add tests (#26335) 2023-08-05 04:34:59 +00:00
sync Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
system Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
templates Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
test test(util): MockProtect when mocking multiple times 2024-06-02 15:24:06 +02:00
testlogger Add codespell support and fix a good number of typos with its help (#3270) 2024-05-09 13:49:37 +00:00
timeutil Remove the time-since class (#29826) 2024-03-20 08:46:30 +01:00
translation Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
turnstile Add new captcha: cloudflare turnstile (#22369) 2023-02-05 15:29:03 +08:00
typesniffer Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
updatechecker Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
uri Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
user test: enforce tenv usage in tests 2024-07-17 23:07:41 +02:00
util Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
validation added validation fixes 2024-05-14 08:31:34 +02:00
web Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
webhook Add support for workflow_dispatch (#3334) 2024-06-28 05:17:11 +00:00
zstd Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00