a627b885c7
Support compression for Actions logs to save storage space and bandwidth. Inspired by https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015 The biggest challenge is that the compression format should support [seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md). So when users are viewing a part of the log lines, Gitea doesn't need to download the whole compressed file and decompress it. That means gzip cannot help here. And I did research, there aren't too many choices, like bgzip and xz, but I think zstd is the most popular one. It has an implementation in Golang with [zstd](https://github.com/klauspost/compress/tree/master/zstd) and [zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go), and what is better is that it has good compatibility: a seekable format zstd file can be read by a regular zstd reader. This PR introduces a new package `zstd` to combine and wrap the two packages, to provide a unified and easy-to-use API. And a new setting `LOG_COMPRESSION` is added to the config, although I don't see any reason why not to use compression, I think's it's a good idea to keep the default with `none` to be consistent with old versions. `LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as an extension to the file name, so Gitea can determine if it needs decompression according to the file name when reading. Old files will keep the format since it's not worth converting them, as they will be cleared after #31735. <img width="541" alt="image" src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9"> (cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588) Conflicts: assets/go-licenses.json go.mod go.sum resolved with make tidy |
||
---|---|---|
.. | ||
fixtures | ||
migration-test | ||
schemas | ||
actions_commit_status_test.go | ||
actions_route_test.go | ||
actions_trigger_test.go | ||
admin_config_test.go | ||
admin_user_test.go | ||
api_actions_artifact_test.go | ||
api_actions_artifact_v4_test.go | ||
api_activitypub_actor_test.go | ||
api_activitypub_person_test.go | ||
api_activitypub_repository_test.go | ||
api_admin_org_test.go | ||
api_admin_test.go | ||
api_block_test.go | ||
api_branch_test.go | ||
api_comment_attachment_test.go | ||
api_comment_test.go | ||
api_feed_plain_text_titles_test.go | ||
api_feed_user_test.go | ||
api_forgejo_root_test.go | ||
api_forgejo_version_test.go | ||
api_fork_test.go | ||
api_gitignore_templates_test.go | ||
api_gpg_keys_test.go | ||
api_health_test.go | ||
api_helper_for_declarative_test.go | ||
api_httpsig_test.go | ||
api_issue_attachment_test.go | ||
api_issue_config_test.go | ||
api_issue_label_test.go | ||
api_issue_milestone_test.go | ||
api_issue_pin_test.go | ||
api_issue_reaction_test.go | ||
api_issue_stopwatch_test.go | ||
api_issue_subscription_test.go | ||
api_issue_templates_test.go | ||
api_issue_test.go | ||
api_issue_tracked_time_test.go | ||
api_keys_test.go | ||
api_label_templates_test.go | ||
api_license_templates_test.go | ||
api_nodeinfo_test.go | ||
api_notification_test.go | ||
api_oauth2_apps_test.go | ||
api_org_avatar_test.go | ||
api_org_test.go | ||
api_packages_alpine_test.go | ||
api_packages_arch_test.go | ||
api_packages_cargo_test.go | ||
api_packages_chef_test.go | ||
api_packages_composer_test.go | ||
api_packages_conan_test.go | ||
api_packages_conda_test.go | ||
api_packages_container_test.go | ||
api_packages_cran_test.go | ||
api_packages_debian_test.go | ||
api_packages_generic_test.go | ||
api_packages_goproxy_test.go | ||
api_packages_helm_test.go | ||
api_packages_maven_test.go | ||
api_packages_npm_test.go | ||
api_packages_nuget_test.go | ||
api_packages_pub_test.go | ||
api_packages_pypi_test.go | ||
api_packages_rpm_test.go | ||
api_packages_rubygems_test.go | ||
api_packages_swift_test.go | ||
api_packages_test.go | ||
api_packages_vagrant_test.go | ||
api_private_serv_test.go | ||
api_pull_commits_test.go | ||
api_pull_review_test.go | ||
api_pull_test.go | ||
api_push_mirror_test.go | ||
api_quota_management_test.go | ||
api_quota_use_test.go | ||
api_releases_test.go | ||
api_repo_activities_test.go | ||
api_repo_archive_test.go | ||
api_repo_avatar_test.go | ||
api_repo_branch_test.go | ||
api_repo_collaborator_test.go | ||
api_repo_compare_test.go | ||
api_repo_edit_test.go | ||
api_repo_file_create_test.go | ||
api_repo_file_delete_test.go | ||
api_repo_file_get_test.go | ||
api_repo_file_helpers.go | ||
api_repo_file_update_test.go | ||
api_repo_files_change_test.go | ||
api_repo_get_contents_list_test.go | ||
api_repo_get_contents_test.go | ||
api_repo_git_blobs_test.go | ||
api_repo_git_commits_test.go | ||
api_repo_git_hook_test.go | ||
api_repo_git_notes_test.go | ||
api_repo_git_ref_test.go | ||
api_repo_git_tags_test.go | ||
api_repo_git_trees_test.go | ||
api_repo_hook_test.go | ||
api_repo_languages_test.go | ||
api_repo_lfs_locks_test.go | ||
api_repo_lfs_migrate_test.go | ||
api_repo_lfs_test.go | ||
api_repo_raw_test.go | ||
api_repo_secrets_test.go | ||
api_repo_tags_test.go | ||
api_repo_teams_test.go | ||
api_repo_test.go | ||
api_repo_topic_test.go | ||
api_repo_variables_test.go | ||
api_settings_test.go | ||
api_team_test.go | ||
api_team_user_test.go | ||
api_token_test.go | ||
api_twofa_test.go | ||
api_user_avatar_test.go | ||
api_user_email_test.go | ||
api_user_follow_test.go | ||
api_user_heatmap_test.go | ||
api_user_info_test.go | ||
api_user_org_perm_test.go | ||
api_user_orgs_test.go | ||
api_user_search_test.go | ||
api_user_secrets_test.go | ||
api_user_star_test.go | ||
api_user_variables_test.go | ||
api_user_watch_test.go | ||
api_wiki_test.go | ||
archived_labels_display_test.go | ||
attachment_test.go | ||
auth_ldap_test.go | ||
auth_token_test.go | ||
avatar.png | ||
benchmarks_test.go | ||
block_test.go | ||
branches_test.go | ||
change_default_branch_test.go | ||
cmd_admin_test.go | ||
cmd_forgejo_actions_test.go | ||
cmd_forgejo_f3_test.go | ||
cmd_keys_test.go | ||
codeowner_test.go | ||
compare_test.go | ||
cors_test.go | ||
create_no_session_test.go | ||
csrf_test.go | ||
db_collation_test.go | ||
delete_user_test.go | ||
doctor_packages_nuget_test.go | ||
download_test.go | ||
dump_restore_test.go | ||
easymde_test.go | ||
editor_test.go | ||
empty_repo_test.go | ||
eventsource_test.go | ||
explore_code_test.go | ||
explore_repos_test.go | ||
explore_user_test.go | ||
forgejo_confirmation_repo_test.go | ||
forgejo_git_test.go | ||
git_clone_wiki_test.go | ||
git_helper_for_declarative_test.go | ||
git_push_test.go | ||
git_smart_http_test.go | ||
git_test.go | ||
goget_test.go | ||
gpg_git_test.go | ||
html_helper.go | ||
incoming_email_test.go | ||
integration_test.go | ||
issue_subscribe_test.go | ||
issue_test.go | ||
issues_comment_labels_test.go | ||
last_updated_time_test.go | ||
lfs_getobject_test.go | ||
lfs_local_endpoint_test.go | ||
lfs_view_test.go | ||
linguist_test.go | ||
links_test.go | ||
markup_external_test.go | ||
markup_test.go | ||
migrate_test.go | ||
milestone_test.go | ||
mirror_pull_test.go | ||
mirror_push_test.go | ||
new_org_test.go | ||
nonascii_branches_test.go | ||
oauth_test.go | ||
opengraph_test.go | ||
org_count_test.go | ||
org_nav_test.go | ||
org_project_test.go | ||
org_team_invite_test.go | ||
org_test.go | ||
private-testing.key | ||
privateactivity_test.go | ||
proctected_branch_test.go | ||
project_test.go | ||
pull_compare_test.go | ||
pull_create_test.go | ||
pull_diff_test.go | ||
pull_merge_test.go | ||
pull_reopen_test.go | ||
pull_request_task_test.go | ||
pull_review_test.go | ||
pull_status_test.go | ||
pull_summary_test.go | ||
pull_test.go | ||
pull_update_test.go | ||
pull_wip_convert_test.go | ||
quota_use_test.go | ||
README.md | ||
README_ZH.md | ||
release_test.go | ||
remote_test.go | ||
rename_branch_test.go | ||
repo_activity_test.go | ||
repo_archive_test.go | ||
repo_archive_text_test.go | ||
repo_badges_test.go | ||
repo_branch_test.go | ||
repo_collaborator_test.go | ||
repo_commits_search_test.go | ||
repo_commits_test.go | ||
repo_delete_test.go | ||
repo_flags_test.go | ||
repo_fork_test.go | ||
repo_generate_test.go | ||
repo_mergecommit_revert_test.go | ||
repo_migrate_test.go | ||
repo_migration_ui_test.go | ||
repo_pagination_test.go | ||
repo_search_test.go | ||
repo_settings_hook_test.go | ||
repo_settings_test.go | ||
repo_signed_tag_test.go | ||
repo_starwatch_test.go | ||
repo_tag_test.go | ||
repo_test.go | ||
repo_topic_test.go | ||
repo_view_test.go | ||
repo_watch_test.go | ||
repo_webhook_test.go | ||
repo_wiki_test.go | ||
repofiles_change_test.go | ||
session_test.go | ||
setting_test.go | ||
signin_test.go | ||
signout_test.go | ||
signup_test.go | ||
size_translations_test.go | ||
ssh_key_test.go | ||
timetracking_test.go | ||
user_avatar_test.go | ||
user_count_test.go | ||
user_dashboard_test.go | ||
user_profile_activity_test.go | ||
user_profile_follows_test.go | ||
user_profile_test.go | ||
user_test.go | ||
version_test.go | ||
view_test.go | ||
webfinger_test.go | ||
webhook_test.go | ||
xss_test.go |
Integration tests
Integration tests can be run with make commands for the appropriate backends, namely:
make test-sqlite
make test-pgsql
make test-mysql
Make sure to perform a clean build before running tests:
make clean build
Run tests via local act_runner
Run all jobs
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
Warning: This file defines many jobs, so it will be resource-intensive and therefore not recommended.
Run single job
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
You can list all job names via:
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
Run sqlite integration tests
Start tests
make test-sqlite
Run MySQL integration tests
Setup a MySQL database inside docker
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)
Start tests based on the database container
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
Run pgsql integration tests
Setup a pgsql database inside docker
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
Start tests based on the database container
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
Running individual tests
Example command to run GPG test:
For SQLite:
make test-sqlite#GPG
For other databases (replace mysql
to pgsql
):
TEST_MYSQL_HOST=localhost:1433 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=sa TEST_MYSQL_PASSWORD=MwantsaSecurePassword1 make test-mysql#GPG
Setting timeouts for declaring long-tests and long-flushes
We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.
You can either:
- Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
- Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite