Template
1
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo synced 2024-11-22 09:54:24 +01:00
forgejo/modules/setting
Jason Song a627b885c7
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">

(cherry picked from commit 33cc5837a655ad544b936d4d040ca36d74092588)

Conflicts:
	assets/go-licenses.json
	go.mod
	go.sum
  resolved with make tidy
2024-08-13 06:51:49 +02:00
..
config Customizable "Open with" applications for repository clone (#29320) 2024-03-23 11:58:54 +01:00
actions.go Support compression for Actions logs (#31761) 2024-08-13 06:51:49 +02:00
actions_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
admin.go fix(services): set SendNotificationEmailOnNewUser 2024-05-25 16:27:44 +02:00
admin_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
api.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
asset_dynamic.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
asset_static.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
attachment.go feat: allow .webp attachments by default 2024-07-21 13:01:08 +05:00
attachment_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
badges.go [GITEA] Add support for shields.io-based badges 2024-02-05 16:09:42 +01:00
cache.go Always enable caches (#28527) 2023-12-19 09:29:05 +00:00
camo.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
config.go Customizable "Open with" applications for repository clone (#29320) 2024-03-23 11:58:54 +01:00
config_env.go Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
config_env_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
config_provider.go [gitea] Remove unnecessary parameter (#29092) 2024-02-10 10:53:43 +01:00
config_provider_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
cors.go Refactor CORS handler (#28587) 2023-12-25 20:13:18 +08:00
cron.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
cron_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
database.go Limit database max connections by default 2024-04-23 00:47:50 +02:00
database_sqlite.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
database_test.go Fix incorrect PostgreSQL connection string for Unix sockets (#28865) 2024-01-20 16:04:47 +00:00
f3.go fix(f3): do not run the F3 CLI if F3 is disabled 2024-06-16 13:33:33 +02:00
federation.go activitypub: Sign the Host header too 2024-08-04 23:57:48 +02:00
forgejo_storage_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
git.go Use [git.config] for reflog cleaning up (#24958) 2023-05-28 01:07:14 +00:00
git_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
highlight.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
i18n.go i18n: restore Malayalam and Serbian files, remove ml-IN from the language selector (#4576) 2024-07-22 14:08:15 +00:00
incoming_email.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-05-05 08:38:16 +01:00
incoming_email_test.go setting: Infer [email.incoming].PORT from .USE_TLS 2024-04-22 16:30:35 +02:00
indexer.go [FEAT] Support Include/Exclude Filters for Grep (#3058) 2024-04-06 13:25:39 +00:00
indexer_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
lfs.go Port "Use general token signing secret" 2024-02-19 20:49:37 +01:00
lfs_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
log.go log: journald integration (#2869) 2024-08-09 07:49:13 +00:00
log_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
mailer.go Add option to change mail from user display name (#31528) 2024-07-22 15:44:13 +02:00
mailer_test.go setting: Allow aliases for some email settings 2024-04-22 16:09:01 +02:00
markup.go Add setting to restrict count of lines being displayed & only highlight those lines 2024-03-25 16:05:01 +01:00
metrics.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
migrations.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mime_type_map.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mirror.go Avoid polluting the config (#25345) 2023-06-18 16:10:44 +00:00
oauth2.go [FEAT] Enable INVALIDATE_REFRESH_TOKENS 2024-07-22 20:45:13 +02:00
oauth2_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
other.go Add [other].SHOW_FOOTER_POWERED_BY setting to hide Powered by (#30253) 2024-04-07 15:40:31 +02:00
packages.go Arch packages implementation (#4785) 2024-08-04 06:16:29 +00:00
packages_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
path.go [BRANDING] alias {FORGEJO,GITEA}_{CUSTOM,WORK_DIR} 2024-02-05 16:05:01 +01:00
path_test.go [BRANDING] alias {FORGEJO,GITEA}_{CUSTOM,WORK_DIR} 2024-02-05 16:05:01 +01:00
picture.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
project.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
proxy.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
queue.go Increase queue length (#27555) 2023-10-10 18:47:49 +08:00
quota.go feat(quota): Humble beginnings of a quota engine 2024-08-02 11:10:34 +02:00
repository.go Change the default SSH clone url to the ssh:// style 2024-04-17 11:04:48 +02:00
repository_archive.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
repository_archive_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
security.go Remember login for a month by default (#30150) 2024-03-30 07:17:31 +01:00
server.go Add slogan config (#3752) 2024-06-07 17:12:48 +00:00
server_test.go Add slogan config (#3752) 2024-06-07 17:12:48 +00:00
service.go [GITEA] add option for banning dots in usernames 2024-02-05 16:05:50 +01:00
service_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
session.go Fix incorrect cookie path for AppSubURL (#29534) 2024-03-06 12:10:46 +08:00
setting.go log: journald integration (#2869) 2024-08-09 07:49:13 +00:00
setting_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
ssh.go Expanded minimum RSA Keylength to 3072 (#26604) 2023-08-28 00:53:16 +00:00
storage.go Enable unparam linter (#31277) 2024-06-16 13:42:58 +02:00
storage_test.go Add testifylint to lint checks (#4535) 2024-07-30 19:41:10 +00:00
task.go handle deprecated settings (#22992) 2023-02-20 16:18:26 -06:00
time.go Enable more revive linter rules (#30608) 2024-04-28 15:39:00 +02:00
ui.go Add MAX_ROWS option for CSV rendering (#30268) 2024-06-09 16:04:57 +02:00
webhook.go [REFACTOR] simplify checkHookType 2024-03-27 22:26:55 +01:00