For #4082.
~~Per the discussion in the issue, the current plan will likely involve duplicating the redis library calling code once for each cacher, as neither garnet nor redict guarantee continued compatibility with redis.~~
See discussion below for details.
## Tasklist
- [x] Write workflow to run cache-specific unit test(s) only (cache, session, queue, nosql) for each cacher
- [x] Check whether garnet and redict pass unit tests with no code modification (gauge required work)
- both passed, but that is because there were very few tests that test the remote cache store
### Out of scope for this PR
- Improve test coverage
- `modules/cache` against a server
- `modules/session` against a server (also needs tests in general)
- _(?) Duplicate implementation for each cacher_
- _Restructure redis usage in `modules/cache` and `modules/settings/cache`_
- _Restructure `modules/session` and its settings_
- _Restructure `modules/queue` and its settings_
- _Restructure `modules/nosql` and its settings_
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4138
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Elias Elwyn <a@jthv.ai>
Co-committed-by: Elias Elwyn <a@jthv.ai>
Uses `gopls check <files>` as a linter. Tested locally and brings up 149
errors currently for me. I don't think I want to fix them in this PR,
but I would like at least to get this analysis running on CI.
List of errors:
```
modules/indexer/code/indexer.go:181:11: impossible condition: nil != nil
routers/private/hook_post_receive.go:120:15: tautological condition: nil == nil
services/auth/source/oauth2/providers.go:185:9: tautological condition: nil == nil
services/convert/issue.go:216:11: tautological condition: non-nil != nil
tests/integration/git_test.go:332:9: impossible condition: nil != nil
services/migrations/migrate.go:179:24-43: unused parameter: ctx
services/repository/transfer.go:288:48-69: unused parameter: doer
tests/integration/api_repo_tags_test.go:75:41-61: unused parameter: session
tests/integration/git_test.go:696:64-74: unused parameter: baseBranch
tests/integration/gpg_git_test.go:265:27-39: unused parameter: t
tests/integration/gpg_git_test.go:284:23-29: unused parameter: tmpDir
tests/integration/gpg_git_test.go:284:31-35: unused parameter: name
tests/integration/gpg_git_test.go:284:37-42: unused parameter: email
```
(cherry picked from commit 816222243af523316041692622be6f48ef068693)
Conflicts:
Makefile
trivial context conflict and also ask renovate to watch over it
do not include it in lint-backend because the errors are not fixed
We're stuck on an old version of golang deadcode. Renovate is confused by it's rename in https://codeberg.org/forgejo/forgejo/pulls/4043.
I don't remember how to use this tool to test it, let's see how the CI reacts to this PR.
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/4048
Reviewed-by: Victoria <efertone@noreply.codeberg.org>
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: 0ko <0ko@noreply.codeberg.org>
Co-committed-by: 0ko <0ko@noreply.codeberg.org>
Summary:
- Move existing test under a `testify` Suite as `baseRedisWithServerTestSuite`
- Those tests require real redis server.
- Add `go.uber.org/mock/mockgen@latest` as dependency
- as a tool (Makefile).
- in the `go.mod` file.
- Mock redis client lives under a `mock` directory under the queue module.
- That mock module has an extra hand-written mock in-memory redis-like struct.
- Add tests using the mock redis client.
- Changed the logic around queue provider creation.
- Now the `getNewQueue` returns a Queue provider directly, not an init
function to create it.
The whole Queue module is close to impossible to test properly because
everything is private, everything goes through a struct route. Because
of that, we can't test for example what keys are used for given queue.
To overcome this, as a first step I removed one step from that hard
route by allowing custom calls to create new queue provider. To achieve
this, I moved the creation logic into the `getNewQueue` (previously it
was `getNewQueueFn`). That changes nothing on that side, everything goes
as before, except the `newXXX` call happens directly in that function
and not outside that.
That made it possible to add extra provider specific parameters to those
function (`newXXX`). For example a client on redis. Calling it through
the `getNewQueue` function, it gets `nil`.
- If the provided client is not `nil`, it will use that instead of the
connection string.
- If it's `nil` (default behaviour), it creates a new redis client as it
did before, no changes to that.
The rest of the provider code is unchanged. All these changes were
required to make it possible to generate mock clients for providers and
use them.
For the tests, the existing two test cases are good with redis server,
and they need some extra helpers, for example to start a new redis
server if required, or waiting on a redis server to be ready to use.
These helpers are only required for test cases using real redis server.
For better isolation, moved existing test under a testify Suite, and
moved them into a new test file called `base_redis_with_server_test.go`
because, well they test the code with server. These tests do exactly the
same as before, calling the same sub-tests the same way as before, the
only change is the structure of the test (remove repetition, scope
server related helper functions).
Finally, we can create unit tests without redis server. The main focus of
this group of tests are higher level overview of operations. With the
mock redis client we can set up expectations about used queue names,
received values, return value to simulate faulty state.
These new unit test functions don't test all functionality, at least
it's not aimed for it now. It's more about the possibility of doing that
and add extra tests around parts we couldn't test before, for example
key.
What extra features can test the new unit test group:
- What is the received key for given queue? For example using `prefix`,
or if all the `SXxx` calls are expected to use `queue_unique` if
it's a unique queue.
- If it's not a unique queue, no `SXxx` functions are called, because
those sets are used only to check if a value is unique or not.
- `HasItem` return `false` always if it's a non-unique queue.
- All functions are called exactly `N` times, and we don't have any
unexpected calls to redis from the code.
Signed-off-by: Victoria Nadasdi <victoria@efertone.me>
More about codespell: https://github.com/codespell-project/codespell .
I personally introduced it to dozens if not hundreds of projects already and so far only positive feedback.
```
❯ grep lint-spell Makefile
@echo " - lint-spell lint spelling"
@echo " - lint-spell-fix lint spelling and fix issues"
lint: lint-frontend lint-backend lint-spell
lint-fix: lint-frontend-fix lint-backend-fix lint-spell-fix
.PHONY: lint-spell
lint-spell: lint-codespell
.PHONY: lint-spell-fix
lint-spell-fix: lint-codespell-fix
❯ git grep lint- -- .forgejo/
.forgejo/workflows/testing.yml: - run: make --always-make -j$(nproc) lint-backend checks-backend # ensure the "go-licenses" make target runs
.forgejo/workflows/testing.yml: - run: make lint-frontend
```
so how would you like me to invoke `lint-codespell` on CI? (without that would be IMHO very suboptimal and let typos sneak in)
Reviewed-on: https://codeberg.org/forgejo/forgejo/pulls/3270
Reviewed-by: Earl Warren <earl-warren@noreply.codeberg.org>
Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>
Co-committed-by: Yaroslav Halchenko <debian@onerussian.com>
there are no tests but since Gitea uses @v1 since last month and Gitea
maintainers rely on make watch, it is safe to assume that upgrading is
not broken. Switching to v1 would require less scrutiny on the
upgrades. Even if there is breakage, it can be fixed with minimal
impact on the developer workflow.
When you cross-compile Gitea and you specify one of the envrionment
variables related to C flags, cgo will fail to build the generator
programs (e.g. generate-bindata) because GOOS and GOARCH are unset, but
those additional flags variables are not unset together with those.
To solve this issue, the simplest way that I've found is to disable cgo
in the `go generate` command as it's not really used there.
For example, I've had this problem with cross-compiling Gitea on FreeBSD
x86_64 to ARMv7 where it's necessary to pass `--target` to `clang` via
`CGO_CFLAGS`:
```
GOOS=freebsd \
GOARCH=arm \
GGOARM=7 \
CGO_ENABLED=1 \
SYSROOT=/usr/local/freebsd-sysroot/armv7 \
CC=clang \
CGO_CFLAGS="--target=armv7-unknown-freebsd13.2-gnueabihf" \
TAGS="bindata sqlite sqlite_unlock_notify" \
make SHELL='sh -x' build
```
```
Running go generate...
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/migration/schemas_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/options/options_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/public/public_bindata.go:8: running "go": exit status 1
# runtime/cgo
In file included from gcc_freebsd_amd64.c:9:
In file included from /usr/include/signal.h:42:
/usr/include/sys/_ucontext.h:44:2: error: unknown type name 'mcontext_t'
modules/templates/templates_bindata.go:8: running "go": exit status 1
gmake[1]: *** [Makefile:781: generate-go] Error 1
*** Error code 2
Stop.
```
But with this fix Gitea compiles successfully.
(cherry picked from commit d11133b83652238023b52576e0d3e57a4f4b21c9)
