Commit graph

7 commits

Author SHA1 Message Date
Loïc Dachary
15e6485ae1
[CI] tests/pgsql.ini.tmpl: do not use minio
f92e0a4018 added minio to the tests/pgsql.ini.tmpl and Forgejo CI does
not run a minio server. It will if there are external storage related
changes at some point but it is not the case now so it is not worth
the burden.

(cherry picked from commit 4cfbf4718d)
(cherry picked from commit 9bd644d601)
(cherry picked from commit 604636c7c4)
(cherry picked from commit 341cdb8540)
(cherry picked from commit 57bdc91de5)
(cherry picked from commit c6ec6517b4)
(cherry picked from commit 2f03fc1b29)
(cherry picked from commit aaecd10966)
(cherry picked from commit 7025ff7b06)
2023-05-23 09:01:09 +02:00
Loïc Dachary
208498caaa
[CI] set PASSWORD_HASH_ALGO = argon2 for integration tests
(cherry picked from commit 1d7ce2a39c)
(cherry picked from commit 3af8757583)
(cherry picked from commit 0b81815209)
(cherry picked from commit 7180a3222a)
(cherry picked from commit 069044f4a6)
(cherry picked from commit dd1438f389)
(cherry picked from commit 4ae6b5d64d)
(cherry picked from commit c572666d78)
(cherry picked from commit de0a4480bc)
(cherry picked from commit 60272ef4f7)
(cherry picked from commit a68fdde4be)
(cherry picked from commit ff448cf3a6)
2023-05-23 07:30:26 +02:00
FuXiaoHei
c757765a9e
Implement actions artifacts (#22738)
Implement action artifacts server api.

This change is used for supporting
https://github.com/actions/upload-artifact and
https://github.com/actions/download-artifact in gitea actions. It can
run sample workflow from doc
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts.
The api design is inspired by
https://github.com/nektos/act/blob/master/pkg/artifacts/server.go and
includes some changes from gitea internal structs and methods.

Actions artifacts contains two parts:

- Gitea server api and storage (this pr implement basic design without
some complex cases supports)
- Runner communicate with gitea server api (in comming)

Old pr https://github.com/go-gitea/gitea/pull/22345 is outdated after
actions merged. I create new pr from main branch.


![897f7694-3e0f-4f7c-bb4b-9936624ead45](https://user-images.githubusercontent.com/2142787/219382371-eb3cf810-e4e0-456b-a8ff-aecc2b1a1032.jpeg)

Add artifacts list in actions workflow page.
2023-05-19 21:37:57 +08:00
wxiaoguang
6f9c278559
Rewrite queue (#24505)
# ⚠️ Breaking

Many deprecated queue config options are removed (actually, they should
have been removed in 1.18/1.19).

If you see the fatal message when starting Gitea: "Please update your
app.ini to remove deprecated config options", please follow the error
messages to remove these options from your app.ini.

Example:

```
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]`
2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options
```

Many options in `[queue]` are are dropped, including:
`WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`,
`BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed
from app.ini.

# The problem

The old queue package has some legacy problems:

* complexity: I doubt few people could tell how it works.
* maintainability: Too many channels and mutex/cond are mixed together,
too many different structs/interfaces depends each other.
* stability: due to the complexity & maintainability, sometimes there
are strange bugs and difficult to debug, and some code doesn't have test
(indeed some code is difficult to test because a lot of things are mixed
together).
* general applicability: although it is called "queue", its behavior is
not a well-known queue.
* scalability: it doesn't seem easy to make it work with a cluster
without breaking its behaviors.

It came from some very old code to "avoid breaking", however, its
technical debt is too heavy now. It's a good time to introduce a better
"queue" package.

# The new queue package

It keeps using old config and concept as much as possible.

* It only contains two major kinds of concepts:
    * The "base queue": channel, levelqueue, redis
* They have the same abstraction, the same interface, and they are
tested by the same testing code.
* The "WokerPoolQueue", it uses the "base queue" to provide "worker
pool" function, calls the "handler" to process the data in the base
queue.
* The new code doesn't do "PushBack"
* Think about a queue with many workers, the "PushBack" can't guarantee
the order for re-queued unhandled items, so in new code it just does
"normal push"
* The new code doesn't do "pause/resume"
* The "pause/resume" was designed to handle some handler's failure: eg:
document indexer (elasticsearch) is down
* If a queue is paused for long time, either the producers blocks or the
new items are dropped.
* The new code doesn't do such "pause/resume" trick, it's not a common
queue's behavior and it doesn't help much.
* If there are unhandled items, the "push" function just blocks for a
few seconds and then re-queue them and retry.
* The new code doesn't do "worker booster"
* Gitea's queue's handlers are light functions, the cost is only the
go-routine, so it doesn't make sense to "boost" them.
* The new code only use "max worker number" to limit the concurrent
workers.
* The new "Push" never blocks forever
* Instead of creating more and more blocking goroutines, return an error
is more friendly to the server and to the end user.

There are more details in code comments: eg: the "Flush" problem, the
strange "code.index" hanging problem, the "immediate" queue problem.

Almost ready for review.

TODO:

* [x] add some necessary comments during review
* [x] add some more tests if necessary
* [x] update documents and config options
* [x] test max worker / active worker
* [x] re-run the CI tasks to see whether any test is flaky
* [x] improve the `handleOldLengthConfiguration` to provide more
friendly messages
* [x] fine tune default config values (eg: length?)

## Code coverage:

![image](https://user-images.githubusercontent.com/2114189/236620635-55576955-f95d-4810-b12f-879026a3afdf.png)
2023-05-08 19:49:59 +08:00
wxiaoguang
5727056ea1
Make minio package support legacy MD5 checksum (#23768)
A feedback from discord:
https://discord.com/channels/322538954119184384/561007778139734027/1090185427115319386

Some storages like:

 * https://developers.cloudflare.com/r2/api/s3/api/
 * https://www.backblaze.com/b2/docs/s3_compatible_api.html

They do not support "x-amz-checksum-algorithm" header

But minio recently uses that header with CRC32C by default. So we have
to tell minio to use legacy MD5 checksum.

I guess this needs to be backported because IIRC we 1.19 and 1.20 are
using similar minio package.


The minio package code for SendContentMD5 looks like this:

<details>

<img width="755" alt="image"
src="https://user-images.githubusercontent.com/2114189/228186768-4f2f6f67-62b9-4aee-9251-5af714ad9674.png">

</details>
2023-03-28 11:10:24 -04:00
John Olheiser
f92e0a4018
Split CI pipelines (#23385)
- This PR attempts to split our various DB tests into separate
pipelines.
- It splits up some of the extra feature-related tests rather than
having most of them in the MySQL test.
- It disables the race detector for some of the pipelines as well, as it
can cause slower runs and is mostly redundant when the pipelines just
swap DBs.
- It builds without SQLite support for any of the non-SQLite pipelines.
- It moves the e2e test to using SQLite rather than PG (partially
because I moved the minio tests to PG and that mucked up the test
config, and partially because it avoids another running service)
- It splits up the `go mod download` task in the Makefile from the tool
installation, as the tools are only needed in the compliance pipeline.
(Arguably even some of the tools aren't needed there, but that could be
a follow-up PR)
- SQLite is now the only arm64 pipeline, moving PG back to amd64 which
can leverage autoscaler

Should resolve #22010 - one thing that wasn't changed here but is
mentioned in that issue, unit tests are needed in the same pipeline as
an integration test in order to form a complete coverage report (at
least as far as I could tell), so for now it remains in a pipeline with
a DB integration test.

Please let me know if I've inadvertently changed something that was how
it was on purpose.

---

I will say sometimes it's hard to pin down the average time, as a
pipeline could be waiting for a runner for X minutes and that brings the
total up by X minutes as well, but overall this does seem to be faster
on average.

---------

Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2023-03-10 01:13:17 -05:00
Kyle D
c8ded77680
Kd/ci playwright go test (#20123)
* Add initial playwright config

* Simplify Makefile

* Simplify Makefile

* Use correct config files

* Update playwright settings

* Fix package-lock file

* Don't use test logger for e2e tests

* fix frontend lint

* Allow passing TEST_LOGGER variable

* Init postgres database

* use standard gitea env variables

* Update playwright

* update drone

* Move empty env var to commands

* Cleanup

* Move integrations to subfolder

* tests integrations to tests integraton

* Run e2e tests with go test

* Fix linting

* install CI deps

* Add files to ESlint

* Fix drone typo

* Don't log to console in CI

* Use go test http server

* Add build step before tests

* Move shared init function to common package

* fix drone

* Clean up tests

* Fix linting

* Better mocking for page + version string

* Cleanup test generation

* Remove dependency on gitea binary

* Fix linting

* add initial support for running specific tests

* Add ACCEPT_VISUAL variable

* don't require git-lfs

* Add initial documentation

* Review feedback

* Add logged in session test

* Attempt fixing drone race

* Cleanup and bump version

* Bump deps

* Review feedback

* simplify installation

* Fix ci

* Update install docs
2022-09-02 15:18:23 -04:00
Renamed from integrations/pgsql.ini.tmpl (Browse further)