The main purpose is to refactor the legacy `unknwon/com` package.
1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com`
2. Use golangci's depguard to process denied packages
3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"`
4. Use correctly escaped content for `go-import` and `go-source` meta tags
5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
The service worker causes a lot of issues with JS errors after instance
upgrades while not bringing any real performance gain over regular HTTP
caching.
Disable it by default for this reason. Maybe later we can remove it
completely, as I simply see no benefit in having it.
- Use a better and more curated list of Ciphers and KeyExchanges, these roughly follows OpenSSH's default.
- Remove some cryptography values which were deprecated.
WebAuthn may cause a security exception if the provided APP_ID is not allowed for the
current origin. Therefore we should reattempt authentication without the appid
extension.
Also we should allow [u2f] as-well as [U2F] sections.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Several users run Gitea in situations whereby `bash` is not available.
If the `SCRIPT_TYPE` is not changed this will cause hooks to fail.
A simple test to check if the provided type is on the PATH should be
sufficient to warn them about this problem.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Make router logger more friendly, show the related function name/file/line.
[BREAKING]
This PR substantially changes the logging format of the router logger. If you use this logging for monitoring e.g. fail2ban you will need to update this to match the new format.
Migrate from U2F to Webauthn
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
It appears that several versions of sendmail require that the mail is sent to them with
LF line endings instead of CRLF endings - which of course they will then convert back
to CRLF line endings to comply with the SMTP standard.
This PR adds another setting SENDMAIL_CONVERT_CRLF which will pass the message writer
through a filter. This will filter out and convert CRLFs to LFs before writing them
out to sendmail.
Fix#18024
Signed-off-by: Andrew Thornton <art27@cantab.net>
Save a bit of bandwidth by only requesting 3-times the rendered avatar
size. Factor 4 is only really beneficial on a handful of mobile phones
and I don't think they are the primary device we design for.
Configurability contributed by zeripath.
Fixes: https://github.com/go-gitea/gitea/pull/17422
Fixes: https://github.com/go-gitea/gitea/issues/16287
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* allways set a message-id on mails
* Add unit tests for mailer & Message-ID
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Refactor install page (db type)
* set correct default DB HOST for different DB TYPE
* remove legacy TiDB from documents
* unify the usage of DB TYPE, in code we only use "mysql". "MySQL" is only shown to users for friendly name.
* Gitea can use TiDB via MySQL protocol
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve install code to avoid low-level mistakes.
If a user tries to do a re-install in a Gitea database, they gets a warning and double check.
When Gitea runs, it never create empty app.ini automatically.
Also some small (related) refactoring:
* Refactor db.InitEngine related logic make it more clean (especially for the install code)
* Move some i18n strings out from setting.go to make the setting.go can be easily maintained.
* Show errors in CLI code if an incorrect app.ini is used.
* APP_DATA_PATH is created when installing, and checked when starting (no empty directory is created any more).
This PR adds another option to app.ini make co-committed-by and co-authored-by trailers
optional on a per server basis.
Fix#17194
Signed-off-by: Andrew Thornton <art27@cantab.net>
Make relative unix sockets absolute by making them absolute against the AppWorkPath
Fix#17833
## ⚠️ BREAKING ⚠️
Prior to this PR relative unix sockets would have been asserted to be relative to the current working directory that gitea, gitea serv, hook and manager etc were running in. Hooks and Serv would have failed to work properly under this situation so we expect that although this is a technically breaking change the previous situation was already broken.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use hostmacher to replace matchlist.
And we introduce a better DialContext to do a full host/IP check, otherwise the attackers can still bypass the allow/block list by a 302 redirection.
* Add settings to allow different SMTP envelope from address
Sometimes it may be advisable to hide or alias the from address on an SMTP mail
envelope. This PR adds two new options to the mailer to allow setting of an overriding
from address.
Fix#17477
Signed-off-by: Andrew Thornton <art27@cantab.net>
Gitea writes its own AppPath into git hook scripts. If Gitea's AppPath changes, then the git push will fail.
This PR:
* Introduce an AppState module, it can persist app states into database
* During GlobalInit, Gitea will check if the current AppPath is the same as last one. If they don't match, Gitea will sync git hooks.
* Refactor some code to make them more clear.
* Also, "Detect if gitea binary's name changed" #11341 is related, we call models.RewriteAllPublicKeys to update ssh authorized_keys file
Convert the old mirror syncing queue to the more modern queue format.
Fix a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead.
The documentation is also updated to add information about the deprecated configuration values.
Signed-off-by: Andrew Thornton <art27@cantab.net>
core.protectNTFS protects NTFS from files which may be difficult to remove or interact
with using the win32 api, however, it also appears to prevent such files from
being entered into the git indexes - fundamentally causing breakages with PRs that
affect these files. However, deliberately setting this to false may cause security
issues due to the remain sparse checkout of files in the merge pipeline.
The only sensible option therefore is to provide an optional setting which admins
could set which would forcibly switch this off if they are affected by this issue.
Fix#17092
Signed-off-by: Andrew Thornton <art27@cantab.net>
Nodeinfo is a way to expose certain metadata about a server for use of discovery regarding functionality of its federation capabilities.
Two endpoints are required:
1. `/.well-known/nodeinfo` which informs client where it can find the location of the location of its metadata (including which version of the schema is used)
2. the endpoint which exposes the metadata in json format according to schema.
Notes:
* `openRegistrations` is a required field, but I propose to set to false as default in case someone writes a crawler to discover "open" gitea instances
* to limit data leakage I also propose to not include the `usage` field (note it is required so it should be included, but left as empty).
More info:
https://github.com/jhass/nodeinfohttps://github.com/jhass/nodeinfo/tree/main/schemas/2.1http://nodeinfo.diaspora.software/protocol.html
Add a new default theme `auto`, which will automatically switch between
`gitea` (light) and `arc-green` (dark) themes depending on the user's
operating system settings.
Closes: #8183
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Rename`[repository]` `DISABLE_MIRRORS` to `[mirror]` `DISABLE_NEW_PULL` and add `ENABLED` and `DISABLE_NEW_PUSH` with the below meanings:
- `ENABLED`: **true**: Enables the mirror functionality. Set to **false** to disable all mirrors.
- `DISABLE_NEW_PULL`: **false**: Disable the creation of **new** mirrors. Pre-existing mirrors remain valid.
- `DISABLE_NEW_PUSH`: **false**: Disable the creation of **new** push mirrors. Pre-existing mirrors remain valid.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
* Add proxy settings and support for migration and webhook
* Fix default value
* Add newline for example ini
* Add lfs proxy support
* Fix lint
* Follow @zeripath's review
* Fix git clone
* Fix test
* missgin http requests for proxy
* use empty
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
One of the issues holding back performance of the API is the problem of hashing.
Whilst banning BASIC authentication with passwords will help, the API Token scheme
still requires a PBKDF2 hash - which means that heavy API use (using Tokens) can
still cause enormous numbers of hash computations.
A slight solution to this whilst we consider moving to using JWT based tokens and/or
a session orientated solution is to simply cache the successful tokens. This has some
security issues but this should be balanced by the security issues of load from
hashing.
Related #14668
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Allow setting X-FRAME-OPTIONS
This PR provides a mechanism to set the X-FRAME-OPTIONS header.
Fix#7951
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
* Make Mermaid.js limit configurable
Add `MERMAID_MAX_SOURCE_CHARACTERS` to `[markup]` settings
to make the maximum size of a mermaid render configurable.
Fix#16513
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fixup! Make Mermaid.js limit configurable
* Update custom/conf/app.example.ini
Co-authored-by: silverwind <me@silverwind.io>
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* Update templates/admin/hook_new.tmpl
Co-authored-by: a1012112796 <1012112796@qq.com>
* Update services/webhook/wechatwork.go
Co-authored-by: a1012112796 <1012112796@qq.com>
* 修善wechatwork
* 修善wechatwork
* fix
* Update locale_cs-CZ.ini
fix
* fix build
* fix
* fix build
* make webhooks.zh-cn.md
* delet unnecessary blank line
* delet unnecessary blank line
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* Update templates/admin/hook_new.tmpl
Co-authored-by: a1012112796 <1012112796@qq.com>
* Update services/webhook/wechatwork.go
Co-authored-by: a1012112796 <1012112796@qq.com>
* 修善wechatwork
* 修善wechatwork
* fix
* fix build
* fix
* fix build
* make webhooks.zh-cn.md
* delet unnecessary blank line
* delet unnecessary blank line
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* fix
* fix
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* fix wechat
* fix wechat
* fix wechat
* fix wechat
* Fix invalid params and typo of email templates (#16394)
Signed-off-by: Meano <meanocat@gmail.com>
* Add LRU mem cache implementation (#16226)
The current default memory cache implementation is unbounded in size and number of
objects cached. This is hardly ideal.
This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea.
The cache is limited by the number of objects stored in the cache (rather than size)
for simplicity. The default number of objects is 50000 - which is perhaps too small
as most of our objects cached are going to be much less than 1kB.
It may be worth considering using a different LRU implementation that actively limits
sizes or avoids GC - however, this is just a beginning implementation.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* [skip ci] Updated translations via Crowdin
* Replace `plugins/docker` with `techknowlogick/drone-docker`in ci (#16407)
* plugins/docker -> techknowlogick/drone-docker
* It is multi-arch
* docs: rewrite email setup (#16404)
* Add intro for both the docs page and mailer methods
* Fix numbering level in SMTP section
* Recommends implicit TLS
Signed-off-by: Bagas Sanjaya <bagasdotme@gmail.com>
* Validate Issue Index before querying DB (#16406)
* Fix external renderer (#16401)
* fix external renderer
* use GBackground context as fallback
* no fallback, return error
Co-authored-by: Lauris BH <lauris@nix.lv>
* Add checkbox to delete pull branch after successful merge (#16049)
* Add checkbox to delete pull branch after successful merge
* Omit DeleteBranchAfterMerge field in json
* Log a warning instead of error when PR head branch deleted
* Add DefaultDeleteBranchAfterMerge to PullRequestConfig
* Add support for delete_branch_after_merge via API
* Fix for API: the branch should be deleted from the HEAD repo
If head and base repo are the same, reuse the already opened ctx.Repo.GitRepo
* Don't delegate to CleanupBranch, only reuse branch deletion code
CleanupBranch contains too much logic that has already been performed by the Merge
* Reuse gitrepo in MergePullRequest
Co-authored-by: Andrew Thornton <art27@cantab.net>
* [skip ci] Updated translations via Crowdin
* Detect encoding changes while parsing diff (#16330)
* Detect encoding changes while parsing diff
* Let branch/tag name be a valid ref to get CI status (#16400)
* fix #16384#
* refactor: move shared helper func to utils package
* extend Tests
* use ctx.Repo.GitRepo if not nil
* fix
* fix
* 企业微信webhook
* 企业微信webhook
* 企业微信webhook
* fix build
* fix build
* Apply suggestions from code review
Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: myheavily <myheavily>
Co-authored-by: zhaoxin <gitea@fake.local>
Co-authored-by: Meano <Meano@foxmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: GiteaBot <teabot@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Bagas Sanjaya <bagasdotme@gmail.com>
Co-authored-by: Norwin <noerw@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Jimmy Praet <jimmy.praet@telenet.be>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* add configuration option to restrict users by default
* default IsRestricted permission only set on sign up
setting this in the model messes with other workflows (e.g. syncing LDAP users) where the IsRestricted permission needs to be explicitly set and not overridden by a config value
* fix formatting
* Apply suggestions from code review
* ensure newly created user is set to restricted
* ensure imports are in the correct order
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
The current default memory cache implementation is unbounded in size and number of
objects cached. This is hardly ideal.
This PR proposes creating a TwoQueue LRU cache as the underlying cache for Gitea.
The cache is limited by the number of objects stored in the cache (rather than size)
for simplicity. The default number of objects is 50000 - which is perhaps too small
as most of our objects cached are going to be much less than 1kB.
It may be worth considering using a different LRU implementation that actively limits
sizes or avoids GC - however, this is just a beginning implementation.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Following the merging of https://github.com/go-git/go-git/pull/330 we
can now add a setting to avoid go-git reading and caching large objects.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Now that #16069 is merged, some sites may wish to enforce that users are all public, limited or private, and/or disallow users from becoming private.
This PR adds functionality and settings to constrain a user's ability to change their visibility.
Co-authored-by: zeripath <art27@cantab.net>
One of the repeatedly reported issues has been that gitea produces too much console
logging during set up even if the console logger is turned off.
Fundamentally this is due to some otherwise very helpful logging that has to occur
before logging is set up. This has come to a head with the merging of #16243 where
otherwise potentially helpful Trace logging in the git module now appears on the
console.
This PR proposes three things:
1. Change the initial default logger to Info not Trace.
2. Change the logging for the AppPath things to Info in recompense.
3. Add two new command line options to gitea web: --quiet and --verbose
`gitea web -q` or `gitea web --quiet` will only log Fatal level initially.
`gitea web -verbose` will log at Trace.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
You can limit or hide organisations. This pull make it also posible for users
- new strings to translte
- add checkbox to user profile form
- add checkbox to admin user.edit form
- filter explore page user search
- filter api admin and public user searches
- allow admins view "hidden" users
- add app option DEFAULT_USER_VISIBILITY
- rewrite many files to use Visibility field
- check for teams intersection
- fix context output
- right fake 404 if not visible
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Andrew Thornton <art27@cantab.net>
* Added support for gopher URLs.
* Add setting and make this user settable instead
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
* Ensure settings for Service and Mailer are read on the install page
NewContext does not set the mailer or service settings so add
a new function that will run this.
Fix#15894
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are a couple of settings in `[indexer]` relating to the `issue_indexer` queue
which override settings in unpredictable ways. This PR adjusts this hierarchy and makes
explicit that these settings are deprecated.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Set self-adjusting deadline for connection writing
In #16055 it appears that the simple 5s deadline doesn't work for large
file writes. Now we can't - or at least shouldn't just set no deadline
as go will happily let these connections block indefinitely. However,
what seems reasonable is to set some minimum rate we expect for writing.
This PR suggests the following algorithm:
* Every write has a minimum timeout of 5s (adjustable at compile time.)
* If there has been a previous write - then consider its previous
deadline, add half of the minimum timeout + 2s per kb about to written.
* If that new deadline is after the minimum timeout use that.
Fix#16055
* Linearly increase timeout
* Make PerWriteTimeout, PerWritePerKbTimeouts configurable
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
As revealed by #15964 there is inconsistent use of filepath Join and path Join
for these directories. The best thing to do is to use filepath.Join but then ToSlash
them for consistency.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
This PR suggests a change to the default configuration for queues:
* Use a common DATADIR for the queues
* Set starting workers to 0 and make boost a single worker
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add a new internal hook to save ssh log
as title, when a ssh error ocure like #15785.
only when switch ``RUN_MODE`` to dev can we
found which error is ocure. But this way is
not a good idea for production envirment.
this changes try save ssh error mesage to the
log file like other log by a new internal hook.
I think it's usefull for find error message
in production envirment. Thanks.
Signed-off-by: a1012112796 <1012112796@qq.com>
* rename and fix nit
* Update modules/private/hook.go
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Refactored handleOAuth2SignIn in routers/user/auth.go
The function handleOAuth2SignIn was called twice but some code path could only
be reached by one of the invocations. Moved the unnecessary code path out of
handleOAuth2SignIn.
* Refactored user creation
There was common code to create a user and display the correct error message.
And after the creation the only user should be an admin and if enabled a
confirmation email should be sent. This common code is now abstracted into
two functions and a helper function to call both.
* Added auto-register for OAuth2 users
If enabled new OAuth2 users will be registered with their OAuth2 details.
The UserID, Name and Email fields from the gothUser are used.
Therefore the OpenID Connect provider needs additional scopes to return
the coresponding claims.
* Added error for missing fields in OAuth2 response
* Linking and auto linking on oauth2 registration
* Set default username source to nickname
* Add automatic oauth2 scopes for github and google
* Add hint to change the openid connect scopes if fields are missing
* Extend info about auto linking security risk
Co-authored-by: Viktor Kuzmin <kvaster@gmail.com>
Signed-off-by: Martin Michaelis <code@mgjm.de>
* _ to unused func options
* rm useless brakets
* rm trifial non used models functions
* rm dead code
* rm dead global vars
* fix routers/api/v1/repo/issue.go
* dont overload import module
Implements request #14320 The rendering of CSV files does match the diff style.
* Moved CSV logic into base package.
* Added method to create a tabular diff.
* Added CSV compare context.
* Added CSV diff template.
* Use new table style in CSV markup.
* Added file size limit for CSV rendering.
* Display CSV parser errors in diff.
* Lazy read single file.
* Lazy read rows for full diff.
* Added unit tests for various CSV changes.
in previouse if a log subsetcion not set level
it will use ``info`` as default value.
this pr will make default value (``[log] -> LEVEL``) useable.
example config:
```INI
[log]
MODE = console
LEVEL = Trace
[log.console]
LEVEL =
STDERR = false
```
previous result:
```JSON
// console:
{
"level": "info",
...................
}
```
after change:
```JSON
// console:
{
"level": "track",
...................
}
```
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Add reverse proxy configuration support for remote IP address validation
* Trust all IP addresses in containerized environments by default
* Use single option to specify networks and proxy IP addresses. By default trust all loopback IPs
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This is an alternative PR to #13687.
Add `[ui.explore]` settings to allow restricting the
explore pages to logged in users only and to disable the users explore page.
The two proposed settings are:
- `REQUIRE_SIGNIN_VIEW`: Only allows access to the explore pages if the
user is signed in. Also restricts
- `/api/v1/user/search`
- `/api/v1/users/{username}`
- `/api/v1/users/{username}/repos`
- but does not restrict `/api/v1/users/{username}/heatmap`
- `DISABLE_USERS_PAGE`: Disables the /explore/users page
Fix#2908Close#13687
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Make SSH server host key path configurable
* make it possible to have multiple keys
* Make gitea.rsa the default key
* Add some more logging
Signed-off-by: Andrew Thornton <art27@cantab.net>
Add SameSite setting for cookies and rationalise the cookie setting code. Switches SameSite to Lax by default.
There is a possible future extension of differentiating which cookies could be set at Strict by default but that is for a future PR.
Fix#5583
Signed-off-by: Andrew Thornton <art27@cantab.net>
