1
1
mirror of https://github.com/go-gitea/gitea synced 2024-11-18 08:04:25 +00:00
Commit Graph

169 Commits

Author SHA1 Message Date
silverwind
d4f28fd4ad
Fix URL of gitea emoji (#15770)
Fixes regression from #15219
2021-05-07 17:34:33 +02:00
Lunny Xiao
9d99f6ab19
Refactor renders (#15175)
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
Lunny Xiao
66f0fd0959
Add NeedPostProcess for Parser interface to improve performance of csv render (#15153) 2021-04-13 15:06:31 +08:00
zeripath
c680eb2cc7
Links in markdown should be absolute to the repository not the server (#15088)
* Links in markdown should be absolute to the repository not the server

Fix #15075

Signed-off-by: Andrew Thornton <art27@cantab.net>

* match github

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
2021-04-11 00:26:28 +08:00
zeripath
b9ed3cbc26
Upgrade to bluemonday 1.0.7 (#15379)
* Upgrade to bluemonday 1.0.7

Fix #15349

Signed-off-by: Andrew Thornton <art27@cantab.net>

* resolve unit test

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-09 19:13:06 -04:00
6543
9c4601bdf8
Code Formats, Nits & Unused Func/Var deletions (#15286)
* _ to unused func options

* rm useless brakets

* rm trifial non used models functions

* rm dead code

* rm dead global vars

* fix routers/api/v1/repo/issue.go

* dont overload import module
2021-04-09 09:40:34 +02:00
zeripath
04196b7658
Update to bluemonday-1.0.6 (#15294)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-04-05 17:38:31 -04:00
KN4CK3R
0c6137617f
Add Tabular Diff for CSV files (#14661)
Implements request #14320 The rendering of CSV files does match the diff style.

* Moved CSV logic into base package.

* Added method to create a tabular diff.

* Added CSV compare context.

* Added CSV diff template.

* Use new table style in CSV markup.

* Added file size limit for CSV rendering.

* Display CSV parser errors in diff.

* Lazy read single file.

* Lazy read rows for full diff.

* Added unit tests for various CSV changes.
2021-03-29 22:44:28 +02:00
zeripath
b68eb54f95
Clusterfuzz found another way (#15160)
Clusterfuzz found another way so I found another way to stop it

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-26 01:21:00 +02:00
zeripath
a587a28434
Fix another clusterfuzz identified issue (#15096)
* Fix another clusterfuzz identified issue

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-22 02:45:01 +01:00
zeripath
c0c052bdbb
another clusterfuzz spotted issue (#15032)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 22:37:11 +01:00
zeripath
71aca93dec
Remove extraneous logging (#15020)
Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-18 04:25:52 +01:00
zeripath
ed31ddc29a
Fix several render issues (#14986)
* Fix an issue with panics related to attributes
* Wrap goldmark render in a recovery function
* Reduce memory use in render emoji
* Use a pipe for rendering goldmark - still needs more work and a limiter

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2021-03-16 00:20:05 +01:00
zeripath
164e35ead3
Make sure sibling images get a link too (#14979)
* Make sure sibling images get a link too

Due a problem with the ast.Walker in the our transformer in goldmark
an image with a sibling image will not be transformed to gain a parent
link. This PR fixes this.

Fix #12925

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-03-14 16:36:51 +00:00
zeripath
aa4f9180e4
Clarify the suffices and prefixes of setting.AppSubURL and setting.AppURL (#12999)
Also removes some unnecessary uses of fmt.Sprintf and adds documentation
strings

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-19 22:36:43 +01:00
zeripath
7ab6c77b41
Remove NULs byte arrays passed to PostProcess (#14587)
PostProcess is supposed to be parsing and handling HTML
fragments, but on fuzzing it appears that there is a weird
issue with NUL elements that could cause a memory address
error in downstream libraries.

The simplest solution is to strip out the weird NULs - they
should not be there in any case and would be stripped out
anyway.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-18 02:32:14 +01:00
zeripath
172229966c
Prevent panic on fuzzer provided string (#14405)
* Prevent panic on fuzzer provided string

The fuzzer has found that providing a <body> tag with an attribute to
PostProcess causes a panic. This PR removes any rendered html or body
tags from the output.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Placate lint

* placate lint again

Signed-off-by: Andrew Thornton <art27@cantab.net>

* minor cleanup

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-01-20 23:10:50 +08:00
Nuno Silva
44ff1d6a1e
Render links for commit hashes followed by comma (#14224)
Regex test cases: https://regex101.com/r/mVbPxM/2/

fixes #14223
2021-01-03 23:11:10 +08:00
Lunny Xiao
11555d850b
Fix bug of link query order on markdown render (#14156)
* Fix bug of link query order on markdown render

* Fix bluemonday bug and fix one wrong test

Co-authored-by: 6543 <6543@obermui.de>
2020-12-29 00:28:27 +08:00
a1012112796
34df4e5df5
Add mentionable teams to tributeValues and change team mention rules to gh's style (#13198)
* Add mentionable teams to tributeValues

Signed-off-by: a1012112796 <1012112796@qq.com>

* Apply suggestions from code review

Co-authored-by: silverwind <me@silverwind.io>

* Change team mention rules to gh's style

* use org's avator as team avator in ui

Signed-off-by: a1012112796 <1012112796@qq.com>

* Update modules/markup/html.go

* Update models/issue.go

Co-authored-by: Lauris BH <lauris@nix.lv>

* Update models/issue.go

* fix a small nit and update test code

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
2020-12-21 16:39:28 +01:00
silverwind
41e8b834af
Markdown task list improvements (#13952)
* Markdown task list improvements

- Remove `.ui` class and wrappers to prevent fomantic from messing with it.
- Change rendered HTML to match GitHub.
- Add custom styling for the checkboxes.

* fix unittest

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-12-13 09:05:50 +08:00
mrsdizzie
3512c7e40f
Fix crash in short link processor (#13839)
Fixes #13819
2020-12-04 03:01:42 +01:00
Jimmy Praet
056b8f5b52
Clickable links in pull request (and issue) titles (#13695)
* clickable links in pull request (and issue) titles #13658

reuses the existing logic to render clickable links in commit messages

* dedicated RenderIssueTitle function

applied patch from @mrsdizzie
2020-12-03 11:50:47 +01:00
kolaente
64133126cd
Update golangci-lint to version 1.31.0 (#13102)
This PR updates golangci-lint to the latest version 1.31.0.

The upgrade introduced a new check for which I've fixed or disabled most cases.

Signed-off-by: kolaente <k@knt.li>
2020-10-11 21:27:20 +01:00
Pranav Nachnekar
1827f892de
fix: media links in org files not liked to media files (#12997)
* fix: media links in org files not liked to media files

* fix: write directly to io.Writer r

as suggested by code review

Co-authored-by: zeripath <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2020-10-01 11:22:34 -04:00
John Olheiser
b2bda11595
Fix markdown meta parsing (#12817)
* Fix meta parsing and add minimal test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add license

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Require first separator and add more tests

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Make fmt

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Go lets this work gracefully 🙏

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-09-12 21:48:47 -04:00
zeripath
f91bb745d6
Don't replace underscores in auto-generated IDs in goldmark (#12805)
Fix #12196

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-09-12 12:00:48 -04:00
John Olheiser
26c4a049da
Issue templates directory (#11450)
* Issue templates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add some comments, appease the linter

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add docs and re-use dir candidates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add default labels to issue templates

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Generate swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Suggested changes

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Update issue.go

* Suggestions

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Extract metadata from legacy if possible

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-09-11 10:48:39 -04:00
zeripath
9fdb4f887b
Update to go-org 1.3.2 (#12728)
* Update to go-org 1.3.2

Fix #12727

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Fix unit test

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-09-05 11:45:10 -04:00
silverwind
ee047312a1
Fix emoji replacements, make emoji images consistent (#12567)
- Fix emoji not being replaced in issue title change text
- Make the image attributes consistent, add alt, remove align

Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-08-23 21:44:53 -04:00
zeripath
74bd9691c6
Re-attempt to delete temporary upload if the file is locked by another process (#12447)
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.

Fix #12339

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
2020-08-11 21:05:34 +01:00
zeripath
e770c2b850
Detect full references to issues and pulls in commit messages (#12399)
Fix #10269

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-08-06 20:20:05 +01:00
silverwind
e61c09ed73
Add loading spinners and mermaid error handling (#12358)
- Add loading spinners on editor and mermaid renderers
- Add error handling and inline error box for mermaid
- Fix Mermaid rendering by using the .init api
2020-08-04 20:56:37 +01:00
mrsdizzie
ea1ed802a3
Fix emoji detection in certain cases (#12320)
* Fix emoji detection certain cases

Previous tests weren't complicated enough so there were some situations where emojis were't detected properly. Find the earliest occurance in addition to checking for the longest combination.

Fixes #12312

* ok spell bot

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-25 16:40:04 +03:00
mrsdizzie
af7ffaa279
Server-side syntax highlighting for all code (#12047)
* Server-side syntax hilighting for all code

This PR does a few things:

* Remove all traces of highlight.js
* Use chroma library to provide fast syntax hilighting directly on the server
* Provide syntax hilighting for diffs
* Re-style both unified and split diffs views
* Add custom syntax hilighting styling for both regular and arc-green

Fixes #7729
Fixes #10157
Fixes #11825
Fixes #7728
Fixes #3872
Fixes #3682

And perhaps gets closer to #9553

* fix line marker

* fix repo search

* Fix single line select

* properly load settings

* npm uninstall highlight.js

* review suggestion

* code review

* forgot to call function

* fix test

* Apply suggestions from code review

suggestions from @silverwind thanks

Co-authored-by: silverwind <me@silverwind.io>

* code review

* copy/paste error

* Use const for highlight size limit

* Update web_src/less/_repository.less

Co-authored-by: Lauris BH <lauris@nix.lv>

* update size limit to 1MB and other styling tweaks

* fix highlighting for certain diff sections

* fix test

* add worker back as suggested

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-01 00:34:03 +03:00
silverwind
2447ffc74a
Disable all typographic replacements in markdown renderer (#11871)
* Disable all typographic replacements in markdown renderer

Previously we only disabled some of them. This disables all the default
replacements that goldmark's typographer extension offers, matching
GitHub's renderer.

Ref: https://github.com/yuin/goldmark#typographer-extension
Fixes: https://github.com/go-gitea/gitea/issues/11001

* remove typographer extension completely

* fix test

* really fix test

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-06-13 02:10:17 -04:00
mrsdizzie
4c1ff57f1a
Update emoji regex (#11584)
When matching emoji, use a regex built from the data we have instead of something generic using unicode ranges. A generic regex can't tell the difference between two separate emoji next to each other or one emoji that is built out of two separate emoji next to each other.

This means that emoji that are next to each other without space in between will be now accurately spanned individually with proper title etc...
2020-05-29 17:08:36 +01:00
zeripath
814ca9ffea
Allow different HardBreaks settings for documents and comments (#11515)
GH has different HardBreaks behaviour for markdown comments and documents.

Comments have hard breaks and documents have soft breaks - therefore Gitea's rendering will always be different from GH's if we only provide one setting.

Here we split the setting in to two - one for documents and one for comments and other things.

Signed-off-by: Andrew Thornton art27@cantab.net

Changes to index.js as per @silverwind 
Co-authored-by: silverwind <me@silverwind.io>

Changes to docs as per @guillep2k 
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-05-24 09:14:26 +01:00
zeripath
742e26f5a5
Prevent 500 with badly formed task list (#11328)
Fix #11317

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-05-11 00:14:49 +01:00
zeripath
ce66ca7f9f
Restore checkbox rendering and prevent poor sanitization of spans (#11277)
* Add test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Restore checkbox rendering and prevent poor sanitization of spans

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also fix preview context

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also fix preview context

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-05-03 17:17:24 -03:00
mrsdizzie
4ed7d2a2bb
Remove last traces of has-emoji class (#11263)
Now that emojify.js has been removed, get rid of all instances of has-emoji class that was only used for that. Support for rendering shortcodes should remain in all of these places so it should still work the same.

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-05-01 20:58:45 +03:00
mrsdizzie
4563eb873d
Support unicode emojis and remove emojify.js (#11032)
* Support unicode emojis and remove emojify.js

This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.

This works in a few ways:

First it adds emoji parsing support into gitea itself. This allows us to

 * Render emojis from valid alias (😄)
 * Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
 * Easily allow for custom "emoji"
 * Support all emoji rendering and features without javascript
 * Uses plain unicode and lets the system render in appropriate emoji font
 * Doesn't leave us relying on external sources for updates/fixes/features

That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)

For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.

The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.

I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.

I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.

Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.

Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130

* add new shared function emojiHTML

* don't increase emoji size in issue title

* Update templates/repo/issue/view_content/add_reaction.tmpl

Co-Authored-By: 6543 <6543@obermui.de>

* Support for emoji rendering in various templates

* Render code and review comments as they should be

* Better way to handle mail subjects

* insert unicode from tribute selection

* Add template helper for plain text when needed

* Use existing replace function I forgot about

* Don't include emoji greater than Unicode Version 12

Only include emoji and aliases in JSON

* Update build/generate-emoji.go

* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have

* final updates

* code review

* code review

* hard code gitea custom emoji to match previous behavior

* Update .eslintrc

Co-Authored-By: silverwind <me@silverwind.io>

* disable preempt

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
2020-04-28 15:05:39 -03:00
zeripath
0e799c26ba
Prevent panic during wrappedConn close at hammertime (#11219)
* Prevent panic during wrappedConn close at hammertime

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/graceful/server.go

* Fix extraneous debug in goldmark.go

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-04-26 21:01:06 -03:00
zeripath
9f959ac064
Make TaskCheckBox render correctly (#11214)
* Fix checkbox rendering

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Normalize checkbox rendering

Signed-off-by: Andrew Thornton <art27@cantab.net>

* set the checkboxes to readonly instead of disabled

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-04-26 02:09:08 -03:00
zeripath
812cfd0ad9
Use markdown frontmatter to provide Table of contents, language and frontmatter rendering (#11047)
* Add control for the rendering of the frontmatter
* Add control to include a TOC
* Add control to set language - allows control of ToC header and CJK glyph choice.

Signed-off-by: Andrew Thornton art27@cantab.net
2020-04-24 14:22:36 +01:00
silverwind
7e20f1cb5b
enable ENABLE_HARD_LINE_BREAK by default (#11162)
enable ENABLE_HARD_LINE_BREAK by default

This matches GitHub's rendering.

Fix #11155
2020-04-21 23:13:56 +01:00
zeripath
0be25e2150
Handle yaml frontmatter (#11016)
Add goldmark-meta to render yaml frontmatter as a table

Fix #5377

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-04-09 11:54:50 +01:00
techknowlogick
d00ebf445b
upgrade to most recent bluemonday (#11007)
* upgrade to most recent bluemonday

* make vendor

* update tests for bluemonday

* update tests for bluemonday

* update tests for bluemonday
2020-04-07 23:08:47 +03:00
zeripath
cf4f2cfa3e
Fix internal server error on checkboxes (#10845)
Annoyingly goldmarks SetAttributeString requires that
the value of the attribute is still a []byte but does
not make it clear in the documentation.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-03-26 23:47:34 +00:00
zeripath
f036de5a94
Completely fix task-list checkbox styling (#10798)
Continuing on from #10668 this PR makes goldmark apply the task-list styling to task-lists.
2020-03-22 22:25:38 +00:00
Alexander Scheel
b8551f8532
Fix task-list checkbox styling (#10668)
* Fix task-list checkbox styling

The pandoc renderer will append the class "task-list" to the ul element
wrapping a li with one or more check-boxes. This allows us to select for
them, removing their list-style-type property. However, goldmark and the
gfm spec doesn't specify the "task-list" class name, so we can't use
that to fix the issue there.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Update to goldmark v1.1.25

This version adds the missing space after a checkbox.

Resolves: #9656

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

Co-authored-by: zeripath <art27@cantab.net>
2020-03-08 21:17:03 +02:00
zeripath
154b137b6d
Relax sanitization as per https://github.com/jch/html-pipeline (#10527)
Looking at github/markup#245 it is clear that GH uses https://github.com/jch/html-pipeline to sanitize. This PR relaxes our sanitization to more closely match this.

Fixes #10471
and likely others...
2020-02-28 20:05:12 +00:00
zeripath
9ad2aa8f4d
Use media links for img in post-process (#10515)
* use media links for img in post-process

* do not visit text of anchors
2020-02-28 01:16:05 -03:00
6543
e57ac841de
Fix potential bugs (#10513)
* use e if it is an option
* potential nil so check err first
* check err first
* m == nil already checked
2020-02-27 20:10:27 -03:00
mrsdizzie
b5f28d154d
Don't manually replace whitespace during render (#10291)
* Don't manually replace whitespace during render

For historical reasons Gitea manually alters the urlPrefix and replaces
a whitespace with a +. This Works for URLs, but we're also passing
urlPrefix to git calls and adding the + is breaking the tree path.

Goldmark will automatically convert a white space to the proper %20, so
we should leave the string as is which lets us pass it to git unmodified
and then let Goldmark fix it.

Also fixed separate bug in URLJoin I noticed while testing where it will
silently discard sections of a path that have # in them (possibly
others). We should just escape it first.

Fixes 10156

* Escape elems as well

* Revert "Escape elems as well"

This reverts commit 8bf49596fe.

* restart ci

* remove changes to URLJoin

* restart ci

Co-authored-by: techknowlogick <matti@mdranta.net>
2020-02-17 07:46:51 +02:00
John Olheiser
53f9dbfc7b Don't convert ellipsis in markdown (#9905)
* Don't convert ellipsis

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Formatting

Co-Authored-By: zeripath <art27@cantab.net>

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-01-22 16:30:20 +01:00
John Olheiser
7d7ab1eeae Issue/PR Context Popups (#9822)
* Add data-index attribute to issue anchors

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Init JS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add required data to anchor

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Finish popup

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Revert changes to html.go

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Better octicon contexts

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Split out popup function for re-use

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Style changes, test fixes, and cross-reference support

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Prefer em to px

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move label margin to base CSS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to separate file.

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move JS to features and fix module

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Remove query-string and hash

Co-Authored-By: silverwind <me@silverwind.io>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
2020-01-19 23:39:21 -05:00
zeripath
948dec3d75
Allow hyphen in language name (#9873) 2020-01-19 21:36:00 +00:00
Moritz
f740943df4 Fix markdown anchor links (#9673)
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-01-16 12:23:48 +01:00
zeripath
27757714d0 Change markdown rendering from blackfriday to goldmark (#9533)
* Move to goldmark

Markdown rendering moved from blackfriday to the goldmark.

Multiple subtle changes required to the goldmark extensions to keep
current rendering and defaults.

Can go further with goldmark linkify and have this work within markdown
rendering making the link processor unnecessary.

Need to think about how to go about allowing extensions - at present it
seems that these would be hard to do without recompilation.

* linter fixes

Co-authored-by: Lauris BH <lauris@nix.lv>
2019-12-31 03:53:28 +02:00
John Olheiser
e6ceb6880a Prefix all user-generated IDs in markup (#9477)
* Prefix all user-generated IDs in markup
* Add user-content- to IDs in unit-tests
* fixup markdown_test.go
* update the hrefs for the wiki test
* Add blackfriday extension regex

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-23 22:38:50 +00:00
Alexander Scheel
ee7df7ba8c Markdown: Sanitizier Configuration (#9075)
* Support custom sanitization policy

Allowing the gitea administrator to configure sanitization policy allows
them to couple external renders and custom templates to support more
markup. In particular, the `pandoc` renderer allows generating KaTeX
annotations, wrapping them in `<span>` elements with class `math` and
either `inline` or `display` (depending on whether or not inline or
block mode was requested).

This iteration gives the administrator whitelisting powers; carefully
crafted regexes will thus let through only the desired attributes
necessary to support their custom markup.

Resolves: #9054

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Document new sanitization configuration

 - Adds basic documentation to app.ini.sample,
 - Adds an example to the Configuration Cheat Sheet, and
 - Adds extended information to External Renderers section.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Drop extraneous length check in newMarkupSanitizer(...)

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>

* Fix plural ELEMENT and ALLOW_ATTR in docs

These were left over from their initial names. Make them singular to
conform with the current expectations.

Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
2019-12-07 14:49:04 -05:00
John Olheiser
9a33adcdd3 Allow kbd tags (#9245)
* Allow kbd tags

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Add test

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-03 14:02:41 -05:00
guillep2k
6a90c7e3dd Alternate syntax for cross references (#9116)
* Add support for local vs. remote xrefs

* Add doc for references

* Docs: fix cases not currently supported

* One more doc fix

* Doc: mentions for teams and orgs

* Change !num ref concept, no change in functionality

* Fix test

* Improve table of issue reference types

* Fix paragraph mark
2019-12-01 15:57:05 +02:00
guillep2k
f25fd5c8eb Fix team links in HTML rendering (#9127)
* Fix team links in HTML rendering

* Fix check and lint
2019-11-24 11:34:44 -05:00
zeripath
675f275237
Fix race condition in ReplaceSanitizer (#9123) 2019-11-22 14:46:39 +00:00
guillep2k
b15f26b1cf Close/reopen issues by keywords in titles and comments (#8866)
* Add close/reopen from comment functionality

* Fix comment

* Rewrite closing/reopening template

* Check xref permissions, move action to services/pull

* Fix RefIsPull field

* Add xref tests

* Fix xref unique filter

* Only highlight keywords for actionable xrefs

* Fix xref neuter filter

* Fix check return status

* Restart CI
2019-11-18 15:13:07 +02:00
guillep2k
7b97e04555 Convert EOL to UNIX-style to render MD properly (#8925)
* Convert EOL to UNIX-style to render MD properly

* Update modules/markup/markdown/markdown.go

Co-Authored-By: zeripath <art27@cantab.net>

* Fix lint optimization

* Check for empty content before conversion

* Update modules/util/util.go

Co-Authored-By: zeripath <art27@cantab.net>

* Improved checks and tests

* Add paragraph render test

* Improve speed even more, improve tests

* Small improvement by @gary-kim

* Fix test for DOS

* More improvements

* Restart CI
2019-11-13 03:27:11 +01:00
Lauris BH
05e7715c4b Update go-org to optimize code (#8824) 2019-11-05 16:39:03 +08:00
Lauris BH
086a46994a Rewrite markdown rendering to blackfriday v2 and rewrite orgmode rendering to go-org (#8560)
* Rewrite markdown rendering to blackfriday v2.0

* Fix style

* Fix go mod with golang 1.13

* Fix blackfriday v2 import

* Inital orgmode renderer migration to go-org

* Vendor go-org dependency

* Ignore errors :/

* Update go-org to latest version

* Update test

* Fix go-org test

* Remove unneeded code

* Fix comments

* Fix markdown test

* Fix blackfriday regression rendering HTML block
2019-10-31 01:06:25 +00:00
guillep2k
cea8ea5ae6 Support inline rendering of CUSTOM_URL_SCHEMES (#8496)
* Support inline rendering of CUSTOM_URL_SCHEMES

* Fix lint

* Add tests

* Fix lint
2019-10-15 02:31:09 +01:00
guillep2k
15809d81f7 Rewrite reference processing code in preparation for opening/closing from comment references (#8261)
* Add a markdown stripper for mentions and xrefs

* Improve comments

* Small code simplification

* Move reference code to modules/references

* Fix typo

* Make MarkdownStripper return [][]byte

* Implement preliminary keywords parsing

* Add FIXME comment

* Fix comment

* make fmt

* Fix permissions check

* Fix text assumptions

* Fix imports

* Fix lint, fmt

* Fix unused import

* Add missing export comment

* Bypass revive on implemented interface

* Move mdstripper into its own package

* Support alphanumeric patterns

* Refactor FindAllMentions

* Move mentions test to references

* Parse mentions from reference package

* Refactor code to implement renderizable references

* Fix typo

* Move patterns and tests to the references package

* Fix nil reference

* Preliminary rendering attempt of closing keywords

* Normalize names, comments, general tidy-up

* Add CSS style for action keywords

* Fix permission for admin and owner

* Fix golangci-lint

* Fix golangci-lint
2019-10-13 23:29:10 +01:00
jaqra
7e17424c7e Make link last commit massages in repository home page and commit tables (#8006)
* Make link last commit massages in repository home page and commit tables

* Use RenderCommitMessageLink instead surround with a

* deleted __debug_bin file

* Exclude email to link from latest commit title

* Exclude email processor from commit table

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Add class parameter to a html element creator functions.
Make links underline dashed that are not commit

* fix tests

* Show dashed underline when also not hovered
2019-09-10 12:03:30 +03:00
Rinat
b660a732ae feat: highlight issue references with : (#8101)
* feat: highlight issue references with :

e.g. #1287: my commit msg
e.g. ABC-1234: my commit msg

* ref: update model regex to consistent with issueNumericPattern

* test: check highlight issue with : in commits messages
2019-09-05 20:39:54 -04:00
Tamal Saha
171b359877 Use gitea forked macaron (#7933)
Signed-off-by: Tamal Saha <tamal@appscode.com>
2019-08-23 12:40:29 -04:00
Norwin
867f46f78e Detect delimiter in CSV rendering (#7869)
* detect csv delimiter in csv rendering

fixes #7868

* make linter happy

* fix failing testcase & use ints where possible

* expose markup type to template

previously all markup had the .markdown class, which is incorrect,
as it applies markdown CSS & JS logic to CSV rendering

* fix build (missing `make css`)

* ignore quoted csv content for delimiter scoring

also fix html generation
2019-08-16 01:09:50 +03:00
Gary Kim
7eed11e5e9 Check commit message hashes before making links (#7713)
* Check commit message hashes before making links

Previously, when formatting commit messages, anything
that looked like SHA1 hashes was turned into a link
using regex. This meant that certain phrases or numbers
such as `777777` or `deadbeef` could be recognized as a commit
even if the repository has no commit with those hashes.

This change will make it so that anything that looks
like a SHA1 hash using regex will then also be checked
to ensure that there is a commit in the repository
with that hash before making a link.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use gogit to check if commit exists

This commit modifies the commit hash check
in the render for commit messages to use
gogit for better performance.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Make code cleaner

Signed-off-by: Gary Kim <gary@garykim.dev>

* Use rev-parse to check if commit exists

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add and modify tests for checking hashes in html link rendering

Signed-off-by: Gary Kim <gary@garykim.dev>

* Return error in sha1CurrentPatternProcessor

Co-Authored-By: mrsdizzie <info@mrsdizzie.com>

* Import Gitea log module

Signed-off-by: Gary Kim <gary@garykim.dev>

* Revert "Return error in sha1CurrentPatternProcessor"

This reverts commit 28f561cac4.

Signed-off-by: Gary Kim <gary@garykim.dev>

* Add debug logging to sha1CurrentPatternProcessor

This will log errors by the git command run in
sha1CurrentPatternProcessor if the error is one
that was unexpected.

Signed-off-by: Gary Kim <gary@garykim.dev>
2019-08-14 16:04:55 +08:00
Christian Muehlhaeuser
54d96c79b5 Removed unnecessary conversions (#7557)
No need to convert to the same type.
2019-07-23 19:50:39 +01:00
Lunny Xiao
8ad64574ee Fix markdown invoke sequence (#7513) 2019-07-18 16:23:27 -04:00
silverwind
b209531959 fix hljs unintenionally highlighting commit links (#7244)
* fix hljs unintenionally highlighting commit links

* fix unit tests
2019-06-18 18:31:31 -04:00
kolaente
f9ec2f89f2 Add golangci (#6418) 2019-06-12 15:41:28 -04:00
Lunny Xiao
355ab0c62c Fix wrong init dependency on markup extensions (#7038)
* fix wrong init dependency on markup extensions
2019-05-25 18:15:39 +01:00
mrsdizzie
0064535ad2 Fix domain name pattern in email regex (#6739)
Fixes #6735
2019-04-24 21:53:41 -04:00
mrsdizzie
1bce1894f5 Use ctx.metas for SHA hash links (#6645)
Since #6273 was merged, we now have access to proper context metas
always. Update SHA generated links to use these instead of urlPrefix.

Update tests as well.

Fixes #4536.
2019-04-16 08:53:57 +01:00
mrsdizzie
3ff0a126e1 Improve issue autolinks (#6273)
* Improve issue autolinks

Update autolinks to match what github does here:

Issue in same repo: #1
Issue in different repo: org/repo#1

Fixes #6264

* Use setting.AppURL when parsing URL

Using setting.AppURL here is a more reliable way of parsing the current
URL and what other functions in this file seem to use.

* Make ComposeMetas always return a valid context

* Add per repository markdown renderers for better context

* Update for use of context metas

Now that we include the user and repo name inside context metas, update
various code and tests for this new logic
2019-04-12 08:53:34 +03:00
silverwind
8e949db3b5 Render SHA1 links as code blocks (#6546) 2019-04-09 06:18:48 +03:00
mrsdizzie
6293736d02 Use stricter boundaries for auto-link detection (#6522)
* Use stricter boundaries for auto-link detection

Currently autolinks use \W for boundary detection which creates many
situations of inserting links into places they don't belong (paths,
URLs, UUIDs, etc...)

This fixes that by replacing \W and only allowing these matches to touch
an open paren or bracket (matching what seems to be Github behavior) in
addition to whitespace and start of line. Similar for ending boundary as
well.

Fixes #6149
(and probably others)

* Update test

Replace incorrect test with a value that is a valid username, based on:

"Username should contain only alphanumeric, dash ('-'), underscore ('_')
and dot ('.') characters."

* Also allow for period at the end

Matching Github behavior

* Fix email regex to work properly with specificed boundaries

Create a specific capture group for email address and then use
FindStringSubmatchIndex to allow for non-matching patterns as
boundaries.

* Add Tests

Add tests for new behavior -- including tests for email addresses which
were absent before.
2019-04-07 12:18:16 +01:00
silverwind
2242a9f82e Improve SHA1 link detection (#6526)
This improves the SHA1 link detection to not pick up extraneous
non-whitespace characters at the end of the URL. The '.' is a special
case handled in code itself because of missing regexp lookahead
support.

Regex test cases: https://regex101.com/r/xUMlqh/3
2019-04-06 21:28:45 +03:00
zeripath
704da08fdc
Better logging (#6038) (#6095)
* Panic don't fatal on create new logger

Fixes #5854

Signed-off-by: Andrew Thornton <art27@cantab.net>

* partial broken

* Update the logging infrastrcture

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Reset the skip levels for Fatal and Error

Signed-off-by: Andrew Thornton <art27@cantab.net>

* broken ncsa

* More log.Error fixes

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Remove nal

* set log-levels to lowercase

* Make console_test test all levels

* switch to lowercased levels

* OK now working

* Fix vetting issues

* Fix lint

* Fix tests

* change default logging to match current gitea

* Improve log testing

Signed-off-by: Andrew Thornton <art27@cantab.net>

* reset error skip levels to 0

* Update documentation and access logger configuration

* Redirect the router log back to gitea if redirect macaron log but also allow setting the log level - i.e. TRACE

* Fix broken level caching

* Refactor the router log

* Add Router logger

* Add colorizing options

* Adjust router colors

* Only create logger if they will be used

* update app.ini.sample

* rename Attribute ColorAttribute

* Change from white to green for function

* Set fatal/error levels

* Restore initial trace logger

* Fix Trace arguments in modules/auth/auth.go

* Properly handle XORMLogger

* Improve admin/config page

* fix fmt

* Add auto-compression of old logs

* Update error log levels

* Remove the unnecessary skip argument from Error, Fatal and Critical

* Add stacktrace support

* Fix tests

* Remove x/sync from vendors?

* Add stderr option to console logger

* Use filepath.ToSlash to protect against Windows in tests

* Remove prefixed underscores from names in colors.go

* Remove not implemented database logger

This was removed from Gogs on 4 Mar 2016 but left in the configuration
since then.

* Ensure that log paths are relative to ROOT_PATH

* use path.Join

* rename jsonConfig to logConfig

* Rename "config" to "jsonConfig" to make it clearer

* Requested changes

* Requested changes: XormLogger

* Try to color the windows terminal

If successful default to colorizing the console logs

* fixup

* Colorize initially too

* update vendor

* Colorize logs on default and remove if this is not a colorizing logger

* Fix documentation

* fix test

* Use go-isatty to detect if on windows we are on msys or cygwin

* Fix spelling mistake

* Add missing vendors

* More changes

* Rationalise the ANSI writer protection

* Adjust colors on advice from @0x5c

* Make Flags a comma separated list

* Move to use the windows constant for ENABLE_VIRTUAL_TERMINAL_PROCESSING

* Ensure matching is done on the non-colored message - to simpify EXPRESSION
2019-04-02 08:48:31 +01:00
mrsdizzie
c8650aef0a Change order that PostProcess Processors are run (#6445)
Make sure Processors that work on full links are run first so that
something matching another pattern doesn't alter a link before we get to
it, for example:

 https://stackoverflow.com/questions/2896191/what-is-go-used-fore

Fixes #4813
2019-03-27 11:37:54 -04:00
Mura Li
d77176912b Use Go1.11 module (#5743)
* Migrate to go modules

* make vendor

* Update mvdan.cc/xurls

* make vendor

* Update code.gitea.io/git

* make fmt-check

* Update github.com/go-sql-driver/mysql

* make vendor
2019-03-27 19:15:23 +08:00
Roland Koebler
57ecf09787 Markdown: enable some more extensions (#6362)
* Markdown: enable some more extensions

Improve Markdown-rendering by enabling some extensions:

- enable definitions lists
- enable footnotes
- enable header-ids and automatically generate header-ids (for linking to README-sections or creating table-of-contents for larger READMEs)

* Markdown: update and exted tests

Update and add tests for additionally enabled Markdown-extensions.
2019-03-21 15:53:06 +02:00
zeripath
b2e9894988 Fix reported issue in repo description (#6306) 2019-03-11 22:23:34 -04:00
mrsdizzie
f2de5dc8c8 Replace linkRegex with xurls library (#6261)
* Replace linkRegex with xurls library

Rather than maintaining a complicated regex to match URLs for
autolinking, gitea can use this existing go library that takes care of
the matching with very little code change to gitea itself. After
spending a while trying to find the perfect regex for all cases this library
still works better as it is more flexible than a single regex ever will be.

This will also fix the following issues: #5844 #3095 #3381

This passes all our current tests and I've added new ones mentioned in
those issues as well.

* Use xurls.StrictMatchingScheme instead of xurls.Strict

This is much faster and we only care about https? links to preserve
existing behavior.
2019-03-07 15:12:01 -05:00
mrsdizzie
020075e12f Remove visitLinksForShortLinks features (#6257)
The visitLinksForShortLinks feature would look inside of an <a> tag and
run shortLinkProcessorFull on any text, which attempts to create links
out of potential 'short links' like [[test]] [[link|example]] etc...
This makes no sense because you can't have nested links within an <a>
tag. Specifically, the html5 standard says <a> tags can't include
interactive content if they contain the href attribute:

 http://w3c.github.io/html/single-page.html#the-a-element

And also defines an <a> element with a href attribute as interactive:

 http://w3c.github.io/html/single-page.html#interactive-content

Therefore you can't really put a link inside of another link. In
practice none of this works anyways since browsers won't render it, it
would probably be broken if they tried, and it is causing a bug
(#4946). No current tests rely on this behavior either.

This removes the feature and also explicitly excludes the
current visitNodeForShortLinks from looking in <a> tags.
2019-03-07 14:13:44 -05:00
mrsdizzie
4a2e92bcd1 Modify linkRegex to require http|https (#6171)
Modify the current linkRegex to require http|https which appears to be
the intended behavior based on the comments. Right now, it also matches
anything starting with www as well. Also add testing for linkRegex
2019-02-28 20:31:53 +08:00
zeripath
2a03e96bce
Allow markdown files to read from the LFS (#5787)
This PR makes it possible for the markdown renderer to render images and media straight from the LFS.

Fix #5746

Signed-off-by: Andrew Thornton [art27@cantab.net](mailto:art27@cantab.net)
2019-02-12 15:09:43 +00:00
zeripath
0823791d17 Recover panic in orgmode.Render if bad orgfile (#4982) (#5903)
This PR protects against the panic referred to in chaseadmsio/goorgeous#82
by recovering from the panic and just returning the raw bytes if
there is an error.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-01-30 17:17:43 -05:00
Khaled Hamed
bd75965296 Prioritize "readme.md" (#5691)
* prioritize readme.md

* Improve IsReadmeFile

* Add more tests
2019-01-14 14:15:06 -05:00
Lunny Xiao
cef0f12c51 support envs on external render commands (#5278) 2018-11-20 17:11:21 -05:00