Feature Request: Add Marked.js Support

[hyrious]

Should be possible by rewriting Marked.js's emStrong tokenizer method. 🙏

It's strikethough (del) method works differently and seems already CJK friendly.

[tats-u]

I didn't know it complies to the latest CommonMark spec. (https://github.com/markedjs/marked/releases/tag/v12.0.0)
I couldn't immediately think of a way to fetch the two previous character.
I want to prioritize ports to non-JS impelementations as for now.

[tats-u]

https://github.com/markedjs/marked/blob/c25dcf94bb31c4d2d40fb206f62d49de06bd2d26/src/Lexer.ts#L450-L454

I don't think it's easy. Doesn't fetch a previous supplementary character properly and will be difficult to make it fetch the 2 previous character according to this design.

[tats-u]

If someone manages to create a plugin, let us know here.

[tats-u]

https://marked.js.org/demo/?text=a**a%E2%88%87**a%0A%0Aa**%E2%88%87a**a%0A%0Aa**a%F0%9D%9C%B5**a%0A%0Aa**%F0%9D%9C%B5a**a&options=%7B%0A%20%22async%22%3A%20false%2C%0A%20%22breaks%22%3A%20false%2C%0A%20%22extensions%22%3A%20null%2C%0A%20%22gfm%22%3A%20true%2C%0A%20%22hooks%22%3A%20null%2C%0A%20%22pedantic%22%3A%20false%2C%0A%20%22silent%22%3A%20false%2C%0A%20%22tokenizer%22%3A%20null%2C%0A%20%22walkTokens%22%3A%20null%0A%7D&version=17.0.1

Marked seems to recognize supplementary characters adjacent to **.

[tats-u]

https://github.com/markedjs/marked/blob/master/src/rules.ts
https://github.com/markedjs/marked/blob/a05d722cb5e14b7b53577078b208517cd274d5ba/src/Tokenizer.ts#L725-L744

Marked seems to delegate the offending process to regexs. I am not good at modifying such regexs as long as pasta.
Someone who are going to implement an extension should be careful of ReDoS.

[tats-u]

It's strikethough (del) method works differently and seems already CJK friendly.

It is considered as a bug by Marked: markedjs/marked#3863

[hyrious]

Yes, this is quite complex. I'm thinking of an easy way to solve common cases (instead of going with the full spec in this repo), at least a hack for Marked.js users:

The idea is that, CJK punctuation characters should not be treated the same as western puncts. The later is more like a "space", and there's a basic rule: There should not be spaces surrounding an emphasis. So, I tried to disable all unicode flags in Marked.js rules, it turns out to be working well in trivial cases:

import { marked } from 'marked'

console.log(marked(`**このアスタリスクは強調記号として認識されず、そのまま表示されます。**この文のせいで。

**该星号不会被识别，而是直接显示。**这是因为它没有被识别为强调符号。

**이 별표는 강조 표시로 인식되지 않고 그대로 표시됩니다(이 괄호 때문에)**이 문장 때문에.`, { async: false }))

console.log(marked('这是一个\\*\\***“会引起”**渲染错误的**“已知问题”**，当加重符号\\*\\*遇到某些中文标点时，可能就会出现**“识别不了”**的情况。就如这句话展现的一样。', { async: false }))

console.log(marked('**ChatGPT**の本質は、**「文章を扱える知性」**にあります。', { async: false }))

Outputs:

<p><strong>このアスタリスクは強調記号として認識されず、そのまま表示されます。</strong>この文のせいで。</p>
<p><strong>该星号不会被识别，而是直接显示。</strong>这是因为它没有被识别为强调符号。</p>
<p><strong>이 별표는 강조 표시로 인식되지 않고 그대로 표시됩니다(이 괄호 때문에)</strong>이 문장 때문에.</p>

<p>这是一个**<strong>“会引起”</strong>渲染错误的<strong>“已知问题”</strong>，当加重符号**遇到某些中文标点时，可能就会出现<strong>“识别不了”</strong>的情况。就如这句话展现的一样。</p>

<p><strong>ChatGPT</strong>の本質は、<strong>「文章を扱える知性」</strong>にあります。</p>

The hack is written as a bun/esbuild plugin and should be easy to port to other bundlers.

// src/hack.ts, as a Bun plugin
import fs from 'node:fs';
import { plugin } from 'bun';

plugin({
	name: 'Replace Marked.js "gu" with "g"',
	setup(build) {
		build.onLoad({ filter: /marked\.(esm|umd)\.js$/ }, args => {
			let contents = fs.readFileSync(args.path, 'utf8'); {
			contents = contents.replaceAll('"gu"', '"g" '); // Space for not breaking source map
			contents = contents.replaceAll('"u")', '"" )');
			contents = contents.replaceAll('/u', '/ ');
			return { contents };
		});
	}
});

Update: It passed all cases provided in those closed issues mentioned above.