Syntaxt highlighting is extremely slow. Can this be improved in my code?

[zach-cloud]

I'm writing an application which includes a syntax highlighter for a language called JASS. The syntax highlighting works properly and performs great on small code files. However this language compiles all its code into one file and so regularly has code files that range between 50,000 and 100,000 lines.

When the syntax highlighter runs on these files, it can take 15 seconds or more to highlight the entire thing, and since this runs whenever the code changes, the program is unusable. For now, I have set the syntax highlighting to be toggleable to avoid this problem.

Is this a limitation of the project or am I doing something wrong in my code? Is there a known best practice for this?

My code currently looks like this:

    private void setupKeywords() {
        addKeywords(blizzardLoaderService.loadCommon());
        addKeywords(blizzardLoaderService.loadBlizzard());
        natives.add("string");
        natives.add("integer");
        natives.add("real");
        natives.add("boolean");
        String[] array = natives.toArray(new String[0]);
        String KEYWORDS_PATTERN = "\\b(" + String.join("|", keywords) + ")\\b";
        String NATIVE_PATTERN = "\\b(" + String.join("|", array) + ")\\b";
        String PAREN_PATTERN = "\\(|\\)";
        String BRACKET_PATTERN = "\\[|\\]";
        String STRING_PATTERN = "\"([^\"\\\\]|\\\\.)*\"";
        String COMMENT_PATTERN = "//[^\n]*" + "|" + "/\\*(.|\\R)*?\\*/";

        pattern = Pattern.compile(
                "(?<NATIVE>" + NATIVE_PATTERN + ")"
                        + "|(?<KEYWORD>" + KEYWORDS_PATTERN + ")"
                        + "|(?<PAREN>" + PAREN_PATTERN + ")"
                        + "|(?<BRACKET>" + BRACKET_PATTERN + ")"
                        + "|(?<STRING>" + STRING_PATTERN + ")"
                        + "|(?<COMMENT>" + COMMENT_PATTERN + ")"
        );
    }

    public void setupHighlighting() {
        setupKeywords();
        highlighting = jassCodeEditor
                .multiPlainChanges()
                .successionEnds(Duration.ofMillis(500))
                .subscribe(ignore -> jassCodeEditor.setStyleSpans(0, computeHighlighting(jassCodeEditor.getText())));

        final Pattern whiteSpace = Pattern.compile("^\\s+");
        jassCodeEditor.addEventHandler(KeyEvent.KEY_PRESSED, KE ->
        {
            if (KE.getCode() == KeyCode.ENTER) {
                int caretPosition = jassCodeEditor.getCaretPosition();
                int currentParagraph = jassCodeEditor.getCurrentParagraph();
                Matcher m0 = whiteSpace.matcher(jassCodeEditor.getParagraph(currentParagraph - 1).getSegments().get(0));
                if (m0.find()) Platform.runLater(() -> jassCodeEditor.insertText(caretPosition, m0.group()));
            }
        });
    }

    private StyleSpans<Collection<String>> computeHighlighting(String text) {
        Matcher matcher = pattern.matcher(text);
        int lastKwEnd = 0;
        StyleSpansBuilder<Collection<String>> spansBuilder
                = new StyleSpansBuilder<>();
        while (matcher.find()) {
            String styleClass =
                    matcher.group("NATIVE") != null ? "native" :
                            matcher.group("KEYWORD") != null ? "keyword" :
                                    matcher.group("PAREN") != null ? "paren" :
                                            matcher.group("BRACKET") != null ? "bracket" :
                                                    matcher.group("STRING") != null ? "string" :
                                                            matcher.group("COMMENT") != null ? "comment" :
                                                                    null; /* never happens */
            assert styleClass != null;
            spansBuilder.add(Collections.emptyList(), matcher.start() - lastKwEnd);
            spansBuilder.add(Collections.singleton(styleClass), matcher.end() - matcher.start());
            lastKwEnd = matcher.end();
        }
        spansBuilder.add(Collections.emptyList(), text.length() - lastKwEnd);
        return spansBuilder.create();
    }

[lanthale]

It seems that you are computing the highlights not in a background thread and therefore the UI is blocked. Have a look into the asyncHighlightDemo file in the repository to see how you can do that in the background and the UI stays responsive for the usera again.

[Jugen]

For files that large you should change the computeHighlighting code to just work on the text of the current paragraph that has been edited, and also only computeHighlighting on changes to the visible paragraphs list.

[zach-cloud]

Thanks so much for the help! I will look into those.

[Jugen]

Hi @zach-cloud could you try this out and let me know how it works for you:

class VisibleParagraphStyler<PS, SEG, S> implements Consumer<ListModification>
{
    private final GenericStyledArea<PS, SEG, S> area;
    private final Function<String,StyleSpans<S>> computeStyles;
    private int prevParagraph, prevTextLength;

    public VisibleParagraphStyler( GenericStyledArea<PS, SEG, S> area, Function<String,StyleSpans<S>> computeStyles )
    {
        this.computeStyles = computeStyles;
        this.area = area;
    }

    @Override
    public void accept( ListModification lm )
    {
        if ( lm.getAddedSize() > 0 )
        {
            int paragraph = Math.min( area.firstVisibleParToAllParIndex() + lm.getFrom(), area.getParagraphs().size()-1 ) ;
            String text = area.getText( paragraph, 0, paragraph, area.getParagraphLength( paragraph ) );

            if ( paragraph != prevParagraph || text.length() != prevTextLength )
            {
                int startPos = area.getAbsolutePosition( paragraph, 0 );
                Platform.runLater( () -> area.setStyleSpans( startPos, computeStyles.apply( text ) ) );
                prevTextLength = text.length();
                prevParagraph = paragraph;
            }
        }
    }
}

Use it by replacing the highlighting = .... block in setupHighlighting() with:

jassCodeEditor.getVisibleParagraphs().addModificationObserver
(
    new VisibleParagraphStyler<>( jassCodeEditor, this::computeHighlighting )
);

[zach-cloud]

That works perfectly. Thanks so much! Definitely above and beyond the support I was expecting :)

[Jugen]

You are welcome.

Note, that I did update the code above - so just want to make sure you are not using the original post.

[zach-cloud]

Updated my code :) Thanks.