Class: CodeRay::Scanners::Scanner
- Inherits:
-
StringScanner
- Object
- StringScanner
- CodeRay::Scanners::Scanner
- Extends:
- CodeRay::Plugin
- Includes:
- Enumerable
- Defined in:
- vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb
Overview
Scanner
The base class for all Scanners.
It is a subclass of Ruby’s great StringScanner, which makes it easy to access the scanning methods inside.
It is also Enumerable, so you can use it like an Array of Tokens:
require 'coderay' c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;" for text, kind in c_scanner puts text if kind == :operator end # prints: (*==)++;
OK, this is a very simple example :) You can also use map, any?, find and even sort_by, if you want.
Direct Known Subclasses
C, CPlusPlus, CSS, Debug, Delphi, Diff, HTML, JSON, Java, JavaScript, NitroXHTML, PHP, Plaintext, Python, RHTML, Ruby, Scheme, YAML
Constant Summary
- ScanError =
Raised if a Scanner fails while scanning
Class.new(Exception)
- DEFAULT_OPTIONS =
The default options for all scanner classes.
Define @default_options for subclasses.
{ :stream => false }
- KINDS_NOT_LOC =
[:comment, :doctype]
Class Method Summary
- + (Object) file_extension(extension = nil)
- + (Object) normify(code)
-
+ (Boolean) streamable?
Returns if the Scanner can be used in streaming mode.
Instance Method Summary
- - (Object) column(pos = self.pos)
-
- (Object) each(&block)
Traverses the tokens.
-
- (Scanner) initialize(code = '', options = {}, &block)
constructor
## Excluded for speed reasons; protected seems to make methods slow.
-
- (Object) line
The current line position of the scanner.
- - (Object) marshal_dump
- - (Object) marshal_load(options)
- - (Object) reset
-
- (Boolean) streaming?
Whether the scanner is in streaming mode.
- - (Object) string(code) (also: #code=)
-
- (Object) tokenize(new_string = nil, options = {})
Scans the code and returns all tokens in a Tokens object.
- - (Object) tokens
Methods included from CodeRay::Plugin
helper, included, plugin_host, plugin_id, register_for, title
Constructor Details
- (Scanner) initialize(code = '', options = {}, &block)
## Excluded for speed reasons; protected seems to make methods slow.
# Save the StringScanner methods from being called. # This would not be useful for highlighting. strscan_public_methods = StringScanner.instance_methods - StringScanner.ancestors[1].instance_methods protected(*strscan_public_methods)
Create a new Scanner.
- code is the input String and is handled by the superclass StringScanner.
- options is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.)
- block is the callback for streamed highlighting.
If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.
Else, a Tokens object is used.
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 119 def initialize code='', = {}, &block raise "I am only the basic Scanner class. I can't scan "\ "anything. :( Use my subclasses." if self.class == Scanner = self.class::DEFAULT_OPTIONS.merge super Scanner.normify(code) @tokens = [:tokens] if [:stream] warn "warning in CodeRay::Scanner.new: :stream is set, "\ "but no block was given" unless block_given? raise NotStreamableError, self unless kind_of? Streamable @tokens ||= TokenStream.new(&block) else warn "warning in CodeRay::Scanner.new: Block given, "\ "but :stream is #{@options[:stream]}" if block_given? @tokens ||= Tokens.new end @tokens.scanner = self setup end |
Class Method Details
+ (Object) file_extension(extension = nil)
85 86 87 88 89 90 91 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 85 def file_extension extension = nil if extension @file_extension = extension.to_s else @file_extension ||= plugin_id.to_s end end |
+ (Object) normify(code)
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 69 def normify code code = code.to_s if code.respond_to? :force_encoding debug, $DEBUG = $DEBUG, false begin code.force_encoding 'utf-8' code[/\z/] # raises an ArgumentError when code contains a non-UTF-8 char rescue ArgumentError code.force_encoding 'binary' ensure $DEBUG = debug end end code.to_unix end |
+ (Boolean) streamable?
Returns if the Scanner can be used in streaming mode.
65 66 67 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 65 def streamable? is_a? Streamable end |
Instance Method Details
- (Object) column(pos = self.pos)
202 203 204 205 206 207 208 209 210 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 202 def column pos = self.pos return 0 if pos <= 0 string = string() if string.respond_to?(:bytesize) && (defined?(@bin_string) || string.bytesize != string.size) @bin_string ||= string.dup.force_encoding('binary') string = @bin_string end pos - (string.rindex(?\n, pos) || 0) end |
- (Object) each(&block)
Traverses the tokens.
187 188 189 190 191 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 187 def each &block raise ArgumentError, 'Cannot traverse TokenStream.' if [:stream] tokens.each(&block) end |
- (Object) line
The current line position of the scanner.
Beware, this is implemented inefficiently. It should be used for debugging only.
198 199 200 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 198 def line string[0..pos].count("\n") + 1 end |
- (Object) marshal_dump
212 213 214 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 212 def marshal_dump end |
- (Object) marshal_load(options)
216 217 218 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 216 def marshal_load = end |
- (Object) reset
143 144 145 146 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 143 def reset super reset_instance end |
- (Boolean) streaming?
Whether the scanner is in streaming mode.
182 183 184 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 182 def streaming? !![:stream] end |
- (Object) string=(code) Also known as: code=
148 149 150 151 152 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 148 def string= code code = Scanner.normify(code) super code reset_instance end |
- (Object) tokenize(new_string = nil, options = {})
Scans the code and returns all tokens in a Tokens object.
164 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 164 def tokenize new_string=nil, = {} |
- (Object) tokens
177 178 179 |
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 177 def tokens @cached_tokens ||= tokenize end |