Class: CodeRay::Scanners::Scanner

Inherits:
StringScanner
  • Object
show all
Extends:
CodeRay::Plugin
Includes:
Enumerable
Defined in:
vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb

Overview

Scanner

The base class for all Scanners.

It is a subclass of Ruby’s great StringScanner, which makes it easy to access the scanning methods inside.

It is also Enumerable, so you can use it like an Array of Tokens:

  require 'coderay'

  c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;"

  for text, kind in c_scanner
    puts text if kind == :operator
  end

  # prints: (*==)++;

OK, this is a very simple example :) You can also use map, any?, find and even sort_by, if you want.

Direct Known Subclasses

C, CPlusPlus, CSS, Debug, Delphi, Diff, HTML, JSON, Java, JavaScript, NitroXHTML, PHP, Plaintext, Python, RHTML, Ruby, Scheme, YAML

Constant Summary

ScanError =

Raised if a Scanner fails while scanning

Class.new(Exception)
DEFAULT_OPTIONS =

The default options for all scanner classes.

Define @default_options for subclasses.

{ :stream => false }
KINDS_NOT_LOC =
[:comment, :doctype]

Class Method Summary

Instance Method Summary

Methods included from CodeRay::Plugin

helper, included, plugin_host, plugin_id, register_for, title

Constructor Details

- (Scanner) initialize(code = '', options = {}, &block)

## Excluded for speed reasons; protected seems to make methods slow.

  # Save the StringScanner methods from being called.
  # This would not be useful for highlighting.
  strscan_public_methods =
    StringScanner.instance_methods -
    StringScanner.ancestors[1].instance_methods
  protected(*strscan_public_methods)

Create a new Scanner.

  • code is the input String and is handled by the superclass StringScanner.
  • options is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.)
  • block is the callback for streamed highlighting.

If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.

Else, a Tokens object is used.



119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 119

def initialize code='', options = {}, &block
  raise "I am only the basic Scanner class. I can't scan "\
    "anything. :( Use my subclasses." if self.class == Scanner
  
  @options = self.class::DEFAULT_OPTIONS.merge options

  super Scanner.normify(code)

  @tokens = options[:tokens]
  if @options[:stream]
    warn "warning in CodeRay::Scanner.new: :stream is set, "\
      "but no block was given" unless block_given?
    raise NotStreamableError, self unless kind_of? Streamable
    @tokens ||= TokenStream.new(&block)
  else
    warn "warning in CodeRay::Scanner.new: Block given, "\
      "but :stream is #{@options[:stream]}" if block_given?
    @tokens ||= Tokens.new
  end
  @tokens.scanner = self

  setup
end

Class Method Details

+ (Object) file_extension(extension = nil)



85
86
87
88
89
90
91
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 85

def file_extension extension = nil
  if extension
    @file_extension = extension.to_s
  else
    @file_extension ||= plugin_id.to_s
  end
end

+ (Object) normify(code)



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 69

def normify code
  code = code.to_s
  if code.respond_to? :force_encoding
    debug, $DEBUG = $DEBUG, false
    begin
      code.force_encoding 'utf-8'
      code[/\z/]  # raises an ArgumentError when code contains a non-UTF-8 char
    rescue ArgumentError
      code.force_encoding 'binary'
    ensure
      $DEBUG = debug
    end
  end
  code.to_unix
end

+ (Boolean) streamable?

Returns if the Scanner can be used in streaming mode.

Returns:

  • (Boolean)


65
66
67
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 65

def streamable?
  is_a? Streamable
end

Instance Method Details

- (Object) column(pos = self.pos)



202
203
204
205
206
207
208
209
210
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 202

def column pos = self.pos
  return 0 if pos <= 0
  string = string()
  if string.respond_to?(:bytesize) && (defined?(@bin_string) || string.bytesize != string.size)
    @bin_string ||= string.dup.force_encoding('binary')
    string = @bin_string
  end
  pos - (string.rindex(?\n, pos) || 0)
end

- (Object) each(&block)

Traverses the tokens.

Raises:

  • (ArgumentError)


187
188
189
190
191
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 187

def each &block
  raise ArgumentError,
    'Cannot traverse TokenStream.' if @options[:stream]
  tokens.each(&block)
end

- (Object) line

The current line position of the scanner.

Beware, this is implemented inefficiently. It should be used for debugging only.



198
199
200
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 198

def line
  string[0..pos].count("\n") + 1
end

- (Object) marshal_dump



212
213
214
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 212

def marshal_dump
  @options
end

- (Object) marshal_load(options)



216
217
218
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 216

def marshal_load options
  @options = options
end

- (Object) reset



143
144
145
146
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 143

def reset
  super
  reset_instance
end

- (Boolean) streaming?

Whether the scanner is in streaming mode.

Returns:

  • (Boolean)


182
183
184
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 182

def streaming?
  !!@options[:stream]
end

- (Object) string=(code) Also known as: code=



148
149
150
151
152
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 148

def string= code
  code = Scanner.normify(code)
  super code
  reset_instance
end

- (Object) tokenize(new_string = nil, options = {})

Scans the code and returns all tokens in a Tokens object.



164
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 164

def tokenize new_string=nil, options = {}

- (Object) tokens



177
178
179
# File 'vendor/plugins/coderay-0.9.2/lib/coderay/scanner.rb', line 177

def tokens
  @cached_tokens ||= tokenize
end