Skip to content

Fix Encoding::InvalidByteSequenceError in Aliki on non-UTF-8 locales #1786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ybiquitous wants to merge 1 commit into
base: master
Choose a base branch
from
+27 −3
Open

Fix Encoding::InvalidByteSequenceError in Aliki on non-UTF-8 locales #1786

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions lib/rdoc/generator/aliki.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,10 @@ def resolve_url(rel_prefix, url)

private

def template_encoding
Encoding::UTF_8
end

def build_class_module_entry(klass)
type = case klass
when RDoc::NormalClass then 'class'
Expand Down
10 changes: 7 additions & 3 deletions lib/rdoc/generator/darkfish.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -599,7 +599,7 @@ def refresh_store_data
# used directly.

def assemble_template(body_file)
body = body_file.read
body = body_file.read(encoding: template_encoding)
return body if body =~ /<html/

head_file = @template_dir + '_head.rhtml'
Expand All @@ -609,7 +609,7 @@ def assemble_template(body_file)

<html lang="#{@options.locale&.name || 'en'}">
<head>
#{head_file.read}
#{head_file.read(encoding: template_encoding)}

#{body}
TEMPLATE
Expand Down Expand Up @@ -693,7 +693,7 @@ def template_for(file, page = true, klass = ERB)
template = assemble_template file
erbout = 'io'
else
template = file.read
template = file.read(encoding: template_encoding)
template = template.encode @options.encoding

file_var = File.basename(file).sub(/\..*/, '')
Expand Down Expand Up @@ -805,6 +805,10 @@ def group_classes_by_namespace_for_sidebar(classes)

private

def template_encoding
nil
end
Comment on lines +808 to +810

@ybiquitous ybiquitous Aug 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, this is not a problem. nil means to leave Ruby the default encoding, rather than specifying Encoding.default_external.

nil also works well. Here's some evidence.

Save test_encoding.rb with:

filename = ARGV[0] or abort "specify file"

puts "# Ruby #{RUBY_VERSION}"

f1 = File.read(filename, encoding: nil)
f2 = File.read(filename)
f3 = File.read(filename, encoding: Encoding::UTF_8)

printf "nil == no-arg: %s\n", f1.encoding == f2.encoding && f1 == f2
printf "valid_encoding?: nil-read=%s, utf8-read=%s\n", f1.valid_encoding?, f3.valid_encoding?

begin
  f1.encode(Encoding::UTF_8)
rescue EncodingError => e
  puts "f1.encode(UTF-8) -> #{e.class}"
end

Run (the same output across the Ruby versions):

$ echo '😄' > sample.txt

$ ruby -E US-ASCII test_encoding.rb sample.txt
# Ruby 4.0.6
nil == no-arg: true
valid_encoding?: nil-read=false, utf8-read=true
f1.encode(UTF-8) -> Encoding::InvalidByteSequenceError
$ rbenv shell 3.4.7
$ ruby -E US-ASCII test_encoding.rb sample.txt
# Ruby 3.4.7
nil == no-arg: true
valid_encoding?: nil-read=false, utf8-read=true
f1.encode(UTF-8) -> Encoding::InvalidByteSequenceError
$ rbenv shell 3.3.12
$ ruby -E US-ASCII test_encoding.rb sample.txt
# Ruby 3.3.12
nil == no-arg: true
valid_encoding?: nil-read=false, utf8-read=true
f1.encode(UTF-8) -> Encoding::InvalidByteSequenceError
$ rbenv shell 3.2.11
$ ruby -E US-ASCII test_encoding.rb sample.txt
# Ruby 3.2.11
nil == no-arg: true
valid_encoding?: nil-read=false, utf8-read=true
f1.encode(UTF-8) -> Encoding::InvalidByteSequenceError


def nesting_namespaces_to_class_modules(klass)
tree = {}

Expand Down
16 changes: 16 additions & 0 deletions test/rdoc/generator/aliki_test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,4 +266,20 @@ def test_html_lang_from_locale
content = File.binread('index.html')
assert_include content, '<html lang="ja">'
end

def test_generate_with_non_utf8_default_external_encoding
original_encoding = Encoding.default_external
begin
Encoding.default_external = Encoding::US_ASCII
@g.generate
ensure
Encoding.default_external = original_encoding
end

assert_file 'index.html'

content = File.read('index.html', encoding: Encoding::UTF_8)
assert content.valid_encoding?, 'index.html should be valid UTF-8'
refute content.ascii_only?, 'non-ASCII characters in the template should be kept'
end
end