Hi John,

this comes from #675.

This is about enabling language notation for document parts.

The notation could be (in the spirit of .class and #identifier):

:lang :es :en :de :grc

I think a new element RawSpan would be also needed to add language notation in some passages (not all of them, but some).

Many thanks for your excellent work,

Pablo

Well, this issue has been opened for almost 15 months.

Although the proposal should be more complete, I want to focus in this minimal part: would it be possible that extended markdown has the special attribute :lang?

I think that hardcoding the language tag in HTML should be avoided. And as a special attribute the user has less to type.

Otherwise, a simple example such as:

_Legibility_ is hyphenated differently than _Lesbarkeit_.

should be written now in extended markdown:

_Legibility_ is hyphenated differently than <span lang="de">_Lesbarkeit_</span>.

instead of the simpler proposal:

_Legibility_ is hyphenated differently than _Lesbarkeit_{:de}.

The proposal has the following benefits:

• The user avoids double tagging (that can be seen as punishment for passages in foreign languages).

• It would be easier to parse to other formats than XML. (Right now, hyphenation is ignored when converted to textile, LaTeX, ConTeXt, RTF, OpenDocument, and (I guess) DOCX.

  The issue with TeX implementations is that they have hyphenation enabled by default. Other formats that allow hyphenation disable it by default.

• Even when parsing to HTML, it would be easier to parse language to xml:lang when required.

@jgm, what do you think about this?

Using <span lang="XX">...</span> is butt ugly. I would really like to not have HTML syntax.

Many thanks for your comment, @mpickering.

I think pandoc needs first to be able to set the document language with only one attribute value for YAML lang variable (#1614).

I don't think there is high enough demand for language specific syntax …

I'm not sure either, but whatever the outcome of the discussion on syntax is, I feel that now that we have the mapping from lang to polyglossia-lang, babel-langetc., we should also implement the conversion from <span lang="XX">…</span> and <div lang="XX">…</div> to the appropriate latex commands or environments:

For babel and polyglossia, e.g.:

If/when a consensus on a language specific syntax emerges, support for it could be added easily later.

@nickbart1980 Are the mappings from <span lang="XX"> to \textYYY{…} the same as from lang to babel-lang? What about XeTeX, LuaTeX and ConTeXt?

Are the mappings from <span lang="XX"> to \textYYY{…} the same as from lang to babel-lang?

<span lang="XX"> to \textYYY{…} is for polyglossia, but: yes, AFAIK all the mappings for in-text commands and environments match those currently in LaTeX.hs, both for babel and polyglossia – with one exception for the latter:

“… for Arabic one cannot use the environment arabic, as \arabic is defined internally by LaTeX. In this case we need to use the environment Arabic.” (polyglossia manual)

I edited the table above to include the babel syntax; I have no idea about LuaTeX and ConTeXt, though.

EDIT: It seems we might be able to use the babel syntax for polyglossia, too:

“Some macros defined in babel’s hyphen.cfg (and thus usually compiled into the XƎLATEX and LuaLATEX format) are redefined, but keep a similar behaviour, namely \selectlanguage, \foreignlanguage, and the environment otherlanguage.” (polyglossia manual)

I haven’t tested this, however, and I’m not sure how well this works for language variants etc.

EDIT 2: As I suspected: With polyglossia/xelatex, \foreignlanguage{english}{\today} and \foreignlanguage{german}{\today} work; \foreignlanguage{british}{\today} and \foreignlanguage{austrian}{\today} don’t (no output at all); see report: reutenauer/polyglossia#112.

… or we could define our own commands using BCP47 tags, e.g. \IETFlang{en-GB}{Blah.}.

These could be defined in the latex template, using xparse, along these lines (this is an example for polyglossia only):

\ExplSyntaxOn
\NewDocumentCommand{\IETFlang}{ m m }
 {
  \str_case:nnn { #1 }
   {
    { ar    } { \textarabic{#2} }
    { de-DE } { \textgerman{#2} }
    { de-AT } { \textgerman[variant=austrian]{#2} }
    { en-US } { \textenglish{#2} }
    { en-GB } { \textenglish[variant=british]{#2} }
    { fr-FR } { \textfrench{#2} }
        % others
   }
   {
    #2 % I~don't~know~what~to~do~with~`#1'
   }
 }
\ExplSyntaxOff

If I recall correctly, in Context you use

\language[es]
¡hola!
\language[en]

It would be better to not pollute the templates with too much redefining and we already have the BCP47-to-polyglossia/babel functions in the LaTeX writer. But the problem with the other LaTeX engines is bothersome. Maybe we'll just have to decide that when people start writing <span lang="es">... in their markdown documents they'll have to go with XeLaTeX and polyglossia, since that's what everyone using multiple languages etc. does anyhow.

It would be better to not pollute the templates …

I agree. But what else can we do if we want pandoc to create engine-agnostic LaTeX documents?

Maybe we'll just have to decide that when people start writing <span lang="es">... in their markdown documents they'll have to go with XeLaTeX and polyglossia …

Fine with me. And if I understand it correctly, LuaLaTeX works with polyglossia, too. So we’d just map from <span lang="es">... to the polyglossia syntax.

I tend to favour the approach of just outputting the polyglossia commands. The only major downside I can think of if someone does pandoc -f html -o out.pdf and there are lang attributes on spans/divs in that HTML, then the PDF generation will fail (rather unexpectedly) since the default latex-engine is pdflatex. Was there some discussion on changing the default pdf engine? Maybe it's time to switch?

Thoughts anyone?

Also, should the otherlangs variable be automatically populated with the values of all lang attributes in all spans and divs, or is it better to leave it as it is and have authors specify it manually in the YAML metadata? The first option means less manual work, the second potentially more control (although I guess we could still allow the variable to be overridden by the YAML).

@jgm: Sounds all good to me.

Since we have language support with babel for non-xelatex.

Well, we have support for the main language with babel, but not multilingual. But yeah... if we emit define commands to map babel to polyglossia, the question then is which one to emit and define. Seems cleaner to emit polyglossia and introduce some hacks for babel (which users that are serious about languages won't use anyway), on the other hand polyglossia seems to imply in the manual that they aim to support babel's command but this doesn't work right now?

@nickbart1980 do you know which way has the easier/simpler LaTeX definitions? \begin{otherlanguage}{british} <-> \begin{english}[variant=british]

I’m not sure how to define polyglossia wrappers for the the babel commands though.

What would work, relatively easy and transparent, is defining our own commands, e.g. \textbcpfourseven{en-GB}{…}; see my ideas using xparse above.

The latex writer could put all required definitions into one pandoc variable, so they won’t clutter up the template (though they will of course appear in the document itself).

I've started on this... @jgm is there a way to extract all lang attributes from both divs and spans without resorting to query extractDivLang blocks ++ query extractSpanLang blocks The problem being that there is no supertype a for both Block and Inlines to write extractLang :: a -> [String].

Thanks jgm, makes sense...

About the mapping... I implemented outputting the polyglossia commands, now trying to come up with LaTeX mappings from polyglossia to babel. I have it working for most languages with e.g.:

\newcommand{\textspanish}[2][]{\foreignlanguage{spanish}{#2}}
\newenvironment{spanish}[1]{\begin{otherlanguage}{spanish}}{\end{otherlanguage}}

until I realized that for spanish, babel itself defines a \textspanish command already and for some reason using \renewcommand goes into infinite recursion (TeX capacity exceeded, sorry [grouping levels=255].) If someone with LaTeX skills can get this to work I'd appreciate that. Otherwise I guess we'll have to go with @nickbart1980 proposal of rolling our own textBCP47{en-GB} command and redefine it for both babel and polyglossia users...

FWIW, a 2009 version of the babel manual, http://www.pvv.ntnu.no/~berland/latex/docs/babel.pdf contains references to \textspanish (and \textgalician, too). I didn’t spot any other \textLANG commands in that document so far. The most recent babel manual (Version 3.9m 2015/08/03) seems to contain no such references.

Good tip, I seem to have babel 2014/03/24 3.9k The Babel package installed (from MacTex 2014), gonna give MacTex 2015 a try :)

I’m afraid \textspanish and \textgalician are still in texlive 2015 (just not in the babel manual): in
/usr/local/texlive/2015/texmf-dist/tex/generic/babel-spanish/spanish.ldf and
/usr/local/texlive/2015/texmf-dist/tex/generic/babel-galician/galician.ldf

Maybe you could use \textSpanish and \textGalician, then just two additional \newcommands are needed when using polyglossia.

@nickbart1980 yeah, although I still think it should be possible to renewcommand textspanish etc... Can you reproduce the error with pdflatex? which version/distribution? (see example doc: I asked at tex.stackexchange and was told it works for him).

Edit: never mind, got a great answer over there that solves the issue.

pandoc now emits \textArabic – this is wrong: only the environment name should be capitalised (\begin{Arabic} … \end{Arabic}), but not the command, which has to be \textarabic{…}.

@nickbart1980 thanks for the correction, is in pull #2481

For non-Latin scripts, it seems we need to add sensible font defaults and methods for overriding them.

I’ve just been looking at xelatex/polyglossia so far: The default font used by xelatex (some form of Computer Modern, it seems), e.g., supports neither Greek nor Arabic - example:

pandoc --latex-engine=xelatex -o test.pdf << EOT

العَرَبِية

---
lang: ar
...
EOT

Result:

! Package polyglossia Error: The current roman font does not contain the Arabic script!
(polyglossia)                Please define \arabicfont with \newfontfamily.

See the polyglossia package documentation for explanation.
Type  H <return>  for immediate help.
...  

l.69 \begin{document}

pandoc: Error producing PDF from TeX source

Adding, e.g.,

\newfontfamily\arabicfont[Script=Arabic]{Amiri}

to default.latex fixes this.

Interestingly enough, the scrartcl class also needs \newfontfamily\arabicfontsf[Script=Arabic]{Amiri} when the input document contains headers.

My suggestion is to solve this issue by adding sensible defaults and also introducing pandoc variables like arabicfont or greekfont.

For Arabic, which does not have separate sf and tt variants, we might use something like the following in default.latex:

$if(arabicfont)$
\newfontfamily\arabicfont[Script=Arabic]{$arabicfont$}
\newfontfamily\arabicfontsf[Script=Arabic]{$arabicfont$}
\newfontfamily\arabicfonttt[Script=Arabic]{$arabicfont$}
$else$
\newfontfamily\arabicfont[Script=Arabic]{Amiri}
\newfontfamily\arabicfontsf[Script=Arabic]{Amiri}
\newfontfamily\arabicfonttt[Script=Arabic]{Amiri}
$endif$

For other scripts we will probably need separate variables, e.g., greekfont, greekfontsf and greekfonttt.

We could also try to use, by default, fonts that support more scripts; Times New Roman, e.g., seems to contain both Greek and Arabic (though here, again, an extra definition \newfontfamily\arabicfontsf[Script=Arabic]{Times New Roman} was needed with scrartcl).

I’m not sure why scrartcl seems to require an extra arabicfontsf definition, or how to simplify this in general. Any hints are appreciated.

scrartcl needs a sans-serif font for headings. Your suggestion how to set up $(arabicfont) in your previous comment looks fine to me.

BTW, my solution to the script problem in XeTeX (I'm still using pandoc 1.12.3.3) is to use the ucharclasses package, which automatically switches the script depending on the Unicode block – no need to explicitly specify a script! For example, I have this in my LaTeX template file:

\ifxetex
  \usepackage{ucharclasses}

  \newfontfamily{\arabicfont}[Script=Arabic]{Amiri}
  \newfontfamily{\devanagarifont}[Script=Devanagari]{FreeSerif}
  \newfontfamily{\laofont}[Script=Lao]{NotoSerifLao}
  \newfontfamily{\telugufont}[Script=Telugu]{Pothana2000}
  \newfontfamily{\thaifont}[Script=Thai]{FreeSerif}

  \setTransitionTo{Arabic}{\begingroup\arabicfont}
  \setTransitionFrom{Arabic}{\endgroup}
  \setTransitionTo{Devanagari}{\begingroup\devanagarifont}
  \setTransitionFrom{Devanagari}{\endgroup}
  \setTransitionTo{Lao}{\begingroup\laofont}
  \setTransitionFrom{Lao}{\endgroup}
  \setTransitionTo{Telugu}{\begingroup\telugufont}
  \setTransitionFrom{Telugu}{\endgroup}
  \setTransitionTo{Thai}{\begingroup\thaifont}
  \setTransitionFrom{Thai}{\endgroup}
\fi

I'm mentioning this just for reference, since a solution that understands and uses language tags is preferable for many reasons. (Note that I don't have non-latin scripts in the headers, so there aren't proper commands for setting up sans-serif fonts.)

If you want to see this in action, have a look at the ttfautohint package (the PDF and the constructed pandoc input files are part of the release tarball only).

For monolingual documents, setting the mainfont to a font that supports all the characters in the doc should solve the issue, right? And everyone who's serious about typesetting bilingual documents will need to manually select some fonts that work together anyway and can include the necessary \newfontfamily definitions in the header-includes variable, like:

---
lang: ar
mainfont: ArialUnicodeMS
header-includes: "\\newfontfamily\\arabicfont[Script=Arabic]{Amiri}"
---

العَرَبِية

This may not be ideal (maybe we should mention it in the README). But if we were to include something like \newfontfamily\arabicfont[Script=Arabic]{$arabicfont$} for every variation of script/language/bold/italics/bold-italics/typewriter etc.—that would be well over a hundred lines polluting the template. And we cannot generate those since we wouldn't know what $arabicfont$ should be (I think template variable substitution happens only once). We could generate \newfontfamily definitions for the languages used and set it to some default font, but the user couldn't change that then.

For ConTeXt it seems we could specify fallback fonts for certain unicode ranges. Unfortunately, while ucharclasses provides the same functionality for Polyglossia, it seems the \newfontfamily definitions are still necessary?

So, I'm not sure there's much we can do (except mentioning this in the README, and maybe set up something for ConTeXt), although recommendations are welcome.

And everyone who's serious about typesetting bilingual documents will need to manually select some fonts that work together anyway …

I’d agree as far as serious work is concerned, but I’m more than a little worried if casual users trying to use a non-Latin script do not get any output but just an error message. In other words, I’d expect pandoc to generate some at least halfway decent output even if a user does not actively specify any non-Latin fonts at all.

Couldn’t we introduce at least one polyglossia-fonts variable to which the latex writer adds sensible default font definitions for non-Latin scripts, based on the language tags found in the document?

So, whenever a document contains the language tag ar,

\newfontfamily\arabicfont[Script=Arabic]{Amiri}
\newfontfamily\arabicfontsf[Script=Arabic]{Amiri}
\newfontfamily\arabicfonttt[Script=Arabic]{Amiri}

would be added to polyglossia-fonts; and $if(polyglossia-fonts)$$polyglossia-fonts$$endif$ would of course be included in default.latex.

These definitions could then still be overridden by, e.g.,

---
header-includes: "\\newfontfamily\\arabicfont[Script=Arabic]{Scheherazade}\\newfontfamily\\arabicfontsf[Script=Arabic]{Scheherazade}"
...

I’d expect pandoc to generate some at least halfway decent output even if a user does not actively specify any non-Latin fonts at all.

Indeed that would be nice, however currently not the case either: e.g. arab characters require you to specify --latex-engine xelatex (or you get ! Package inputenc Error: Unicode char \u8:ا not set up for use with LaTeX.) and even then the PDF is currently empty for me since the default font doesn't have the required characters. Personally, I think sooner or later we should change the default pdf engine to either XeLaTeX, ConTeXt or even plain TeX anyway...

Couldn’t we introduce at least one polyglossia-fonts variable to which the latex writer adds sensible default font definitions for non-Latin scripts, based on the language tags found in the document? [...]
These definitions could then still be overridden by, e.g., header-includes: "\\newfontfamily...

I see, indeed if they can be overriden (and they can, just tested) then I'm in favour as well. So I suppose now we need a good list of fonts widely available for lots of languages...

@nickbart1980, the default font in XeTeX and LuaTeX (Latin Modern) is set by fontspec. Perhaps an issue for adding defaults for other languages should be added to its repository, to see whether a broader solution can be found?

Could we also discuss the syntax for the language attribute?

:lang is fine for me. It is borrowed from Textile, but it is fine, since it is the CSS selector for pseudo-classes (and languages in CSS are a pseudo-class).

If we could agree on this, the language special attribute could be already implemented in the elements that allow it. These are mainly titles and code.

At least for code, when writing technical documents in other languages than English, it is extremely useful to be able to tag inline code as being written in English. Otherwise hyphenation for that part will be highly probable wrong.

And when issue #168 will be solved, we would benefit a lot from the special syntax for language attributes in text divisions and spans.

So, could we reach an agreement about the syntax for the language attribute?

@jgm, after issue #168 is fixed, could we discuss this issue?

This issue is older than #168. It comes comes from #162, [which I originally reported at https://code.google.com/archive/p/pandoc/issues/201 (more than six years ago).

I think that :lang is the natural syntax for languages, since it is the CSS selector. This would be the same syntax for #id and #class.

And a comment on the issue: it is about the syntax (or at least, that was my original report). As I reported the original issue at Google Code, you discarded it because it looked like recreating LaTeX in pandoc (see comment 7 there).

Well, time flies. And the vast majority of comments in this issue explain how LaTeX (its babel and polyglossia packages) deals with languages.

I think we have to set a special language syntax first.

I have tried to discuss it at the mailing list, but I guess this should be the proper place to discuss it (since I got no reply there).

Special language syntax is needed to have different document sections in different languages, such as in:

# The US Constitution {:en}

[English text]

# Das deutsche Grundgesetz {:de}

[deutscher Text]

I think it is clear we need that special syntax for the language attribute. it is essential for multilingual documents.