Migrating Applications to Scintilla 5 with Lexilla.

Introduction

With Scintilla 5.0, all lexers were moved from Scintilla into the Lexilla library which is a separate project. Lexilla may be either a static library that is linked into an application or a shared library that is loaded at runtime.

Lexilla has its own documentation.

Basics

With Scintilla 4.x, it was most common for applications to set a lexer either by lexer name or lexer ID (SCLEX_...) by calling SCI_SETLEXERLANGUAGE("<name>") or SCI_SETLEXER(SCLEX_*).

With Scintilla 5, the normal technique is to call Lexilla's CreateLexer function with a lexer name, then apply the result with Scintilla's SCI_SETILEXER method:
ILexer5 *pLexer = CreateLexer("<name>") SCI_SETILEXER(pLexer)

Lexer names are now strongly preferred to lexer IDs and applications should switch where possible. Some lexers will not have lexer IDs but all will have names.

Applications may be written in C++ or C; may contain a statically linked Lexilla library or load a Lexilla shared library; and may use the raw Lexilla API or the LexillaAccess C++ helper module.

From C++: LexillaAccess

The easiest technique is to implement in C++ using LexillaAccess and load a Lexilla shared library.

LexillaAccess simplifies use of Lexilla, hides operating system differences, and allows loading multiple libraries that support the Lexilla protocol. It is defined in lexilla/access/LexillaAccess.h and the source code is in lexilla/access/LexillaAccess.cxx. Add these to the build dependencies of the project or build file.

Both SciTE and TestLexers (used to test Lexilla) in lexilla/test use LexillaAccess. TestLexers is much simpler than SciTE so can be a good example to examine.

Building

Header files for lexers and for using Lexilla will be included so build files will need to reference these new locations. LexillaAccess.cxx should be added to the set of source files. In make, for example, this may appear similar to:


    LEXILLA_DIR ?= $(srcdir)/../../lexilla

    INCLUDES += -I $(LEXILLA_DIR)/include  -I $(LEXILLA_DIR)/access

    SOURCES += $(LEXILLA_DIR)/access/LexillaAccess.cxx

Steps in code

Set the directory to search for Lexilla when full paths aren't provided. This may be a known good system or user directory or the directory of the application depending on the operating system conventions.


    std::string homeDirectory = "/Users/Me/bin";

    Lexilla::SetDefaultDirectory(homeDirectory);

Load Lexilla or another library that implements the Lexilla protocol. The name "." means the standard name and extension (liblexilla.so / liblexilla.dylib / lexilla.dll) in the default directory so is often a good choice. Names without extensions have the operating system's preferred shared library extension added. A full path can also be used. Multiple libraries can be loaded at once by listing them separated by ';'. Something like one of these lines:


    Lexilla::Load(".");

    Lexilla::Load("lexilla;lexpeg;XMLexers");

    Lexilla::Load("/usr/lib/liblexilla.so;/home/aardvark/bin/libpeg.so");

Choose the name of the lexer to use. This may be determined from the file extension or some other mechanism.


    std::string lexerName = "cpp";

Create a lexer and apply it to a Scintilla instance.


    Scintilla::ILexer5 *pLexer = Lexilla::MakeLexer(lexerName);

    CallScintilla(scintilla, SCI_SETILEXER, 0, pLexer);

Some applications may use integer lexer IDs from SciLexer.h with an "SCLEX_" prefix like SCLEX_CPP. These can be converted to a name with NameFromID before passing to MakeLexer.


    Scintilla::ILexer5 *pLexer = Lexilla::MakeLexer(Lexilla::NameFromID(SCLEX_CPP));

From C: Lexilla.h and system APIs

Applications written in C or C++ can use the raw Lexilla API which is defined in lexilla/include/Lexilla.h. Since the ILexer interface is difficult to define for C, C applications should treat the result of CreateLexer as a void* that is simply passed through to Scintilla. If there is a need to call methods on the lexer before passing it to Scintilla then it would be best to use some C++ code although it may be possible for a sufficiently motivated developer to call methods on the lexer from C.

There is an example for using Lexilla from C in examples/CheckLexilla.

Steps in code

Include the system header for loading shared objects. This depends on the operating system: <windows.h> for Windows or <dlfcn.h> for Unix.

#include <windows.h>

#include <dlfcn.h>

Define a path to the Lexilla shared library. This may be a known good system or user directory or the directory of the application depending on the operating system conventions.


    #include "Lexilla.h"

    char szLexillaPath[] = "../../bin/" LEXILLA_LIB LEXILLA_EXTENSION;

Load Lexilla using the appropriate operating system function: either LoadLibrary on Windows or dlopen on Unix.

HMODULE lexillaLibrary = LoadLibrary(szLexillaPath);

void *lexillaLibrary = dlopen(szLexillaPath, RTLD_LAZY);

Find the CreateLexer function inside the shared library using either GetProcAddress on Windows or dlsym on Unix.

FARPROC fun = GetProcAddress(lexillaLibrary, LEXILLA_CREATELEXER);

void *fun = dlsym(lexillaLibrary, LEXILLA_CREATELEXER);

Cast this to the correct type.

CreateLexerFn lexerCreate = (CreateLexerFn)(fun);

Choose the name of the lexer to use. This may be determined from the file extension or some other mechanism.

char lexerName[] = "cpp";

Create a lexer and apply it to a Scintilla instance.


    void *pLexer = lexerCreate(lexerName);

    CallScintilla(scintilla, SCI_SETILEXER, 0, pLexer);

If the application uses integer lexer IDs then find the LEXILLA_LEXERNAMEFROMID function, cast to LexerNameFromIDFn then call with the ID before using the result to call CreateLexer.

FARPROC funName = GetProcAddress(lexillaLibrary, LEXILLA_LEXERNAMEFROMID);

void *funName = dlsym(lexillaLibrary, LEXILLA_LEXERNAMEFROMID);

LexerNameFromIDFn lexerNameFromID = (LexerNameFromIDFn)(funName);

const char *name = lexerNameFromID(lexerID);

Static linking

Lexilla may be linked directly into an application or built into a static library that is then linked into the application, then there is no need to load a shared library and CreateLexer can be directly called.

It is possible to link Lexilla into an application and then dynamically load other shared libraries that implement the Lexilla protocol. SciTE on Windows implements this as an option when built with STATIC_BUILD defined.