NMODL development
This is an introduction for making changes to the NMODL codebase.
NMODL components
To translate a mod file into a C++ source file, NMODL employs several stages that we will go through in varying degrees of detail. The relevant stages of translation are:
lexer and parser
visitors
codegen
Lexer and parser
Note
Normally, you will not need to modify anything regarding the lexer and parser, since the grammar of NMODL itself is fixed.
The content of a given mod file is initially converted to a list of tokens using a lexer (flex). All of the code in charge of lexing a mod file is in src/nmodl/lexer. The code is then converted to an abstract syntax tree (AST), which is composed of nodes, using a parser (bison). This is basically done using the following C++ code:
/// driver object creates lexer and parser, just call parser method
NmodlDriver driver;
/// parse mod file and construct AST
const auto& ast = driver.parse_string(content);
Note
The code above may look like it creates an immutable AST, but due to the use of smart pointers, the tree itself can actually be modified using various visitors. More on that later.
As a result, we will get an AST which can be traversed and modified using various visitors.
The visitors
NMODL employs the visitor pattern to traverse the AST and perform operations on its nodes. The basic operations that can be performed on the AST are:
insertion of a new node
modification of an existing node
deletion of a node
Before we get into the details of the visitors, it is helpful to understand which types of nodes are there, and which properties they have. All of the various types of NMODL nodes are actually generated from a single YAML file, src/nmodl/language/nmodl.yaml. For instance, this is the definition of a FUNCTION block in the YAML file:
- FunctionBlock:
nmodl: "FUNCTION "
members:
- name:
brief: "Name of the function"
type: Name
node_name: true
- parameters:
brief: "Vector of the parameters"
type: Argument
vector: true
prefix: {value: "(", force: true}
suffix: {value: ")", force: true}
separator: ", "
getter: {override: true}
- unit:
brief: "Unit if specified"
type: Unit
optional: true
prefix: {value: " "}
suffix: {value: " ", force: true}
- statement_block:
brief: "Block with statements vector"
type: StatementBlock
getter: {override: true}
The above provides us with the following information:
the NMODL statement corresponding to a
FunctionBlockisFUNCTION ``, and a ``FunctionBlocknode has a name (indicated bynode_name: true), meaning that we can’t have a nameless function, and the name of the function has a node typeNamethe function can take a vector of parameters (indicated by
vector: true), and each argument is a node of typeArgument. The list of arguments must be preceded by an opening paren (() and succeeded by a closing paren (``)``), otherwise the lexer or parser will throw an error. Furthermore, the arguments must be separated by a semicolon (,)the function can optionally contain units
the function must always contain a statement block (even if the statement block itself is empty)
Visiting a node
Assuming that we want to visit a function definition (or any other node for that matter), and maybe do some manipulation of its children, how can we accomplish this? NMODL provides us with the nmodl::ast::AstVisitor class, which we can override to visit a particular node:
// in a header file
#include "visitors/ast_visitor.hpp"
#include "ast/function_block.hpp"
namespace nmodl {
namespace visitor {
class MyFunctionVisitor : public ast::AstVisitor {
public:
void visit_function_block(ast::FunctionBlock&) override;
};
}
}
Next, you need to write the actual implementation:
// in the implementation file
#include "visitors/my_function_visitor.hpp"
namespace nmodl {
namespace visitor {
void MyFunctionVisitor::visit_function_block(ast::FunctionBlock& node) {
// implementation goes here
}
}
}
In general, the naming convention is snake_case for the methods and header files, and TitleCase for the classes themselves.
Note
When overriding a given method, the const-ness of the method signature corresponds to whether or not the visitor it inherits from is AstVisitor or ConstAstVisitor. In the above example, if we inherited from ConstAstVisitor, then the signature of visit_function_block would be const ast::FunctionBlock&.
Warning
Don’t forget to add your visitor implementation file to the visitor library in the CMake build code (currently in src/nmodl/visitors/CMakeLists.txt)!
Visiting nested nodes
Sometimes you may want to visit multiple nested nodes; for instance, if your node can appear globally, i.e. anywhere, but has a special meaning inside of a FUNCTION block. NMODL provides the visit_children method for this, that performs the recursive descent into the children, which should be called as:
void MyFunctionVisitor::visit_function_block(ast::FunctionBlock& node) {
// some code
node.visit_children(*this);
// other code that needs to run _after_ the children have been visited
}
A common pattern used troughout the codebase is a toggle flag to notify the children node that it is inside of some parent node:
void MyFunctionVisitor::visit_function_block(ast::FunctionBlock& node) {
// assuming ``in_function`` is a private member of ``MyFunctionVisitor``
in_function = true;
node.visit_children(*this);
in_function = false;
}
You can also override the accept method, which allows you to visit the current node itself, but not any of the children.
Warning
It may seem tempting to add C++ code directly to the AST by inserting VERBATIM blocks; this mixes implementation details (C++ code) with the overall abstraction (the AST), so it should only be done sparsely.
Performing the transformation with the visitor
To actually perform any transformations you implemented, you need to create an instance of the class, and then call visit_program:
const auto& my_visitor = MyFunctionVisitor();
my_visitor.visit_program(*ast);
Tip
The constructor of the visitor can take any arguments you specify; this is typically useful if you only want to perform changes to the AST depending on whether some CLI flag is present or not.
Warning
If you are performing changes to the AST that also involve adding additional symbols (for instance, a LOCAL variable), in order to keep the symbol table synchronized with the contents of the AST, you should call SymtabVisitor().visit_program(*ast) afterwards.
Adding a new node type
On certain occasions, it may be necessary to add new nodes to NMODL; this is usually due to some internal requirements for codegen, and the right place to add it is in src/nmodl/language/codegen.yaml, i.e. as a codegen node type. For example, the following node type was added for keeping track of the information required for CVODE codegen:
- CvodeBlock:
nmodl: "CVODE_BLOCK "
members:
- name:
brief: "Name of the block"
type: Name
node_name: true
suffix: {value: " "}
- n_odes:
brief: "number of ODEs to solve"
type: Integer
prefix: {value: "["}
suffix: {value: "]"}
- non_stiff_block:
brief: "Block with statements of the form Dvar = f(var), used for updating non-stiff systems"
type: StatementBlock
- stiff_block:
brief: "Block with statements of the form Dvar = Dvar / (1 - dt * J(f)), used for updating stiff systems"
type: StatementBlock
brief: "Represents a block used for variable timestep integration (CVODE) of DERIVATIVE blocks"
If a mod file can use CVODE time integration method, the above will be used to generate an intermediate file which contains something like:
CVODE_BLOCK myblock [3] {
: statements for non-stiff block
}
{
: statements for stiff block
}
Helper functions
Some helper functions that are frequently used when manipulating the AST are listed below (for a full list, consult the src/nmodl/visitor/visitor_utils.hpp file):
nmodl::collect_nodes(node, types): starting from a given node, collect given node type(s), and return the result as a vector of smart pointersnmodl::to_nmodl(node): convert a given node to a string representing the NMODL statement associated with that nodenmodl::visitor::create_statement(statement): convert a given NMODL string to an AST node, and return a smart pointer to it
Warning
The create_statement is a convenience function, but does not always work properly because it wraps the string into a PROCEDURE block (so anything that is not a valid statement in a PROCEDURE block will fail). The “proper” way of creating a given statement is via calling the various AST node constructors manually.
Special visitors
Below we will list some “special” visitors.
Visitors requiring Python
There are several visitors which make use of Python, i.e. they require executing Python code itself. While there is nothing inherently special about them, they do require some additional setup:
the Python interpreter must be initialized first; this is achieved with the
initialize_interpreterfunctionafter any visitors requiring Python have completed their work, the interpreter must be finalized; this is achieved with the
finalize_interpreterfunction
Visitors that perform I/O
Some of the visitors perform I/O, notably:
NmodlPrintVisitor(filename): output the current AST to a file in NMODL format tofilenameJSONVisitor(filename): output the current AST to a file in JSON format tofilenamePerfVisitor(filename): output the performance of NMODL (i.e. how many times NMODL itself visited a given node) in JSON format tofilenameCodegen*Visitor: various visitors which output C++ code
Code generation
After performing any manipulations on the AST, it is time to generate actual code that a given compiler can use. NMODL has 3 code backends:
the NEURON code backend for generating C++ code compatible with NEURON (
codegen_neuron_cpp_visitor.cpp)the coreNEURON code backend for generating C++ code compatible with coreNEURON using CPUs (
codegen_coreneuron_cpp_visitor.cpp)the coreNEURON code backend for generating C++ code compatible with coreNEURON using GPUs (
codegen_acc_visitor.cpp)
The above all inherit from CodegenCppVisitor ( codegen_cpp_visitor.cpp), which overrides many visit_* methods with a custom implementation, and use the print_* methods to output any code. A common pattern used is:
void print_something(const ast::SomeType& node) {
// actual code to output
}
void visit_something(const ast::SomeType& node) {
print_something(node);
}
Note that print_something methods are often declared with the virtual specifier since the code emitted is highly dependent on the backend (coreNEURON or NEURON).
Debugging issues
Now that we implemented our changes, and the code compiles, it is possible that they do not work quite as intended. Possible scenarios include, but are not limited to:
segfaults or errors when running NMODL on an input file
modifications via visitors not being applied to the AST
the modifications result in an invalid AST
the resulting C++ file does not compile
the resulting C++ file compiles, but the mechanism segfaults at runtime
To aid in debugging the above without having to resort to using a debugger (yet!), the NMODL CLI comes with several useful features. They can also be useful for figuring out what visitor methods need to be implemented.
The option passes --nmodl-ast writes the intermediate AST to a file (in NMODL format) after a visitor performs its pass. This is implemented in the main code as:
SomeVisitor().visit_program(*ast);
ast_to_nmodl(*ast, filepath("some"));
Almost all visitors output NMODL files after they visit the AST using the above pattern, and the output filename is <modfile>.<number>.<suffix>.mod, where <number> is the current ordinal number of the visitor, and <suffix> is the argument to filepath above.
Tip
When writing a new visitor, this pattern is very useful to verify that your changes work as intended.
Warning
The resulting intermediate NMODL file should not be re-used for input to NMODL, because it may contain invalid NMODL blocks (such as those used for conveying codegen information).
The option passes --json-ast writes an AST to a file (in JSON format). Note that the resulting AST is not exactly equivalent to the input AST (because some minor visitors run beforehand), but it provides a good starting point for figuring out which visitors need to be implemented, and how they should handle the nodes.
The option --verbose [info,debug,trace] provides varying degrees of debugging information that is stored in the logger. This can be useful for quickly diagnosing where a segfault could be occurring, or where the code gets stuck, without resorting to a full-blown debugger.
The option blame --line <number> is used for creating a backtrace of how a given line in the C++ file ended up there. For instance, the output of nmodl --neuron src/nrnoc/hh.mod blame --line 511 could be something like:
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 711, in print_statement_block
710: !statement->is_mutex_unlock() && !statement->is_protect_statement()) {
> 711: printer->add_indent();
712: }
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 963, in visit_var_name
962: const auto& index = node.get_index();
> 963: name->accept(*this);
964: if (at_index) {
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 713, in print_statement_block
712: }
> 713: statement->accept(*this);
714: if (need_semicolon(*statement)) {
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 963, in visit_var_name
962: const auto& index = node.get_index();
> 963: name->accept(*this);
964: if (at_index) {
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 1070, in visit_binary_expression
1069: printer->add_text(" " + op + " ");
> 1070: rhs->accept(*this);
1071: }
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 963, in visit_var_name
962: const auto& index = node.get_index();
> 963: name->accept(*this);
964: if (at_index) {
Source: "/nrn/src/nmodl/codegen/codegen_neuron_cpp_visitor.cpp", line 267, in print_function_or_procedure
> 267: print_statement_block(*node.get_statement_block(), false, false);
268: printer->fmt_line("return ret_{};", name);
Source: "/nrn/src/nmodl/codegen/codegen_cpp_visitor.cpp", line 718, in print_statement_block
717: if (!statement->is_mutex_lock() && !statement->is_mutex_unlock()) {
> 718: printer->add_newline();
719: }
For an even larger backtrace, you can also add the --detailed flag (output omitted for brevity from this document).
Note that the blame option is only available if NEURON is compiled with the NRN_ENABLE_BACKTRACE=ON CMake option.
Note
NMODL uses backward-cpp for printing the backtrace; however, other libraries are in charge of a) walking the stack, and b) retrieving the debugging information from the executable. This complicates things quite a bit, and so far this only works reliably on Linux. The libdwarf-devel, elfutils-devel, and libunwind-devel DNF packages have been shown to work on Fedora 42, and libdw-dev and binutils-dev APT packages have been shown to work on Ubuntu 24.04 LTS. If NEURON is adapted to work with C++23, there is the possibility of using the stacktrace from the STL as an alternative, which should not require any external dependencies.
Writing tests
There are several kinds of tests to ensure correctness in NMODL:
unit tests, where we test that a given NMODL feature (such as a visitor, or the parser) works as intended (located in
test/nmodl/transpiler/unit)integration tests, where we test that NMODL can translate a given mod file to a C++ file (located in
test/nmodl/transpiler/integration)usecase (more commonly known as end-to-end) tests, where we test that NMODL can translate, and NEURON can compile and link, the resulting C++ file, and that the resulting outputs from running a simulation under NEURON are correct (located in
test/nmodl/transpiler/usecase)
Unit tests
Most of the NMODL unit tests are written in Catch2, a C++ testing framework. While describing all of Catch2 is outside the scope of this document, the workflow for writing a new unit test is roughly as follows:
write a minimum working example (MWE) of an NMODL file that is affected by your changes
run the parser on the input string to obtain the AST
run any visitors on the AST
use Catch2 functionality to compare the resulting AST to the expected AST
add the new test to the CMake build code
Note
The AST classes currently do not have operator== implemented, which means that a direct comparison will not work due to compiler errors. The workaround for this is to convert the AST to NMODL again using to_nmodl, and applying nmodl::test_utils::reindent_text on the output, which standardizes the indentation, and can then be compared with the expected NMODL result.
Usecase tests
The usecase tests check that NMODL can translate mod files to C++ files, and that the resulting files can be compiled and linked to NEURON, as well as that the resulting mechanisms have correct results. The usecase tests are specific to the NEURON codegen backend, and are used to verify matching behavior between the old NMODL compiler (nocmodl) and the new one (nmodl). To create a new usecase test:
add a new directory in
test/nmodl/transpiler/usecasesadd any mod files that should be compiled there
add any Python files in the same directory that run NEURON simulations. Note that the Python files must be called either
simulate.py, or start with thetest_prefix (this is due to legacy reasons and the file naming requirement may be removed in a future refactoring).add the new directory to the
NMODL_USECASE_DIRSlist intest/nmodl/transpiler/usecases/CMakeLists.txt