Monotone Framework

This section is a very brief recap of the dataflow framework.
For more detailed and certainly more correct explanation consider watching these lectures or diving even deeper with the SPA book (chapters 4 and 5).

The goal of a dataflow analysis is to collect some facts and information about the program, so that these facts can be used later for further analyses or transformations.

MLIR’s dataflow analysis is based on the classical monotone framework.
Conceptually, it’s a very simple idea: run the analysis iteratively until no more new facts can be derived:

let analysis = DataflowAnalysis();
let worklist = ParseProgram(sourceCode);
while (!worklist.empty()) {
  let node = worklist.pop();
  let knownFacts = analysis.getFacts(node.predecessors());
  if (analysis.canDeriveNewFacts(node, knownFacts)) {
     let newFacts = analysis.deriveNewFacts(node);
     analysis.mergeFacts(newFacts, knownFacts);
     worklist.add(node.successors());
  }
}

The pseudocode above shows this idea in action: try deriving new facts for each node in the program, if new facts are indeed found, then update the existing information and add the successor nodes for further analysis.
The last step is crucial: the newly derived facts “invalidate” facts for the subsequent nodes, so some nodes might be analysed multiple times.

The invalidation part is important for at least two reasons:

• loops and cycles: if the program has loops in it (which most programs do), then we may need to re-analyze the loop body several times to accumulate all the facts

• composability: it’s possible to run several analyses at the same time, in which case combined facts may yield more information. E.g., an analysis A didn’t find any facts for node X and completed, later an analysis B found some facts for node X, which triggers analysis A on the node X again, but now with the new facts.

Subscribe now

Sparse vs Dense Analysis

While MLIR has a generic implementation of the monotone framework, it also comes with several predefined implementations: DenseForward, DenseBackward, SparseForward, and SparseBackward analyses. Whenever you want to build your own analysis, these are perhaps the ones you should consider as a starting point.

While the difference between Forward vs Backward is clear, the Dense vs Sparse might be confusing at first.

The main difference between these two flavours is the way the information or facts are propagated.

MLIR’s dataflow framework attaches facts about a program to the abstract LatticeAnchors. The abstract anchor can be either an mlir::ProgramPoint, an mlir::Value, or an mlir::GenericLatticeAnchor.

The ProgramPoint represents the location of a fact within the program, e.g. “before an operation“ or “after an operation.” Thus, the facts are propagated from one operation to another via (implicit) control-flow edges. This is the essence of the Dense Analysis.

The Value can be used to attach the facts to specific SSA-values. The facts are propagated from one value to another via data flow edges.

The generic anchor can be used to encode anything that doesn’t fit a ProgramPoint or a Value. For example, DeadCodeAnalysis uses it to represent a CFGEdge.

Below is an illustration of the difference between Dense and Sparse analyses for the following program:

func.func @do_stuff(%a: i32, %b: i32) -> i32 {
  %two = arith.constant 2 : i32
  %three = arith.constant 3 : i32
  %t1 = arith.muli %a, %two : i32
  %t2 = arith.muli %b, %three : i32
  %res = arith.addi %t1, %t2 : i32
  return %res : i32
}

In the case of the Dense forward analysis, any change of facts of the op %three triggers new analysis for all the ops “below” it.
In the case of the Sparse forward analysis though, the same change would only affect “half” of the program.

The APIs of both analyses reflect the lattice anchors:

// Dense Analysis
LogicalResult visitOperation(Operation *op,
                             const AnalysisState &before,
                             AnalysisState *after);

// Sparse Analysis
LogicalResult visitOperation(Operation *op,
                             ArrayRef<const AnalysisState*> before,
                             ArrayRef<AnalysisState *> after);

These are precisely the transfer/merge functions from the monotone framework, and it’s the analysis authors responsibility to implement them.

Gotchas

There is a very important detail one must keep in mind!

Looking at the documentation around the Dataflow Analysis Framework you can find the following lines at the top of both SparseAnalysis.h and DenseAnalysis.h (emphasis mine):

This file implements sparse data-flow analysis using the data-flow analysis framework. The analysis is forward and conditional and uses the results of dead code analysis to prune dead code during the analysis.

What the documentation doesn’t say however, is that by default everything is considered dead.
If you fail to include DeadCodeAnalysis in the solver, then all of the operations will be skipped as “dead”.
I observed similar behaviour due to the lack of SparseConstantPropagation as some of the nested regions (coming from scf) were ignored, even though they were in fact live and reachable.

At the time of writing (LLVM 20), this is the way to run your own analysis:

mlir::DataFlowSolver solver;

// These two are strictly required for the complete analysis to work
solver.load<mlir::dataflow::DeadCodeAnalysis>();
solver.load<mlir::dataflow::SparseConstantPropagation>();

solver.load<MyFancyAnalysis>();
solver.initializeAndRun(op);

This is the part you should be looking at very carefully, especially when upgrading to a newer version of LLVM/MLIR: at some point some new implicitly “required” analysis might be introduced, which may lead to an incomplete view of the program.

More information

I’m still learning about the framework internals, so if you know of any examples of out-of-tree analyses - any pointers would be highly appreciated 🙌

I highly recommend watching these two talks to get a better idea of the motivation behind the current design and real-world use-cases.

• 2023 EuroLLVM - MLIR Dataflow Analysis

• 2024 EuroLLVM - Efficient Data-Flow Analysis on Region-Based Control Flow in MLIR

If you want to build such a tool, then you obviously need to somehow “connect” your code to LLVM or MLIR libraries.

In this article I’m not going to cover how to do the build itself (I believe there are plenty of great resources out there already), but rather focus on various ways to actually obtain those LLVM libraries and what kinds of features those options bring with them.

I’m also considering the simplest integration: CMake and C++, no fancy build systems, no fancy languages. Different build systems and languages would require different considerations.

Effectively, this article is organized as a table with different ways to get LLVM/MLIR on one axis, and various available features on another.

The actual table is at the very end.

Features

Here is a non-exhaustive list of different features that I consider important.
If you believe something is missing, please leave a comment.

Leave a comment

(Fast) Build Times

Obviously, everyone wants to have fast build times. There are two slightly different angles to this story: if you decide to build LLVM from scratch, it would obviously take long time. But even if you don’t build LLVM from scratch, you may still have to wait for way too long due to the static linking.

Also, building LLVM/MLIR from scratch without caching is going to be a huge bottleneck on the CI.

Debugging Experience

Once in a while things go south, so you need to debug not only your code, but also look into what’s “wrong” inside of LLVM.
What I mean here is not just having debug info and assertions enabled, but also facilities like -debug-only=.
One example from MLIR is debugging long conversion pipelines/pattern matching, when things don’t quite work the way you’d expect.

Testing Infrastructure

Both LLVM and MLIR heavily rely on integration testing using lit and filecheck.
None of these are part of the “official distribution” unfortunately1. While the official lit can be installed as a separate python package, for filecheck your best bet is third-party solutions, which are actually pretty good starting points if you don’t need very advanced filecheck features (e.g. mull-project/filecheck.py or AntonLydike/filecheck).

Bleeding Edge

This is also an important factor. As a starting point, you can just use whatever is available from your default OS package manager (e.g. apt or homebrew), but at some point you may need to pick something much newer due to bugfixes or new features.

Dynamic Linking

This is more of a niche feature, but it is very important if you are working on any kind of plugins, or if you don’t want to deal with long static linking time during development.

Different LLVM distributions

Here I’m considering more or less cross-platform solutions, so I’m not covering Debian/Ubuntu specific repo. Which leaves us with three options: (semi-)official versions from an OS package manager, precompiled binaries (submitted by volunteers), and BYOB: “bring your own build” story.

(Semi-)official OS packages

These are the packages maintained by the OS maintainers and not necessarily by LLVM maintainers. These packages are the easiest way to start: just call apt/brew install llvm and you are done.

The packages come with dynamic libraries, which enables both fast build times and plugin support. The packages usually contain everything that is needed for testing, but they of course lack the debugging story.

The other inconvenience might be the age of the package: depending on the OS and its stability guarantees, the package might be way too old for your use case.
For LLVM it’s probably fine, but it gets trickier for MLIR as the APIs are less stable across the recent versions.

Precompiled packages

These packages are available as the release artifacts, for example 20.1.4 or 18.1.8.

On one hand, this is the most convenient way to get those binaries: the most recent binaries appear there just a few days after the official release.
On the other hand, some packages are prepared by volunteers, so some releases might be missing the build for your specific OS/version, and the presence of e.g. LLVM-20.1.4-Linux-X64.tar.xz build doesn’t guarantee compatibility with e.g. Ubuntu 20.04 due to the the “old” glibc.

Just as with the official OS packages, the debugging story is not there: the packages are built in the release mode.

In general, these packages are kinda the “best effort”: if it works - great, if not - well, you are out of luck.

Build your own LLVM

This is obviously the most flexible approach: you can build any version/commit on any supported OS, you get the debugging facilities if you wish so, all the testing infrastructure is there, it’s your choice whether to use dynamic or static linking.

But of course the price is the long build times, especially if you want to get more than just LLVM (e.g, MLIR or clang libraries).

Summary

As a conclusion, the exact option depends on your use case.
Just to start with, you can pick the official package available on your OS and then decide whether you need more.

If you need the newest version, then the precompiled packages from LLVM releases page is your best bet, especially when it comes to CI integration.

However, at least at some point, you may consider building your own version of LLVM/MLIR libraries for local development, but still stick to the precompiled packages for CI checks.

To wrap it up, here is a table that sums it all up.

Subscribe now

However, if you want to build plugins for LLVM (or any other project really), then you should link against the specific versions installed on the user’s system.

As I’m working on such a plugin, it’s been a long “dream” of mine to migrate to Bazel for the many benefits it provides. Over time, the existing build system (CMake) has grown its capabilities and I have certain requirements for how the builds should work.
Namely:

• the plugin must work on different versions of OS (Ubuntu 20.xx-24.xx, macOS)

• the plugin must support different versions of LLVM, which are different on each OS (e.g., LLVM 12 on Ubuntu 20.04, LLVM 16, 17, 18 on Ubuntu 24.04 etc)

• the plugin must be linking against the system libraries due to the ABI requirements

• the build system should support multiple versions at the same time

None of these are necessarily hard or impossible with Bazel, but the devil is always in the details.

What follows is my take on solving this problem.

Source code is available here https://github.com/AlexDenisov/bazel-llvm-plugin.

Following the Cunningham's Law I claim that there is no better way to do it.

Detecting available LLVM versions

Third-party dependencies in Bazel are typically coming in a form of external repositories, thus all supported LLVM versions must be defined in MODULE.bazel upfront. However, what happens if the version is not supported or not installed on the host OS? In this case, these repositories must be defined dynamically.

To do so, first we need to define a custom dynamic repository which will check which versions are installed on the host OS and store this information in a global variable available for later use by different parts of the build system:

Which must be defined in MODULE.bazel:

Defining LLVM repositories

Now, as we know which versions are available installed on the host system, we can define LLVM repositories which will expose libLLVM.so and all the needed headers.

This part requires a dynamic module extension which will either define a real repository, or will define a “fake” empty repo. This is needed so that all the repositories can be later defined in MODULE.bazel safely.

How we can tell Bazel that these repos are available for consumption:

Defining plugin targets

Now, the rest is rather trivial. We can define all the plugin libraries depending on the LLVM versions available on the host OS:

Defining test targets

Obviously, we must have tests for the plugin. This is also relatively trivial, we need to define a test case for each available LLVM versions as well, thus producing NxM tests where N is the number of tests and M is the number of LLVM versions.

Conclusion

With all the little pieces above, the builds are now completely transparent and smooth for the end user:

Full working example can be found here: https://github.com/AlexDenisov/bazel-llvm-plugin

Here, I want to share some links that would help to learn compilers.

The list could not be exhaustive - everyone is busy, and no one has time to read the Dragon Book.

The main criteria behind each link:

• I can personally recommend the material as I went through it

• each entry should be relatively short and can be consumed in a reasonable time

I'm a big fan of learning through practicing. Thus the main focus is on LLVM, as you can go and do something cool with real-world programs!

The list consists of four groups: general theory, front-end, middle-end, and back-end.

At the first run, you can take the first item from each group, and it should put you on a solid ground.

Disclaimer

There are a lot of excellent resources out there!
Some of them are not on the list because of my subjective judgment, and the others are not here because I've never seen them!
Please, share your favourite resource in the comments.

Leave a comment

General Theory / Introduction

• AOSA book: LLVM
  This is a chapter from the Architecture of Open Source Applications book.
  It is written by Chris Lattner and covers high-level LLVM design.

• Compilers
  The course is taught by Alex Aiken. In this course, you build a compiler for a real programming language from scratch. It covers the whole compilation pipeline: parsing, type-checking, optimizations, code generation. Besides practical parts, it also dives into the theory.

• Automata Theory
  The course is taught by Jeffrey Ullman. This one is pretty heavy on theory. It starts with relatively simple topics like state machines and finite automata (deterministic and otherwise). It gradually moves on to more complex things like Turing-machines, computational complexity, famous P vs. NP, etc.

or

• Theory of Computation
  This course is taught by Michael Sipser. It is similar to the one above but delivered in a different style. It goes into more details on specific topics.

Front-end

The compiler front-end is where the interaction with the actual source code happens. The compiler parses the source code into an Abstract Syntax Tree (AST), does semantic analysis and type-checking, and converts it into the intermediate representation (IR).

The Compilers course from the above covers the general parts.

Here are some links specific to Clang:

• Understanding the Clang AST
  This article is written by Jonas Devlieghere. It goes into detail and touches implementation details of Clang's AST. It also has a lot of excellent links to dive deeper into the subject.

• clang-tutor
  This repository maintained by Andrzej Warzyński. It contains several Clang plugins covering various topics, from simple AST traversals to more involved subjects such as automatic refactoring and obfuscation.

Middle-end

The middle-end is a place where various optimizations happen. Typically, the middle-ends use some intermediate representation. The intermediate representation of LLVM is usually referred to as LLVM IR or LLVM Bitcode.

In a nutshell, it is a human-readable assembly language for a pseudo-machine (i.e., the IR does not target any specific CPU).

The LLVM IR maintains certain properties: it is in a Static Single Assignment (SSA) form organized as a Control-Flow Graph (CFG).

• LLVM IR Tutorial - Phis, GEPs and other things, oh my!
  This is a great talk by Vince Bridgers and Felipe de Azevedo Piovezan.

• Introduction to LLVM
  A one-hour-long talk/tutorial from LLVM Developers meeting given by Eric Christopher and Johannes Doerfert. Another great tutorial that better builds on top of the previous video.

• CS 6120: Advanced Compilers
  The course is taught by Adrian Sampson. The title says "advanced," but it covers what one would expect in a modern production-grade compiler: SSA, CFG, optimizations, various analyses.

• Bitcode Demystified
  This one is from me. It gives a high-level description of what's the LLVM Bitcode/IR is.

• llvm-tutor
  This one is also from Andrzej Warzyński. It covers LLVM plugins (so-called passes) that allow one to analyze and transform the programs in the LLVM IR form.

Back-end

The last phase of the compilation is a back-end. This phase aims to convert the intermediate representation into a machine code (zeros and ones). The zeros and ones later can be run on the CPU. Therefore, to understand the back-end, you need to understand the machine code and how CPUs work.

• Build a Modern Computer from First Principles: From Nand to Tetris
  Taught by Shimon Schocken and Noam Nisan. This course starts backward: first, you build the logic gates (and, or, xor, etc.), then use the logic gates to construct Arithmetic-Logic Unit (ALU), and then use the ALU to build the CPU. Then you learn how to control the CPU with zeros and ones (machine code), and eventually, you develop your assembler to convert the human-readable assembly into the machine code.

• Parsing Mach-O files
  This is a short article written by me. It shows how to parse object files on macOS (Mach-O). If you are on Linux or Windows, search for similar articles on `elf` and `PE/COFF` files, respectively.

• Performance Analysis and Tuning on Modern CPUs
  The book by Denis Bakhvalov. While it is about performance, it gives an excellent introduction to how CPUs work.

Bonus points

Here are some more LLVM related channels I recommend looking at:

• LLVM's YouTube channel
  Here you can find a lot of talks from developer meetings.

• LLVM Weekly
  A weekly newsletter run by Alex Bradbury. This is the single newsletter I am aware of that doesn't have ads!

• LLVM Blog
  This is, well, LLVM's blog.

• LLVM Tutorials
  Good starting points, even if you know nothing about compilers.

• Embedded in academia
  John Regehr's blog has lots of goodies when it comes to LLVM and compilers!

Strings attached

As I mentioned in the beginning, Compilers is a huge field!

If you go through the material above, you will learn a lot, but you will still have a few knowledge gaps in the whole compilation pipeline (I certainly do).

But the good thing is - you'd know what the gaps are and how to address them!

Good luck!

Share

In this article, I want to make a short intro on where to start if you have a similar issue. It is by no means an exhaustive guide, but rather a starting point that could have saved me a few hours would I have known this information before.

Inspect the dyld from inside

1. Source code
   To understand what’s going on within DYLD you’d need to look at the source code, but also at the disassembly.
   The source code is generally available on GitHub, but might be slightly outdated if you are running the latest version of macOS.

2. Binaries
   The binaries you are interested in are /usr/bin/dyld and /usr/lib/system/libdyld.dylib. Since macOS switched do dyld cache, you’d need to use dyld-shared-cache-extractor to extract all the binaries. Once you get the libraries you can disassemble them (Hopper ftw) and start debugging.

3. LLDB
   Run your binary and add a breakpoint based on a regex (e.g. `br set -r dyld`) and hit `run`

During debugging, please be ready to jump a lot between the source code, /usr/lib/dyld and /usr/lib/system/libdyld.dylib

Inspect the dyld from the outside

There are a few other options to observe the behavior of dyld without looking at the code, source or binary.
You will also need to disable SIP if you want to exercise it on systems apps.

All the options are controlled via environment variables. These are the ones I found the most useful:

• DYLD_PRINT_APIS: documented, prints a nice trace of almost everything that is happening inside of dyld. Here is an example output:
  
  _dyld_register_func_for_add_image(0x7fff7696ab92)

  _dyld_get_image_slide(0x1000f1000)

  _dyld_register_func_for_add_image(0x7fff7689cd98)

  _dyld_get_image_slide(0x1000f1000)

  _dyld_register_func_for_add_image(0x7fff76be67cb)

  dyld_image_path_containing_address(0x7fff75221000)

It looks cryptic, but it greatly helps to understand the program execution flow.

• DYLD_PRINT_LIBRARIES: documented, prints all the dynamic libraries that are being loaded during the app startup. Here is an example output:
  
  dyld: loaded: /usr/lib/libiconv.2.dylib

  dyld: loaded: /System/Library/Frameworks/Security.framework/Versions/A/Security

  dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation

  dyld: loaded: /usr/lib/libz.1.dylib

  dyld: loaded: /usr/lib/libSystem.B.dylib

  dyld: loaded: /usr/lib/libresolv.9.dylib

  dyld: loaded: /usr/lib/system/libcache.dylib

  dyld: loaded: /usr/lib/system/libcommonCrypto.dylib

  dyld: loaded: /usr/lib/system/libcompiler_rt.dylib


• DYLD_*_PATH: changes the order of directories where dyld will search for dynamic libraries. The nice side effect of using these variables is that their presence disables the dyld3 closure cache. So if your suspect is dyld3 closures, then export any of the DYLD_*_PATH variables to disable them. Some examples:
  
  export DYLD_FRAMEWORK_PATH=

  export DYLD_LIBRARY_PATH=
  

For more info, please consult the dyld man page (man dyld) or dig through the code, source or binary.

Summary

Debugging such a thing as dyld is a nontrivial task, but it is indeed possible. If you know any other hints or tricks, please share them.

Happy debugging!