Introduction

This project provides a toolkit for efficiently processing bibliographic records encoded in MARC 21, which is a popular file format used to exchange bibliographic data between libraries. In particular, the command line tool marc21 allows efficient filtering of records and extraction of data into a rectangular schema. Since the extracted data is in tabular form, it can be processed with popular frameworks such as Polars or Tidyverse.

marc21-rs is developed by the Metadata Department of the German National Library (DNB). It is used for data analysis and for automating metadata workflows (data engineering) as part of automatic content indexing.

The source code is licensed under the European Union Public License 1.2.

Getting Started

Installation

Binaries

In order to install the marc21 binary, archives with a precompiled binary are available for Windows, macOS and Linux.

Two variants are available for Linux: a dynamically linked version and a fully statically linked version (MUSL). In most cases, the statically linked version should be preferred, as it is independent of the glibc version of the host system. The following commands install the binary into the /usr/local/bin directory:

$ tar xfz marc21-0.5.0-x86_64-unknown-linux-musl.tar.gz
$ sudo install -Dm755 marc21-0.5.0-x86_64-unknown-linux-musl/marc21 \
   /usr/local/bin/marc21

From Source

If a Rust toolchain is available, marc21 can also be installed using the Rust package manager cargo. The project requires a Rust compiler with a minimum version of 1.93. Use the following command to install the program with the default features:

$ cargo install marc21-cli

The binary can be built with the following features as needed:

build: Commands and functions that are only needed during the build process or packaging are activated with the build feature. This includes, for example, the commands for generating man pages (build-man) and shell completions (build-completion).
performant: This feature activates optimizations aimed at improving performance. This includes, for example, the activation of SIMD or a more aggressive inline strategy. Since the main goal of the project is high performance, the feature is enabled by default.
unstable: New features that are still in the testing phase can be activated using the unstable feature. Keep in mind that these functions may change at any time.

First Steps

This section provides an overview of working with the marc21 command line tool. It demonstrates important commands using simple use cases. An in-depth explanation of the concepts, in particular the structure of filter expressions, has been omitted for brevity.

The marc21 tool provides various commands for processing MARC 21 records (see marc21 --help for a complete list of available commands).

Concatenate Multiple Files

The concat command can be used to combine multiple files into a single output. In the following example, the authority data files from the Integrated Authority Files (GND) are concatenated into the single file GND.mrc.gz.

$ marc21 concat -ps authorities-gnd-*.mrc.gz -o GND.mrc.gz
10,122,437 records, 0 invalid | 49,035 records/s, elapsed: 00:03:19

The --skip-invalid (-s) option is used to skip invalid records that could not be decoded. If the option is not specified, processing will abort at the first invalid record. In addition, the processing progress can be displayed with the --progress (-p) option.

Filtering Records

The filter command extracts those records that fulfill a specified condition. For example, all records with status z and at least one field 100 with indicators 1 and # (space) can be filtered as follows:

$ marc21 filter -s 'ldr.status == "z" && 100/1#?' DUMP.mrc.gz -o out.mrc

Operators

The comparison operators ==, !=, >=, >, <=, and < can be used for values in selected leader fields, values in control fields, and values in subfields. Here are a few examples

$ marc21 filter -s '100/1#.a == "Lovelace, Ada"' DUMP.mrc.gz -o out.mrc
$ marc21 filter -s '100/*.a != "Curie, Marie"' DUMP.mrc.gz -o out.mrc
$ marc21 filter -s '001 == "119232022"' DUMP.mrc.gz -o out.mrc
$ marc21 filter -s 'ldr.length > 3000' DUMP.mrc.gz -o out.mrc
$ marc21 filter -s 'ldr.status == "z"' DUMP.mrc.gz -o out.mrc

To check whether a value (control field or data field) comes from a specified list, the in operator is used. In contrast, the not in operator checks whether a value is not contained in the list. The following example tests whether a field 100 exists that has a subfield a with the value “Curie, Marie” or “Lovelace, Ada”:

$ marc21 filter -s '100/*.a in ["Lovelace, Ada", "Curie, Marie"]' \
    DUMP.mrc.gz -o out.mrc

The =? operator and, in negated form, !? perform a substring search on subfield values. These operators allow simultaneous searching for multiple patterns by using the []-notation:

$ marc21 filter -s '100/*.a =? ["Hate", "Love"]' DUMP.mrc.gz -o out.mrc.gz
$ marc21 filter -s '100/1#.a =? "Love"' DUMP.mrc.gz -o out.mrc.gz

Subfield values can be checked against one or a set of regular expressions. The filter expression uses the =~ operator or the !~ operator in negated form. The underlying regex engine does not support all regex features; please refer to the specification to learn more about the syntax and possible limitations. The following example searches for all records with a field 533 that contains a subfield n whose value matches the regular expression for an ISBN.

$ marc21 filter -s \
    '533.n =~ "(?i)ISBN(?:-1[03])?(?::?\\s*)?\\s(?:97[89][-\ ]?)?\\d{1,5}[-\\ ]?(?:\\d+[-\\ ]?){2}(?:\\d|X)"' \
    DUMP.mrc.gz -o out.mrc.gz

To test whether a subfield value begins with a prefix or not, the =^ operator or, in its negated form, the !^ operator is used:

$ marc21 filter -s '400/1#.a =^ "Love"' DUMP.mrc.gz -o out.mrc.gz
$ marc21 filter -s '400/1#.a =^ ["Hate", "Love"]' DUMP.mrc.gz -o out.mrc.gz
$ marc21 filter -s '400/1#{ [ac] =^ "Count" }' DUMP.mrc.gz -o out.mrc.gz

In contrast, the =$ operator can be used to check whether a subfield value ends with a specific suffix. Keep in mind that the $ character often has a special meaning on the command line and may need to be escaped.

$ marc21 filter -s '548.4 =$ "/gnd#dateOfBirthAndDeath"' DUMP.mrc.gz -o out.mrc.gz
$ marc21 filter -s '401/1#.a !$ "Ada"' DUMP.mrc.gz -o out.mrc.gz

Similarity comparisons between character strings are performed using the =* operator (in negated form !*). The normalized Levenshtein distance is calculated between the subfield value and the comparison value. If this is greater than the specified threshold value, the comparison is considered a match. The default threshold value is 0.8 and can be changed using the command line option --strsim-threshold:

$ marc21 filter -s --strsim-threshold 0.9 '100/1#.a =* "Lovelace, Bda"' \
    DUMP.mrc.gz -o out.mrc.gz

Transforming records into CSV/TSV format

In the fields of data science and data engineering, it is essential that data be organized in a rectangular table schema (similar to a relational database). If the data to be analyzed is available in this format, efficient tools such as Polars can be used to perform data analysis on the underlying data. Using the select command, records can be efficiently transformed into a tabular format. By default, the output is written in CSV format.

The following example demonstrates how to create a table in CSV format, where the first column (cn) contains the control number of the record, the second column (label) contains the name of the authority record, and the third column (gndsys) contains the GND classification. Since multiple notations from the GND classification can be assigned to a single authority record, the output generates multiple rows for these records.

$ marc21 select -ps --header 'cn,label,gndsys' \
    '001, 150.a, 065{ a | 2 == "sswd" }' DUMP.mrc.gz -o out.csv
207,505 records, 0 invalid | 102,139 records/s, elapsed: 00:00:01  

$ cat out.csv
cn,label,gndsys
040000028,A 302 D,31.9b
040000230,Aargauer,17.1
040000303,Abakus,28
040000443,Abbildung,28
040000540,ABC-Schutz,7.15a
040000540,ABC-Schutz,8.4
040000567,ABC-Waffen,8.4
040000656,Abdichtung,31.3b
040000656,Abdichtung,31.6
...

Summary Statistics

The frequency command (alias freq) is used to calculate frequency tables based on the values (columns) of a query expression. The output is in CSV/TSV format and sorted in descending order.

The following example generates a frequency table of the combinations gndgen and gndspec (subfield 2) found in subfield b of field 065:

$ marc21 frequency -ps -H 'gndgen,gendspec,count' \
    '065{ b | 2 == "gndgen" }, 065{ b | 2 == "gndspec" }' GND.mrc.gz \
    -o out.csv.gz
10,220,897 records, 0 invalid | 495,993 records/s, elapsed: 00:00:20

$ zcat out.csv.gz | head -10
gndgen,gndspec,count
p,piz,6522289
b,kiz,910734
f,vie,807018
b,,375426
u,wim,330810
u,wit,242153
g,gik,180608
s,saz,129500
b,kio,115566

Counting Records

The number of records contained in the input can be determined using the count command:

$ marc21 count GND.mrc.gz
10329438

The --where option can be used to count only those records that match a certain criterion:

$ marc21 count GND.mrc.gz --where 'ldr.type == "z" && 075{ b == "gik" && 2 == "gndspec" }'
179672

Print Records

The print command output records in a human-readable format. The leader, control and data fields are written on a separate line. Consecutive records are divided by a blank line. The output of the command can be used in combination with standard UNIX tools such as grep, cut or sed. In the following example, a single data record is printed on the console:

$ marc21 print tests/data/ada.mrc --where '100/*.a =? "Love"'
LDR 03612nz  a2200589nc 4500
001 119232022
003 DE-101
005 20250720173911.0
008 950316n||azznnaabn           | aaa    |c
024/7# $a 119232022 $0 http://d-nb.info/gnd/119232022 $2 gnd
035 $a (DE-101)119232022
035 $a (DE-588)119232022
035 $z (DE-588)172642531
035 $z (DE-588a)172642531 $9 v:zg
035 $z (DE-588a)119232022 $9 v:zg
035 $z (DE-588c)4370325-2 $9 v:zg
040 $a DE-386 $c DE-386 $9 r:DE-576 $b ger $d 1841
042 $a gnd1
043 $c XA-GB
065 $a 28p $2 sswd
065 $a 9.5p $2 sswd
075 $b p $2 gndgen
075 $b piz $2 gndspec
079 $a g $q f $q s $q z $u w $u k $u v
100/1# $a Lovelace, Ada $d 1815-1852
375 $a 2 $2 iso5218
400/1# $a Lovelace, Augusta Ada of $d 1815-1852
400/1# $a Lovelace, Ada Augusta of $d 1815-1852
400/1# $a Byron, Ada $d 1815-1852
400/1# $a Byron King, Augusta Ada $d 1815-1852
400/1# $a King, Augusta Ada $d 1815-1852
400/1# $a King, Ada $d 1815-1852
400/1# $a King-Noel, Augusta Ada $c Countess of Lovelace $d 1815-1852
...

Partitioning

The input can be split into different subsets based on the values of a field or subfield using the partition command. For example, the following command partitions the authority records based on the GND classifications (field 065):

$ marc21 partition -ps '065{ a | 2 == "sswd" }' \
    authorities-gnd-sachbegriff_dnbmarc.mrc.gz -o out
207,505 records, 0 invalid | 100,033 records/s, elapsed: 00:00:01

$ tree out
out
├── 00.mrc
├── 10.10.mrc
├── 10.11a.mrc
├── 10.11b.mrc
├── 10.11c.mrc
...
├── 9.5b.mrc
├── 9.5c.mrc
└── 9.5p.mrc

1 directory, 346 files

Since the path expression for a record can produce multiple values, the partitions are generally not disjoint. If a value occurs multiple times for a record, the record is written to the respective partition only once.

Concepts

Tag Matcher

In MARC21, variable fields are uniquely identified by tags. A tag matcher is an expression used to filter those fields that match a specific tag.

In its simplest form, only the three numerical digits of a tag are specified. A match with a tag only exists if these digits exactly match those of the tag:

$ marc21 count tests/data/ada.mrc --where '001 == "119232022"'
1

$ marc21 count tests/data/ada.mrc --where '002 == "xyz"'
0

In order to identify more than one field, a pattern-based comparison must be performed. Each numerical digit of a tag can be specified by one of the following variants.

First, a digit can be represented by the wildcard character . that accepts all possible values from 0 to 9. For example, the following tag matcher accepts all fields that contains at least one field that begins with 0 and ends with 8. The middle position can contain any digit.

$ marc21 count tests/data/ada.mrc --where '0.8?'
1

Furthermore, a digit can also be represented by specifying a class of possible digits. In the following example, all fields that start with a zero, have either a two, three, or five in the second position, and end with a 5 are accepted.

$ marc21 count tests/data/ada.mrc --where '0[235]5?'
1

Similar to the character classes of a regular expression, several consecutive digits within a class can be combined into a range. The range is inclusive, and the upper interval limit must be greater than the lower limit. Note that a class can consist of more than one range expression.

$ marc21 count tests/data/ada.mrc --where '0[2-5]5?'
1

A class can also be specified in negated form (^). In this case, the matcher checks that the digit in the corresponding position of the tag does not originate from the class digits:

$ marc21 count tests/data/ada.mrc --where '04[^0-3]?'
0

After all, every options can be used in all positions of a tag matcher. In the most extreme case, the expressions ... and [0-9][0-9][0-9] accept every field, while the expression [^0-9][^0-9][^0-9] accepts no fields. In the following example, all fields that begin with any digit not followed by a 0and end with a 1, 2, 3, 5, or 6 are taken into account.

$ marc21 count tests/data/ada.mrc --where '.[^0][1-356].2 == "sswd"'
1

Indicator Matcher

Data fields are distinguished not only by the tag, but also by an indicator consisting of two lowercase ASCII alphabetic, numeric, or blank characters. The indicator matcher checks whether a data field matches the specified value. It always appears in combination with a tag matcher and is preceded by the prefix /.

The indicator matcher distinguishes between three types:

explicit matching (/10, /#1),
pattern based matching (/1[23], /1[2-5]),
wildcard matching (/*)

Note

If no indicator matcher is specified, only the fields that contain a space in both positions are taken into account.

Explicit Matching

The simplest form is to explicitly specify the two indicator positions, preceded by the prefix /. A space is not a valid value and must be replaced with #.

Examples

$ marc21 count tests/data/ada.mrc --where '400/1#?'
1

$ marc21 count tests/data/ada.mrc --where '400/11?'
0

Pattern Based Matching

If you need to specify more than one specific value for an indicator, the pattern-based approach can be helpful. In this case, a position can be either an explicit value, a class of values ([1-3]), or any value (.). The elements of a class are specified by listing the allowed values enclosed in square brackets (e.g., [136]). A class can also contain one or more ranges, such as [13-56-9]. Negation is also supported by prefixing the class with ^ ([^234]); this class matches all positions that are not 2, 3, or 4.

Examples

/2. — Indicators that begin with 2, followed by any value (e.g., 25 or 2#)
/2[35] — Indicators that begin with 2, followed by a 3 or 5 (22 or 25)
/2[3-5] — Indicators that begin with 2, followed by a 3, 4, or 5 (23, 24, or 25)
/2[^35] — Indicators that begin with 2 and are not followed by a 3 or 5 (e.g., 21, 2#, 29)
/[12][1#] — Indicators that begin with 1 or 2, followed by 1 or a blank (#) (e.g., 11, 12, 1#)
/.. or /[0-9#][0-9#] — Accepts all indicators

$ marc21 count tests/data/minna.mrc --where '083/0.?'
1

$ marc21 count tests/data/minna.mrc --where '083/0[34]?'
1

$ marc21 count tests/data/minna.mrc --where '083/0[3-5]?'
1

$ marc21 count tests/data/minna.mrc --where '083/0[^5-9#]?'
1

$ marc21 count tests/data/minna.mrc --where '083/[30][34]?'
1

$ marc21 count tests/data/minna.mrc --where '100/[0-9#][0-9#]?'
1

$ marc21 count tests/data/minna.mrc --where '083/..?'
1

Wildcard Matching

If you want to accept all possible indicators associated with a data field, you could use the wildcard expression /*:

$ marc21 count tests/data/ada.mrc --where '100/*.a =? "Ada"'
1

Subfield Matcher

A subfield matcher is an expression that is applied to a list of subfields (of a data field) and checks whether that list meets the specified criteria. The matcher is primarily used as part of a field matcher and as a constraint in query or path expressions.

The following elementary variants are distinguished:

The exists matcher ? checks whether a specific subfield is present
The count matcher # checks the number of occurrences a subfield
The comparison matcher compares the value of a subfield against a reference value
The contains matcher =? checks whether a subfield contains a specific phrase
The in matcher in checks whether the value of a subfield comes from a reference list
The starts-with matcher =^ checks whether the value of a subfield begins with a prefix
The ends-with matcher =$ checks whether the value of a subfield ends with a suffix
The similarity matcher =* checks whether the value of a subfield is similar to a reference value
The regex matcher =~ checks whether the value of a subfield matches a regular expression

Exists Matcher

The ? operator is used to check whether a data field has one or more subfields. The specific value of the subfield is irrelevant in this context. In the following example, the field 079 is checked to see if a subfield named u exists:

$ marc21 count tests/data/ada.mrc --where '079.a?'
1

By negating the expression, you can check whether a field does not contain a specific subfield:

$ marc21 count tests/data/ada.mrc --where '079{ !u? }'
0

$ marc21 count tests/data/ada.mrc --where '!079.x?'
1

By specifying a code class, you can check whether one of the specified subfields is present. The following example checks whether the field 079 contains either the subfield p or q:

$ marc21 count tests/data/ada.mrc --where '079{ [pq]? }'
1

$ marc21 count tests/data/ada.mrc --where '079.[pq]?'
1

Count Matcher

The count matcher (#) counts the number of occurrences of one or more subfields and compares that number against a reference value. The available comparison operators are ==, !=, >=, >, <=, and <. The count matcher can only be used in the long form of a field matcher.

$ marc21 count tests/data/ada.mrc --where '079{ #u > 2 }'
1

$ marc21 count tests/data/ada.mrc --where '079{ #[aq-w] >= 7 }'
1

Comparison Matcher

The comparison matcher compares the value of a subfield to a reference value. The available comparison operators are ==, !=, >=, >, <=, and <.

$ marc21 count tests/data/ada.mrc --where '075{ b == "piz" }'
1

$ marc21 count tests/data/ada.mrc --where '075.b != "piz"'
1

Optionally, the statement can be quantified using the universal quantifier ALL or the existential quantifier ANY. By default, the existential quantifier is used; that is, the existence of at least one subfield that corresponds to the reference value according to the operator is sufficient for the statement to be true. Quantified expressions can’t be used in the short form of a field matcher.

$ marc21 count tests/data/ada.mrc --where '079{ ALL u >= "k" }'
1

$ marc21 count tests/data/ada.mrc --where '079{ ANY u != "k" }'
1

Contains Matcher

The contains matcher checks whether the specified substring is contained in the specified subfield. Internally, the matcher uses the Aho–Corasick algorithm (with SIMD acceleration in some cases) to enable efficient substring searches.

$ marc21 count tests/data/ada.mrc --where '400/1#{ a =? "Augusta" }'
1

$ marc21 count tests/data/ada.mrc --where '400/1#.a =? "Augusta"'
1

The matcher also allows you to search for multiple substrings at once by specifying a list of substrings:

$ marc21 count tests/data/ada.mrc --where '400/1#.a =? ["Hate", "Love"]'
1

To check whether a substring (or a list of substrings) is not contained, use the !? operator:

$ marc21 count tests/data/ada.mrc --where '400/1#.a !? "Curie"'
1

Finally, the statements can be quantified using the universal quantifier ALL or the existential quantifier ANY. A quantifier can’t be used in the short form of a field matcher.

$ marc21 count tests/data/ada.mrc --where '035{ ALL [az] =? "DE-" }'
1

In Matcher

The in matcher checks whether the value of a subfield comes from a reference list. The values are specified as a non-empty, comma-separated list enclosed in square brackets. If you want to check whether the value of a field does not come from a list, use the not in operator instead of in.

$ marc21 count tests/data/ada.mrc --where '075{ b in ["p", "s", "u"] }'
1

$ marc21 count tests/data/ada.mrc --where '075.b in ["p", "s", "u"]'
1

$ marc21 count tests/data/ada.mrc --where '075.b not in ["b", "f", "g"]'
1