Example Usage Scenario¶

A typical scenario of Bibolamazi usage might be:

• You use a bibliography manager, such as Mendeley, to store all your references. You have maybe configured e.g. Mendeley to keep a BibTeX file Documents/bib/MyLibrary.bib in sync with your library;

• You’re working, say on a document mydoc.tex, which cites entries from MyLibrary.bib;

• You like to keep URLs in your entries in your Mendeley library, because it lets you open the journal page easily, but you don’t want the URLs to be displayed in the bibliography of your document mydoc.tex. But you’ve gone through all the bibliography styles, and really, the one you prefer unfortunatly does display those URLs.

• You don’t want to edit the file MyLibrary.bib, because it would just be overwritten again the next time you open Mendeley. The low-tech solution (what people generally do!) would then be to export the required citations from Mendeley to a new bibtex file, or copy MyLibrary.bib to a new file, and edit that file manually.

• To avoid having to perform this tedious task manually, you can use Bibolamazi to prepare the BibTeX file as you would like it to be. For this specific task, for example, you would perform the following steps:

  • Create a bibolamazi file, say, mydoc.bibolamazi.bib;

  • Specify as a source your original MyLibrary.bib:

    src: ~/Documents/bib/MyLibrary.bib
    

  • Give the following filter command:

    filter: url -dStrip
    

    which instructs to strip all urls (check out the documentation of the url filter in the Help & Reference Browser)

  • Run bibolamazi.

  • Use this file as your bibtex bibliography, i.e. in your LaTeX document, use:

    \bibliography{mydoc.bibolamazi}
    

  Note that you can then run Bibolamazi as many times as you like, to update your file, should there have been changes to your original MyLibrary.bib, for example.

Teaser: Features¶

The most prominent features of Bibolamazi include:

• A duplicates filter allows you to efficiently collaborate on LaTeX documents: in your shared LaTeX document, each collaborator may cite entries in his own bibliography database (each a source in the bibolamazi file). Then, if instructed to do so, bibolamazi will detect when two entries are duplicates of each other, merge their information, and produce LaTeX definitions such that the entries become aliases of one another. Then both entry keys will refer to the same entry in the bibliography.

  Catch: there is one catch to this, though, which we can do nothing about: if two entries in two different database share the same key, but refer to different entries. This may happen, for example, if you have automatic citation keys of the form AuthorYYYY, and if the author published several papers that same year.

• A powerful arxiv filter, which can normalize the way entries refer to the arXiv.org online preprint repository. It can distinguish between published and unpublished entries, and its output is highly customizable.

• A general-purpose fixes filter provides general fixes that are usually welcome in a BibTeX files. For example, revtex doesn’t like Mendeley’s way of exporting swedish ‘Å’, for example in Åberg, as \AA berg, and introduces a space between the ‘Å’ and the ‘berg’. This filter allows you to fix this.

• Many more! Check out the filter list in the Help & Reference Browser window of Bibolamazi!

The Bibolamazi Application¶

If you’re unsure which flavor to get, this is the one you’re looking for. It’s straightfoward to download, there is no installation required, and the application is easy to use.

Download the latest release from our releases page:

Download Release: https://github.com/phfaist/bibolamazi/releases

These binaries don’t need any installation, you can just download them, place them wherever you want, and run them.

You may now start using Bibolamazi normally. To read more on bibolamazi, skip to Using the Bibolamazi Application.

Installing the Command-Line Interface¶

Bibolamazi runs with Python 2.7 (this is there by default on most linux and Mac systems).

Additionally, the graphical user interface requires PyQt4. If you’re on a linux distribution, it’s most probably in your distribution packages. Note you only need PyQt4 to run the graphical user interface: the command-line version will happily run without.

The easy way: via PIP

The recommended way to install Bibolamazi command line and gui interfaces is via pip:

pip install bibolamazi        # for the command-line interface
pip install bibolamazigui     # if you want the GUI interface

After that, you’ll find the bibolamazi (respectively bibolamazi_gui) executables in your PATH:

> bibolamazi --help           # command-line interface
(...)
> bibolamazi_gui              # to launch the GUI
(...)

The less easy way: From Source

You may, alternatively, download and compile the packages from source.

• First, clone this repository on your computer (don’t download the prepackaged ZIP/Tarball proposed by github, because there will be missing submodules):

  > cd somewhere/where/Ill-keep-bibolamazi/
  ...> git clone --recursive https://github.com/phfaist/bibolamazi
  

  Note the --recursive switch which will also retrieve all required submodules.

• Then, run the setup script to install the package and script (see Installing Python Modules):

  > python setup.py install
  

  After that, you should find the bibolamazi executable in your PATH automatically:

  > bibolamazi --help
  

• If you want to install the GUI Application, you need to do that seperately. Go into the gui/ directory of the source code, and run the python setup script there:

  > cd gui/
  gui> python setup.py install
  

  After that, you should find the bibolamazi_gui executable in your PATH automatically:

  > bibolamazi_gui
  

Bibolamazi Operating Mode¶

Bibolamazi works by reading your reference bibtex files—the ‘sources’, which might for example have been generated by your favorite bibliography manager or provided by your collaborators—and merging them all into a new file, applying specific rules, or ‘filters’, such as turning all the first names into initials or normalizing the way arxiv IDs are presented.

The Bibolamazi file is this new file, in which all the required bibtex entries will be merged. When you prepare you LaTeX document, you should create a new bibolamazi file, and provide that bibolamazi file as the bibtex file for the bibliography.

When you open a bibolamazi file, you will be prompted to edit its configuration. This is the set of rules which will tell bibolamazi where to look for your bibtex entries and how to handle them. You first need to specify all your sources, and then all the filters.

The bibolamazi file is then a valid BibTeX file to include into your LaTeX document, so if your bibolamazi file is named main.bibolamazi.bib, you would include the bibliography in your document with a LaTeX command similar to:

\bibliography{main.bibolamazi}

The Bibolamazi Configuration Section¶

If you open the Bibolamazi application and open your bibolamazi file (or create a new one), you’ll immediately be prompted to edit its configuration section.

% src: mysource.bib

% src: /home/philippe/bibtexfiles/mylibrary.bib /Users/philippe/bibtexfiles/mylibrary.bib

% src: file1.bib [alternativefile1.bib ...]
% src: file2.bib [alternativefile2.bib ...]
% [...]

% filter: filtername [options and arguments]

% filter: arxiv --mode=strip --unpublished-mode=eprint

> bibolamazi --help arxiv

> bibolamazi --help <filtername>

> bibolamazi --list-filters

% filter: myfilterpackage:myfiltername --option1=val1  ...

Example/Template Configuration Section¶

%% BIBOLAMAZI configuration section.
%% Additional two leading percent signs indicate comments in the configuration.

%% **** SOURCES ****

%% The _first_ accessible file in _each_ source list will be read and filtered.

src:   <source file 1> [ <alternate source file 1> ... ]
src:   <source file 2> [ ... ]

%% Add additional sources here. Alternative files are useful, e.g., if the same
%% file must be accessed with different paths on different machines.

%% **** FILTERS ****

%% Specify filters here. Specify as many filters as you want, each with a `filter:'
%% directive. See also `bibolamazi --list-filters' and `bibolamazi --help <filter>'.

filter: filter_name  <filter options>

%% Example:
filter: arxiv -sMode=strip -sUnpublishedMode=eprint

%% Finally, if your file is in a VCS, sort all entries by citation key so that you don't
%% get huge file differences for each commit each time bibolamazi is run:
filter: orderentries

Available Filters¶

You can get a full list of available filters if you open the bibolamazi help & reference browser window (from the main application startup window). You can click on the various filters displayed to view their documentation on how to use them.

Filter Packages¶

Filters are organized into filter packages. All built-in filters are in the package named filters. If you want to write your own filters, or use someone else’s own filters, then you can install further filter packages.

A filter package is a Python package, i.e. a directory containing a __init__.py file, which contains python modules that implement the bibolamazi filter API.

If you develop your own filters, it is recommended to group them in a filter package, and not for example fiddle with the built-in filter package. Put your filters in a directory called, say, myfilters, and place an additional empty file in it called __init__.py. This will create a python package named myfilters with your filters as submodules.

To register the filter packages so that bibolamazi knows where to look for your filters, open the settings dialog, and click “Add filter package ...”; choose the directory corresponding to your filter package (e.g. myfilters). Now you can refer in your bibolamazi file to the filters within your filter package with the syntax myfilters:filtername or simply filtername (as long as the filter name does not clash with another filter of the same name in a different filter package).

First Steps With Bibolamazi Command-Line¶

Once you’ve installed bibolamazi as described in Installing the Command-Line Interface, you may start using it! Here are a couple of commands to get you started playing around. But it’s important to understand how Bibolamazi works: for that, read the following sections of this manual carefully.

• To compile a bibolamazi bibtex file, you should run bibolamazi in general as:

  > bibolamazi bibolamazibibtexfile.bibolamazi.bib
  

• To quickly get started with a new bibolamazi file, the following command will create the given file and produce a usable template which you can edit:

  > bibolamazi --new newfile.bibolamazi.bib
  

• For an example to study, look at the test files test/testX.bibolamazi.bib in the source code. To compile them, run:

  > bibolamazi test/test0.bibolamazi.bib
  

• For a help message with a list of possible options, run:

  > bibolamazi --help
  

  To get a list of all available filters along with their description, run:

  > bibolamazi --list-filters
  

  To get information about a specific filter, simply use the command:

  > bibolamazi --help <filter>
  

Bibolamazi Operating Mode¶

Bibolamazi works by reading a bibtex file (say main.bibolamazi.bib) with a special bibolamazi configuration section at the top. These describe on one hand sources, and on the other hand filters. Bibolamazi first reads all the entries in the given sources (say source1.bib and source2.bib), and then applies the given filters to them. Then, the main bibtex file (in our example main.bibolamazi.bib) is updated, such that:

• Any content that was already present in the main bibtex file before the configuration section is restored unchanged;
• The configuration section is restored as it was;
• All the filtered entries (obtained from, e.g., source1.bib and source2.bib) are then dumped in the rest of the file, overwriting the rest of main.bibolamazi.bib (which logically contained output of a previous run of bibolamazi).

The bibolamazi file main.bibolamazi.bib is then a valid BibTeX file to include into your LaTeX document, so you would include the bibliography in your document with a LaTeX command similar to:

\bibliography{main.bibolamazi}

The Bibolamazi Configuration Section¶

The main bibtex file should contain a block of the following form:

%%%-BIB-OLA-MAZI-BEGIN-%%%
%
%   ... bibolamazi configuration section ...
%
%%%-BIB-OLA-MAZI-END-%%%

The configuration section is started by the string %%%-BIB-OLA-MAZI-BEGIN-%%% on its own line, and is terminated by the string %%%-BIB-OLA-MAZI-END-%%%, also on its own line. The lines between these two markers are the body of the configuration section, and are where you should specify sources and filters. Leading percent signs on these inner lines are ignored. Comments can be specified in the configuration body with two additional percent signs, e.g.:

% %% This is a comment

Content of the Configuration Section¶

The content of the configuration section is the same as described in The Bibolamazi Configuration Section. Of course, you’ll probably want to prefix all lines by an additional ‘%’ to make sure it gets interpreted as a bibtex comment (see example below).

Example Full Bibolamazi File¶

Here is a minimal example of a bibolamazi bibtex file:

.. Additionnal stuff here will not be managed by bibolamazi. It will also not be
.. overwritten. You can e.g. temporarily add additional references here if you
.. don't have bibolamazi installed.


%%%-BIB-OLA-MAZI-BEGIN-%%%
%
% %% BIBOLAMAZI configuration section.
% %% Additional two leading percent signs indicate comments in the configuration.
%
% %% **** SOURCES ****
%
% %% The _first_ accessible file in _each_ source list will be read and filtered.
%
% src:   <source file 1> [ <alternate source file 1> ... ]
% src:   <source file 2> [ ... ]
%
% %% Add additional sources here. Alternative files are useful, e.g., if the same
% %% file must be accessed with different paths on different machines.
%
% %% **** FILTERS ****
%
% %% Specify filters here. Specify as many filters as you want, each with a `filter:'
% %% directive. See also `bibolamazi --list-filters' and `bibolamazi --help <filter>'.
%
% filter: filter_name  <filter options>
%
% %% Example:
% filter: arxiv -sMode=strip -sUnpublishedMode=eprint
%
% %% Finally, if your file is in a VCS, sort all entries by citation key so that you don't
% %% get huge file differences for each commit each time bibolamazi is run:
% filter: orderentries
%
%%%-BIB-OLA-MAZI-END-%%%
%
%
% ALL CHANGES BEYOND THIS POINT WILL BE LOST NEXT TIME BIBOLAMAZI IS RUN.
%

... bibolamazi filtered entries ...

Querying Available Filters and Filter Documentation¶

A complete list of available filters, along with a short description, is obtained by:

> bibolamazi --list-filters

Run that command to get an up-to-date list. At the time of writing, the list of filters is:

> bibolamazi --list-filters

List of available filters:
--------------------------

Package `filters':

  arxiv         ArXiv clean-up filter: normalizes the way each biblographic
                entry refers to arXiv IDs.
  citearxiv     Filter that fills BibTeX files with relevant entries to cite
                with \cite{1211.1037}
  citekey       Set the citation key of entries in a standard format
  duplicates    Filter that detects duplicate entries and produces rules to make
                one entry an alias of the other.
  fixes         Fixes filter: perform some various known fixes for bibtex
                entries
  nameinitials  Name Initials filter: Turn full first names into only initials
                for all entries.
  only_used     Filter that keeps only BibTeX entries which are referenced in
                the LaTeX document
  orderentries  Order bibliographic entries in bibtex file
  url           Remove or add URLs from entries according to given rules, e.g.
                whether DOI or ArXiv ID are present

--------------------------

Filter packages are listed in the order they are searched.

Use  bibolamazi --help <filter>  for more information about a specific filter
and its options.

Specifying Filter Packages¶

The command-line bibolamazi by default only knows the built-in fitler package filters. You may however specify additional packages either by command-line options or with an environment variable.

You can specify additional filter packages with the command-line option --filter-package:

> bibolamazi myfile.bibolamazi.bib --filter-package 'package1=/path/to/filter/pack'

The argument to --filter-package is of the form ‘packagename=/path/to/the/filter/package’. Note that the path is which path must be added to python’s sys.path in order to import the filterpackagename package itself, i.e. the last item of the path must not be the package directory.

This option may be repeated several times to import different filter packages. The order is relevant; the packages specified last will be searched for first.

You may also set the environment variable BIBOLAMAZI_FILTER_PATH. The format is filterpack1=/path/to/somewhere:filterpack2=/path/to/otherplace:..., i.e. a list of filter package specifications separated by ‘:’ (Linux/Mac) or ‘;’ (Windows). Each filter package specification has the same format as the command-line option argument. In the environment variable, the first given filter packages are searched first.

Developing Custom filters¶

Writing filters is straightforward. An example is provided here: Example of a custom filter. Look inside the bibolamazi/filters/ directory at the existing filters for further examples, e.g. arxiv.py, duplicates.py or url.py. They should be rather simple to understand.

A filter can either act on individual entries (e.g. the arxiv.py filter), or on the whole database (e.g. duplicates.py).

For your organization, it is recommended to develop your filter(s) in a custom filter package which you keep a repository e.g. on github.com, so that the filter package can be easily installed on the different locations you would like to run bibolamazi from.

Don’t forget to make use of the bibolamazi cache, in case you fetch or compute values which you could cache for further reuse. You should access caches through the BibUserCacheAccessor class. Look at for the documentation for the bibusercache module. Look at examples most of all!! (TODO: add documentation about caches)

There are a couple utilities provided for the filters, check the bibolamazi.filters.util module. In particular check out the arxivutil and auxfile modules.

Feel free to contribute filters, it will only make bibolamazi more useful!

The Filter Module¶

There are two main objects your module should define at the very least:

• a filter class, subclass of BibFilter.

• a method called bibolamazi_filter_class(), which should return the filter class object. For example:

  def bibolamazi_filter_class():
      return ArxivNormalizeFilter;
  

You may want to have a look at Example of a custom filter for an example of a custom filter.

Your filter should log error, warning, information and debug messages to a logger obtained via Python’s logging mechanism, as demonstrated in the example.

Passing Arguments to the Filter¶

Command line arguments passed to the filter in the user’s bibolamazi config section are parsed into Python arguments to the filter class’ constructor. The translation is rather intuitive: each argument to the filter may be specified as an option, either using the syntax --use-uppercase=value or --use-uppercase value, where underscores are replaced by dashes, or using the Ghostscript-like syntax -dUseUppercase or -dUseUppercase=false, or for other types -sMode=fixed.

Some remarks:

• to each filter argument corresponds a command-line option starting with --, where underscores are replaced by dashes. The command-line takes a single mandatory argument (except for arguments declared as booleans in their arg-docs, see Argdocs: Filter Argument Documentation below).
• to each filter argument, corresponds a command-line option starting with -d or -s, using the syntax -dFilterOptionName, -dFilterOptionName=Value or -sFilterOptionName=Value. The -d variant is used to specify boolean option values, the -s variant any other type. The FilterOptionName is obtained by camel-casing the filter python argument: for example, if the filter constructor accepts an argument named use_uppercase_chars, then the corresponding camel-cased version will be UseUppercaseChars. (See note below on case sensitivity.)
• each filter argument may be documented using Argdocs: Filter Argument Documentation. This information will appear in the filter help text.
• if the filter constructor accepts a **kwargs, then any additional option-value pairs given as -sKey=Value or -dKey or -dKey=Value are passed on to the filter constructor’s kwargs.
• if the filter constructor accepts a *args, then any additional positional arguments on the command line is passed to that *args parameter. The ordering of positional and optional arguments on the command-line make no difference. (Note that this also works this way if not all the previous declared arguments are specified. There’s some python hacking in there ;) )

Filter General Help Documentation¶

The filter class should declare the members helpauthor, helpdescription and helptext with meaningful help text:

• helpauthor should be a short one-line description of the filter and contributor with license. E.g.:

  ArXiv clean-up filter by Philippe Faist, (C) 2013, GPL 3+
  

• helpdescription is a brief description of what the filter does. This is displayed right after the Usage section in the help text, and before the filter arguments description.

• helptext is a long description of what the filter exactly does, how to use it, the advantages, tricks, pitfalls, etc.

In the built-in filters, as well as the examples, the text is declared outside of the class (see HELP_AUTHOR etc.) so that we don’t have to deal with the indentation (and in the class, we only have helpauthor=HELP_AUTHOR etc.). That’s perfectly fair and completely optional.

Argdocs: Filter Argument Documentation¶

The docstring of the filter constructor is parsed in a special way. Documentation of the function arguments are specially parsed: they should have the form:

- argument_name(type): Description of the argument. The description may
  span over several lines.
- other_argument_name: Description of the other option. Notice that the
  type is optional and will default to a simple string.

This information will be displayed when running bibolamazi --help filtername.

If a type is specified, it should be a name of a python type, or a type which is available in the namespace of the filter module. The filter factory will attempt to convert the given string to the specified type when calling the filter constructor. If the given type is a custom type, and it has a docstring, then the docstring is included in the “Note on Filter Options Syntax” section of the help text.

There are some convenient predefined types for filter arguments, all defined in the module bibolamazi.bibfilter.argtypes:

• CommaStrList: a comma-separated list of strings. This type may directly be used as a list type.
• enum_class(): a function which returns a custom class which represents an enumeration value of several options.

Maybe look at the built-in filters and other examples to get an idea.

More doc should come here at some point in the future..........

Customizing Default Behavior¶

There are several other functions the module may define, although they are not mandatory.

• parse_args() should parse an argument string, and return a tuple (args, kwargs) of how the filter constructor should be called. If the module does not provide this function, a very powerful default automatic filter option processor (based on python’s argparse module) is built using the filter argument names as options names.

• format_help() should return a string with full detailed information about how to use the filter, and which options are accepted. If the module does not provide this function, the default automatic filter option processor is used to format a useful help text (which should be good enough for most of your purposes, especially if you don’t want to reinvent the wheel).

  Note: the helptext attribute of your BibFilter subclass is only used by the default automatic filter option processor; so if you implement format_help() manually, the helptext attribute will be ignored.

Module contents¶

Core bibolamazi module.

See bibolamazi.core.bibfilter, bibolamazi.core.bibolamazifile, bibolamazi.core.bibusercache for the main core modules.

Subpackages¶

bibolamazi.core.argparseactions module¶

This module defines callbacks and actions for parsing the command-line arguments for bibolamazi. You’re most probably not interested in this API. (Not mentioning that it might change if I feel the need for it.)

bibolamazi.core.argparseactions.help_list_filters()[source]¶

bibolamazi.core.argparseactions.helptext_prolog()[source]¶

class bibolamazi.core.argparseactions.opt_action_help(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.opt_action_version(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.opt_init_empty_template(nargs=1, **kwargs)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.opt_list_filters(nargs=0, **kwargs)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.opt_set_fine_log_levels(nargs=1, **kwargs)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.opt_set_verbosity(nargs=1, **kwargs)[source]¶

Bases: argparse.Action

bibolamazi.core.argparseactions.run_pager(text)[source]¶

Call pydoc.pager() in a unicode-safe way.

class bibolamazi.core.argparseactions.store_key_bool(option_strings, dest, nargs=1, const=True, exception=<type 'exceptions.ValueError'>, **kwargs)[source]¶

Bases: argparse.Action

Handles an ghostscript-style option of the type ‘-dBoolKey’ or ‘-dBoolKey=0’.

class bibolamazi.core.argparseactions.store_key_const(option_strings, dest, nargs=1, const=True, **kwargs)[source]¶

Bases: argparse.Action

class bibolamazi.core.argparseactions.store_key_val(option_strings, dest, nargs=1, exception=<type 'exceptions.ValueError'>, **kwargs)[source]¶

Bases: argparse.Action

Handles an ghostscript-style option of the type ‘-sBoolKey=some-value’.

class bibolamazi.core.argparseactions.store_or_count(option_strings, dest, nargs='?', **kwargs)[source]¶

Bases: argparse.Action

bibolamazi.core.bibolamazifile module¶

The Main bibolamazifile module: this contains the BibolamaziFile class definition.

bibolamazi.core.bibolamazifile.AFTER_CONFIG_TEXT = "%\n%\n% ALL CHANGES BEYOND THIS POINT WILL BE LOST NEXT TIME BIBOLAMAZI IS RUN.\n%\n\n\n%\n% This file was generated by BIBOLAMAZI __BIBOLAMAZI_VERSION__ on __DATETIME_NOW__\n%\n% https://github.com/phfaist/bibolamazi\n%\n% Bibolamazi collects bib entries from the sources listed in the configuration section\n% above, and merges them all into this file while applying the defined filters with\n% the given options. Your sources will not be altered.\n%\n% Any entries ABOVE the configuration section will be preserved as is, which means that\n% if you don't want to install bibolamazi or if it not installed, and you want to add\n% a bibliographic entry to this file, add it AT THE TOP OF THIS FILE.\n%\n%\n\n\n\n\n\n"¶

Some text which is inserted immediately after the config section when saving bibolamazi files. Includes a warning about losing any changes.

bibolamazi.core.bibolamazifile.BIBOLAMAZIFILE_INIT = 0¶

Bibolamazi file load state: freshly initialized, no data read. See doc for BibolamaziFile.

bibolamazi.core.bibolamazifile.BIBOLAMAZIFILE_LOADED = 3¶

Bibolamazi file load state: data read and parsed, filters instanciated and data from sources loaded. See doc for BibolamaziFile.

bibolamazi.core.bibolamazifile.BIBOLAMAZIFILE_PARSED = 2¶

Bibolamazi file load state: data read and parsed, filters instanciated but no sources loaded. See doc for BibolamaziFile.

bibolamazi.core.bibolamazifile.BIBOLAMAZIFILE_READ = 1¶

Bibolamazi file load state: data read, not parsed. See doc for BibolamaziFile.

bibolamazi.core.bibolamazifile.BIBOLAMAZI_FILE_ENCODING = 'utf-8'¶

The encoding used to read and write bibolamazi files. Don’t change this.

class bibolamazi.core.bibolamazifile.BibolamaziFile(fname=None, create=False, load_to_state=3, use_cache=True, default_cache_invalidation_time=None)[source]¶

Bases: object

Represents a Bibolamazi file.

This class provides an API to read and parse bibolamazi files, as well as load data defined in its configuration section and interact with its filters.

A BibolamaziFile object may be in different load states:

• BIBOLAMAZIFILE_INIT: The BibolamaziFile object is initialized to an empty state. The file name (fname()) may be set already, but is None by default.
• BIBOLAMAZIFILE_READ: Data has been read from a given file, but not parsed. You may call certain methods such as rawHeader() or configData(), but e.g. configCmds() will return an invalid value.
• BIBOLAMAZIFILE_PARSED: Data has been read from a bibolamazi file and parsed, and filter objects have been instanciated. Methods such as filters() or sourceLists() may be called.
• BIBOLAMAZIFILE_LOADED: The bibolamazi file has been read and parsed, filter objects have been instanciated and bibtex data from the sources has been loaded. This is the “maximally loaded” state.

You may query the load state with getLoadState() and load a bibolamazi file either from the constructor or by calling explicitly load(). Some methods on this object may only be called if the object has reached a certain load state. These methods are documented as such.

The bibliography database is accessed with bibliographyData(). You may change the entries in the database via direct access (using the pybtex API), or using the method setEntries().

To create a new bibolamazi file template, you may specify create=True to the constructor with a valid file name, and save the object.

Create a BibolamaziFile object.

If fname is provided, the file is fully loaded (unless create is also provided).

If create is given and set to True, then an empty template is loaded and the internal file name is set to fname. The internal state will be set to BIBOLAMAZIFILE_LOADED and calling saveToFile() will result in writing this template to the file fname.

If load_to_state is given, then the file is only loaded up to the given state. See load() for more details. The state should be one of BIBOLAMAZIFILE_INIT, BIBOLAMAZIFILE_READ, BIBOLAMAZIFILE_PARSED or BIBOLAMAZIFILE_LOADED.

If use_cache is True (default), then when loading this file, we’ll attempt to load a corresponding cache file if it exists. Note that even if use_cache is False, then cache will still be written when calling saveToFile().

If default_cache_invalidation_time is given, then the default cache invalidation time is set before loading the cache.

bibliographyData()[source]¶

Return the pybtex.database.BibliographyData object which stores all the bibliography entries.

This object is only instanciated and initialized once in the BIBOLAMAZIFILE_LOADED state. If getLoadState() != BIBOLAMAZIFILE_LOADED, then this function returns None.

bibliographydata()[source]¶

Deprecated since version 2.0: Use bibliographyData() instead!

cacheAccessor(klass)[source]¶

Returns the cache accessor instance corresponding to the given class.

See documentation of bibolamazi.core.bibusercache for more information about the bibolamazi cache.

cacheFileName()[source]¶

The file name where the cache will be stored. You don’t need to access this directly, the cache will be loaded and saved automatically.

Filters should only access the cache through cache accessors. See cacheAccessor().

configCmds()[source]¶

Return a list of parsed commands from the configuration section.

This returns a list of BibolamaziFileCmd objects.

This may be called in the state BIBOLAMAZIFILE_PARSED.

configData()[source]¶

Returns the configuration commands, with leading percent signs stripped, and without the begin and end tags.

This may be called in the state BIBOLAMAZIFILE_READ.

configLineNo(filelineno)[source]¶

Utility to convert file line number to config line number

Returns the line number in the config data corresponding to line filelineno in the file. Opposite of fileLineNo().

This may be called in the state BIBOLAMAZIFILE_READ.

fdir()[source]¶

Returns the directory name in which this bibolamazi file resides, always as a full path (using os.path.realpath, resolving symlinks). The value is cached, so you may call this function several times with little performance overhead.

If fname() is None (this is only possible if the load state is BIBOLAMAZIFILE_INIT), then None is returned.

fileLineNo(configlineno)[source]¶

Utility to convert config line number to file line number

Returns the line number in the bibolamazi file corresponding to the config line number configlineno. The configlineno refers to the line number INSIDE the config section, where line number 1 is right after the begin config tag CONFIG_BEGIN_TAG.

This may be called in the state BIBOLAMAZIFILE_READ.

filters()[source]¶

Return a list of filter instances

This returns the list of all filter commands given in the bibolamazi config section. The instances have already been instanciated with the proper options. The order of this list is exactly the order of the filters in the config section.

If in the config section the same filter is invoked several times, then separate instances are returned in this list with the appropriate ordering, as you’d expect.

fname()[source]¶

Returns the file name this object refers to.

If the state is any other than BIBOLAMAZIFILE_INIT, then this function will never return None.

getLoadState()[source]¶

Returns the state of the BibolamaziFile object. One of BIBOLAMAZIFILE_INIT, BIBOLAMAZIFILE_READ, BIBOLAMAZIFILE_PARSED, or BIBOLAMAZIFILE_LOADED.

load(fname=[], to_state=3)[source]¶

Load the given file into the current object.

If fname is None, then reset the object to an empty state. If fname is not given or an empty list, then use any previously loaded fname and its state.

This function may be called several times with different states to incrementally load the file, for example:

bibolamazifile.reset()
# load up to 'parsed' state
bibolamazifile.load(fname="somefile.bibolamazi.bib", to_state=BIBOLAMAZIFILE_PARSED)
# continue loading up to fully 'loaded' state
bibolamazifile.load(fname="somefile.bibolamazi.bib", to_state=BIBOLAMAZIFILE_LOADED)

If to_state is given, will only attempt to load the file up to that state. This can be useful, e.g., in a config editor which needs to parse the sections of the file but does not need to worry about syntax errors. The state should be one of BIBOLAMAZIFILE_INIT, BIBOLAMAZIFILE_READ, BIBOLAMAZIFILE_PARSED or BIBOLAMAZIFILE_LOADED.

rawConfig()[source]¶

Return the raw configuration section. The returned value will NOT have the leading percent signs removed.

This may be called in the state BIBOLAMAZIFILE_READ.

rawHeader()[source]¶

Return any content above the configuration section.

This may be called in the state BIBOLAMAZIFILE_READ.

rawRest()[source]¶

Return all the contents after the config section at the moment the file was read from the disk. This includes the begin and end config section tags (CONFIG_BEGIN_TAG and CONFIG_END_TAG).

Any changes to the bibliography data will not be reflected here, even if you call saveToFile().

This may be called in the state BIBOLAMAZIFILE_READ.

rawStartConfigDataLineNo()[source]¶

Returns the line number on which the begin config tag CONFIG_BEGIN_TAG is located. Line numbers start at 1 at the top of the file like in any reasonable editor.

This may be called in the state BIBOLAMAZIFILE_READ.

reset()[source]¶

Reset the current object to an empty state and unset the file name. This will reset the object to the state BIBOLAMAZIFILE_INIT.

resolveSourcePath(path)[source]¶

Resolves a path (for example corresponding to a source file) to an absolute file location.

This function expands ‘~/foo/bar’ to e.g. ‘/home/someone/foo/bar’; it also expands shell variables, e.g. ‘$HOME/foo/bar’ or ‘${MYBIBDIR}/foo/bar.bib’.

If the path is relative, it is made absolute by interpreting it as relative to the location of this bibolamazi file (see fdir()).

Note: path should not be an URL.

saveToFile()[source]¶

Save the current bibolamazi file object to disk.

This will write to the file fname() in order:

• the raw header data (rawHeader()) unchanged
• the config section text (rawConfig()) unchanged
• the bibliography data contained in bibliographyData(), saved in BibTeX format.

A warning message is included after the config section that the remainder of the file was automatically generated.

As the file fname is expected to already exist, it is always silently overwritten (so be careful).

setBibliographyData(bibliographydata)[source]¶

Set the bibliographydata database object directly.

The object bibliographydata should be of instance pybtex.database.BibliographyData.

Warning

Filters should NOT set a different bibliographydata object: caches might have kept a pointer to this object (see, for example EntryFieldsTokenChecker). Please use setEntries() instead.

setConfigData(configdata)[source]¶

Store the given data configdata in memory as the configuration section of this file.

This function cleanifies the configdata a bit by adding leading percent signs and forcing a final newline, adds the config section begin and end tags, and then directly calls setRawConfig().

setDefaultCacheInvalidationTime(time_delta)[source]¶

A timedelta object giving the amount of time for which data in cache is consdered valid (by default).

Note that this function should be called BEFORE the data is loaded. If you just call, for example the default constructor, this might be too late already. If you use the load() function, set the default cache invalidation time before you load up to the state BIBOLAMAZIFILE_LOADED.

Note that you may also use the option in the constructor default_cache_invalidation_time, which has the same effect as this funtion called at the appropriate time.

setEntries(bibentries)[source]¶

Replace all the entries in the current bibliographydata object by the given entries.

Arguments:

• bibentries: the new entries to set. bibentries should be an iterable of (key, entry) (or, more precisely, any valid argument for pybtex.database.BibliographyData.add_entries()).

Warning

This will remove any existing entries in the database.

This function alters the current bibliographyData() object, and does not replace it by a new object. (I.e., if you kept a reference to the bibliographyData() object, the reference is still valid after calling this function.)

setRawConfig(configblock)[source]¶

Store the given configblock in memory as the raw configuration section of the bibolamazi file. We must be at least in state BIBOLAMAZIFILE_READ to call this function; this function will also reset to state back to BIBOLAMAZIFILE_READ (as the configuration might have changed).

Note that configblock is expected to start and end with the appropriate config section tags (CONFIG_BEGIN_TAG and CONFIG_END_TAG).

After calling this function, configData() will return the new configuration data. Call load() to re-instanciate filters and re-load sources.

sourceLists()[source]¶

Return a list of source lists, in the order they are specified in the configuration section.

Each item in the returned list is itself a list of alternative sources to consider.

This may be called in the state BIBOLAMAZIFILE_PARSED.

sources()[source]¶

Return a list of sources which have been read.

This is a list of strings. Each item in the returned list is one of the items in the corresponding list from sourceLists() (the one that was actually found and read). If no corresponding item in sourceLists() was readable, then the corresponding item in this list is None. For example:

# suppose that we have the following instructions in the bibolamazi file:
#
#     src: src1.bib
#     src: a.bib b.bib c.bib
#     src: x/x.bib y/y.bib
#
# we would then have:
#
f.sourceLists() == [["src1.bib"], ["a.bib", "b.bib", "c.bib"], ["x/x.bib", "y/y.bib"]]

# suppose that "src1.bib" exists, "a.bib" doesn't exist but "b.bib" exists, and neither
# "x/x.bib" nor "y/y.bib" don't exist.
#
# Then, after loading this object, we get:
#
f.sources() == ["src1.bib", "b.bib", None]

This function may be called in the state BIBOLAMAZIFILE_LOADED.

class bibolamazi.core.bibolamazifile.BibolamaziFileCmd(cmd=None, text='', lineno=-1, linenoend=-1, info={})[source]¶

A command in the bibolamazi file configuration

Stores the command name (e.g. ‘src’ or ‘filter’), additional text (the options), line number information and possible additional information.

Object Properties:

• cmd: the command name. Currently this is ‘src’ or ‘filter’
• text: the text following the command. This is e.g. the sources list, or a filter name followed by options. In general, it is anything following the ‘src:’ or ‘filter:’ commands.
• lineno: the line number at which this command is specified in the bibolamazi file, relative to the top of the file. The first line of the file is line number 1.
• linenoend: the line number at which the command ends.
• info: a dictionary with possible additional information which is available at parse time. For example, the filter name for ‘filter’ commands is stored when parsing commands.

See also bibolamazifile.configCmds().

Construct a BibolamaziFileCmd with the given cmd, text, lineno, linenoend and info.

exception bibolamazi.core.bibolamazifile.BibolamaziFileParseError(msg, fname=None, lineno=None)[source]¶

Bases: bibolamazi.core.butils.BibolamaziError

bibolamazi.core.bibolamazifile.CONFIG_BEGIN_TAG = '%%%-BIB-OLA-MAZI-BEGIN-%%%'¶

The line which defines the beginning of a config section in a bibolamazi file.

bibolamazi.core.bibolamazifile.CONFIG_END_TAG = '%%%-BIB-OLA-MAZI-END-%%%'¶

The line which defines the end of a config section in a bibolamazi file.

exception bibolamazi.core.bibolamazifile.NotBibolamaziFileError(msg, fname=None, lineno=None)[source]¶

Bases: bibolamazi.core.bibolamazifile.BibolamaziFileParseError

This error is raised to signify that the file specified is not a bibolamazi file—most probably, it does not contain a valid configuration section.

bibolamazi.core.blogger module¶

Set up a logging framework for logging debug, information, warning and error messages.

Modules should get their logger using Python’s standard logging mechanism:

import logging
logger = logging.getLogger(__name__)

This allows for the user to be rather specific about which type of messages she/he would like to see.

class bibolamazi.core.blogger.BibolamaziConsoleFormatter(ttycolors=False, show_pos_info_level=None, **kwargs)[source]¶

Bases: logging.Formatter

Format log messages for console output. Customized for bibolamazi.

format(record)[source]¶

setShowPosInfoLevel(level)[source]¶

class bibolamazi.core.blogger.BibolamaziLogger(name, level=0)[source]¶

Bases: logging.Logger

A Logger used in Bibolamazi.

This logger class knows about an additional log level, LONGDEBUG.

Initialize the logger with a name and an optional level.

getSelfLevel()[source]¶

Returns the level that was set on this logger. If no specific level was set, then returns logging.NOTSET. In this respect, this is NOT the same as getEffectiveLevel().

longdebug(msg, *args, **kwargs)[source]¶

Produce a log message at level LONGDEBUG.

class bibolamazi.core.blogger.ConditionalFormatter(defaultfmt=None, datefmt=None, **kwargs)[source]¶

Bases: logging.Formatter

A formatter class.

Very much like logging.Formatter, except that different formats can be specified for different log levels.

Specify the different formats to the constructor with keyword arguments. E.g.:

ConditionalFormatter('%(message)s',
                     DEBUG='DEBUG: %(message)s',
                     INFO='just some info... %(message)s')

This will use ‘%(message)s’ as format for all messages except with level other thand DEBUG or INFO, for which their respective formats are used.

do_format(record, fmt)[source]¶

format(record)[source]¶

bibolamazi.core.blogger.logger = <bibolamazi.core.blogger.BibolamaziLogger object>¶

(OBSOLETE) The main logger object. This is a logging.Logger object.

Deprecated since version 2.1: This object is still here to keep old code functioning. New code should use the following idiom somewhere at the top of their module:

import logging
logger = logging.getLogger(__name__)

(Just make sure the logging mechanism has been set up correctly already, see doc for blogger module.)

This object has an additional method longdebug() (which behaves similarly to debug()), for logging long debug output such as dumping the database during intermediate steps, etc. This corresponds to bibolamazi command-line verbosity level 3.

bibolamazi.core.blogger.setup_simple_console_logging(logger=<logging.RootLogger object>, stream=<open file '<stderr>', mode 'w'>)[source]¶

Sets up the given logger object for simple console output.

The main program module may for example invoke this function on the root logger to provide a basic logging mechanism.

bibolamazi.core.butils module¶

Various utilities for use within all of the Bibolamazi Project.

exception bibolamazi.core.butils.BibolamaziError(msg, where=None)[source]¶

Bases: exceptions.Exception

Root bibolamazi error exception.

See also BibFilterError and BibUserCacheError.

bibolamazi.core.butils.call_with_args(fn, *args, **kwargs)[source]¶

Utility to call a function fn with *args and **kwargs.

fn(*args) must be an acceptable function call; beyond that, additional keyword arguments which the function accepts will be provided from **kwargs.

This function is meant to be essentially fn(*args, **kwargs), but without raising an error if there are arguments in kwargs which the function doesn’t accept (in which case, those arguments are ignored).

bibolamazi.core.butils.get_version()[source]¶

Return the version string version_str, unchanged.

bibolamazi.core.butils.get_version_split()[source]¶

Return a 4-tuple (maj, min, rel, suffix) resulting from parsing the version obtained via version.version_str.

............ TODO: FIXME: CURRENTLY, the elements are strings! why not integers? If not there, they will/should be empty or None?

bibolamazi.core.butils.getbool(x)[source]¶

Utility to parse a string representing a boolean value.

If x is already of integer or boolean type (actually, anything castable to an integer), then the corresponding boolean convertion is returned. If it is a string-like type, then it is matched against something that looks like ‘t(rue)?’, ‘1’, ‘y(es)?’ or ‘on’ (ignoring case), or against something that looks like ‘f(alse)?’, ‘0’, ‘n(o)?’ or ‘off’ (also ignoring case). Leading or trailing whitespace is ignored. If the string cannot be parsed, a ValueError is raised.

bibolamazi.core.butils.guess_encoding_decode(dat, encoding=None)[source]¶

bibolamazi.core.butils.parse_timedelta(in_s)[source]¶

Note: only positive timedelta accepted.

bibolamazi.core.butils.quotearg(x)[source]¶

bibolamazi.core.butils.resolve_type(typename, in_module=None)[source]¶

Returns a type object corresponding to the given type name typename, given as a string.

..... TODO: MORE DOC .........

bibolamazi.core.butils.warn_deprecated(classname, oldname, newname, modulename=None, explanation=None)[source]¶

bibolamazi.core.main module¶

This module contains the code that implements Bibolamazi’s main functionality. It also provides the basic tools for the command-line interface.

class bibolamazi.core.main.AddFilterPackageAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)[source]¶

Bases: argparse.Action

class bibolamazi.core.main.ArgsStruct(bibolamazifile, use_cache, cache_timeout)¶

Bases: tuple

bibolamazifile¶

Alias for field number 0

cache_timeout¶

Alias for field number 2

use_cache¶

Alias for field number 1

exception bibolamazi.core.main.BibolamaziNoSourceEntriesError[source]¶

Bases: bibolamazi.core.butils.BibolamaziError

bibolamazi.core.main.get_args_parser()[source]¶

bibolamazi.core.main.main(argv=['-T', '-b', 'readthedocssinglehtmllocalmedia', '-D', 'language=en', '.', '_build/localmedia'])[source]¶

bibolamazi.core.main.run_bibolamazi(bibolamazifile, **kwargs)[source]¶

bibolamazi.core.main.run_bibolamazi_args(args)[source]¶

bibolamazi.core.main.setup_filterpackage_from_argstr(argstr)[source]¶

Add a filter package definition and path to filterfactory.filterpath from a string that is a e.g. a command-line argument to –filterpath or a part of the environment variable BIBOLAMAZI_FILTER_PATH.

bibolamazi.core.main.setup_filterpackages_from_env()[source]¶

bibolamazi.core.main.verbosity_logger_level(verbosity)[source]¶

Simple mapping of ‘verbosity level’ (used, for example for command line options) to correspondig logging level (logging.DEBUG, logging.ERROR, etc.).

bibolamazi.core.version module¶

bibolamazi.core.version.version_str = '3.0'¶

The version string. This is increased upon each release.

bibolamazi.filters.util.arxivutil Module¶

class bibolamazi.filters.util.arxivutil.ArxivFetchedAPIInfoCacheAccessor(**kwargs)[source]¶

Bases: bibolamazi.core.bibusercache.BibUserCacheAccessor

A BibUserCacheAccessor for fetching and accessing information retrieved from the arXiv API.

fetchArxivApiInfo(idlist)[source]¶

Populates the given cache with information about the arXiv entries given in idlist. This must be, yes you guessed right, a list of arXiv identifiers that we should fetch.

This function performs a query on the arXiv.org API, using the arxiv2bib library. Please note that you should avoid making rapid fire requests in a row (this should normally not happen anyway thanks to our cache mechanism). However, beware that if we get a 403 Forbidden HTTP answer, we should not continue or else arXiv.org might interpret our requests as a DOS attack. If a 403 Forbidden HTTP answer is received this function raises BibArxivApiFetchError with a meaningful error text.

Only those entries in idlist which are not already in the cache are fetched.

idlist can be any iterable.

getArxivApiInfo(arxivid)[source]¶

Returns a dictionary:

{
  'reference':  <arxiv2bib.Reference>,
  'bibtex': <bibtex string>
}

for the given arXiv id in the cache. If the information is not in the cache, returns None.

Don’t forget to first call fetchArxivApiInfo() to retrieve the information in the first place.

Note the reference part may be a arxiv2bib.ReferenceErrorInfo, if there was an error retreiving the reference.

initialize(cache_obj, **kwargs)[source]¶

class bibolamazi.filters.util.arxivutil.ArxivInfoCacheAccessor(**kwargs)[source]¶

Bases: bibolamazi.core.bibusercache.BibUserCacheAccessor

A BibUserCacheAccessor for fetching and accessing information retrieved from the arXiv API.

complete_cache(bibdata, arxiv_api_accessor)[source]¶

Makes sure the cache is complete for all items in bibdata.

getArXivInfo(entrykey)[source]¶

Get the arXiv information corresponding to entry citekey entrykey. If the entry is not in the cache, returns None. Call complete_cache() first!

initialize(cache_obj, **kwargs)[source]¶

rebuild_cache(bibdata, arxiv_api_accessor)[source]¶

Clear and rebuild the entry cache completely.

revalidate(bibolamazifile)[source]¶

Re-validates the cache (with validate()), and calls again complete_cache() to fetch all missing or out-of-date entries.

exception bibolamazi.filters.util.arxivutil.BibArxivApiFetchError(msg)[source]¶

Bases: bibolamazi.core.bibusercache.BibUserCacheError

bibolamazi.filters.util.arxivutil.detectEntryArXivInfo(entry)[source]¶

Extract arXiv information from a pybtex.database.Entry bibliographic entry.

Returns upon success a dictionary of the form:

{ 'primaryclass': <primary class, if available>,
  'arxivid': <the (minimal) arXiv ID (in format XXXX.XXXX  or  archive/XXXXXXX)>,
  'archiveprefix': value of the 'archiveprefix' field
  'published': True/False <whether this entry was published in a journal other than arxiv>,
  'doi': <DOI of entry if any, otherwise None>
  'year': <Year in preprint arXiv ID number. 4-digit, string type.>
}

Note that ‘published’ is set to True for PhD and Master’s thesis. Also, the arxiv.py filter handles this case separately and explicitly, the option there -dThesesCountAsPublished=0 has no effect here.

If no arXiv information was detected, then this function returns None.

bibolamazi.filters.util.arxivutil.get_arxiv_cache_access(bibolamazifile)[source]¶

bibolamazi.filters.util.arxivutil.setup_and_get_arxiv_accessor(bibolamazifile)[source]¶

bibolamazi.filters.util.arxivutil.stripArXivInfoInNote(notestr)[source]¶

Assumes that notestr is a string in a note={} field of a bibtex entry, and strips any arxiv identifier information found, e.g. of the form ‘arxiv:XXXX.YYYY’ (or similar).

bibolamazi.filters.util.auxfile Module¶

Utilities (actually for now, utility) to parse .aux files from LaTeX documents.

bibolamazi.filters.util.auxfile.get_all_auxfile_citations(jobname, bibolamazifile, filtername, search_dirs=None, callback=None, return_set=True)[source]¶

Get a list of bibtex keys that a specific LaTeX document cites, by inspecting its .aux file.

Look for the file <jobname>.aux in the current directory, or in the search directories search_dirs if given. Parse that file for commands of the type \citation{..}, and collect all the arguments of such commands. These commands are generated by calls to the \cite{} command in the LaTeX document.

This effectively gives a list of entries that a particular document cites.

Note: latex/pdflatex must have run at least once on the document already.

Copyright¶

Copyright (c) 2014 Philippe Faist

Bibolamazi is developed and maintained by Philippe Faist. It is distributed under the GNU General Public License (GPL), Version 3 or higher.

Credits and Third-Party Code¶

This project also contains the following 3rd party code.

PybTeX is used as python library for parsing and writing BibTeX files.

Copyright (c) 2006, 2007, 2008, 2009, 2010, 2011 Andrey Golovizin

Full copyright notice is availble in this repo in the file 3rdparty/pybtex/COPYING.

Arxiv2Bib is a tool for querying the arxiv.org API for preprint details and for parsing its results.

Copyright (c) 2012, Nathan Grigg

Full copyright notice is available in this repo in the first lines of the file 3rdparty/arxiv2bib/arxiv2bib.py.

Contact¶

Please contact me for any bug reports, or if you want to contribute.