3. Revision history

This page summarizes some of the more important changes between releases.

3.1. Current version

3.1.1. v2.4.20

Released: July 27, 2016

  • convert2bed
    • Increased memory allocation for maximum number of per-read CIGAR operations in BAM and SAM conversion to help improve stability. Thanks to Adam Freedman for the report!
    • Improved reliability of gene ID parsing from GTF input, where gene_id field may be positioned at start, middle, or end of attributes string, or may be empty. Thanks to blaiseli for the report!

3.2. Previous versions

3.2.1. v2.4.19

Released: May 9, 2016

  • convert2bed
    • Fixed bug in BAM and SAM parallel conversion scripts (*_gnuParallel and *_sge) with inputs containing chromosome names without chr prefix. Thanks to Eric Haugen for the bug report!
  • Starch C++ API
    • Fixed bug with extraction of bzip2- and gzip-backed archives with all other non-primary Starch tools (all tools except starch, unstarch, starchcat, and sort-bed). Thanks to Eric Haugen for the bug report!

3.2.2. v2.4.18

Released: April 28, 2016

  • convert2bed
    • Fixed compile warnings.
    • Fixed bug in BAM and SAM conversion with optional field line overflow. Thanks to Jemma Nelson for the bug report!
  • General documentation improvements
    • Updated OS X Installer and Github release instructions
    • Added thank-you to Feng Tian for bug report

3.2.3. v2.4.17

Released: April 26, 2016

  • bam2bed and sam2bed
    • Improved parsing of non-split BAM and SAM inputs.
  • Docker container build target added for Debian
    • Thanks to Leo Comitale (Poldo) for writing a Makefile target and spec for creating a BEDOPS Docker container for the Debian target.
  • Starch C++ API
    • Fixed bug with extraction of bzip2- and gzip-backed archives with all other non-primary Starch tools (all tools except starch, unstarch, starchcat, and sort-bed). Thanks to Feng Tian for reports.

3.2.4. v2.4.16

Released: April 5, 2016

  • bedmap
    • Added new --echo-ref-row-id option to report reference row ID elements.
  • Starch C++ API
    • Fixed bug with extraction of archives made with starch --gzip (thanks to Brad Gulko for the bug report and Paul Verhoeven and Peter Weir for compile and testing assistance).
  • General improvements
    • Small improvements to build cleanup targets.

3.2.5. v2.4.15

Released: January 21, 2016

  • Docker container build target added for CentOS 7
    • Thanks to Leo Comitale (Poldo) for writing a Makefile target and spec for creating a BEDOPS Docker container for CentOS 7.
  • convert2bed
    • Fixed buffer overflows in convert2bed to improve conversion reliability for VCF files (thanks to Jared Andrews and Kousik Kundu for bug reports).
  • General improvements
    • Improved OS X 10.11 build process.

3.2.6. v2.4.14

Released: April 21, 2015

  • convert2bed
    • Fixed missing samtools variable references in cluster conversion scripts (thanks to Brad Gulko for the bug report).
  • General suite-wide improvements
    • Fixed exception error message for stdin check (thanks to Brad Gulko for the bug report).

3.2.7. v2.4.13

Released: April 20, 2015

  • bedops
    • Resolved issue in using --ec with bedops when reading from stdin (thanks to Brad Gulko for the bug report).
  • General suite-wide improvements
    • Addressed inconsistency with constants defined for the suite at the extreme end of the limits we allow for coordinate values (thanks again to Brad Gulko for the report).

3.2.8. v2.4.12

Released: March 13, 2015

  • bedops
    • Checks have been added to determine if an integer argument is a file in the current working directory, before interpreting that argument as an overlap criterion for -e and -n options. To reduce ambiguity, if an integer is used as a file input, bedops issues a warning of the interpretation and provides guidance on how to force that value to instead be used as an overlap specification, if desired (thanks to E. Rynes for the pointer).
  • bedmap
    • Added support for --prec/--sci with --min-element and --max-element operations (thanks to E. Rynes for the pointer).
  • bedops | bedmap | closest-features
    • Added support for bash process substitution/named pipes with specification of --chrom and/or --ec options (thanks to B. Gulko for the bug report).
    • Fixed code that extracts gzip-backed Starch archives from bedops and other core tools (thanks again to B. Gulko for the bug report).
  • convert2bed
    • Switched matches and qSize fields in order of psl2bed output. Refer to documentation for new field order.
    • Added null sentinel to GTF ID value.
    • To help reduce the chance of buffer overflows, the convert2bed tool increases the maximum field length from 8191 to 24575 characters to allow parsing of inputs with longer field length, such as very long attributes from mosquito GFF3 data (thanks to T. Karginov for the bug report).

3.2.9. v2.4.11

Released: February 24, 2015

  • convert2bed
    • Fixed bug in psl2bed where matches column value was truncated by one character. Updated unit tests. Thanks to M. Wirthlin for the bug report.

3.2.10. v2.4.10

Released: February 23, 2015

  • starch
    • In addition to checking chromosome interleaving, the starch tool now enforces sort-bed sort ordering on BED input and exits with an EINVAL error if the data are not sorted correctly.
  • convert2bed
    • Added --zero-indexed option to wig2bed and wig2starch wrappers and convert2bed binary, which converts WIG data that are zero-indexed without any coordinate adjustments. This is useful for WIG data sourced from the UCSC Kent tool bigWigToWig, where the bigWig data can potentially be sourced from 0-indexed BAM- or bedGraph-formatted data.
    • If the WIG input contains any element with a start coordinate of 0, the default use of wig2bed, wig2starch and convert2bed will exit early with an error condition, suggesting the use of --zero-indexed.
    • Updated copyright date range of wrapper scripts

3.2.11. v2.4.9

Released: February 17, 2015

  • sort-bed
    • Added support for --check-sort to report if input is sorted (or not)
  • Starch
    • Improved support for starch --header, where header contains tab-delimited fields
  • Starch C++ API
    • Fixed bug with starch --header functionality, such that BEDOPS core tools (bedops, etc.) would be unable to extract correct data from headered Starch archive

3.2.12. v2.4.8

Released: February 7, 2015

  • Mac OS X packaging
  • Linux packaging
    • SHA1 hashes of each tarball are now part of the BEDOPS Releases description page, going forwards
  • Updated copyright dates in source code

3.2.13. v2.4.7

Released: February 2, 2015

  • convert2bed fixes and improvements
    • Fixed –split support in psl2bed (thanks to Marco A.)
    • Fixed compilation warning regarding comparison of signed and unsigned values
    • Fixed corrupted psl2bed test inputs

3.2.14. v2.4.6

Released: January 30, 2015

  • convert2bed fixes and improvements
    • Added support for conversion of the GVF file format, including wrapper scripts and unit tests. Refer to the gvf2bed documentation for more information.
    • Fixed bug in string copy of zero-length element attribute for gff2bed and gtf2bed (GFF and GTF) formats
  • General fixes and improvements
    • Fixed possibly corrupt bzip2, Jansson and zlib tarballs (thanks to rekado, Shane N. and Richard S.)
    • Fixed typo in bedextract documentation
    • Fixed broken image in Overview
    • Removed 19 MB _build intermediate result directory (which should improve overall git clone time considerably!)

3.2.15. v2.4.5

Released: January 28, 2015

  • convert2bed improvements
    • Addition of RepeatMasker annotation output (.out) file conversion support, rmsk2bed and rmsk2starch wrappers, and unit tests

3.2.16. v2.4.4

Released: January 25, 2015

  • Documentation improvements
    • Implemented substantial style changes via A Better Sphinx Theme and various customizations. We also include responsive web style elements to help improve browsing on mobile devices.
    • Fixes to typos in conversion and other documents.

3.2.17. v2.4.3

Released: December 18, 2014

  • Compilation improvements
    • Shane Neph put in a great deal of work to enable parallel builds (e.g., make -j N to build various targets in parallel). Depending on the end user’s environment, this can speed up compilation time by a factor of 2, 4 or more.
    • Fixed numerous compilation warnings of debug builds of starch toolkit under RHEL6/GCC and OS X 10.10.1/LLVM.
  • New bedops features
    • Added --chop and --stagger options to “melt” inputs into contiguous or staggered disjoint regions of equivalent size.
    • For less confusion, arguments for --element-of, --chop and other bedops operations that take numerical modifiers no longer require a leading hyphen character. For instance, --element-of 1 is now equivalent to the former usage of --element-of -1.
  • New bedmap features
    • The --sweep-all option reads through the entire map file without early termination and can help deal with SIGPIPE errors. It adds to execution time, but the penalty is not as severe as with the use of --ec. Using --ec alone will enable error checking, but will now no longer read through the entire map file. The --ec option can be used in conjunction with --sweep-all, with the associated time penalties. (Another method for dealing with issue this is to override how SIGPIPE errors are caught by the interpreter (bash, Python, etc.) and retrapping them or ignoring them. However, it may not a good idea to do this as other situations may arise in production pipelines where it is ideal to trap and handle all I/O errors in a default manner.)
    • New --echo-ref-size and --echo-ref-name operations report genomic length of reference element, and rename the reference element in chrom:start-end (useful for labeling rows for input for matrix2png or R or other applications).
  • bedextract
    • Fixed upper bound bug that would cause incorrect output in some cases
  • conversion scripts
    • Brand new C99 binary called convert2bed, which wrapper scripts (bam2bed, etc.) now call. No more Python version dependencies, and the C-based rewrite offers massive performance improvements over old Python-based scripts.
    • Added Parallel bam2starch script, which parallelizes creation of Starch archive from very large BAM files in SGE environments.
    • Added bug fix for missing code in starchcluster.gnu_parallel script, where the final collation step was missing.
    • The vcf2bed script now accepts the --do-not-split option, which prints one BED element for all alternate alleles.
  • Starch archival format and compression/extraction tools
    • Added duplicate- and nested-element flags in v2.1 of Starch metadata, which denote if a chromosome contains one or more duplicate and/or nested elements. BED files compressed with starch v2.5 or greater, or Starch archives updated with starchcat v2.5 or greater will include these values in the archive metadata. The unstarch extraction tool offers --has-duplicate and --has-nested options to retrieve these flag values for a specified chromosome (or for all chromosomes).
    • Added --is-starch option to unstarch to test if specified input file is a Starch v1 or v2 archive.
    • Added bug fix for compressing BED files with starch, where the archive would not include the last element of the BED input, if the BED input lacked a trailing newline. The compression tools now include a routine for capturing the last line, if there is no newline.
  • Documentation improvements
    • Remade some image assets throughout the documents to support Retina-grade displays

3.2.18. v2.4.2

Released: April 10, 2014

  • conversion scripts
    • Added support for sort-bed --tmpdir option to conversion scripts, to allow specification of alternative temporary directory for sorted results when used in conjunction with --max-mem option.
    • Added support for GFF3 files which include a FASTA directive in gff2bed and gff2starch (thanks to Keith Hughitt).
    • Extended support for Python-based conversion scripts to support use with Python v2.6.2 and forwards, except for sam2bed and sam2starch, which still require Python v2.7 or greater (and under Python3).
    • Fixed --insertions option in vcf2bed to now report a single-base BED element (thanks to Matt Maurano).

3.2.19. v2.4.1

Released: February 26, 2014

  • bedmap

    • Added --fraction-both and --exact (--fraction-both 1) to list of compatible overlap options with --faster.
    • Added 5% performance improvement with bedmap operations without --faster.
    • Fixed scenario that can yield incorrect results (cf. Issue 43).
  • sort-bed

    • Added --tmpdir option to allow specification of an alternative temporary directory, when used in conjunction with --max-mem option. This is useful if the host operating system’s standard temporary directory (e.g., /tmp on Linux or OS X) does not have sufficient space to hold intermediate results.
  • All conversion scripts

    • Improvements to error handling in Python-based conversion scripts, in the case where no input is specified.
    • Fixed typos in gff2bed and psl2bed documentation (cf. commit a091e18).
  • OS X compilation improvements

    • We have completed changes to the OS X build process for the remaining half of the BEDOPS binaries, which now allows direct, full compilation with Clang/LLVM (part of the Apple Xcode distribution).

      All OS X BEDOPS binaries now use Apple’s system-level C++ library, instead of GNU’s libstdc++. It is no longer required (or recommended) to use GNU gcc to compile BEDOPS on OS X.

      Compilation is faster and simpler, and we can reduce the size and complexity of Mac OS X builds and installer packages. By using Apple’s C++ library, we also eliminate the likelihood of missing library errors.

      In the longer term, this gets us closer to moving BEDOPS to using the CMake build system, to further abstract and simplify the build process.

  • Cleaned up various compilation warnings found with clang/clang++ and GCC kits.

3.2.20. v2.4.0

Released: January 9, 2014

  • bedmap
    • Added new --echo-map-size and --echo-overlap-size options to calculate sizes of mapped elements and overlaps between mapped and reference elements.
    • Improved performance for all --echo-map-* operations.
    • Updated documentation.
  • Major enhancements and fixes to sort-bed:
    • Improved performance.
    • Fixed memory leak.
    • Added support for millions of distinct chromosomes.
    • Improved internal estimation of memory usage with --max-mem option.
  • Added support for compilation on Cygwin (64-bit). Refer to the installation documentation for build instructions.
  • starchcat
    • Fixed embarassing buffer overflow condition that caused segmentation faults on Ubuntu 13.
  • All conversion scripts
    • Python-based scripts no longer use temporary files, which reduces file I/O and improves performance. This change also reduces the need for large amounts of free space in a user’s /tmp folder, particularly relevant for users converting multi-GB BAM files.
    • We now test for ability to locate starch, sort-bed, wig2bed_bin and samtools in user environment, quitting with the appropriate error state if the dependencies cannot be found.
    • Improved documentation. In particular, we have added descriptive tables to each script’s documentation page which describe how columns map from original data input to BED output.
    • bam2bed and sam2bed
      • Added --custom-tags <value> command-line option to support a comma-separated list of custom tags (cf. Biostars discussion), i.e., tags which are not part of the original SAMtools specification.
      • Added --keep-header option to preserve header and metadata as BED elements that use _header as the chromosome name. This now makes these conversion scripts fully “non-lossy”.
    • vcf2bed
      • Added new --snvs, --insertions and --deletions options that filter VCF variants into three separate subcategories.
      • Added --keep-header option to preserve header and metadata as BED elements that use _header as the chromosome name. This now makes these conversion scripts fully “non-lossy”.
    • gff2bed
      • Added --keep-header option to preserve header and metadata as BED elements that use _header as the chromosome name. This now makes these conversion scripts fully “non-lossy”.
    • psl2bed
      • Added --keep-header option to preserve header and metadata as BED elements that use _header as the chromosome name. This now makes these conversion scripts fully “non-lossy”.
    • wig2bed
      • Added --keep-header option to wig2bed binary and wig2bed/wig2starch wrapper scripts, to preserve header and metadata as BED elements that use _header as the chromosome name. This now makes these conversion scripts fully “non-lossy”.
  • Added OS X uninstaller project to allow end user to more easily remove BEDOPS tools from this platform.
  • Cleaned up various compilation warnings found with clang/clang++ and GCC kits.

3.2.21. v2.3.0

Released: October 2, 2013

  • Migration of BEDOPS code and documentation from Google Code to Github.

    • Due to changes with Google Code hosting policies at the end of the year, we have decided to change our process for distributing code, packages and documentation. While most of the work is done, we appreciate feedback on any problems you may encounter. Please email us at bedops@stamlab.org with details.
    • Migration to Github should facilitate requests for code by those who are familiar with git and want to fork our project to submit pull requests.
  • bedops

    • General --ec performance improvements.
  • bedmap

    • Adds support for the new --skip-unmapped option, which filters out reference elements which do not have mapped elements associated with them. See the end of the score operations section of the bedmap documentation for more detail.
    • General --ec performance improvements.
  • starch

    • Fixed bug with starch where zero-byte BED input (i.e., an “empty set”) created a truncated and unusable archive. We now put in a “dummy” chromosome for zero-byte input, which unstarch can now unpack.

      This should simplify error handling with certain pipelines, specifically where set or other BEDOPS operations yield an “empty set” BED file that is subsequently compressed with starch.

  • unstarch

    • Can now unpack zero-byte (“empty set”) compressed starch archive (see above).
    • Changed unstarch --list option to print to stdout stream (this was previously sent to stderr).
  • starch metadata library

  • All conversion scripts

    • Python scripts require v2.7+ or greater.
    • Improved (more “Pythonic”) error code handling.
    • Disabled support for --max-mem sort parameter until sort-bed issue is resolved. Scripts will continue to sort, but they will be limited to available system memory. If you are processing files larger than system memory, please contact us at bedops@stamlab.org for details of a temporary workaround.
  • gff2bed conversion script

    • Resolved IndexError exceptions by fixing header support, bringing script in line with v1.21 GFF3 spec.
  • bam2bed and sam2bed conversion scripts

    • Rewritten bam2* and sam2* scripts from bash into Python (v2.7+ support).
    • Improved BAM and SAM input validation against the v1.4 SAM spec.
    • New --split option prints reads with N CIGAR operations as separated BED elements.
    • New --all-reads option prints all reads, mapped and unmapped.
  • bedextract

  • New documentation via readthedocs.org.

    • Documentation is now part of the BEDOPS distribution, instead of being a separate download.
    • We use readthedocs.org to host indexed and searchable HTML.
    • PDF and eBook documents are also available for download.
    • Documentation is refreshed and simplified, with new installation and compilation guides.
  • OS X compilation improvements

    • We have made changes to the OS X build process for half of the BEDOPS binaries, which allows direct compilation with Clang/LLVM (part of the Apple Xcode distribution). Those binaries now use Apple’s system-level C++ library, instead of GNU’s libstdc++.

      This change means that we require Mac OS X 10.7 (“Lion”) or greater—we do not support 10.6 at this time.

      Compilation is faster and simpler, and we can reduce the size and complexity of Mac OS X builds and installer packages. By using Apple’s C++ library, we also reduce the likelihood of missing library errors. When this process is completed for the remaining binaries, it will no longer be necessary to install GCC 4.7+ (by way of MacPorts or other package managers) in order to build BEDOPS on OS X, nor will we have to bundle libstdc++ with the installer.

3.2.22. v2.2.0b

  • Fixed bug with OS X installer’s post-installation scripts.

3.2.23. v2.2.0

Released: May 22, 2013

  • Updated packages
    • Precompiled packages are now available for Linux (32- and 64-bit) and Mac OS X 10.6-10.8 (32- and 64-bit) hosts.
  • Starch v2 test suite
    • We have added a test suite for the Starch archive toolkit with the source download. Test inputs include randomized BED data generated from chromosome and bounds data stored on UCSC servers as well as static FIMO search results. Tests put starch, unstarch and starchcat through various usage scenarios. Please refer to the Starch-specific Makefiles and the test target and subfolder’s README doc for more information.
  • starchcat
    • Resolves bug with --gzip option, allowing updates of gzip -backed v1.2 and v1.5 archives to the v2 Starch format (either bzip2 - or gzip -backed).
  • unstarch
    • Resolves bug with extraction of Starch archive made from BED files with four or more columns. A condition where the total length of additional columns exceeds a certain number of characters would result in extracted data in those columns being cut off. As an example, this could affect Starch archives made from the raw, uncut output of GTF- and GFF- conversion scripts.
  • conversion scripts
    • We have partially reverted wig2bed, providing a Bash shell wrapper to the original C binary. This preserves consistency of command-line options across the conversion suite, while making use of the C binary to recover performance lost from the Python-based v2.1 revision of wig2bed (which at this time is no longer supported). (Thanks to Matt Maurano for reporting this issue.)

3.2.24. v2.1.1

Released: May 3, 2013

  • bedmap
    • Major performance improvements made in v2.1.1, such that current bedmap now operates as fast or faster than the v1.2.5 version of bedmap!
  • bedops
    • Resolves bug with --partition option.
  • conversion scripts
    • All v2.1.0 Python-based scripts now include fix for SIGPIPE handling, such that use of head or other common UNIX utilities to process buffered standard output no longer yields IOError exceptions. (Thanks to Matt Maurano for reporting this bug.)
  • 32-bit Linux binary support
    • Pre-built Linux binaries are now available for end users with 32-bit workstations.

Other issues fixed:

  • Jansson tarball no longer includes already-compiled libraries that could potentially interfere with 32-bit builds.
  • Minor changes to conversion script test suite to exit with useful error code on successful completion of test.

3.2.25. v2.1.0

Released: April 22, 2013

  • bedops
    • New --partition operator efficiently generates disjoint segments made from genomic boundaries of all overlapping inputs.
  • conversion scripts
    • All scripts now use sort-bed behind the scenes to output sorted BED output, ready for use with BEDOPS utilities. It is no longer necessary to pipe data to or otherwise post-process converted data with sort-bed.
    • New psl2bed conversion script, converting PSL-formatted UCSC BLAT output to BED.
    • New wig2bed conversion script written in Python.
    • New *2starch conversion scripts offered for all *2bed scripts, which output Starch v2 archives.
  • closest-features
    • Replaced --shortest option name with --closest, for clarity. (Old scripts which use --shortest will continue to work with the deprecated option name for now. We advise editing pipelines, as needed.)
  • starch
    • Improved error checking for interleaved records. This also makes use of *2starch conversion scripts with the --do-not-sort option safer.
  • Improved Mac OS X support
    • New Mac OS X package installer makes installation of BEDOPS binaries and scripts very easy for OS X 10.6 - 10.8 hosts.
    • Installer resolves fatal library errors seen by some end users of older OS X BEDOPS releases.

3.2.26. v2.0.0b

Released: February 19, 2013

3.2.27. v2.0.0a

Released: February 7, 2013

  • bedmap
    • Takes in Starch-formatted archives as input, as well as raw BED (i.e., it is no longer required to extract a Starch archive to an intermediate, temporary file or named pipe before applying operations).
    • New --chrom operator jumps to and operates on information for specified chromosome only.
    • New --echo-map-id-uniq operator lists unique IDs from overlapping mapping elements.
    • New --max-element and --min-element operators return the highest or lowest scoring overlapping map element.
  • bedops
    • Takes in Starch-formatted archives as input, as well as raw BED.
    • New --chrom operator jumps to and operates on information for specified chromosome only.
  • closest-features
    • Takes in Starch-formatted archives as input, as well as raw BED.
    • New --chrom operator jumps to and operates on information for specified chromosome only.
  • sort-bed and bbms
    • New --max-mem option to limit system memory on large BED inputs.
    • Incorporated bbms functionality into sort-bed with use of --max-mem operator.
  • starch, starchcat and unstarch
    • New metadata enhancements to Starch-format archival and extraction, including: --note, --elements, --bases, --bases-uniq, --list-chromosomes, --archive-timestamp, --archive-type and --archive-version (see --help to starch, starchcat and unstarch binaries, or view the documentation for these applications for more detail).
    • Adds 20-35% performance boost to creating Starch archives with starch utility.
    • New documentation with technical overview of the Starch format specification.
  • conversion scripts
    • New gtf2bed conversion script, converting GTF (v2.2) to BED.
  • Scripts are now part of main download; it is no longer necessary to download the BEDOPS companion separately.

3.2.28. v1.2.5b

Released: January 14, 2013

  • Adds support for Apple 32- and 64-bit Intel hardware running OS X 10.5 through 10.8.
  • Adds README for companion download.
  • Removes some obsolete code.

3.2.29. v1.2.5

Released: October 13, 2012

  • Fixed unusual bug with unstarch, where an extra (and incorrect) line of BED data can potentially be extracted from an archive.
  • Updated companion download with updated bam2bed and sam2bed conversion scripts to address 0-indexing error with previous revisions.

3.2.30. v1.2.3

Released: August 17, 2012

  • Added --indicator option to bedmap.
  • Assorted changes to conversion scripts and associated companion download.