Commit 981e22cf authored by Hal Finkel's avatar Hal Finkel
Browse files

Upgrade to latest blosc library

blosc git: e394f327ccc78319d90a06af0b88bce07034b8dd
parent cda87e9f
......@@ -63,10 +63,12 @@ $(FEDIR)/%.o: thirdparty/blosc/%.c | $(FEDIR)
$(FEDIR)/%.o: %.cxx | $(FEDIR)
$(CXX) $(FE_CFLAGS) $(FE_CPPFLAGS) -c -o $@ $<
$(FEDIR)/GenericIOPrint: $(FEDIR)/GenericIOPrint.o $(FEDIR)/GenericIO.o $(FEDIR)/blosc.o $(FEDIR)/blosclz.o $(FEDIR)/shuffle.o
FE_BLOSC_O := $(FEDIR)/blosc.o $(FEDIR)/blosclz.o $(FEDIR)/shuffle.o $(FEDIR)/bitshuffle-generic.o $(FEDIR)/shuffle-generic.o
$(FEDIR)/GenericIOPrint: $(FEDIR)/GenericIOPrint.o $(FEDIR)/GenericIO.o $(FE_BLOSC_O)
$(CXX) $(FE_CFLAGS) -o $@ $^
$(FEDIR)/GenericIOVerify: $(FEDIR)/GenericIOVerify.o $(FEDIR)/GenericIO.o $(FEDIR)/blosc.o $(FEDIR)/blosclz.o $(FEDIR)/shuffle.o
$(FEDIR)/GenericIOVerify: $(FEDIR)/GenericIOVerify.o $(FEDIR)/GenericIO.o $(FE_BLOSC_O)
$(CXX) $(FE_CFLAGS) -o $@ $^
FE_UNAME := $(shell uname -s)
......@@ -98,16 +100,18 @@ $(MPIDIR)/%.o: thirdparty/blosc/%.c | $(MPIDIR)
$(MPIDIR)/%.o: %.cxx | $(MPIDIR)
$(MPICXX) $(MPI_CFLAGS) $(MPI_CPPFLAGS) -c -o $@ $<
$(MPIDIR)/GenericIOPrint: $(MPIDIR)/GenericIOPrint.o $(MPIDIR)/GenericIO.o $(MPIDIR)/blosc.o $(MPIDIR)/blosclz.o $(MPIDIR)/shuffle.o
MPI_BLOSC_O := $(MPIDIR)/blosc.o $(MPIDIR)/blosclz.o $(MPIDIR)/shuffle.o $(MPIDIR)/bitshuffle-generic.o $(MPIDIR)/shuffle-generic.o
$(MPIDIR)/GenericIOPrint: $(MPIDIR)/GenericIOPrint.o $(MPIDIR)/GenericIO.o $(MPI_BLOSC_O)
$(MPICXX) $(MPI_CFLAGS) -o $@ $^
$(MPIDIR)/GenericIOVerify: $(MPIDIR)/GenericIOVerify.o $(MPIDIR)/GenericIO.o $(MPIDIR)/blosc.o $(MPIDIR)/blosclz.o $(MPIDIR)/shuffle.o
$(MPIDIR)/GenericIOVerify: $(MPIDIR)/GenericIOVerify.o $(MPIDIR)/GenericIO.o $(MPI_BLOSC_O)
$(MPICXX) $(MPI_CFLAGS) -o $@ $^
$(MPIDIR)/GenericIOBenchmarkRead: $(MPIDIR)/GenericIOBenchmarkRead.o $(MPIDIR)/GenericIO.o $(MPIDIR)/blosc.o $(MPIDIR)/blosclz.o $(MPIDIR)/shuffle.o
$(MPIDIR)/GenericIOBenchmarkRead: $(MPIDIR)/GenericIOBenchmarkRead.o $(MPIDIR)/GenericIO.o $(MPI_BLOSC_O)
$(MPICXX) $(MPI_CFLAGS) -o $@ $^
$(MPIDIR)/GenericIOBenchmarkWrite: $(MPIDIR)/GenericIOBenchmarkWrite.o $(MPIDIR)/GenericIO.o $(MPIDIR)/blosc.o $(MPIDIR)/blosclz.o $(MPIDIR)/shuffle.o
$(MPIDIR)/GenericIOBenchmarkWrite: $(MPIDIR)/GenericIOBenchmarkWrite.o $(MPIDIR)/GenericIO.o $(MPI_BLOSC_O)
$(MPICXX) $(MPI_CFLAGS) -o $@ $^
frontend-progs: $(FEDIR)/GenericIOPrint $(FEDIR)/GenericIOVerify
......
===============================================================
Announcing Blosc 1.2.3
A blocking, shuffling and lossless compression library
Announcing c-blosc 1.10.0
A blocking, shuffling and lossless compression library for C
===============================================================
What is new?
============
New `blosc_init()` and `blosc_destroy()` functions have been added so
that the global lock can be initialized safely. These new functions
will also allow for other kind of initializations/destructions in the
future.
This release introduces support for the new Zstd codec. Zstd is meant to
achieve larger compression ratios than Zlib, but with higher speeds. We
are talking about a well-balanced codec that should see a lot of use
among Blosc users. There is a blog about what you can expect of it in:
Existing applications using Blosc do not need to start using the new
functions right away, as long as they calling `blosc_set_nthreads()`
previous to anything else. However, using them is highly recommended.
Thanks to Oscar Villellas for the init/destroy suggestion, it is a
nice idea indeed!
http://blosc.org/blog/zstd-has-just-landed-in-blosc.html
For more info, please see the release notes in:
https://github.com/FrancescAlted/blosc/wiki/Release-notes
https://github.com/Blosc/c-blosc/blob/master/RELEASE_NOTES.rst
What is it?
===========
Blosc (http://www.blosc.org) is a high performance compressor
Blosc (http://www.blosc.org) is a high performance meta-compressor
optimized for binary data. It has been designed to transmit data to
the processor cache faster than the traditional, non-compressed,
direct memory fetch approach via a memcpy() OS call.
Blosc is the first compressor (that I'm aware of) that is meant not
only to reduce the size of large datasets on-disk or in-memory, but
also to accelerate object manipulations that are memory-bound.
Blosc has internal support for different compressors like its internal
BloscLZ, but also LZ4, LZ4HC, Snappy and Zlib. This way these can
automatically leverage the multithreading and pre-filtering
(shuffling) capabilities that comes with Blosc.
There is also a handy command line for Blosc called Bloscpack
(https://github.com/esc/bloscpack) that allows you to compress large
binary datafiles on-disk. Although the format for Bloscpack has not
stabilized yet, it allows you to effectively use Blosc from you
favorite shell.
Download sources
================
......@@ -49,11 +41,12 @@ http://www.blosc.org/
and proceed from there. The github repository is over here:
https://github.com/FrancescAlted/blosc
https://github.com/Blosc
Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for
details.
Mailing list
============
......@@ -64,10 +57,3 @@ http://groups.google.es/group/blosc
Enjoy Data!
.. Local Variables:
.. mode: rst
.. coding: utf-8
.. fill-column: 70
.. End:
Bitshuffle - Filter for improving compression of typed binary data.
Copyright (c) 2014 Kiyoshi Masui (kiyo@physics.ubc.ca)
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Blosc - A blocking, shuffling and lossless compression library
Copyright (C) 2009-2012 Francesc Alted <faltet@gmail.com>
Copyright (C) 2013 Francesc Alted <faltet@gmail.com>
Copyright (C) 2009-2016 Francesc Alted <francesc@blosc.org>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......@@ -20,4 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
LZ4 - Fast LZ compression algorithm
Copyright (C) 2011-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- LZ4 source repository : http://code.google.com/p/lz4/
Copyright 2011, Google Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Copyright (c) 2006-2008 Alexander Chemeris
ISO C9x compliant stdint.h for Microsoft Visual Studio
Based on ISO/IEC 9899:TC2 Committee draft (May 6, 2005) WG14/N1124
Copyright (c) 2006-2013 Alexander Chemeris
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
......@@ -10,8 +13,9 @@ modification, are permitted provided that the following conditions are met:
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. The name of the author may be used to endorse or promote products
derived from this software without specific prior written permission.
3. Neither the name of the product nor the names of its contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
......@@ -22,4 +26,4 @@ PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
\ No newline at end of file
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Copyright notice:
(C) 1995-2013 Jean-loup Gailly and Mark Adler
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu
......@@ -3,8 +3,22 @@
===============================================================
:Author: Francesc Alted
:Contact: faltet@gmail.com
:Contact: francesc@blosc.org
:URL: http://www.blosc.org
:Gitter: |gitter|
:Travis CI: |travis|
:Appveyor: |appveyor|
.. |gitter| image:: https://badges.gitter.im/Blosc/c-blosc.svg
:alt: Join the chat at https://gitter.im/Blosc/c-blosc
:target: https://gitter.im/Blosc/c-blosc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |travis| image:: https://travis-ci.org/Blosc/c-blosc.svg?branch=master
:target: https://travis-ci.org/Blosc/c-blosc
.. |appveyor| image:: https://ci.appveyor.com/api/projects/status/3mlyjc1ak0lbkmte?svg=true
:target: https://ci.appveyor.com/project/FrancescAlted/c-blosc/branch/master
What is it?
===========
......@@ -17,41 +31,57 @@ that is meant not only to reduce the size of large datasets on-disk or
in-memory, but also to accelerate memory-bound computations.
It uses the blocking technique (as described in [2]_) to reduce
activity on the memory bus as much as possible. In short, this
activity on the memory bus as much as possible. In short, this
technique works by dividing datasets in blocks that are small enough
to fit in caches of modern processors and perform compression /
decompression there. It also leverages, if available, SIMD
instructions (SSE2) and multi-threading capabilities of CPUs, in order
to accelerate the compression / decompression process to a maximum.
You can see some recent benchmarks about Blosc performance in [3]_
instructions (SSE2, AVX2) and multi-threading capabilities of CPUs, in
order to accelerate the compression / decompression process to a
maximum.
Blosc is actually a metacompressor, that meaning that it can use a range
of compression libraries for performing the actual
compression/decompression. Right now, it comes with integrated support
for BloscLZ (the original one), LZ4, LZ4HC, Snappy, Zlib and Zstd. Blosc
comes with full sources for all compressors, so in case it does not find
the libraries installed in your system, it will compile from the
included sources and they will be integrated into the Blosc library
anyway. That means that you can trust in having all supported
compressors integrated in Blosc in all supported platforms.
You can see some benchmarks about Blosc performance in [3]_
Blosc is distributed using the MIT license, see LICENSES/BLOSC.txt for
details.
.. [1] http://www.blosc.org
.. [2] http://blosc.org/docs/StarvingCPUs-CISE-2010.pdf
.. [3] http://blosc.org/trac/wiki/SyntheticBenchmarks
.. [3] http://blosc.org/synthetic-benchmarks.html
Meta-compression and other advantages over existing compressors
===============================================================
Blosc is not like other compressors: it should rather be called a
C-Blosc is not like other compressors: it should rather be called a
meta-compressor. This is so because it can use different compressors
and pre-conditioners (programs that generally improve compression
ratio). At any rate, it can also be called a compressor because it
happens that it already integrates one compressor and one
pre-conditioner, so it can actually work like so.
Currently it uses BloscLZ, a compressor heavily based on FastLZ
(http://fastlz.org/), and a highly optimized (it can use SSE2
instructions, if available) Shuffle pre-conditioner. However,
different compressors or pre-conditioners may be added in the future.
Blosc is in charge of coordinating the compressor and pre-conditioners
so that they can leverage the blocking technique (described above) as
well as multi-threaded execution (if several cores are available)
automatically. That makes that every compressor and pre-conditioner
and filters (programs that generally improve compression ratio). At
any rate, it can also be called a compressor because it happens that
it already comes with several compressor and filters, so it can
actually work like so.
Currently C-Blosc comes with support of BloscLZ, a compressor heavily
based on FastLZ (http://fastlz.org/), LZ4 and LZ4HC
(https://github.com/Cyan4973/lz4), Snappy
(https://github.com/google/snappy) and Zlib (http://www.zlib.net/), as
well as a highly optimized (it can use SSE2 or AVX2 instructions, if
available) shuffle and bitshuffle filters (for info on how and why
shuffling works, see slide 17 of
http://www.slideshare.net/PyData/blosc-py-data-2014). However,
different compressors or filters may be added in the future.
C-Blosc is in charge of coordinating the different compressor and
filters so that they can leverage the blocking technique (described
above) as well as multi-threaded execution (if several cores are
available) automatically. That makes that every compressor and filter
will work at very high speeds, even if it was not initially designed
for doing blocking or multi-threading.
......@@ -59,85 +89,104 @@ Other advantages of Blosc are:
* Meant for binary data: can take advantage of the type size
meta-information for improved compression ratio (using the
integrated shuffle pre-conditioner).
* Small overhead on non-compressible data: only a maximum of 16
additional bytes over the source buffer length are needed to
compress *every* input.
integrated shuffle and bitshuffle filters).
* Maximum destination length: contrarily to many other
compressors, both compression and decompression routines have
support for maximum size lengths for the destination buffer.
* Small overhead on non-compressible data: only a maximum of (16 + 4 *
nthreads) additional bytes over the source buffer length are needed
to compress *any kind of input*.
* Replacement for memcpy(): it supports a 0 compression level that
does not compress at all and only adds 16 bytes of overhead. In
this mode Blosc can copy memory usually faster than a plain
memcpy().
* Maximum destination length: contrarily to many other compressors,
both compression and decompression routines have support for maximum
size lengths for the destination buffer.
When taken together, all these features set Blosc apart from other
similar solutions.
Compiling your application with Blosc
=====================================
Compiling your application with a minimalistic Blosc
====================================================
Blosc consists of the next files (in blosc/ directory)::
The minimal Blosc consists of the next files (in `blosc/ directory
<https://github.com/Blosc/c-blosc/tree/master/blosc>`_)::
blosc.h and blosc.c -- the main routines
blosclz.h and blosclz.c -- the actual compressor
shuffle.h and shuffle.c -- the shuffle code
blosc.h and blosc.c -- the main routines
shuffle*.h and shuffle*.c -- the shuffle code
blosclz.h and blosclz.c -- the blosclz compressor
Just add these files to your project in order to use Blosc. For
information on compression and decompression routines, see blosc.h.
information on compression and decompression routines, see `blosc.h
<https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h>`_.
To compile using GCC (4.4 or higher recommended) on Unix:
To compile using GCC (4.9 or higher recommended) on Unix:
.. code-block:: console
$ gcc -O3 -msse2 -o myprog myprog.c blosc/*.c -lpthread
$ gcc -O3 -mavx2 -o myprog myprog.c blosc/*.c -Iblosc -lpthread
Using Windows and MINGW:
.. code-block:: console
$ gcc -O3 -msse2 -o myprog myprog.c blosc\*.c
$ gcc -O3 -mavx2 -o myprog myprog.c -Iblosc blosc\*.c
Using Windows and MSVC (2008 or higher recommended):
Using Windows and MSVC (2013 or higher recommended):
.. code-block:: console
$ cl /Ox /Femyprog.exe myprog.c blosc\*.c
$ cl /Ox /Femyprog.exe /Iblosc myprog.c blosc\*.c
A simple usage example is the benchmark in the bench/bench.c file.
Also, another example for using Blosc as a generic HDF5 filter is in
the hdf5/ directory.
In the `examples/ directory
<https://github.com/Blosc/c-blosc/tree/master/examples>`_ you can find
more hints on how to link your app with Blosc.
I have not tried to compile this with compilers other than GCC, MINGW,
Intel ICC or MSVC yet. Please report your experiences with your own
platforms.
I have not tried to compile this with compilers other than GCC, clang,
MINGW, Intel ICC or MSVC yet. Please report your experiences with your
own platforms.
Adding support for other compressors with a minimalistic Blosc
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Testing Blosc
=============
The official cmake files (see below) for Blosc try hard to include
support for LZ4, LZ4HC, Snappy, Zlib inside the Blosc library, so
using them is just a matter of calling the appropriate
`blosc_set_compressor() API call
<https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h>`_. See
an `example here
<https://github.com/Blosc/c-blosc/blob/master/examples/many_compressors.c>`_.
Go to the test/ directory and issue:
Having said this, it is also easy to use a minimalistic Blosc and just
add the symbols HAVE_LZ4 (will include both LZ4 and LZ4HC),
HAVE_SNAPPY and HAVE_ZLIB during compilation as well as the
appropriate libraries. For example, for compiling with minimalistic
Blosc but with added Zlib support do:
.. code-block:: console
$ make test
$ gcc -O3 -msse2 -o myprog myprog.c blosc/*.c -Iblosc -lpthread -DHAVE_ZLIB -lz
These tests are very basic, and only valid for platforms where GNU
make/gcc tools are available. If you really want to test Blosc the
hard way, look at:
In the `bench/ directory
<https://github.com/Blosc/c-blosc/tree/master/bench>`_ there a couple
of Makefile files (one for UNIX and the other for MinGW) with more
complete building examples, like switching between libraries or
internal sources for the compressors.
http://blosc.org/trac/wiki/SyntheticBenchmarks
Supported platforms
~~~~~~~~~~~~~~~~~~~
where instructions on how to intensively test (and benchmark) Blosc
are given. If while running these tests you get some error, please
report it back!
Blosc is meant to support all platforms where a C89 compliant C
compiler can be found. The ones that are mostly tested are Intel
(Linux, Mac OSX and Windows) and ARM (Linux), but exotic ones as IBM
Blue Gene Q embedded "A2" processor are reported to work too.
Compiling the Blosc library with CMake
======================================
Blosc can also be built, tested and installed using CMake_.
Blosc can also be built, tested and installed using CMake_. Although
this procedure might seem a bit more involved than the one described
above, it is the most general because it allows to integrate other
compressors than BloscLZ either from libraries or from internal
sources. Hence, serious library developers are encouraged to use this
way.
The following procedure describes the "out of source" build.
Create the build directory and move into it:
......@@ -147,46 +196,109 @@ Create the build directory and move into it:
$ mkdir build
$ cd build
Configure Blosc in release mode (enable optimizations) specifying the
installation directory:
Now run CMake configuration and optionally specify the installation
directory (e.g. '/usr' or '/usr/local'):
.. code-block:: console
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=INSTALL_DIR \
PATH_TO_BLOSC_SOURCE_DIR
$ cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory ..
Please note that configuration can also be performed using UI tools
provided by CMake_ (ccmake or cmake-gui):
CMake allows to configure Blosc in many different ways, like prefering
internal or external sources for compressors or enabling/disabling
them. Please note that configuration can also be performed using UI
tools provided by CMake_ (ccmake or cmake-gui):
.. code-block:: console
$ cmake-gui PATH_TO_BLOSC_SOURCE_DIR
$ ccmake .. # run a curses-based interface
$ cmake-gui .. # run a graphical interface
Build, test and install Blosc:
.. code-block:: console
$ make
$ make test
$ make install
$ cmake --build .
$ ctest
$ cmake --build . --target install
The static and dynamic version of the Blosc library, together with
header files, will be installed into the specified INSTALL_DIR.
header files, will be installed into the specified
CMAKE_INSTALL_PREFIX.
.. _CMake: http://www.cmake.org
Once you have compiled your Blosc library, you can easily link your
apps with it as shown in the `example/ directory
<https://github.com/Blosc/c-blosc/blob/master/examples>`_.
Adding support for other compressors (LZ4, LZ4HC, Snappy, Zlib) with CMake
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The CMake files in Blosc are configured to automatically detect other
compressors like LZ4, LZ4HC, Snappy or Zlib by default. So as long as
the libraries and the header files for these libraries are accessible,
these will be used by default. See an `example here
<https://github.com/Blosc/c-blosc/blob/master/examples/many_compressors.c>`_.
*Note on Zlib*: the library should be easily found on UNIX systems,
although on Windows, you can help CMake to find it by setting the
environment variable 'ZLIB_ROOT' to where zlib 'include' and 'lib'
directories are. Also, make sure that Zlib DDL library is in your
'\Windows' directory.
However, the full sources for LZ4, LZ4HC, Snappy and Zlib have been
included in Blosc too. So, in general, you should not worry about not
having (or CMake not finding) the libraries in your system because in
this case, their sources will be automatically compiled for you. That
effectively means that you can be confident in having a complete
support for all the supported compression libraries in all supported
platforms.
If you want to force Blosc to use external libraries instead of
the included compression sources:
.. code-block:: console
$ cmake -DPREFER_EXTERNAL_LZ4=ON ..
You can also disable support for some compression libraries:
.. code-block:: console
$ cmake -DDEACTIVATE_SNAPPY=ON ..
Mac OSX troubleshooting
~~~~~~~~~~~~~~~~~~~~~~~
If you run into compilation troubles when using Mac OSX, please make
sure that you have installed the command line developer tools. You
can always install them with:
.. code-block:: console
$ xcode-select --install
Wrapper for Python
==================
Blosc has an official wrapper for Python. See:
https://github.com/FrancescAlted/python-blosc
https://github.com/Blosc/python-blosc
Command line interface and serialization format for Blosc
=========================================================
Blosc can be used from command line by using Bloscpack. See:
https://github.com/Blosc/bloscpack
Filter for HDF5
===============
For those that want to use Blosc as a filter in the HDF5 library,
there is a sample implementation in the hdf5/ directory.
For those who want to use Blosc as a filter in the HDF5 library,
there is a sample implementation in the blosc/hdf5 project in:
https://github.com/Blosc/hdf5
Mailing list