embulk

Embulk is an Open-source Pluggable Bulk Data Loader
to/from varieties of storages, file formats, databases, cloud services, and else.

Latest Versions

Stable
v0.11.0
Note that v0.11 is not fully compatible with v0.9. Several plugins for v0.9 would not work with v0.11.
Take a look at this article for details.
Past stable
v0.9.25

Recent Articles

Embulk is a bulk data loader. It helps data transfer between types of databases, storages, file formats, cloud services, and else.

Embulk supports:

  • Combination of input and output from varieties of plugins
  • Plugins released in Maven and Ruby gem repositories
  • Automated guess of input file formats
  • Parallel execution to deal with big data sets
  • Transaction control to guarantee all-or-nothing

Community

For Users
User Forum: GitHub Discussions
For Developers
Chat: Zulip at https://embulk-dev.zulipchat.com/
Developer Resources: https://dev.embulk.org/

Quick Start

Embulk runs on Java. Make sure Java Runtime is installed.

Embulk v0.9 (past stable) run only on Java 8.

Embulk v0.11 officially supports only Java 8, but expected to work somehow with Java 11, and 17. We are going to test v0.11 with Java 11 and 17 so that we can declare an official support for them.

Linux & macOS

curl --create-dirs -o ~/.embulk/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar"
chmod +x ~/.embulk/bin/embulk
echo 'export PATH="$HOME/.embulk/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Windows

PowerShell -Command "& {[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12; Invoke-WebRequest http://dl.embulk.org/embulk-latest.jar -OutFile embulk.bat}"

Set up with JRuby

Download your preferred version 9.x.y of JRuby Complete jar from JRuby Downloads, and put it in your favorite directory.

Embulk is tested with old JRuby 9.1.15.0 as of now. (However, older JRuby has an issue in installing Ruby gems. You may need to try newer JRuby.)

Then, create ~/.embulk/embulk.properties as follows.

jruby=file:///path/to/jruby-complete-9.x.y.z.jar

Finally, install some Ruby gems.

embulk gem install embulk -v 0.11.0 # Must be the same Embulk version
embulk gem install msgpack
embulk gem install bundler # if you need Bundler
embulk gem install liquid  # if you need Liquid

Next steps

The embulk example command generates an example CSV file.

You can try Embulk quickly by:

embulk example ./try1
embulk guess ./try1/seed.yml -o config.yml
embulk preview config.yml
embulk run config.yml

Presentation