Embulk plugins

Embulk plugins by category

List of Plugins by Category

INPUT

Star Name Author About Download
postgresql
$ embulk gem install embulk-input-postgresql
Sadayuki Furuhashi Selects records from a table. 14138
redshift
$ embulk gem install embulk-input-redshift
Sadayuki Furuhashi Selects records from a table. 12928
s3
$ embulk gem install embulk-input-s3
Sadayuki Furuhashi Reads files stored on Amazon S3 12368
mysql
$ embulk gem install embulk-input-mysql
Sadayuki Furuhashi Selects records from a table. 11774
jdbc
$ embulk gem install embulk-input-jdbc
Sadayuki Furuhashi Selects records from a table. 9280
marketo
$ embulk gem install embulk-input-marketo
uu59, yoshihara Loads records from Marketo. 7454
gcs
$ embulk gem install embulk-input-gcs
Satoshi Akama Reads files stored on Google Cloud Storage (Standard, Durable Reduced Availability or Nearline) 7361
hdfs
$ embulk gem install embulk-input-hdfs
Civitaspo Reads files stored on Hdfs. 6880
sfdc
$ embulk gem install embulk-input-sfdc
yoshihara, uu59 Loads sObjects using SOQL from Salesforce.com 6018
mixpanel
$ embulk gem install embulk-input-mixpanel
yoshihara, uu59 Loads records from Mixpanel. 5852
jira
$ embulk gem install embulk-input-jira
uu59, yoshihara Loads records from Jira. 5557
http
$ embulk gem install embulk-input-http
Takuma kanari Fetch data via http 4843
riak_cs
$ embulk gem install embulk-input-riak_cs
Sadayuki Furuhashi Reads files stored on Riak CS 4093
remote
$ embulk gem install embulk-input-remote
Shinichi Ishimura Reads files stored on Remote hosts by SCP. 3741
oracle
$ embulk gem install embulk-input-oracle
Sadayuki Furuhashi Selects records from a table. 3425
pcapng-files
$ embulk gem install embulk-input-pcapng-files
enukane Pcapng Files input plugin for Embulk 3360
command
$ embulk gem install embulk-input-command
Sadayuki Furuhashi Executes a command and reads a file from its STDOUT. 3311
mongodb
$ embulk gem install embulk-input-mongodb
Kazuyuki Honda Loads records from Mongodb. 3287
sqlserver
$ embulk gem install embulk-input-sqlserver
Sadayuki Furuhashi Selects records from a table. 2860
azure_blob_storage
$ embulk gem install embulk-input-azure_blob_storage
Satoshi Akama Reads files stored on Microsoft Azure Blob Storage. 2815
zendesk
$ embulk gem install embulk-input-zendesk
uu59, muga, sakama Loads records from Zendesk. 2692
ftp
$ embulk gem install embulk-input-ftp
Sadayuki Furuhashi Reads files stored on a FTP server. 2563
dynamodb
$ embulk gem install embulk-input-dynamodb
Daisuke Higashi "Loads records from Dynamodb." 2486
filesplit
$ embulk gem install embulk-input-filesplit
Hitoshi Tanaka Embulk plugin for splitting input file 1969
presto
$ embulk gem install embulk-input-presto
toyama0919 Facebook Presto input plugin for Embulk. 1458
slack-history
$ embulk gem install embulk-input-slack-history
Akihiro YAGASAKI Embulk input plugin for Slack chat history 1440
vertica
$ embulk gem install embulk-input-vertica
Naotoshi Seo Loads records from Vertica. 1345
sftp
$ embulk gem install embulk-input-sftp
Satoshi Akama Reads files stored on remote server using SFTP. 1152
random
$ embulk gem install embulk-input-random
KUMAZAKI Hiroki Create dummy table 1100
twitterarchive
$ embulk gem install embulk-input-twitterarchive
takuya sato Loads records from Twitterarchive. 1044
elasticsearch
$ embulk gem install embulk-input-elasticsearch
toyama0919 Loads records from Elasticsearch. parallel query support. 973
sql
$ embulk gem install embulk-input-sql
kakusuke "Loads records from Sql." 861
apache-dummy-log
$ embulk gem install embulk-input-apache-dummy-log
Hiroyuki Sato Apache Dummy Log input plugin is an Embulk plugin that loads records from Apache Dummy Log so that any output plugins can receive the records. Search the output plugins by 'embulk-output' keyword. 841
healthplanet
$ embulk gem install embulk-input-healthplanet
Masahiro Yoshizawa Load records from Health Planet through Health Planet API v1. 808
redis
$ embulk gem install embulk-input-redis
Mitsunori Komatsu Embulk input plugins for Redis 792
jstat
$ embulk gem install embulk-input-jstat
KUBOTA Yuji Embulk input plugin for Java Virtual Machine statistics by jstat command. 788
slack_message
$ embulk gem install embulk-input-slack_message
yuemori Loads records from Slack Message. 426
google_analytics
$ embulk gem install embulk-input-google_analytics
uu59 Loads records from Google Analytics. 409
googlespreadsheet
$ embulk gem install embulk-input-googlespreadsheet
yang-xu Fetches data from Googlespreadsheet. 323
sequence
$ embulk gem install embulk-input-sequence
tomykaira Generate sequential number. 285
inline
$ embulk gem install embulk-input-inline
toyama0919 Loads records from Yaml Inline. 202
rediskeys
$ embulk gem install embulk-input-rediskeys
dokuma Loads records from Redis. 198
swift
$ embulk gem install embulk-input-swift
yuuzi41 Reads files stored on Swift. 196

OUTPUT

Star Name Author About Download
bigquery
$ embulk gem install embulk-output-bigquery
Satoshi Akama, Naotoshi Seo Embulk plugin that insert records to Google BigQuery. 33994
mysql
$ embulk gem install embulk-output-mysql
Sadayuki Furuhashi Inserts or updates records to a table. 28629
postgresql
$ embulk gem install embulk-output-postgresql
Sadayuki Furuhashi Inserts or updates records to a table. 18850
redshift
$ embulk gem install embulk-output-redshift
Sadayuki Furuhashi Inserts or updates records to a table. 18242
vertica
$ embulk gem install embulk-output-vertica
Naotoshi Seo, Eiji Sekiya Dump records to vertica 17032
oracle
$ embulk gem install embulk-output-oracle
Sadayuki Furuhashi Inserts or updates records to a table. 11573
jdbc
$ embulk gem install embulk-output-jdbc
Sadayuki Furuhashi Inserts or updates records to a table. 10243
elasticsearch
$ embulk gem install embulk-output-elasticsearch
Muga Nishizawa Elasticsearch output plugin is an Embulk plugin that loads records to Elasticsearch read by any input plugins. Search the input plugins by "embulk-input" keyword. 8932
td
$ embulk gem install embulk-output-td
Muga Nishizawa TreasureData output plugin is an Embulk plugin that loads records to TreasureData read by any input plugins. Search the input plugins by 'embulk-output' keyword. 8707
postgres-udf
$ embulk gem install embulk-output-postgres-udf
kakusuke Dumps records to PostgreSQL via user-defined function. 8604
hdfs
$ embulk gem install embulk-output-hdfs
Civitaspo Stores files on Hdfs. 4428
command
$ embulk gem install embulk-output-command
Sadayuki Furuhashi Executes commands and writes files to STDIN. 4274
s3
$ embulk gem install embulk-output-s3
Manabu Takayama Stores files on S3. 4178
aerospike
$ embulk gem install embulk-output-aerospike
Takeru Sato Dumps records to Aerospike. 3337
sftp
$ embulk gem install embulk-output-sftp
Civitaspo Stores files on Sftp. 3209
bigobject
$ embulk gem install embulk-output-bigobject
Cheng-Ching Huang Dumps records to Bigobject. 2622
gcs
$ embulk gem install embulk-output-gcs
Kazuyuki Honda "Dumps records to Google Cloud Storage." 2330
azure_blob_storage
$ embulk gem install embulk-output-azure_blob_storage
Satoshi Akama Stores files on Microsoft Azure Blob Storage. 2231
salesforce
$ embulk gem install embulk-output-salesforce
Makoto Tajitsu Dumps records to Salesforce. 2141
parquet
$ embulk gem install embulk-output-parquet
OKUNO Akihiro Parquet output plugin is an Embulk plugin that loads records to Parquet read by any input plugins. Search the input plugins by "embulk-input" keyword. 2139
mailchimp
$ embulk gem install embulk-output-mailchimp
uu59, takkanm Dumps records to Mailchimp. 1884
google_spreadsheets
$ embulk gem install embulk-output-google_spreadsheets
Noriaki Katayama Dumps records to Google Spreadsheets. 1559
s3_per_record
$ embulk gem install embulk-output-s3_per_record
tomykaira Upload a column's value to S3 as one S3 object per row. 1325
sqlserver
$ embulk gem install embulk-output-sqlserver
Sadayuki Furuhashi Inserts or updates records to a table. 1201
influxdb
$ embulk gem install embulk-output-influxdb
joker1007 Dumps records to InfluxDB. 1089
dynamodb
$ embulk gem install embulk-output-dynamodb
Satoshi Akama Dumps records to AWS DynamoDB. 942
maprfs
$ embulk gem install embulk-output-maprfs
Katsunori Kanda Stores files on MapR-FS. 835
postgres-json
$ embulk gem install embulk-output-postgres-json
Sadayuki Furuhashi Embulk output for PostgreSQL json and jsonb column 823
sqlite3
$ embulk gem install embulk-output-sqlite3
Yuichi Takada SQLite3 output plugin for Embulk 803
redis
$ embulk gem install embulk-output-redis
Mitsunori Komatsu Embulk output plugins for Redis 794
groonga
$ embulk gem install embulk-output-groonga
Hiroyuki Sato Dumps records to Groonga. 491
ftp
$ embulk gem install embulk-output-ftp
Satoshi Akama Store files using FTP. 448
analytics_cloud
$ embulk gem install embulk-output-analytics_cloud
Makoto Tajitsu Dumps records to Analytics Cloud. 403
elasticsearch_1.x
$ embulk gem install embulk-output-elasticsearch_1.x
Muga Nishizawa, Shinji Ikeda Elasticsearch 1.x output plugin is an Embulk plugin that loads records to Elasticsearch read by any input plugins. Search the input plugins by "embulk-input" keyword. 376
rediskeys
$ embulk gem install embulk-output-rediskeys
dokuma Dumps records to Redis. 198
swift
$ embulk gem install embulk-output-swift
yuuzi41 Stores files on OpenStack Swift. 195

FILTER

Star Name Author About Download
column
$ embulk gem install embulk-filter-column
Naotoshi Seo A filter plugin for Embulk to filter out columns. 6528
row
$ embulk gem install embulk-filter-row
Naotoshi Seo A filter plugin for Embulk to filter out rows with conditions. 4664
expand_json
$ embulk gem install embulk-filter-expand_json
Civitaspo Expand Json 3325
ruby_proc
$ embulk gem install embulk-filter-ruby_proc
joker1007 Filter each record by ruby proc 3159
speedometer
$ embulk gem install embulk-filter-speedometer
hata Write log message of processed bytes and throughput periodically. 2950
timestamp_format
$ embulk gem install embulk-filter-timestamp_format
Naotoshi Seo A filter plugin for Embulk to change timestamp format. 2870
woothee
$ embulk gem install embulk-filter-woothee
toyama0919 Woothee filter plugin for Embulk 2694
kuromoji
$ embulk gem install embulk-filter-kuromoji
toyama0919 Kuromoji filter plugin for Embulk. Neologd support. 2603
insert
$ embulk gem install embulk-filter-insert
Masahiro Yoshizawa Embulk filter plugin that inserts column(s) at any position (e.g., the top/bottom of the columns, before/after the specified column name) 2311
eval
$ embulk gem install embulk-filter-eval
mgi166 Eval 2223
distinct
$ embulk gem install embulk-filter-distinct
Civitaspo Distinct 1720
hash
$ embulk gem install embulk-filter-hash
Shinichi Ishimura Embulk filter plugin to convert a input to hash. 1614
icu4j
$ embulk gem install embulk-filter-icu4j
toyama0919 Unicode normalize string value. see http://site.icu-project.org/ 1487
query_string
$ embulk gem install embulk-filter-query_string
Minnano Wedding Co., Ltd. The query string filter plugin parses the column contents as query string and insert columns from that field. 1392
split
$ embulk gem install embulk-filter-split
toyama0919 Split filter plugin for Embulk 1390
add_time
$ embulk gem install embulk-filter-add_time
Muga Nishizawa Add time column to the schema 1362
flatten_json
$ embulk gem install embulk-filter-flatten_json
Civitaspo Flatten Json 1189
to_json
$ embulk gem install embulk-filter-to_json
Civitaspo To Json 1188
typecast
$ embulk gem install embulk-filter-typecast
Naotoshi Seo A filter plugin for Embulk to cast column type. 1044
join_file
$ embulk gem install embulk-filter-join_file
Civitaspo Join File 1043
crawler
$ embulk gem install embulk-filter-crawler
toyama0919 Crawler4J filter plugin for Embulk 1028
json_key
$ embulk gem install embulk-filter-json_key
Civitaspo Json Key 994
rearrange
$ embulk gem install embulk-filter-rearrange
Tooru Okano Rearrange 831
stdout
$ embulk gem install embulk-filter-stdout
sonots Embulk filter plugin to print embulk records to stdout 760
script_ruby
$ embulk gem install embulk-filter-script_ruby
shinjiikeda Script Ruby 712
split_column
$ embulk gem install embulk-filter-split_column
yskn67 A filter plugin for Embulk to split one string column to several any type columns. 646
unique
$ embulk gem install embulk-filter-unique
Naohiro Aota Extract unique records 644
timestamp_hs
$ embulk gem install embulk-filter-timestamp_hs
Matsuoka Takashi Convert string to timestamp at high speed. 590
to_csv
$ embulk gem install embulk-filter-to_csv
Civitaspo To Csv 539
encrypt
$ embulk gem install embulk-filter-encrypt
Sadayuki Furuhashi Encrypt 532
mysql
$ embulk gem install embulk-filter-mysql
toyama0919 Mysql filter plugin for Embulk. Execute prepared statements query. 518
script
$ embulk gem install embulk-filter-script
SNakano Embulk filter plugin to external ruby script 479
url_encode
$ embulk gem install embulk-filter-url_encode
Minnano Wedding Co., Ltd. The encode filter plugin encodes a column in the URL encoding format. 468
pherialize
$ embulk gem install embulk-filter-pherialize
cynipe Embulk plugin that deserialize PHP serialized strings to extract values as new column 300
base58
$ embulk gem install embulk-filter-base58
Kevin Fitzgerald Embulk filter plugin to convert a hex value to and from base58. 252
forcepoint_converter
$ embulk gem install embulk-filter-forcepoint_converter
sesame Forcepoint Converter 242
rename_with_gsub
$ embulk gem install embulk-filter-rename_with_gsub
sesame Rename With Gsub 232
concat
$ embulk gem install embulk-filter-concat
toru-takahashi Concat 199
jsoncolumn
$ embulk gem install embulk-filter-jsoncolumn
dokuma Jsoncolumn 172
mask
$ embulk gem install embulk-filter-mask
Tetsuo Yamabe Mask 126

FILE PARSER

Star Name Author About Download
query_string
$ embulk gem install embulk-parser-query_string
yoshihara, uu59 Parses Query String files read by other file input plugins. 5814
poi_excel
$ embulk gem install embulk-parser-poi_excel
hishidama Parses Excel files(xls, xlsx) read by other file input plugins. 3668
jsonl
$ embulk gem install embulk-parser-jsonl
Shunsuke Mikami Parses Jsonl files read by other file input plugins. 3568
msgpack
$ embulk gem install embulk-parser-msgpack
Sadayuki Furuhashi Parses files encoded in MessagePack. 3310
xml
$ embulk gem install embulk-parser-xml
Takuma kanari XML parser plugin is Embulk plugin to fetch entries in xml format. 3266
json
$ embulk gem install embulk-parser-json
Takuma kanari Json parser plugin is Embulk plugin to fetch entries in json format. 2266
regex
$ embulk gem install embulk-parser-regex
Ken Morishita Parses lines using regular-expression in files read by other file input plugins. 1697
apache-custom-log
$ embulk gem install embulk-parser-apache-custom-log
Hiroyuki Sato, Osamu Ishikawa Parses Apache Custom Log files read by other file input plugins. 1595
none
$ embulk gem install embulk-parser-none
Naotoshi Seo Embulk parser plugin not to parse at all. 1406
ltsv
$ embulk gem install embulk-parser-ltsv
toyama0919 Parses Ltsv files read by other file input plugins. 1388
grok
$ embulk gem install embulk-parser-grok
Ikezoe Akihiro, Rishi Kathera Parses Grok files read by other file input plugins. 1243
fluent-s3-log
$ embulk gem install embulk-parser-fluent-s3-log
y-matsuwitter Parses Fluent S3 Log files read by other file input plugins. 880
apache-log
$ embulk gem install embulk-parser-apache-log
Hiroyuki Sato Parses Apache Log files read by other file input plugins. 858
regexp
$ embulk gem install embulk-parser-regexp
sesame Parses Regexp files read by other file input plugins. 846
pcapng
$ embulk gem install embulk-parser-pcapng
enukane Pcapng parser plugin is an Embulk plugin that parses Pcapng file format read by any file input plugins. Search the file input plugins by 'embulk-input file' keywords. 831
jdbc-schema-csv
$ embulk gem install embulk-parser-jdbc-schema-csv
Hitoshi Tanaka This Embulk plugin extends CSV parser to define columns based on database meta data. 791
roo-excel
$ embulk gem install embulk-parser-roo-excel
Hiroyuki Sato Parses Excel files(xlsx) read by other file input plugins. 733
xpath
$ embulk gem install embulk-parser-xpath
Tatsunori Matoba XPath parser plugin is Embulk plugin to fetch entries in xml format use XPath. 504
sisimai
$ embulk gem install embulk-parser-sisimai
Hiroyuki Sato Parses Sisimai Analyzer files read by other file input plugins. 456
fluentd_out_file
$ embulk gem install embulk-parser-fluentd_out_file
Muga Nishizawa Parses Fluentd Out File files read by other file input plugins. 394
forcepoint
$ embulk gem install embulk-parser-forcepoint
sesame Parses Forcepoint files read by other file input plugins. 316
avro
$ embulk gem install embulk-parser-avro
joker1007 Parses Avro files read by other file input plugins. 302
script_ruby
$ embulk gem install embulk-parser-script_ruby
shinjiikeda Parses Script Ruby files read by other file input plugins. 211

FILE DECODER

Star Name Author About Download
commons-compress
$ embulk gem install embulk-decoder-commons-compress
hata Commons Compress decoder plugin is an Embulk plugin that decodes files using Apache Commons Compress library. It is read by any file input plugins. Search the file input plugins by 'embulk-input file' keywords. 2444

FILE FORMATTER

Star Name Author About Download
jsonl
$ embulk gem install embulk-formatter-jsonl
TAKEI Yuya Formats Embulk Formatter Jsonl files for other file output plugins. 4536
single_value
$ embulk gem install embulk-formatter-single_value
Naotoshi Seo Embulk formatter plugin to output values of a single column. 1794
poi_excel
$ embulk gem install embulk-formatter-poi_excel
hishidama Formats Excel files(xls, xlsx) for other file output plugins. 1379
markdown_table
$ embulk gem install embulk-formatter-markdown_table
yuemori Formats Markdown Table files for other file output plugins. 475
html_table
$ embulk gem install embulk-formatter-html_table
yuemori Formats Html Table files for other file output plugins. 453
geojson
$ embulk gem install embulk-formatter-geojson
lewuathe Formats Geojson files for other file output plugins. 410

FILE ENCODER

Star Name Author About Download
commons-compress
$ embulk gem install embulk-encoder-commons-compress
hata Commons Compress encoder plugin is an Embulk plugin that encodes files using Commons Compress so that any file output plugins can store the files. Search the file output plugins by 'embulk-output file' keywords. 1350
encrypted_zip
$ embulk gem install embulk-encoder-encrypted_zip
Ken Morishita Encodes files using Encrypted Zip for other file output plugins. 703
xz
$ embulk gem install embulk-encoder-xz
Sadayuki Furuhashi Compress files using XZ (also know as LZMA) 690
pgp_encryption
$ embulk gem install embulk-encoder-pgp_encryption
Ken Morishita Encodes files using PGP Encryption for other file output plugins. 405

EXECUTOR

Star Name Author About Download
mapreduce
$ embulk gem install embulk-executor-mapreduce
Sadayuki Furuhashi Executes tasks on Hadoop. 10471