Data Engineering
Docker Image for Running Embulk
→ 日本語版を読むOverview
I wanted to easily run Embulk in a Docker container, so I tried building one.
Preparation
Prepare the config file config.yml needed for Embulk to run, and test.csv which will serve as Embulk's input.
config.yml is configured to read a CSV and display it to standard output.
in:
type: file
path_prefix: '/work/test.csv'
parser:
charset: UTF-8
newline: LF
type: csv
delimiter: ','
quote: '"'
escape: '"'
null_string: "NULL"
trim_if_not_quoted: false
skip_header_lines: 1
allow_extra_columns: false
allow_optional_columns: false
default_timezone: 'Asia/Tokyo'
columns:
- {name: id, type: string}
- {name: name, type: string}
- {name: point, type: string}
out: {type: stdout}
test.csv is as follows:
id, name, point
100, Yamada, 80
200, Nishida, 65
Docker Image and Execution
Build a Docker image for running Embulk.
The Dockerfile is as follows:
# Java environment is required to run Embulk
# amazoncorretto is a distribution of OpenJDK provided by AWS
# Embulk only supports Java 8, so specify the corresponding tag (8-al2-jdk)
FROM amazoncorretto:8-al2-jdk
# Download Embulk and place it as /usr/local/bin/embulk
RUN curl --create-dirs -o /usr/local/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar" \
&& chmod +x /usr/local/bin/embulk
# Specify the working directory
WORKDIR /work
# Copy the Embulk config file
COPY config.yml ./
# Copy the CSV file to be read by Embulk
COPY test.csv ./
# Docker containers cannot run .jar files without java -jar. A key gotcha!!!
CMD ["java", "-jar", "/usr/local/bin/embulk", "run", "/work/config.yml"]
Build the Dockerfile, create the container, and start the container.
> docker build . -t embulk:v3 # Build the image. Image name is embulk, tag name is v3
[+] Building 0.2s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 815B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/amazoncorretto:8-al2-jdk 0.0s
=> [1/5] FROM docker.io/library/amazoncorretto:8-al2-jdk 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 59B 0.0s
=> CACHED [2/5] RUN curl --create-dirs -o /usr/local/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar" && chmod +x /usr/local/bin/embulk 0.0s
=> CACHED [3/5] WORKDIR /work 0.0s
=> CACHED [4/5] COPY config.yml ./ 0.0s
=> CACHED [5/5] COPY test.csv ./ 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:c1a5b25ba1706d107a564e4c9ce657ca860fdaeb6eb3acde5e7696d3e9c3f18b 0.0s
=> => naming to docker.io/library/embulk:v3 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
> docker create --name embulk embulk:v3 # Create a container named embulk from the embulk:v3 image
bb895ad56a345ccfee3f0288795779947b28756ea762ca5d2bc6b380f5bb39f2
> docker start -a embulk # Start the container. The -a option displays standard output and standard error
2023-08-11 15:55:44.087 +0000 [DEBUG] (main): /root/.embulk/embulk.properties does not exist. Ignored.
2023-08-11 15:55:44.091 +0000 [INFO] (main): m2_repo is set as a sub directory of embulk_home: /root/.embulk/lib/m2/repository
2023-08-11 15:55:44.092 +0000 [INFO] (main): gem_home is set as a sub directory of embulk_home: /root/.embulk/lib/gems
2023-08-11 15:55:44.092 +0000 [INFO] (main): gem_path is set empty.
2023-08-11 15:55:44.092 +0000 [DEBUG] (main): Embulk system property "default_guess_plugin" is set to: "gzip,bzip2,json,csv"
2023-08-11 15:55:44.273 +0000 [INFO] (main): Started Embulk v0.11.0
2023-08-11 15:55:44.320 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-file
2023-08-11 15:55:44.397 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-stdout
2023-08-11 15:55:44.467 +0000 [INFO] (0001:transaction): Loaded plugin embulk-parser-csv
2023-08-11 15:55:44.594 +0000 [INFO] (0001:transaction): Listing local files at directory '/work' filtering filename by prefix 'test.csv'
2023-08-11 15:55:44.594 +0000 [INFO] (0001:transaction): "follow_symlinks" is set false. Note that symbolic links to directories are skipped.
2023-08-11 15:55:44.595 +0000 [INFO] (0001:transaction): Loading files [/work/test.csv]
2023-08-11 15:55:44.725 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=16 / output tasks 8 = input tasks 1 * 8
2023-08-11 15:55:44.771 +0000 [INFO] (0001:transaction): {done: 0 / 1, running: 0}
100, Yamada, 80
200, Nishida, 65
2023-08-11 15:55:44.832 +0000 [INFO] (0001:transaction): {done: 1 / 1, running: 0}
2023-08-11 15:55:44.839 +0000 [INFO] (main): Committed.
2023-08-11 15:55:44.839 +0000 [INFO] (main): Next config diff: {"in":{"last_path":"/work/test.csv"},"out":{}}