Designing a Data Pipeline for Massive Tables That Scales with Minimal Code
This article was written using AI. In data engineering practice, managing pipelines becomes unmanageable as the number o
Articles
Notes on data engineering, cloud, Python, and more.
132 記事
[Architecture Patterns] Writing the Repository Pattern in Python
This article was written using AI. Without the repository pattern, data access code tends to be written directly into bu
[Python] About Protocol in PEP 544
This article was written using AI. While studying Python type hints, you may have come across the terms "PEP 544" or "Pr
[BigQuery] JSON Technique: Replace Dynamic Keys with a Dummy Value
The other day, I was working with JSON-type data in BigQuery and ran into trouble extracting values when the keys were a
Setting Up Airflow on GKE
Using Terraform and Helm, and following Rodel van Rooijen's Medium article, I create an Airflow cluster on GKE without u
Airflow Custom Trigger
I created a custom Trigger for Airflow in Composer. It was quite challenging, but using it helps reduce resource consump
Reading Notes: Architecture Patterns with Python
This was the best technical book of 2024. You can learn about the practice of DDD and Clean Architecture using Python as
Receiving Email with Amazon SES
1. Verify your domain in Amazon SES 2. Register DKIM records (CNAME records) in GCP Cloud DNS 3. Register MX records in
Reading Notes: Introduction to Domain-Driven Design
Notes from reading Introduction to Domain-Driven Design. There may be mistakes, so please refer to the book for accurate
DNS
I had the opportunity at work to research DNS configuration for a cross-cloud setup between GCP and AWS. Since there wer
Trying MySQL Replication on Docker
Using a Docker Ubuntu image, I will try setting up MySQL 8 replication. First, prepare containers to host MySQL. Since t
Docker Parser Directives
Parser directives are special comments written at the top of a Dockerfile that affect how subsequent lines in the Docker
Docker Multi-stage Builds
In short, this refers to builds that use multiple FROM statements. Each FROM clause starts a new build stage. By using C
Docker Build Context
The first build you learn is probably docker build .. This is called a "Local context". Docker references and incorporat
Docker Best Practices
A summary of Docker best practices I researched. Pin the version of the base image. Using digest references is even bett
Querying All Tables in a Dataset Using a FOR Loop in BigQuery
There are times when I want to compile the record counts of multiple tables into a single table. In such cases, I use a
Access Permissions for BigQuery INFORMATION_SCHEMA
I often use INFORMATIONSCHEMA views to retrieve table and column information for use in table creation and similar tasks
Enum in Python
The Enum class in the standard library allows you to use enumerations. Since enumeration types are used to represent con
Metadata to Manage and Data Quality Tests
Metadata can be divided into three types: business metadata, technical metadata, and operational metadata. Specific exam
4 Interesting Blogs and Videos Related to BigQuery
A title like a light novel. This article introduces how BigQuery manages files internally with clear diagrams. When inse
NULL Sorting in BigQuery
When using an ORDER BY clause in BigQuery, the default NULL ordering differs between ASC and DESC. With ASC, NULLs come
List of Sites I Referenced When MySQL Was Slow
When I tried to extract data from MySQL with Embulk and it was slow, I researched various related topics. For now, I'll
Data Quality Check with Dataplex to Verify Foreign Key Constraints Are Satisfied
Since BigQuery's foreign key constraints are not enforced, users are responsible for ensuring that only values referenci
BigQuery PRIMARY KEY Is Not Enforced
I had no idea, but BigQuery's primary key / foreign key constraints are not enforced... This is something that could cau
Use TRUNCATE When Deleting All Rows in BigQuery
In BigQuery, it is better to use TRUNCATE when deleting all rows. You can restore data with Time Travel, it is faster th
Time Complexity (Big O) in SQL
Retrieving any single row from a table Filtering with a WHERE clause on an indexed column Because this is O(log N), the
Comparing NULL with Comparison Operators in BigQuery
In BigQuery, if either side of a comparison operator is NULL, the result is NULL. Both of the following queries return N
Performing DELETE-INSERT in BigQuery with MERGE ~ ON FALSE
When you want to perform a DELETE-INSERT in BigQuery, you can use: This allows you to perform a DELETE-INSERT, and it is
Blog Posts on Big O in SQL
medium.com www.datacamp.com www.kdnuggets.com www.kdnuggets.com
The allow_quoted_newlines Option for BigQuery External Tables
When creating an external table from a CSV file in BigQuery, make sure you are not missing any required options. One day
IntelliJ Initial Setup
Here I note IntelliJ settings as a reference for myself. I mainly use Terraform and Python, so the settings focus on tho
Multi-Cursor String Selection in IntelliJ
I looked into how to use multiple cursors in IntelliJ. Here I note the ones that seem particularly useful to remember. A
【Linux Tips】Output Command Results to a Datetime-Named File
Outputting command results to a datetime-named file was cumbersome, so I registered an alias to streamline the process.
【Linux Tips】Miscellaneous Notes
Even a file without execute permission can be run if you have read permission by passing it as an argument to bash, sour
Linux iptablels
External communication control for Docker containers is done using iptables. (reference) Since I want to understand exte
nftables: The Successor to iptables
nftables is a packet filtering tool and the successor to iptables, ip6tables, and others. For simple use cases, firewall
Red Hat Kernel Upgrade
The kernel is the core part of the Linux OS that acts as an intermediary (interface) between hardware and software appli
What is glibc?
glibc is a library of programs that form the foundation of the Linux system, including system calls. > An implementation
Steps to Create a Linux Machine on GCE for Starters
Steps Enable APIs (especially easy to forget Cloud Identity-Aware Proxy API) Create VPC network Create GCE instance (ass
Repositories in Linux: yum or dnf
In Linux, a repository is a server that hosts packages (.rpm, .deb, etc.) for download. For example, Nifcloud places var
Object-Oriented Programming in Python
Microsoft Learn had a nicely organized introduction to object-oriented programming in Python, so I'm taking notes on the
Common Processing Implementation When Receiving UDP Packets
When an externally exposed system receives a UDP packet: Ignore the packet. Do nothing. Don't write logs either. Writing
Trying Out Pytest
pytest is a software framework for Python. It discovers written tests by name and runs them automatically. Combined with
Trying Out Airflow
In Airflow, workflows are called DAGs. A DAG consists of multiple Tasks. Tasks execute various processes, and the DAG de
Sending UDP Packets with the nc Command
To send UDP packets with the nc command, use the -u option. Not knowing this, I kept sending with TCP and was wondering
[CircleCI] Setting Up a New Repository
Notes for my own reference on the steps to create a new repository on Bitbucket and integrate it with CircleCI. 1. Creat
Running InSpec by Remotely Logging in to Windows
When setting up a Windows machine, you may want to connect remotely, execute PowerShell commands, and verify that the co
Building a Chef InSpec Execution Environment with a Docker Container
No effort required. There is an official Dockerfile on GitHub. The base image is Ubuntu 22.04, which is personally very
How to Check the Base Image of a Docker Container
I was wondering what OS a container was using, so I casually entered the Docker container and ran uname -r. The host OS
pyproject.toml — I Had No Idea
qiita.com nikkie-ftnext.hatenablog.com
Azure DNS Server 168.63.129.16 and RHEL NetworkManager
When I created a private DNS zone in Azure, I assumed I would also need to set up a DNS server inside the private networ
Issue: Cannot Connect to SAML-Integrated Tableau from Tableau Desktop — Resolved by Signing the Server Certificate with a Private CA
When I tried to connect to a SAML-integrated Tableau Server from Tableau Desktop, I got the following error and could no
Behavioral Differences: Proxy Server vs. Firewall
The other day, an HTTP request sent to the outside via a firewall came back with a 403 error. I assumed the firewall its
Subject Alternative Names
I had no idea Subject Alternative Names were a thing... When using a self-signed certificate with Tableau Prep, the docu
Symbolic Links on Windows
I found out that you can create symbolic links on Windows using Command Prompt or PowerShell, so I noted it down here. I
[Azure] Disk Expansion on RHEL VM
When I built a RHEL 8 VM on Azure, the logical volume (LV) allocated to the partition by default was not sufficient, so
Running Airflow on GKE
Run Airflow on GKE and access the Airflow UI from an external network. Encrypt communication with HTTPS. Use External HT
Visual Studio Code Settings
The default line ending in VS Code is CRLF (\r\n). If CRLF is mixed into Linux scripts, it causes errors, so set the lin
[GCP] Two Types of Firewall Services
GCP has two types of firewall services. They are VPC Firewall Rules and Firewall Policies. The following link explains t
AD Server Setup => Domain Join => GPO Creation
Build an AD server on Windows Server. dev.classmethod.jp Join Windows to a domain. server-network-info.blogspot.com Crea
GCP DevOps Learning Log
> It should contain the followings. Incident timeline List of actions carried out to restore the service Command hierarc
Useful Sites for Studying GCP Professional Cloud DevOps Engineer
www.youtube.com What is DevOps: Research and Solutions | Google Cloud sre.google github.com Google Professional Cloud De
Learnings on GCP DevOps
A memorandum on learning for GCP DevOps certification. DevOps has five goals. 1. reduce organizational silos 2. Accept f
Docker Image for Running Embulk
I wanted to easily run Embulk in a Docker container, so I tried building one. Prepare the config file config.yml needed
Reading Notes: "Linux System Build and Operations for Professionals"
Overview Chapter 1: Linux Server Basics Chapter 2: Server Operations Basics Chapter 3: Linux Storage Management Chapter
Reading Notes: "Azure Active Directory at a Glance"
I'm summarizing what I understood and researched from reading "Azure Active Directory at a Glance, 3rd Edition." The mai
Alerts in Azure
In general, the following three types of alerts are used. Log alerts Activity log alerts Metric alerts Queries are execu
Log Collection in Azure
Overview Types of logs in Azure and log collection methods Resource logs Diagnostic settings Activity logs Azure Active
Linux Study Notes
Since I'm working with Linux, I'm jotting down notes from re-studying it after a long break. Overview ssh Virtual Machin
Terraform for Expressions
I use Terraform's foreach expression frequently, but I've barely used for expressions, so I tried passing a map type to
Trying Out JavaScript
While studying cross-site scripting, JavaScript came up, so I looked into the basics of JavaScript. > JavaScript is a pr
Notes on Creating a Website with WordPress
I set up a VM on GCP and created a website with Wordpress, so I am noting it down here. Since GCP has a free tier, I dec
Security-Related Keywords
CIS Controls and CIS Benchmarks Data Loss Prevention (DLP) DLP (Data Loss Prevention) | Security Glossary | NRI Secure D
[In Progress] PCCSE Course
Overview Prisma Cloud Visibility, Compliance, and Governance: Onboarding and Setup Prisma Cloud Compute Detailed Protect
How to Study Prisma Cloud
I looked into how to study Prisma Cloud. I tried working through the official learning materials related to Prisma Cloud
Investigating Whether a Yoga Lesson Booking Site Can Be Built with WordPress
I looked into whether it is possible to build a booking site with WordPress that allows reservations based on a pre-defi
Ansible Basic Notes
I read the Ansible Practical Guide 3rd Edition and am noting only the very basics here. These are rough notes, so please
Connecting to the Internet via WiFi AP with a Cisco 841M Router
Notes on the initial setup of the Cisco 841M router and the settings required for internet connection. I wanted to try c
Book Notes: SQL Server Transact-SQL Programming Practical Development Guide
Overview Instance Users Instance connection authentication methods Creating each user Schema SQL Server tools SSMS SQL S
SSH Connection Error After Disk Replacement on GCE Linux Instance
On GCP, after replacing an "additional disk" on a GCE Linux instance, SSH connection became impossible, but access was r
Loading JSON Data into BigQuery
With BigQuery, even nested JSON data can be loaded and easily normalized. For example, if data obtained via an API is in
English Words That Need Pronunciation Practice
according to additional additionally ad-hoc after analysis analytics analyze annual application application assumption b
English Expressions Used in IT Projects
A collection of English expressions used in IT projects, to reduce time spent researching expressions in the future. as
Preparing for English Meetings
Notes on English phrases likely to be used in meetings or business trips. Phrases I want to remember are marked with ★.
Book Notes: How Linux Works (in progress)
Notes on important points from "How Linux Works." Layer --- User programs OS external libraries OS libraries Kernel Hard
Native Apps and WebView
When the display portion of a smartphone app is written using a dedicated program, that type of smartphone app is called
Web Tracking
Web tracking refers to recording and tracking the online behavior of users who visit a website. Can determine where user
Azure DevOps
I took the following Udemy course as I might need to use Azure DevOps at work. I'll take notes on the important parts. L
Project Management
Scope Time Cost A change to any one of these will affect the other constraints. Identified risks: Known risks. Risks mus
Book Notes: Practical Introduction to Cloud Native with AWS
Notes on important points from "Practical Introduction to Cloud Native with AWS." This time I only read Chapter 2 "Cloud
Book Notes: Introduction to Design Through Good and Bad Code
Notes on important points from "Introduction to Design Through Good and Bad Code." Some of the code written in the book
AWS Container Services
AWS container services include ECS and Fargate. That said, Fargate is actually one of the launch types within ECS, so th
Azure DP-203 Study Notes
Datalake → Data Warehouse → BI Datalake: Azure Storage Gen2 Data Warehouse: Azure Synapse Analytics BI: PowerBI There ar
AZ-104 Study Notes
Azure Direct Azure EA Azure in Open CSP Split invoices Separation of access rights Separation of quota management Resour
PyCharm Frequently Used Shortcuts
Move caret to beginning of line: Home Move caret to end of line: End Duplicate line: Ctrl + D Run: Shift + F10 Comment o
Trying Out Protocol Buffers
This time I'll try out Protocol Buffers (Protobuf) using these references: tech.raksul.com nansystem.com developers.goog
Reading 'Tableau Thorough Introduction'
I read "Tableau Thorough Introduction" and summarized its contents. However, I only read all of Chapter 1 and part of Ch
Learning JSTQB Through Video
The JSTQB syllabus is a bit hard to understand in places, so I went looking for videos and found the YouTube playlist be
Passed the OSS-DB Silver Exam
I had an opportunity to use PostgreSQL at work, so I decided to take the exam. My study approach was to read through "Po
Backup for Cloud SQL for PostgreSQL
There are several ways to back up Cloud SQL for PostgreSQL. I wasn't sure what was actually covered by each backup metho
How to Pattern Match on PostgreSQL text[] Arrays
Pattern matching cannot be performed directly on text array columns. To pattern match on a text array, use the UNNEST fu
Useful Articles for Version Upgrades
I found a useful article about PostgreSQL version upgrades, so I'm noting it here. techblog.gmo-ap.jp Need to check not
Getting Started with PostgreSQL
I had an opportunity to use PostgreSQL, so I started learning it. I'll limit my notes to things that seem important and
Passed Oracle Master Silver SQL
I passed Oracle Master Silver SQL in October 2022. I think it's a certification testing the minimum knowledge needed to
[Tableau] Missing Data with Incremental Refresh
In Tableau, there are two ways to extract data source updates: full refresh and incremental refresh. By default, full re
What Is the Oracle Date Format "DDSPTH"?
There are display formats DDTH and DDSP, and DDSPTH is a combination of the two. !( docs.oracle.com
Git Commands on Local
!( Let's understand the overall architecture. First, we divide into local PC and cloud. Documents on the local PC are pu
About GCP Cloud APIs
In GCP, each service becomes available by enabling its Cloud API on a per-project basis. Even when using services from t
Can You Assign the Private IP of One CloudSQL Instance to Another CloudSQL Instance?
Can the internal IP assigned to one CloudSQL instance be assigned to a different CloudSQL instance? Here is the backgrou
About GCP Static IPs
GCP offers Static External IP and Static Internal IP as the equivalent of AWS Elastic IP. External IP is a global IP, an
Oracle Master Silver SQL Study Notes
NUMBER: integers and decimals CHAR(n): fixed-length string of n bytes VARCHAR2(n): variable-length string of up to n byt
Trying SQL Zoo ~ SELECT in SELECT and JOIN
SQLZOO is a site where you can learn SQL for free through hands-on practice. I tried the SELECT in SELECT and JOIN secti
Trying SQL Zoo ~ Self Join
I couldn't figure out problem 10.... 8. I couldn't figure this one out....
Trying SQL Zoo ~ Using NULL
The Using NULL section of SQL Zoo had mostly easy problems. I'll leave my answers here for reference.
Trying SQL Zoo — More Join
Notes on just the answers to the harder problems in the More Join section of SQL Zoo. I'd like to compare with other peo
I Didn't Understand Airflow, So I Studied on Udemy — Part 2
Notes from studying the Airflow course on Udemy. This post mainly covers pre-setup and code. The Udemy course uses Docke
I Didn't Understand Airflow, So I Studied on Udemy
The Airflow Tutorial didn't seem like something I could understand quickly, so I tried a Udemy course. It was an English
Trying Out Airflow — Quick Start
Following the Airflow Quick Start, I'll install Airflow on WSL Ubuntu 20.04. The following is excerpted from the Quick S
Passed the Terraform Associate Exam
Today (September 2022), I passed the Terraform Associate exam. I had been wanting to use Terraform and had been experime
I Want to Try WIN32API
> There are two ways to create apps that run on Windows: using the .NET framework, or using WIN32API. > To program for W
Passed GCP Associate Cloud Engineer and Professional Data Engineer
I passed the GCP Associate Cloud Engineer (ACE) and Professional Data Engineer (PDE) exams. I took them in August and Se
Passed the Python Engineer Certification Exam
I recently decided on a whim to take the Python Engineer Certification Exam and passed. I had known about the certificat
Notes: Udemy HashiCorp Certified: Terraform Associate 2022
Overview Terraform version Specifying the Terraform version Provider version constraints and specification .terraform.lo
How Python Logging Works
hackers-high.com python.civic-apps.com
GCP PDE Mock Exam Notes
Overview AutoML Vision Apache Hadoop BigTable Storing CPU and memory usage of millions of computers as time series When
Book Notes: Getting Started with Data Engineering on Google Cloud
Notes from reading "Getting Started with Data Engineering on Google Cloud." Chapters 8–11 covering BigQuery data aggrega
Book Notes: Introduction to Data Analytics Platforms for Engineers
Notes on key points from "Introduction to Data Analytics Platforms for Engineers." A system built specifically for data
Notes: Mastering Azure Basics — Data Fundamentals
Overview Explore Core Data Concepts Explore Core Data Concepts Data formats File formats Avro ORC (Optimized Row Columna
Book Notes: Practical Terraform
Overview Versioning Commands Symbols Files variable Variable types local output data provider Lists Built-in functions M
Book Notes: The Expert's Guide to Database Design (Chapters 1–3)
Notes on key points from "The Expert's Guide to Database Design" for future reference. The book has 9 chapters, but this
Passed the CKA Exam [April 2022]
Udemy alone is not enough — you absolutely must also do the Killer.sh practice exams! December 2021: First touched Kuber
[Study Notes] Kubernetes Udemy CKA Course
Studying for the Certified Kubernetes Administrator (CKA), one of the Kubernetes certification exams. Taking the "Certif