Yisheng's Blog

Context Engineering from the Inside Out

Sun, 15 Mar 2026 00:00:00 +0000

AI agents live and die by their context windows. The system prompt, tool schemas, conversation history, retrieved documents — everything the model sees during a single request shapes what it does next. Context engineering is the art of deciding what goes in, where it goes, and when it gets removed.

Good context engineering makes agents more reliable, faster, and cheaper. This post explores the design philosophy behind features like CLAUDE.md, hooks, skills, subagents, and CLI vs MCP — and why KV cache hit rate is critical to agent system performance.

GPT-OSS Inference from Scratch

Tue, 03 Mar 2026 00:00:00 +0000

In this post, we implement GPT-OSS-20B inference from scratch in PyTorch. Building GPT-OSS shares a lot of common ground with building the Qwen3 family — if you haven’t read my Qwen3 Inference from Scratch and Qwen3 MoE from Scratch posts, I recommend starting there.

GPT-OSS is also a sparse MoE transformer with RMSNorm + GQA + RoPE + SwiGLU experts — the shared components work the same way. This post focuses on things GPT-OSS does differently

Qwen3 MoE Inference from Scratch

Fri, 27 Feb 2026 00:00:00 +0000

In this post, we implement Qwen3 MoE inference from scratch in PyTorch. It builds on the Qwen3 dense architecture — if you haven’t read my Qwen3 Inference from Scratch post, I recommend starting there, as it covers the full dense transformer pipeline (tokenizer, RMSNorm, RoPE, GQA, SwiGLU FFN, KV cache, generation). This post covers the key architectural advancement from dense to MoE: replacing the dense FFN with a Sparse Mixture of Experts layer. MoE is a more efficient architecture — it achieves better performance by scaling total parameters while keeping compute low, activating only a small subset of weights per token. Everything else — attention, normalization, tokenizer, generation loop — is identical.

Qwen3 Inference from Scratch

Sun, 15 Feb 2026 00:00:00 +0000

Better late than never — I’ve started a repo called llm-from-scratch, where I reimplement popular open-source LLMs piece by piece, with a focus on making every component’s structure clear and easy to follow. It’s meant to be a hands-on tutorial for beginners and a personal reference for myself. The first entry is a from-scratch implementation of Qwen3-0.6B in pure PyTorch — no Hugging Face transformers, no vllm, just raw tensor operations. This post walks through each core building block: tokenizer, RMSNorm, RoPE, FFN, GQA, transformer blocks, KV cache, prefill/decode, temperature sampling and batch processing — what each does, why it’s designed that way, and the non-obvious details I learned along the way.

Memorization vs Generalization

Fri, 12 Dec 2025 00:00:00 +0000

Today I revisit the classic LRU cache problem in my playground repo. The standard answer comes to my mind immediately: HashMap + Doubly Linked List. Cool, I remember that. But then I ask myself, what about LFU, B+Tree, and other complex data structures? I can barely remember and have to look up the script I developed when I’m at school.

Yeah, the real problem is that I’m memorizing the answer, not generalization. I suddenly realized that it’s also where LLMs suffer from: they are so good at memorizing answer, but not able to generalize.

工作三年，许多想说

Mon, 01 Sep 2025 00:00:00 +0000

最近读到laike9m <工作三年我学到了什么>，听完那期捕蛇者说博客，觉得自己的视角下有许多不同，想也来写一写自己的感想。我毕业之后就一直在一家硅谷小厂工作，到今年6月也满三年了。前两年的角色大致是DE/SDE，最近一年在转向AI/LLM。这篇想把主题扩展到工作之外，到生活，心态，团队，方方面面，也算一个阶段性的记录。

换方向是常有的事

换方向是常有的事，在小公司里更是。我刚进公司的时候是做维护data pipeline，花了6个月做了一次AWS EMR到databricks的迁移，稳定之后就几乎不需要时间维护了。随后便去接手了后端服务的性能优化，以这个契机接触了python / redis / mysql / elasticsearch后端技术栈，有段时间还去做了一些k8s hpa的调研。然后就是从一年前的夏天开始做AI一直到现在，现在也逐渐走上正路。如果在大公司里，这算不算三年换了两次组呢？我不太知道。但总体来说，我觉得十分满足，给了我太多跨领域学习的机会。

我现在不太会想这样的经历会不会不专注，这是从后视镜看到的看法。在那个当下，我能看到的只是：面前有一个机会，我要不要抓住它？能在某个领域一直做很久并成为那个领域的专家，是一件幸运的事，但在目前的公司里我不太能看到这种可能。这个时代本身就是在变化的，所以我更多把换方向看作为一种常态，毕竟在科技行业，变化是一定会发生的事，我们谁也不敢说能依靠现在的技术栈一直工作到退休。

playground是我的工具箱

在小公司一个比较大的问题是技术栈比较简单，使用的技术一般比较成熟，甚至老旧。带来的问题就是视野和发展受限，“在工作中学习”是一种奢望，很容易处于一种“好像没什么再可以学的”状态，但是回归到人才市场时，却发现自己一无所长。我慢慢想出了一种变通解决的方法，叫做“在工作外学习，在工作中实践”，即选定一些自己感兴趣的知识和领域，慢慢了解它、学习它，并在工作中尝试运用它。一般来说，老板对新技术的采用抱开放态度（毕竟能提升性能或者省钱，都是看得见的成果），至少能申请一些做实验的机会，如果能上到生产环境那就是一次从0到1使用的经验，即使不采纳，也可以为以后的工作做铺垫。我发现在小公司里这样的机会还真不少，如果能证明有显著提升，最终能上到生产环境的概率很高，比如这几年我做了：

[Y] (llm) ReAct agent
[N] (benchmark) valkey vs redis
[N] (benchmark) postgresql vs mysql
[Y] (full stack) 带计费功能的openai api proxy
[Y] (redis - client tracking) redis客户端缓存
[Y] (redis - redis stream) 基于redis的可靠消息队列
[Y] (mysql - replication) 基于mysql binlog的变动数据捕获 (change data capture)
[Y] (k8s - keda) 基于keda的k8s HPA
[N] (python - cpp extension) 用cpp写的trietree包，加速字符串匹配

有idea了，就花时间自己动手探索一下，在我的playground repo把教程跑一遍，或者做一个原型prototype看看情况，效果好的可以直接进行展示，效果不好也积累了知识。基本原则是小量试错，不花太多的时间就能得到初步结论，做一些笔记，然后拿着结论决定是否需要深入研究。这样长久积累下来我的playground repo就变成了一个大工具箱，里面有我的所有学习笔记和测试代码，可以随时clone下来作为知识库搜索。

Redis vs Valkey

Fri, 23 May 2025 00:00:00 +0000

Background

过去一年各大云服务厂商在Redis换license之后，力推使用Valkey代替Redis作为kv内存数据库的使用方案。Valkey作为Redis 7.2.4 fork，不仅兼容redis各个语言的client库，也解决了Redis目前存在的单线程性能瓶颈和内存占用问题，不禁好奇其性能有多大提升，周末找了个时间进行了性能benchmark。

已有youtube博主进行过benchmark，但由于方法不同结果仅供参考，可以与本文结果进行互相比较

关于Valkey相较于Redis的改进，主要有两个，一个是io-thread, 另一个是数据结构内存优化, 可以参见以下两篇blog post:

其中io-thread是最重要的改进，支持了Epoll job并行执行，建议配置

io-thread=<核心数>
events-per-io-thread=2 (default)
io-threads-do-reads=yes (default)

另外，需要注意Redis 7.2.4 之后引入的 feature 如 time series, vector db, valkey 可能没有很好的支持，参考Redis方面的post
Valkey实验版本支持了RDMA，不经过OS和CPU直接进行数据读取，继续提升性能，参考以下两篇:

Method

instance: RaspberryPi 5b, 4vcore 8GiB
kennel: linux 6.14.0-1005-raspi
redis: 8.0.0
valkey: 8.1.0
command: `redis-benchmark -h <host>`
client number: 50 (default)
request number: 100000 (default)

valkey单核: io-threads=1
valkey多核: io-threads=4

Result

Redis 8.0

Why MySQL replication is lagging

Tue, 09 Apr 2024 00:00:00 +0000

Background

Our team planned to upgrade one of our MySQL instance from 8.0.28 to 8.0.36 through AWS RDS blue/green deployment. The whole process includes 4 steps:

set up a new MySQL 8.0.36 instance (slave), recover data from snapshot
new MySQL instance (8.0.36) subscribe to old MySQL instance (8.0.28)
once there’s no replication lags, do switch-over. The downtime is usually < 10s.
check online service. shutdown old instance after checking everything is good.

However, We stuck at step 2. The replication lag kept growing and it seemed never to decrease, which made it impossible to start switch-over.

Lost connection error with aiomysql

Tue, 16 Jan 2024 00:00:00 +0000

Background

When I tried to run a big range query on mysql using aiomysql‘s SSDictCursor, everything looks fine at first but after a period of time (15min to 20min), the query stopped with error. The error message is:

Error Code 2013: Lost connection to MySQL server during query.

Environment

Database
MySQL version: 8.0.29
connect_timeout=10
net_read_timeout=30
net_write_timeout=60

Test Client
Linux kernal version: 6.1
Instance type: AWS t2.micro (1 vCPU, 1GiB RAM)
Python: 3.8.10
aiomysql: 0.1.1
PyMySQL: 1.1.0

Observation

What the code is doing is like

Spark executor OOM: not enough memory to build hash map

Tue, 08 Aug 2023 00:00:00 +0000

Background

One of the pipelines I maintained failed last week. The first try failed but the second try was successful. The pipeline’s structure is quite simple, it contains 3 steps, like most of the ETL pipelines:

Load data from database
Boardcast join a 2GiB dataset
Write to delta table

Error message is There is not enough memory to build the hash map. It happened in step 2.

Environment Details

Databricks Runtime: 11.3 LTS, Scala 2.12, Spark 3.3.0
Cluster: (AWS EC2 instances)
Driver: r5.xlarge · (32GiB memory, 4 vcores)
Workers: i3en.xlarge · 4 workers (32GiB memory, 4 vcores)
Spark config (selected):
spark.driver.memory 24g
spark.driver.maxResultSize 12g
spark.executor.cores 2
spark.executor.memory 12g
spark.executor.memoryOverhead 2048

Observation

Detailed error log and traceback:

Nginx reuseport导致偶发性卡顿

Mon, 05 Jun 2023 00:00:00 +0000

背景

摘抄自plantegg的blog

从2018年开始，我们有个业务陆续接到反馈 Nginx 线上集群经常出现不响应或者偶发性的“超慢”请求。这种卡顿每天都有少量出现。而只有多个集群中的一个出现，其他压力更大的集群皆未出现。

业务结构比较简单：LVS->Nginx->后端，如图

观察到的现象：

出问题前不久升级 Nginx 配置，打开了 reuseport 功能
在压力大的后端（upstream）服务环境不容易出现，后端压力轻对应的Nginx卡顿概率更高
关闭 reuseport 后问题少了很多
失败的请求响应时间都是 0ms（Nginx日志不靠谱了）
从 Nginx 日志上看，所有失败的健康检查请求都是0ms 的499 错误码（健康检查设置超时是2秒），但实际出问题的时候有5s-2分钟没有任何日志输出（Nginx卡了这么久）要么是Nginx卡住没去accept，要么是accept了没响应
所有超时来自同一个worker(一个Nginx服务一般按照机器核数开启多个worker)

并且已知，卡顿的原因是打开 reuseport 后，新进来的请求可以由内核 hash 派发给一个 Nginx woker ，避免了锁争抢以及惊群。但如果网络条件足够好，压力足够低，Nginx worker 一直来不及读完 receive buffer 中的内容时，就无法切换并处理其他的 request，于是在新请求的客户端会观测不间断的卡顿，而压力大的后端由于网络传输慢，经常卡顿，Nginx worker 反而有时间能处理别的请求。在调小 receive buffer 人为制造卡顿后该问题得以解决。

目标

由于所述场景比较复杂，缺乏直接证据，我打算通过构造一个较简单的环境来复现这个问题，并且在这个过程中抓包、观测 Nginx worker 的具体行为，验证这个假设。

术语

快连接和慢连接

短连接：通常是传输时间短、传输量小的连接，耗时通常是ms级别
长连接：通常是传输时间长、传输量大的连接，可以维持传输状态一段时间（如30s, 1min）

在本次场景复现过程中，这两种连接都是短连接，每次请求开始前都需要三次握手建立连接，结束后都需要四次挥手销毁连接

Epoll

Nginx使用了epoll模型，epoll 是多路复用的一种实现。在多路复用的场景下，一个task（process）会批量处理多个socket，哪个来了数据就去读那个。这就意味着要公平对待所有这些socket，不能阻塞在任何socket的”数据读”上，也就是说不能在阻塞模式下针对任何socket调用recv/recvfrom。

epoll 每次循环为O(1) 操作，循环前会得到一个就绪队列，其中包含所有已经准备好的 socket stream（有数据可读），不需要循环全部 socket stream 读取数据，在循环后会将被读取数据的 stream 重新放回睡眠队列。睡眠队列中的 socket stream 有数据可读时，再唤醒加入到就绪队列中。

Avoid using big strings as MySQL index key

Sat, 27 May 2023 00:00:00 +0000

Background

As what is explained in my pervious post, B+ tree depth decides the worst case of MySQL’s query performance, only considering those use index. And with the help of SSD, that should be fine to have a B+ tree deeper than 4 layers.

So the next question is: how bad the performance is possible to be? Here I intentionally design a table using very long string as primary key and index, which results in very deep B+ trees, up to 9 layers, to test its worst performance on SSD, which you are not expected to deal with in production.

Is 20M rows still a soft limit of MySQL table in 2023?

Mon, 22 May 2023 00:00:00 +0000

Rumor

There’s rumor around the internet that we should avoid having > 20M rows in a single MySQL table. Otherwise, the table’s performance will be downgraded, you will find SQL query much slower than usual when it’s above the soft limit. These judgements were made on HDD many years ago. I’m wondering if it’s still true for MySQL on SSD in 2023, and if true, why is that?

Environment

— Database  
MySQL version: 8.0.25  
Instance type: AWS db.r5.large (2vCPUs, 16GiB RAM)  
EBS storage type: General Purpose SSD (gp2)  
— Test Client  
Linux kernal version: 6.1  
Instance type: AWS t2.micro (1 vCPU, 1GiB RAM)

Experiment Design

Create tables with same schema but different in size. I created 9 tables with 100k, 200k, 500k, 1m, 2m, 5m, 10m, 20m, 30m, 50m, 60m of rows respectively.

为什么我的网络传输速度下降了

Sat, 06 May 2023 00:00:00 +0000

背景

这个问题一开始是在进行RDS实验的时候发现的。最初的情景是，多台机器同时对数据库进行select和insert操作时，会发现insert操作会造成select操作qps大幅下降，且insert操作结束之后select操作的qps仍不能回升。起初以为是RDS的问题，但是在复现问题、监控RDS之后发现RDS的压力其实很小。于是开始怀疑是网络的问题，在简化了场景和操作之后，发现能在过去做tcp实验的机器上复现，于是用这个更简单的场景进行问题复现和分析。

环境

linux kernal version: 6.1
linux image: amazon/al2023-ami-2023.0.20230419.0-kernel-6.1-x86_64
instance type: AWS t2.micro (1 vCPU, 1GiB RAM)
tc qdisc: fq_codel

无网络流量情况下：
network rtt: 0.5 ms
network bandwidth: 60 MB/s

操作

开三台 t2.micro 机器，一台做 server，两台做client，在server上放一个 2GB 大小的文件
client1 通过 curl get server 文件
等待一段时间（约20s），client1网速稳定后，client2 通过 curl get server 文件
可以观察到两种情况（如果无法复现请多试几次，就能复现）
正常情况：两个 client 速度稳定在30MB/s左右
降速情况：两个 client 速度降低到3.5MB/s左右
关掉client2，观察到client1恢复到7-8MB/s，但是远低于60MB/s的带宽上限
降速情况发生之后，client1/2重新通过curl 下载 server文件，就会出现起初网络速度在30-50MB/s，但10s后会降速7-8MB/s的情况，需要重启server才能恢复到60MB/s

分析

抓包

通过tcpdump抓包，获得了完整的降速过程中，server和两个client的网络包的传输情况，pcap文件已上传到Github

可以观察到：
1. rtt有明显上升，rtt从降速前的小于1ms上升到降速后的50-100ms
2. 两边rtt不对等，在降速后关掉client2, server->client1 tcp stream的rtt在server端和client端差异很大
猜测A：bufferbloat

Packet capture experiment 2

Sat, 15 Apr 2023 00:00:00 +0000

send/receive buffer

Goal

Get a better understanding of how send/receive buffer size may affect the network rtt and throughput. Usually send/receive buffer size is managed by kernal and application should not hardcode these values. However, real world cases are more rare and complicated. Inapporiate buffer size may be the root case of some network issues. Rare but possible.

Method

Instance

Instance provider: AWS
Instance type: t2.micro, 1vCPU, 1GiB memory, Low to Moderate network
Region: us-east-1b
OS: Amazon Linux 2023 AMI
Python: 3.9

Packet capture experiment 1

Tue, 11 Apr 2023 00:00:00 +0000

packet delay, loss, duplicate, corrupt, out-of-order and bandwidth limit

Goal

The goal of packet capture experiment here is to get a better understanding of TCP protocol. Besides, by reproducing the common network issues in the real world like packet delay, loss, out-of-order and brandwidth limit and saving a snapshot of packet capture result of circumstances above, we can identify and solve these problems more quickly next time.

Method

Instance

Instance provider: AWS
Instance type: t2.micro, 1vCPU, 1GiB memory, Low to Moderate network
Region: us-east-1b
OS: Amazon Linux 2023 AMI
Python: 3.9

Spot termination makes runtime unstable

Sat, 18 Mar 2023 00:00:00 +0000

Observation

Spark Job runtime is not stable

As what can see from the screenshot, the pipeline (spark job) runtime is not quite stable. However, the pipeline deals with almost fixed amount data everyday, the runtime should not be so quite unstable like this.

Besides, the pipeline runtime is quite stable at around 15min before migration to databricks.

Analysis

Spark driver log

23/03/16 19:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20230316185728-0000/2 is now EXITED (Worker shutting down)
23/03/16 19:04:26 WARN DLTDebugger: Failed to talk to RPC endpoint: dlt-debugger
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:454)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:110)
    at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:46)
    at org.apache.spark.debugger.DLTDebugger$.liftedTree1$1(DLTDebugger.scala:258)
    at org.apache.spark.debugger.DLTDebugger$.getDebuggerRef(DLTDebugger.scala:257)
    at org.apache.spark.debugger.DLTDebugger$.report(DLTDebugger.scala:341)
    at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$receive$1.$anonfun$applyOrElse$5(StandaloneAppClient.scala:188)
    at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$receive$1.applyOrElse(StandaloneAppClient.scala:184)
    at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:115)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
    at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
    at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
    at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.$anonfun$run$1(SparkThreadLocalForwardingThreadPoolExecutor.scala:110)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:41)
    at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:74)
    at org.apache.spark.util.threads.SparkThreadLocalCapturingHelper.runWithCaptured$(SparkThreadLocalForwardingThreadPoolExecutor.scala:60)
    at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.runWithCaptured(SparkThreadLocalForwardingThreadPoolExecutor.scala:107)
    at org.apache.spark.util.threads.SparkThreadLocalCapturingRunnable.run(SparkThreadLocalForwardingThreadPoolExecutor.scala:110)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: Cannot find endpoint: spark://dlt-debugger@10.241.97.77:45081
    at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1(NettyRpcEnv.scala:175)
    at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1$adapted(NettyRpcEnv.scala:171)
    at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at org.apache.spark.util.ThreadUtils$$anon$1.execute(ThreadUtils.scala:105)
    at scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
    at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
    at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallback(Promise.scala:316)
    at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:307)
    at scala.concurrent.impl.Promise.transformWith(Promise.scala:40)
    at scala.concurrent.impl.Promise.transformWith$(Promise.scala:38)
    at scala.concurrent.impl.Promise$DefaultPromise.transformWith(Promise.scala:187)
    at scala.concurrent.Future.flatMap(Future.scala:306)
    at scala.concurrent.Future.flatMap$(Future.scala:306)
    at scala.concurrent.impl.Promise$DefaultPromise.flatMap(Promise.scala:187)
    at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:177)
    ... 25 more
23/03/16 19:04:26 INFO StandaloneSchedulerBackend: Executor app-20230316185728-0000/2 removed: Worker shutting down
23/03/16 19:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20230316185728-0000/4 on worker-20230316185728-10.241.99.108-43045 (10.241.99.108:43045) with 4 core(s)
23/03/16 19:04:26 INFO StandaloneSchedulerBackend: Granted executor ID app-20230316185728-0000/4 on hostPort 10.241.99.108:43045 with 4 core(s), 12.0 GiB RAM
23/03/16 19:04:26 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20230316185728-0000/4 is now FAILED (java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.)
23/03/16 19:04:26 INFO StandaloneSchedulerBackend: Executor app-20230316185728-0000/4 removed: java.lang.IllegalStateException: Shutdown hooks cannot be modified during shutdown.

Spark log4j-active log shows that there’s excutor exit. But no reason for that.

Spark driver OOM: Broadcast variable is too big

Wed, 15 Mar 2023 00:00:00 +0000

Observation

Spark cluster master node OOM

Master node shutdown due to OOM. Error log:

INFO Data stored in hdfs:///XXXX
INFO XXXXX updated
INFO Data has XXXXX records
INFO Data stored in hdfs:///XXXX
INFO XXXXX updated
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 *****"...

Spark cluster worker node shutdown

Worker nodes worked well until driver shutdown. Error log:

ERROR YarnCoarseGrainedExecutorBackend: Executor self-exiting due to : Driver ip-***-***-***-***.ec2.internal:***** disassociated! Shutting down.
INFO MemoryStore: MemoryStore cleared
INFO BlockManager: BlockManager stopped
ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

Analysis

Look up driver log

Redis OOM due to big keys

Thu, 16 Feb 2023 00:00:00 +0000

Observation

Queue client performance went down

Everything worked well in the morning until we got monitoring alarm at 12:45 PM EST: <topic_name> Queue lag is too big. The issue happened in a service which is responsible for consuming message from message queue, processing message and writing it to database. The service also uses Redis to cache some objects which can be reused every time it processes the message. Auto-scaling rule is applied to the service so there’ll be tens to hundreds of pods running under heavy workload. We observed that the consumer speed went down by 50%, causing messages to be backlogged in the queue.

MySQL connection deadlock

Fri, 10 Feb 2023 00:00:00 +0000

Observation

CronJob is taking more than 1h to complete

My colleagues told me that one of the cron job stuck in the middle after a random day. They received the warning: CronJob is taking more than 1h to complete. And the pod kept stucking there after a day, which is abnormal. However, another cron job which almost uses the same code works well. No database failure was reported during the period of time.

Why signal file is a bad idea

Fri, 13 Jan 2023 00:00:00 +0000

Observation

Signal file is widely used in Hadoop ecosystem. If you have experience with MapReduce, you’ll notice that by default MapReduce runtime writes an empty _SUCCESS file to mark successful completion of a job to the output folder. AWS DataPipeline and Databricks also support “file arrival” to trigger a downstream job.

Question

Is signal file a good architecture design?

Can I use _SUCCESS created by MapReduce as signal file to trigger downstream job?

About Me

Sun, 01 Jan 2023 00:00:00 +0000

My name is Yisheng Gong.

Software Engineer@YipitData

Based in Santa Clara, California.

Recently working on Agent / RL Infra

I love open source and I’m a committer of sglang, miles and verl

And here’s some personal projects I built during my spare time:

playground - Learning by breaking things
llm-from-scratch - Build LLMs from scratch
writings - Short stories and poetry

Find me here:

Hope you will love my posts