Introduction
One day, Cassandra stop listening for thrift client until restart it manually.
After checking Cassandra log, found it encountered OutOfMemoryError
After checking Cassandra log, found it encountered OutOfMemoryError
ERROR [Thrift-Selector_27] 2020-03-31 06:00:44,020 TDisruptorServer.java (line 391) run() exiting due to uncaught error
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:695)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
at sun.nio.ch.IOUtil.write(IOUtil.java:58)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:164)
at com.thinkaurelius.thrift.util.mem.Buffer.writeTo(Buffer.java:104)
at com.thinkaurelius.thrift.util.mem.FastMemoryOutputTransport.streamTo(FastMemoryOutputTransport.java:112)
at com.thinkaurelius.thrift.Message.write(Message.java:222)
at com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.handleWrite(TDisruptorServer.java:598)
at com.thinkaurelius.thrift.TDisruptorServer$SelectorThread.processKey(TDisruptorServer.java:569)
at com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.select(TDisruptorServer.java:423)
at com.thinkaurelius.thrift.TDisruptorServer$AbstractSelectorThread.run(TDisruptorServer.java:383)
DirectByteBuffer.<init>
DirectByteBuffer is used to handle off heap memory.
We can see it needs reserveMemory in constructor by calling Bits.reserveMemory(size, cap)
We can see it needs reserveMemory in constructor by calling Bits.reserveMemory(size, cap)
// My env: jdk_1.8.0_211
class DirectByteBuffer extends MappedByteBuffer implements DirectBuffer {
...
DirectByteBuffer(int cap) { // package-private
super(-1, 0, cap, cap);
boolean pa = VM.isDirectMemoryPageAligned();
int ps = Bits.pageSize();
long size = Math.max(1L, (long)cap + (pa ? ps : 0));
Bits.reserveMemory(size, cap);
long base = 0;
try {
base = unsafe.allocateMemory(size);
} catch (OutOfMemoryError x) {
Bits.unreserveMemory(size, cap);
throw x;
}
unsafe.setMemory(base, size, (byte) 0);
if (pa && (base % ps != 0)) {
// Round up to page boundary
address = base + ps - (base & (ps - 1));
} else {
address = base;
}
cleaner = Cleaner.create(this, new Deallocator(base, size, cap));
att = null;
...
}
Bits.reserveMemory
Bits.java is used to note memory usage.
Don't allocate memory by unsafe class until making sure Bits shows reserveMemory success.
Don't allocate memory by unsafe class until making sure Bits shows reserveMemory success.
In this code, we will find "totalCapacity" is used to check the capacity is enough or not.
It will be used to compare with "maxMemory"
It will be used to compare with "maxMemory"
The "maxMemory" comes from VM.maxDirectMemory()
The value of "VM.maxDirectMemory" can be configured by "-XX:MaxDirectMemorySize=<size>" when start up JVM
class Bits {
// -- Direct memory management --
// A user-settable upper limit on the maximum amount of allocatable
// direct buffer memory. This value may be changed during VM
// initialization if it is launched with "-XX:MaxDirectMemorySize=<size>".
private static volatile long maxMemory = VM.maxDirectMemory();
private static final AtomicLong reservedMemory = new AtomicLong();
private static final AtomicLong totalCapacity = new AtomicLong();
private static final AtomicLong count = new AtomicLong();
private static volatile boolean memoryLimitSet = false;
// max. number of sleeps during try-reserving with exponentially
// increasing delay before throwing OutOfMemoryError:
// 1, 2, 4, 8, 16, 32, 64, 128, 256 (total 511 ms ~ 0.5 s)
// which means that OOME will be thrown after 0.5 s of trying
private static final int MAX_SLEEPS = 9;
// These methods should be called whenever direct memory is allocated or
// freed. They allow the user to control the amount of direct memory
// which a process may access. All sizes are specified in bytes.
static void reserveMemory(long size, int cap) {
if (!memoryLimitSet && VM.isBooted()) {
maxMemory = VM.maxDirectMemory();
memoryLimitSet = true;
}
// optimist!
if (tryReserveMemory(size, cap)) {
return;
}
final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess();
// retry while helping enqueue pending Reference objects
// which includes executing pending Cleaner(s) which includes
// Cleaner(s) that free direct buffer memory
while (jlra.tryHandlePendingReference()) {
if (tryReserveMemory(size, cap)) {
return;
}
}
// trigger VM's Reference processing
System.gc();
// a retry loop with exponential back-off delays
// (this gives VM some time to do it's job)
boolean interrupted = false;
try {
long sleepTime = 1;
int sleeps = 0;
while (true) {
if (tryReserveMemory(size, cap)) {
return;
}
if (sleeps >= MAX_SLEEPS) {
break;
}
if (!jlra.tryHandlePendingReference()) {
try {
Thread.sleep(sleepTime);
sleepTime <<= 1;
sleeps++;
} catch (InterruptedException e) {
interrupted = true;
}
}
}
// no luck
throw new OutOfMemoryError("Direct buffer memory");
} finally {
if (interrupted) {
// don't swallow interrupts
Thread.currentThread().interrupt();
}
}
}
private static boolean tryReserveMemory(long size, int cap) {
// -XX:MaxDirectMemorySize limits the total capacity rather than the
// actual memory usage, which will differ when buffers are page
// aligned.
long totalCap;
while (cap <= maxMemory - (totalCap = totalCapacity.get())) {
if (totalCapacity.compareAndSet(totalCap, totalCap + cap)) {
reservedMemory.addAndGet(size);
count.incrementAndGet();
return true;
}
}
return false;
}
static void unreserveMemory(long size, int cap) {
long cnt = count.decrementAndGet();
long reservedMem = reservedMemory.addAndGet(-size);
long totalCap = totalCapacity.addAndGet(-cap);
assert cnt >= 0 && reservedMem >= 0 && totalCap >= 0;
}
...
}
VM.maxDirectMemory()
Go to VM.java, will find default maxDirectMemory is 64MB
In VM.java, it shows
- No "sun.nio.MaxDirectMemorySize" configured, use default 64MB
- Config "sun.nio.MaxDirectMemorySize" to -1, use Runtime.getRuntime().maxMemory()
- Except use user specified memory size
- The MaxDirectMemorySize is shared by whole process, which means when there are many threads to create DirectByteBuffer with capacity, OutOfMemoryError will be easy to happen when it exceed 64MB.Can consider increase the size if the loading is expected
public class VM {
...
public static void saveAndRemoveProperties(Properties var0) {
if (booted) {
throw new IllegalStateException("System initialization has completed");
} else {
savedProps.putAll(var0);
String var1 = (String)var0.remove("sun.nio.MaxDirectMemorySize");
if (var1 != null) {
if (var1.equals("-1")) {
directMemory = Runtime.getRuntime().maxMemory();
} else {
long var2 = Long.parseLong(var1);
if (var2 > -1L) {
directMemory = var2;
}
}
}
var1 = (String)var0.remove("sun.nio.PageAlignDirectMemory");
if ("true".equals(var1)) {
pageAlignDirectMemory = true;
}
var1 = var0.getProperty("sun.lang.ClassLoader.allowArraySyntax");
allowArraySyntax = var1 == null ? defaultAllowArraySyntax : Boolean.parseBoolean(var1);
var0.remove("java.lang.Integer.IntegerCache.high");
var0.remove("sun.zip.disableMemoryMapping");
var0.remove("sun.java.launcher.diag");
var0.remove("sun.cds.enableSharedLookupCache");
}
}
...
}
About heap OutOfMemoryError
Can config to do something when OutOfMemoryError
-XX:OnOutOfMemoryError=/restart.sh"
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/yourpath"
