Log directory offline on AWS EBS

Hello,

We are encountering such issues on AWS EBS based system, nothing in AWS
logs.

Is it memory or IO issue, as it states in one line

in dir /srv/kafka/disk1 due to IOException
(kafka.server.LogDirFailureChannel)
java.io.IOException: Map failed

and in second

Caused by: java.lang.OutOfMemoryError: Map failed


> [2022-02-07 21:35:43,046] WARN [ReplicaManager broker=77] Unable to
> describe replica dirs for /srv/kafka/disk1 (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.KafkaStorageException: Log directory
> /srv/kafka/disk1 is offline


[2022-02-07 21:34:54,355] ERROR Failed to clean up log for
> invent_points_of_interest-39 in dir /srv/kafka/disk1 due to IOException
> (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed at
> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938) at kafka.log.
> AbstractIndex.<init>(AbstractIndex.scala:124) at kafka.log.OffsetIndex.<
> init>(OffsetIndex.scala:54) at kafka.log.LazyIndex$.$anonfun$forOffset$1(
> LazyIndex.scala:106) at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex
> .scala:63) at kafka.log.LazyIndex.get(LazyIndex.scala:60) at kafka.log.
> LogSegment.offsetIndex(LogSegment.scala:64) at kafka.log.Cleaner
> .buildOffsetMapForSegment(LogCleaner.scala:932) at kafka.log.Cleaner.
> $anonfun$buildOffsetMap$5(LogCleaner.scala:908) at kafka.log.Cleaner.
> $anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904) at
> scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563) at
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:919) at
> scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889) at
> kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904) at kafka.log.
> Cleaner.doClean(LogCleaner.scala:523) at kafka.log.Cleaner.clean(
> LogCleaner.scala:511) at kafka.log.LogCleaner$CleanerThread.cleanLog(
> LogCleaner.scala:380) at kafka.log.LogCleaner$CleanerThread
> .cleanFilthiestLog(LogCleaner.scala:352) at kafka.log.LogCleaner
> $CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332) at kafka.log.
> LogCleaner$CleanerThread.doWork(LogCleaner.scala:321) at kafka.utils.
> ShutdownableThread.run(ShutdownableThread.scala:96) Caused by: java.lang.
> OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native
> Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935) ... 21
> more

[2022-02-07 21:34:54,356] WARN [ReplicaManager broker=77] Stopping serving
replicas in dir /srv/kafka/disk1 (kafka.server.ReplicaManager) -02-07 21:34:
54,588] WARN [ReplicaManager broker=77] While recording the replica LEO,
the partition multibooks-250 hasn't been created. (kafka.server.
ReplicaManager) [2022-02-07 21:34:54,696] WARN Stopping serving logs in dir
/srv/kafka/disk1 (kafka.log.LogManager)



>