[{"content":"In a Spring Boot service organized by technical layers (controller, service, repository, entity…), the business rule ends up next to JPA, Kafka and Jackson code. To test it, you start Spring, or you mock a large part of the framework.\nThis post shows another way to organize the code, the hexagonal architecture, on a small and complete example: a service that reads sensor readings from Kafka, checks them and computes statistics. The domain and the use cases are plain Java, Spring, Kafka and JPA stay in adapters, and a test makes the build fail if this rule is broken. The code is in the tag phase-1 of the public repository iot-monitoring-platform, presented in the post Kafka and Spring Boot: a complete local project.\nThe idea in short The hexagonal architecture comes from an article by Alistair Cockburn (2005), who also calls it « Ports and Adapters ». The idea fits in three words:\nthe core (the domain and the use cases) holds the rules of the application, in simple code, without a framework; the ports are interfaces: what the core offers (input ports) and what it needs (output ports); the adapters connect the ports to the real world: a REST controller, a Kafka listener, a JPA repository. Dependencies always point to the inside: an adapter knows the core, the core knows no adapter.\nThe hexagonal architecture of the service iot-backend-processing: in the middle, the domain and the use cases in plain Java; at the bottom of the hexagon, the input and output ports; around it, the Kafka, REST and JPA adapters. All dependencies point to the inside. Use cases StoreMessageService ListStatisticsService… Domain SensorReading ParseResult (sealed) SensorStatistics input ports StoreMessageUseCase… output ports MessageRepository StatisticsRepository Input adapters SensorMessageListener reads Kafka MessageController… REST API call Output adapters JpaMessageRepository JpaStatisticsRepository PostgreSQL implement Dependencies point to the inside: the core knows no adapter. The service of the example, iot-backend-processing, is small on purpose: it receives a line of text from Kafka (for example sensor-1 TEMPERATURE 21.5), checks it with a business rule, stores the message, updates the statistics of the sensor, and gives them through a REST API. This is enough to see each part of the architecture, with no noise around it.\nBefore, after Before this work, the service was organized by technical layers (public history of the project, commit 82b4054):\nprocessing/ BkProcessingApplication.java config/ CorsConfig, KafkaConsumerConfig controller/ StatsController dto/ SensorEvent, SensorEventEnvelope, SensorStats entity/ SensorReading kafka/ SensorEventConsumer repository/ SensorReadingRepository service/ AggregationService util/ PodInfo In the tag phase-1:\nprocessing/ ProcessingApplication.java domain/ SensorReading, ParseResult, SensorStatistics, SensorType, ReceivedMessage, MessageStatus application/ port/in/ StoreMessageUseCase, ListLastMessagesUseCase, ListStatisticsUseCase, InvalidLimitException port/out/ MessageRepository, StatisticsRepository StoreMessageService, ListLastMessagesService, ListStatisticsService adapter/ in/kafka/ SensorMessageListener in/web/ MessageController, StatisticsController + responses out/persistence/ JPA entities, Spring Data repositories, adapters config/ ProcessingConfig The two versions do not have the same features: the old one passed pod info and JSON envelopes along, with no business rule; the new one reads and checks each line. So we do not compare the number of classes, only the shape. Before, the name of a package said a technology (kafka, repository); now it says a role.\nThe domain: the rule, in plain Java A valid line has three parts: a sensor, a type and a value inside the range of its type. The rule is in SensorReading.parse:\npublic static ParseResult parse(String text) { if (text == null || text.isBlank()) { return new ParseResult.Rejected(\u0026#34;empty line\u0026#34;); } String[] parts = text.strip().split(\u0026#34;\\\\s+\u0026#34;); if (parts.length != 3) { return new ParseResult.Rejected(\u0026#34;a line needs 3 parts: sensor, type and value\u0026#34;); } String sensorId = parts[0]; if (!SENSOR_ID.matcher(sensorId).matches()) { return new ParseResult.Rejected(\u0026#34;invalid sensor id \u0026#34; + sensorId); } Optional\u0026lt;SensorType\u0026gt; type = SensorType.fromName(parts[1]); if (type.isEmpty()) { return new ParseResult.Rejected(\u0026#34;unknown type \u0026#34; + parts[1]); } if (!DECIMAL.matcher(parts[2]).matches()) { return new ParseResult.Rejected(\u0026#34;invalid value \u0026#34; + parts[2]); } double value = Double.parseDouble(parts[2]); if (!type.get().accepts(value)) { return new ParseResult.Rejected(type.get().name() + \u0026#34; must be between \u0026#34; + type.get().min() + \u0026#34; and \u0026#34; + type.get().max()); } return new ParseResult.Valid(new SensorReading(sensorId, type.get(), value)); } For a service that receives sensor data, a bad line is a normal case, not a bug of the program. So the method does not throw an exception: it returns a result, one of the two cases of a sealed interface (Java 17):\npublic sealed interface ParseResult { record Valid(SensorReading reading) implements ParseResult { } record Rejected(String reason) implements ParseResult { } } The compiler knows all the cases: a switch on a ParseResult (pattern matching for switch, Java 21) must handle Valid and Rejected, or it does not compile.\nThe statistics of a sensor are an immutable record. Each reading gives new statistics, without reading the past readings again:\npublic SensorStatistics add(SensorReading reading) { if (!sensorId.equals(reading.sensorId()) || type != reading.type()) { throw new IllegalArgumentException(\u0026#34;The reading of \u0026#34; + reading.sensorId() + \u0026#34; \u0026#34; + reading.type() + \u0026#34; does not belong to the statistics of \u0026#34; + sensorId + \u0026#34; \u0026#34; + type); } return new SensorStatistics(sensorId, type, count + 1, Math.min(min, reading.value()), Math.max(max, reading.value()), sum + reading.value()); } Here, the exception makes sense: adding the reading of another sensor would be a bug of the program, not bad data.\nIn the whole domain package, there is no import of Spring, Kafka, JPA or Jackson.\nThe ports and the use cases The input port says what the service offers:\npublic interface StoreMessageUseCase { ReceivedMessage store(String text, int partition, long offset); } The output ports say what it needs, without saying how. For example, for the statistics:\npublic interface StatisticsRepository { Optional\u0026lt;SensorStatistics\u0026gt; find(String sensorId, SensorType type); void save(SensorStatistics statistics); List\u0026lt;SensorStatistics\u0026gt; findAll(); } The use case connects the rule and the ports:\npublic class StoreMessageService implements StoreMessageUseCase { private final MessageRepository messageRepository; private final StatisticsRepository statisticsRepository; private final Clock clock; // constructor: the three fields above @Override public ReceivedMessage store(String text, int partition, long offset) { ParseResult result = SensorReading.parse(text); ReceivedMessage stored = messageRepository.save( ReceivedMessage.notStoredYet(text, partition, offset, Instant.now(clock), result)); if (result instanceof ParseResult.Valid valid) { updateStatistics(valid.reading()); } return stored; } private void updateStatistics(SensorReading reading) { SensorStatistics statistics = statisticsRepository.find(reading.sensorId(), reading.type()) .map(current -\u0026gt; current.add(reading)) .orElseGet(() -\u0026gt; SensorStatistics.first(reading)); statisticsRepository.save(statistics); } } No @Service, no @Autowired: it is a Java class. The time comes from a Clock given to the constructor, so a test can fix it. Spring creates it in the config class, with the adapters that implement the ports:\n@Bean public StoreMessageUseCase storeMessageUseCase(MessageRepository messageRepository, StatisticsRepository statisticsRepository, Clock clock) { return new StoreMessageService(messageRepository, statisticsRepository, clock); } All the Spring wiring is in config/. The rest of the core does not know that Spring exists.\nThe adapters The Kafka listener is an input adapter: it turns a Kafka message into a call of the input port.\n@KafkaListener(topics = \u0026#34;${iot.kafka.topic}\u0026#34;) @Transactional public void onMessage(ConsumerRecord\u0026lt;String, String\u0026gt; consumerRecord) { ReceivedMessage stored = storeMessage.store( consumerRecord.value(), consumerRecord.partition(), consumerRecord.offset()); // … log of the result } The transaction is opened here, in the adapter: the message and the statistics are saved together or not at all, and the use case stays plain Java. This works because the Spring Kafka container calls the listener through the Spring bean: in proxy mode (the default mode), Spring intercepts only the calls that go through the proxy.\nThe JPA adapters are output adapters. The JPA entity (StatisticsEntity, with its @Entity, @Column… annotations) is a different class from the domain record. The adapter translates:\n@Override public void save(SensorStatistics statistics) { StatisticsEntity entity = springDataRepository .findBySensorIdAndSensorType(statistics.sensorId(), statistics.type()) .orElseGet(() -\u0026gt; new StatisticsEntity(statistics.sensorId(), statistics.type())); entity.update(statistics.count(), statistics.min(), statistics.max(), statistics.sum()); springDataRepository.save(entity); } In the same way, the REST API returns response records (StatisticsResponse, MessageResponse), not the domain objects: the shape of the JSON can change without touching the domain.\nThe tests this allows The domain, with nothing to start One test line per kind of bad line (excerpt):\n@ParameterizedTest(name = \u0026#34;[{0}] -\u0026gt; {1}\u0026#34;) @CsvSource(delimiter = \u0026#39;|\u0026#39;, value = { \u0026#34;sensor-1 WIND 12 | unknown type WIND\u0026#34;, \u0026#34;sensor-1 TEMPERATURE 21,5 | invalid value 21,5\u0026#34;, \u0026#34;sensor-1 HUMIDITY -1 | HUMIDITY must be between 0 and 100\u0026#34;}) void parse_badLine_returnsRejectionWithReason(String text, String reason) { assertThat(SensorReading.parse(text)).isEqualTo(new ParseResult.Rejected(reason)); } The use case, with fake ports The ports are interfaces: a test gives them an in-memory version, and a fixed clock:\nprivate final FakeMessageRepository messageRepository = new FakeMessageRepository(); private final FakeStatisticsRepository statisticsRepository = new FakeStatisticsRepository(); private final StoreMessageService service = new StoreMessageService(messageRepository, statisticsRepository, Clock.fixed(NOW, ZoneOffset.UTC)); @Test void store_rejectedLine_savesReasonAndKeepsStatistics() { service.store(\u0026#34;sensor-1 TEMPERATURE 20\u0026#34;, 0, 0L); ReceivedMessage stored = service.store(\u0026#34;sensor-1 WIND 12\u0026#34;, 0, 1L); assertThat(stored.status()).isEqualTo(MessageStatus.REJECTED); assertThat(stored.rejectionReason()).isEqualTo(\u0026#34;unknown type WIND\u0026#34;); assertThat(statisticsRepository.findAll()) .containsExactly(new SensorStatistics(\u0026#34;sensor-1\u0026#34;, SensorType.TEMPERATURE, 1, 20, 20, 20)); } No Spring, no Kafka, no database, no Mockito. The tests of the domain and of the use cases:\nmvn -pl iot-backend-processing test -Dtest=\u0026#39;SensorReadingTest,SensorStatisticsTest,StoreMessageServiceTest,ListLastMessagesServiceTest\u0026#39; [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 s -- in dev.aymenjamaaoui.iot.processing.application.StoreMessageServiceTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 s -- in dev.aymenjamaaoui.iot.processing.application.ListLastMessagesServiceTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s -- in dev.aymenjamaaoui.iot.processing.domain.SensorStatisticsTest [INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 s -- in dev.aymenjamaaoui.iot.processing.domain.SensorReadingTest [INFO] Tests run: 35, Failures: 0, Errors: 0, Skipped: 0 35 tests, each one in a few milliseconds. The adapters have their own tests: a @WebMvcTest for the REST API, and an integration test with a real Kafka and a real PostgreSQL (Testcontainers).\nThe rule, checked by the build An architecture rule written only in a README ends up forgotten. The project checks it with ArchUnit:\n@AnalyzeClasses(packages = \u0026#34;dev.aymenjamaaoui.iot.processing\u0026#34;, importOptions = ImportOption.DoNotIncludeTests.class) class ArchitectureTest { @ArchTest static final ArchRule DOMAIN_AND_APPLICATION_HAVE_NO_FRAMEWORK = noClasses() .that().resideInAnyPackage(\u0026#34;..domain..\u0026#34;, \u0026#34;..application..\u0026#34;) .should().dependOnClassesThat().resideInAnyPackage( \u0026#34;org.springframework..\u0026#34;, \u0026#34;org.apache.kafka..\u0026#34;, \u0026#34;jakarta..\u0026#34;, \u0026#34;org.hibernate..\u0026#34;, \u0026#34;com.fasterxml.jackson..\u0026#34;, \u0026#34;tools.jackson..\u0026#34;); } To see it fail, I add @Component (Spring) to the record SensorStatistics on purpose, then I run mvn -pl iot-backend-processing test -Dtest=ArchitectureTest (short output):\nArchitecture Violation [Priority: MEDIUM] - Rule \u0026#39;no classes that reside in any package [\u0026#39;..domain..\u0026#39;, \u0026#39;..application..\u0026#39;] should depend on classes that reside in any package [\u0026#39;org.springframework..\u0026#39;, \u0026#39;org.apache.kafka..\u0026#39;, \u0026#39;jakarta..\u0026#39;, \u0026#39;org.hibernate..\u0026#39;, \u0026#39;com.fasterxml.jackson..\u0026#39;, \u0026#39;tools.jackson..\u0026#39;]\u0026#39; was violated (1 times): Class \u0026lt;dev.aymenjamaaoui.iot.processing.domain.SensorStatistics\u0026gt; is annotated with \u0026lt;org.springframework.stereotype.Component\u0026gt; in (SensorStatistics.java:0) [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 The message gives the rule, the class and the cause. The build fails, so nobody can break the rule without seeing it.\nThe cost, honestly This way of organizing the code has a price:\nmore classes: the service has 28 classes in src/main, with 6 ports (interfaces and one exception) and 2 JPA entities next to the domain records; translation code: JPA entity ↔ domain record ↔ REST response; more files to open to follow a call, from the listener to the database. On such a small example, the structure can look heavy: here it is on purpose, so that each part is visible. It pays when the service grows: when the rules grow, when a technology changes (another database, another way to receive the messages), or when you want to test the rule fast and often. For a service that only copies data from one system to another, with no real rule, a simpler structure is enough.\nTested with Java 25 (Temurin 25.0.4), Spring Boot 4.1.2, ArchUnit 1.5.1, JUnit 5, AssertJ, Maven 3.10.0. Code: tag phase-1, service iot-backend-processing.\nSources Alistair Cockburn, « Hexagonal architecture » (the 2005 article, « Ports and Adapters ») JEP 409: Sealed Classes (Java 17) JEP 441: Pattern Matching for switch (Java 21) ArchUnit, user guide (package dependency checks) Spring Framework: Using @Transactional (proxy mode) Further reading The previous post of the series: Kafka and Spring Boot: a complete local project. The README of the tag phase-1: the business rule, the API and all the tests. ","permalink":"https://aymenjamaaoui.dev/en/blog/hexagonal-architecture-spring-boot/","summary":"\u003cp\u003eIn a Spring Boot service organized by technical layers (\u003ccode\u003econtroller\u003c/code\u003e,\n\u003ccode\u003eservice\u003c/code\u003e, \u003ccode\u003erepository\u003c/code\u003e, \u003ccode\u003eentity\u003c/code\u003e…), the business rule ends up next to\nJPA, Kafka and Jackson code. To test it, you start Spring, or you mock a\nlarge part of the framework.\u003c/p\u003e\n\u003cp\u003eThis post shows another way to organize the code, the hexagonal\narchitecture, on a small and complete example: a service that reads\nsensor readings from Kafka, checks them and computes statistics. The\ndomain and the use cases are plain Java, Spring, Kafka and JPA stay in\nadapters, and a test makes the build fail if this rule is broken. The\ncode is in the tag\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/iot-monitoring-platform/-/tree/phase-1\"\u003e\u003ccode\u003ephase-1\u003c/code\u003e\u003c/a\u003e\nof the public repository iot-monitoring-platform, presented in the post\n\u003ca href=\"/en/blog/kafka-spring-boot-locally/\"\u003eKafka and Spring Boot: a complete local project\u003c/a\u003e.\u003c/p\u003e","title":"Hexagonal architecture with Spring Boot: a simple example"},{"content":"One service receives the readings of sensors; another one must process and store them. If the first one calls the second one over REST, the two are tied: when the second one is stopped or slow, the first one cannot deliver anything. Kafka sits between them: the first one publishes a message, the second one reads it at its own pace.\nThis post builds that link from end to end, locally: Kafka in KRaft mode in Docker, a Spring Boot service that publishes a plain line of text, another one that reads it, and a page that shows the partition and the offset of each message. All the code is in the tag phase-1 of the public repository iot-monitoring-platform.\nThe project in one diagram The flow of the project: the simulator sends a line of text to iot-backend-ingestion, which publishes it to the Kafka topic iot-sensor-events; iot-backend-processing reads the topic, stores the message in PostgreSQL and gives it to the Angular page. Kafka and PostgreSQL run in Docker. Docker (compose.yaml) Simulator iot-backend- ingestion :8081 Kafka 4.4 (KRaft) topic iot-sensor-events partition 0 0 1 2 3 … offsets iot-backend- processing :8082 Angular page PostgreSQL 18 POST text publishes (String) reads (group iot-backend-processing) JPA GET /api/messages iot-backend-ingestion (port 8081) receives a line of text, for example sensor-1 TEMPERATURE 21.5, and publishes it to the topic iot-sensor-events. iot-backend-processing (port 8082) reads the topic, stores each message in PostgreSQL and gives it through a REST API. A simulator sends one line per second, and an Angular page shows the last messages. The message is a plain String, with no key, on a topic with only one partition. This is on purpose: the goal is to see the full path of a message, with as few settings as possible. This post is only about the Kafka part; the business rule and the way the code is organized (hexagonal architecture) are the topic of another post.\nAt the end, you have Kafka and PostgreSQL in Docker, the two services on your machine, and messages whose partition and offset you can follow, from the send to the database.\nKafka in KRaft mode with the official image Since Kafka 4.0, Kafka runs fully without ZooKeeper: in KRaft mode, controller nodes of Kafka manage the metadata of the cluster. On a dev machine, one node can be both broker and controller, so one container is enough.\nThe project uses the official image of the Apache Kafka project, apache/kafka. With its default config, it starts one node, which is both broker and controller, and listens on port 9092. The compose.yaml file of the project (excerpt):\nservices: kafka: # Official Apache Kafka image. With its default config it starts one # node in KRaft mode (no ZooKeeper) and listens on localhost:9092. image: apache/kafka:4.4.0 container_name: iot-kafka ports: - \u0026#34;9092:9092\u0026#34; No environment variable: no KAFKA_ZOOKEEPER_CONNECT, no list of listeners to write. The same file also starts PostgreSQL.\ndocker compose up -d The logs of the container show what the default config does (docker logs iot-kafka, excerpt):\nadvertised.listeners = [PLAINTEXT://localhost:9092] node.id = 1 process.roles = [broker, controller] process.roles = [broker, controller]: one node has both roles. advertised.listeners is the address that Kafka gives to the clients: localhost:9092. The Spring Boot services run on the machine, so this address works for them as it is.\nPublish a String The config The application.yml file of the ingestion service:\nspring: application: name: iot-backend-ingestion kafka: # Spring Boot uses String serializers by default: the message is a plain String. bootstrap-servers: localhost:9092 iot: kafka: topic: iot-sensor-events There is no Kafka config class in the project. Spring Boot creates a KafkaTemplate, and its default serializers are StringSerializer, for the key and for the value. So there is nothing to set to publish text. bootstrap-servers is already localhost:9092 by default; I write it anyway, so you can see where the service connects.\nThe send The adapter that publishes (excerpt of KafkaSensorMessagePublisher):\n@Override public void publish(SensorMessage message) { // No key: in phase 1 the topic has one partition, so the order is // kept anyway. The key comes in phase 2, with several partitions. kafkaTemplate.send(topic, message.text()) .whenComplete((result, exception) -\u0026gt; { // The send is asynchronous: an error only shows in the log. // Error handling (retry, DLT) comes in phase 2. if (exception != null) { LOGGER.error(\u0026#34;Could not publish the message \u0026#39;{}\u0026#39; to {}\u0026#34;, message.text(), topic, exception); } else { LOGGER.info(\u0026#34;Published \u0026#39;{}\u0026#39; to {} (partition {}, offset {})\u0026#34;, message.text(), topic, result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } }); } send does not block: it returns a CompletableFuture\u0026lt;SendResult\u0026gt;. The result comes later, with the partition and the offset that Kafka gave to the message. This is why the REST controller answers 202 Accepted and not 200 OK: the service handed the message to Kafka, but it does not wait for the confirmation. The limit is clear: if the send fails, the error only shows in the log. Error handling comes with phase 2.\nThe topic The topic is declared by a NewTopic bean (excerpt of IngestionConfig):\n@Bean public NewTopic sensorEventsTopic(IotKafkaProperties properties) { return TopicBuilder.name(properties.topic()) .partitions(1) .replicas(1) .build(); } At start, Spring creates the topic if it does not exist; if it already exists, the bean is ignored. So the service does not depend on the automatic creation of topics by the broker. One partition and one replica: enough for a single local broker.\ndocker exec iot-kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic iot-sensor-events Topic: iot-sensor-events\tTopicId: oiLmbWoqSneUtet0x7hxCw\tPartitionCount: 1\tReplicationFactor: 1\tConfigs: min.insync.replicas=1,segment.bytes=1073741824 Topic: iot-sensor-events\tPartition: 0\tLeader: 1\tReplicas: 1\tIsr: 1\tElr: LastKnownElr: Consume The listener On the processing side, the config adds a consumer group:\nspring: kafka: # Spring Boot uses String deserializers by default: the message is a plain String. bootstrap-servers: localhost:9092 consumer: group-id: iot-backend-processing # Read the topic from the start when the group has no offset yet, # so messages sent before the first start are not lost. auto-offset-reset: earliest And the listener (excerpt of SensorMessageListener):\n@KafkaListener(topics = \u0026#34;${iot.kafka.topic}\u0026#34;) @Transactional public void onMessage(ConsumerRecord\u0026lt;String, String\u0026gt; consumerRecord) { ReceivedMessage stored = storeMessage.store( consumerRecord.value(), consumerRecord.partition(), consumerRecord.offset()); // … log of the result } The ConsumerRecord gives the value (our text), and also its place in the topic: the partition and the offset. The service stores them with the message, so we can see them on the page.\nThe group iot-backend-processing is the name under which Kafka keeps where the consumer is: for each partition, a « committed » position, the offset of the next message to read. When it restarts, the service goes on from this position.\nThe trap of auto-offset-reset What does a consumer do when its group has no saved position yet, for example at the very first start? This is the job of auto.offset.reset. Its main values:\nValue With no saved position, the consumer… latest (default) starts at the end of the topic: it reads only the messages that come after earliest starts at the beginning of the topic: it also reads the messages already there none stops with an error The setting matters only in this case. As soon as the group has a position, the consumer goes on from where it was.\nTo see it, two new groups read the same topic, which already holds two messages. First with the default value:\ndocker exec iot-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot-sensor-events --group demo-latest --timeout-ms 5000 [2026-10-24 16:59:56,249] ERROR Error processing message, terminating consumer process: (org.apache.kafka.tools.consumer.ConsoleConsumer) org.apache.kafka.common.errors.TimeoutException Processed a total of 0 messages Then with earliest:\ndocker exec iot-kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic iot-sensor-events --group demo-from-start --command-property auto.offset.reset=earliest --timeout-ms 5000 sensor-1 TEMPERATURE 21.5 sensor-3 HUMIDITY 47.2 [2026-10-24 17:01:09,610] ERROR Error processing message, terminating consumer process: (org.apache.kafka.tools.consumer.ConsoleConsumer) org.apache.kafka.common.errors.TimeoutException Processed a total of 2 messages The TimeoutException is normal: --timeout-ms 5000 stops the tool after five seconds with no new message. (I removed from these outputs a line of information about the KIP-848 protocol.)\nWith latest, the first group read nothing: the two messages were there before it. This is exactly what would happen to the processing service if it started after the ingestion with the default value. Hence auto-offset-reset: earliest in its application.yml.\nRun the project and watch Kafka work Kafka and PostgreSQL are already running (docker compose up -d). In two terminals, from the root of the project:\ncd iot-backend-processing mvn spring-boot:run cd iot-backend-ingestion mvn spring-boot:run Then one message:\ncurl -i -X POST -H \u0026#34;Content-Type: text/plain\u0026#34; --data \u0026#34;sensor-1 TEMPERATURE 21.5\u0026#34; http://localhost:8081/api/messages HTTP/1.1 202 Content-Length: 0 Date: Sat, 24 Oct 2026 16:59:19 GMT The two services tell the path of the message in their logs (after a second message; I keep only the end of the lines, without the date and the thread name):\nPublished \u0026#39;sensor-1 TEMPERATURE 21.5\u0026#39; to iot-sensor-events (partition 0, offset 0) Published \u0026#39;sensor-3 HUMIDITY 47.2\u0026#39; to iot-sensor-events (partition 0, offset 1) Stored \u0026#39;sensor-1 TEMPERATURE 21.5\u0026#39; (partition 0, offset 0) Stored \u0026#39;sensor-3 HUMIDITY 47.2\u0026#39; (partition 0, offset 1) The producer and the consumer see the same position: partition 0, offsets 0 then 1. In a partition, the offset is simply the rank of the message, and it only grows.\nThe tool kafka-consumer-groups.sh shows where the group of the processing service is (short output: without the columns CONSUMER-ID, HOST and CLIENT-ID):\ndocker exec iot-kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group iot-backend-processing GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG iot-backend-processing iot-sensor-events 0 2 2 0 CURRENT-OFFSET is the committed position of the group (the next message to read), LOG-END-OFFSET the end of the partition, and LAG the gap between the two: 0, the service is up to date.\nTo send a steady flow, the project has a simulator (one line per second):\ncd iot-event-simulator mvn compile exec:java The Angular page (npm start in iot-frontend-angular, then http://localhost:4200) then shows the messages with their partition and their offset, which grows with each line.\nSee the topic with Kafdrop kafka-console-consumer.sh is the basic tool. To see the topic in a browser, Kafdrop is a web UI for Kafka (Apache 2.0 license, Java 17 or newer). It is not part of the project: I run it next to it, with its official jar (release 4.3.0), without changing the project:\njava --add-opens=java.base/sun.nio.ch=ALL-UNNAMED \\ -jar kafdrop-4.3.0.jar --kafka.brokerConnect=localhost:9092 Then http://localhost:9000, the topic iot-sensor-events, and its messages:\nWe find everything from above: one partition, offsets that follow each other, and Key: empty on each message, because the service publishes with no key.\nKafdrop runs on the machine and not in a container, for a reason seen above: Kafka advertises localhost:9092. From another container, this address means the container itself, and the connection fails. It would need a second listener in compose.yaml; that is for phase 2.\nWhat is missing, on purpose This project is minimal on purpose. It lacks what a real system would have:\na message key: with no key and one partition, the order is kept; with several partitions, a key (the sensor) will be needed to keep the order of the messages of one sensor; several partitions and several consumers, to process in parallel; error handling: today, a failed send leaves only a log line, and a message that cannot be stored is retried with the default settings of Spring Kafka, then lost; a typed event with a contract, instead of a line of text. This is the plan of phase 2 of the project, with its own posts.\nTested with Java 25 (Temurin 25.0.4), Spring Boot 4.1.2, Apache Kafka 4.4.0 (image apache/kafka:4.4.0), PostgreSQL 18.6, Maven 3.10.0, Docker Desktop on macOS, Kafdrop 4.3.0. Code: tag phase-1.\nSources Apache Kafka 4.0.0 Release Announcement (Kafka without ZooKeeper, KRaft by default) Kafka: Getting Started, Docker (the apache/kafka image and its default config) Kafka: consumer configs, auto.offset.reset Spring Boot: Kafka (the auto-configured KafkaTemplate, the NewTopic beans) Spring for Apache Kafka: Sending Messages (send returns a CompletableFuture) Spring for Apache Kafka: Configuring Topics Kafdrop (README, license) Further reading The README of the tag phase-1: the whole project, the Angular page and its dev proxy. The next post of the series: Hexagonal architecture with Spring Boot: a simple example, how the code of the processing service is organized. ","permalink":"https://aymenjamaaoui.dev/en/blog/kafka-spring-boot-locally/","summary":"\u003cp\u003eOne service receives the readings of sensors; another one must process\nand store them. If the first one calls the second one over REST, the two\nare tied: when the second one is stopped or slow, the first one cannot\ndeliver anything. Kafka sits between them: the first one publishes a\nmessage, the second one reads it at its own pace.\u003c/p\u003e\n\u003cp\u003eThis post builds that link from end to end, locally: Kafka in KRaft\nmode in Docker, a Spring Boot service that publishes a plain line of\ntext, another one that reads it, and a page that shows the partition and\nthe offset of each message. All the code is in the tag\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/iot-monitoring-platform/-/tree/phase-1\"\u003e\u003ccode\u003ephase-1\u003c/code\u003e\u003c/a\u003e\nof the public repository iot-monitoring-platform.\u003c/p\u003e","title":"Kafka and Spring Boot: a complete local project"},{"content":"My Java code stores files in S3 and data in DynamoDB. How do I develop and try it on my machine, without an AWS account, without a bill, and without changing the code that talks to AWS?\nThis post introduces Floci, a local AWS emulator, with an end-to-end example: the AWS CLI creates the resources, a small Java application uses them, and the Floci web console shows them. I contribute to the Floci project.\nWhy a local emulator To try code that talks to AWS, you can use a real account: you need credentials, each call can cost money, and a configuration mistake can touch more than a test environment. A local emulator answers the same calls, on the machine, with no account and no bill.\nLocalStack is the best-known tool in this field. Since March 2026, according to its own blog, it ships as a single image that needs an auth token, with a free account-based plan for individuals and open source projects. Floci works with no account and no token.\nFloci in brief According to its README, Floci is \u0026ldquo;a free, open-source local AWS emulator for development, testing, and CI\u0026rdquo;, under the MIT license. It is part of a family of emulators: floci for AWS (port 4566), and others for Azure, GCP and OCI.\nAll AWS services answer on http://localhost:4566. Any published region works, and the credentials can be any non-empty values.\nAn emulator, not a mock In a test, a mock replaces the AWS client with a fake object. It answers what you told it to answer. The code makes no network call, and the test mostly checks what you believe about AWS.\nFloci is something else: it is a server that speaks the AWS protocol. The code keeps the real AWS client (the one of the AWS CLI, the Java SDK, Terraform…), which sends real HTTP requests, signed like for AWS. Only the address changes: localhost instead of AWS.\nThe README describes its architecture:\nan HTTP router receives the requests in the AWS format; stateless services (SQS, SNS, IAM, STS, KMS, Secrets Manager…); stateful services: S3, DynamoDB and DynamoDB Streams; services that start real Docker containers: Lambda, RDS, ElastiCache, MSK, ECS, EC2, EKS…; the storage of your choice: in memory, persistent, hybrid, or with a write-ahead log. Floci measures its compatibility with test suites per SDK and per tool, published in its repository: for example 1,326 tests for the AWS SDK for Java v2, 205 for the AWS CLI v2, 67 for Terraform.\nStart Floci The simplest way is its CLI, floci-cli (Docker must be running):\nfloci start floci status Floci then runs in a container and answers on port 4566.\nWith the AWS CLI floci env prints the variables that send the AWS CLI to Floci:\nexport AWS_ENDPOINT_URL=\u0026#39;http://localhost.floci.io:4566\u0026#39; export AWS_ACCESS_KEY_ID=\u0026#39;test\u0026#39; export AWS_SECRET_ACCESS_KEY=\u0026#39;test\u0026#39; export AWS_DEFAULT_REGION=\u0026#39;us-east-1\u0026#39; localhost.floci.io is a name that points to 127.0.0.1, like its subdomains: documents.localhost.floci.io too. This is useful for S3, which can put the bucket name in the host name.\nI create the resources of the example: a bucket for the texts, a table for the info.\neval \u0026#34;$(floci env)\u0026#34; aws s3 mb s3://documents aws dynamodb create-table \\ --table-name document-info \\ --attribute-definitions AttributeName=id,AttributeType=S \\ --key-schema AttributeName=id,KeyType=HASH \\ --billing-mode PAY_PER_REQUEST To see that the CLI does nothing special, the --debug option shows the request:\naws s3 ls --debug Endpoint provider result: http://localhost.floci.io:4566 Sending http request: \u0026lt;AWSPreparedRequest ... method=GET, url=http://localhost.floci.io:4566/, headers={... \u0026#39;Authorization\u0026#39;: b\u0026#39;AWS4-HMAC-SHA256 Credential=test/..., SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=...\u0026#39; ...}\u0026gt; Response headers: {\u0026#39;Content-Type\u0026#39;: \u0026#39;application/xml\u0026#39;, ..., \u0026#39;x-amz-request-id\u0026#39;: \u0026#39;9f476e72-...\u0026#39;} (I shortened the lines.) The request is signed with SigV4 (AWS4-HMAC-SHA256), exactly like for AWS, and Floci answers in XML, like S3. So that you do not export these variables in each terminal, a named AWS profile does the same; I explain it in the post about my dev machine.\nWith the Java SDK The example is a small application, published on GitLab: floci-aws-local-demo. It stores « documents »: the text in S3, the info (title, author) in DynamoDB, and the consultation returns both together. It uses the AWS SDK for Java v2; Quarkus is only the web layer.\nEndpoint What it does AWS service POST /documents (text) stores the text, returns an id S3 POST /documents/{id}/info (JSON) stores the title and the author DynamoDB GET /documents/{id} returns the info and the text DynamoDB + S3 GET /documents lists the documents DynamoDB It is an educational example: each class builds its client with visible constants. The S3 client (a simplified part of DocumentTextResource.java):\nprivate static final URI ENDPOINT = URI.create(\u0026#34;http://localhost:4566\u0026#34;); private static final Region REGION = Region.US_EAST_1; // Floci accepts any non-empty credentials. private static final AwsBasicCredentials CREDENTIALS = AwsBasicCredentials.create(\u0026#34;test\u0026#34;, \u0026#34;test\u0026#34;); try (S3Client s3 = S3Client.builder() .endpointOverride(ENDPOINT) .region(REGION) .credentialsProvider(StaticCredentialsProvider.create(CREDENTIALS)) .build()) { PutObjectRequest request = PutObjectRequest.builder() .bucket(\u0026#34;documents\u0026#34;) .key(id + \u0026#34;.txt\u0026#34;) .contentType(MediaType.TEXT_PLAIN) .build(); s3.putObject(request, RequestBody.fromString(text)); } This is the S3 client you would use with AWS. The only difference is endpointOverride: with real AWS, you remove this line and use real credentials. The DynamoDB client is built the same way, in DocumentInfoResource.java.\nThe application runs on the machine, with the Java and Maven versions given by its mise.toml file:\nmise install mvn quarkus:dev Then you store a document and its info, and read it back:\ncurl -X POST localhost:8080/documents -H \u0026#39;Content-Type: text/plain\u0026#39; \\ --data \u0026#39;Floci speaks the AWS protocol on port 4566.\u0026#39; {\u0026#34;id\u0026#34;:\u0026#34;3fa7b64a-10e9-4e1e-b9d3-258a043f8122\u0026#34;} curl -X POST localhost:8080/documents/3fa7b64a-10e9-4e1e-b9d3-258a043f8122/info \\ -H \u0026#39;Content-Type: application/json\u0026#39; -d \u0026#39;{\u0026#34;title\u0026#34;:\u0026#34;Hello Floci\u0026#34;,\u0026#34;author\u0026#34;:\u0026#34;Foulen Foulani\u0026#34;}\u0026#39; curl localhost:8080/documents/3fa7b64a-10e9-4e1e-b9d3-258a043f8122 {\u0026#34;id\u0026#34;:\u0026#34;3fa7b64a-…\u0026#34;,\u0026#34;title\u0026#34;:\u0026#34;Hello Floci\u0026#34;,\u0026#34;author\u0026#34;:\u0026#34;Foulen Foulani\u0026#34;,\u0026#34;createdAt\u0026#34;:\u0026#34;2026-10-24T03:12:58.387781Z\u0026#34;,\u0026#34;text\u0026#34;:\u0026#34;Floci speaks the AWS protocol on port 4566.\u0026#34;} The title comes from DynamoDB, the text from S3. An unknown id answers 404.\nHere, the Floci address is a constant in the code. In a real project, you can keep it out of the code: according to the AWS guide, the SDK for Java 2.x reads AWS_ENDPOINT_URL, or endpoint_url in an AWS profile (the SDK 1.x does not).\nThe web console Floci has a web console: http://localhost:4566/_floci/ui. It does not run at startup. On the first visit, Floci starts a container next to it, with the console, then redirects the browser there (here, to port 4500). This is why Floci needs the Docker socket (floci start takes care of it). The console does more than show: you can also create a bucket, upload a file, add an item to a table or run a PartiQL query.\nThe documents bucket holds the text stored by the application:\nAnd the document-info table holds its info. A PartiQL query shows it as JSON:\nWhat to know An emulator is not AWS. Floci reproduces the protocol and the behavior of the services, and its test suites measure this compatibility; but for a given service, check its place in the list of services and in these tests. The services that start real containers (Lambda, RDS…) need Docker.\nTo go further, Floci also has Testcontainers modules (Java, Node.js, Python, .NET) that start an isolated Floci for each test. Testing your code with Floci will be the topic of a next post.\nTested on macOS with Floci 2.3.0 (CLI 0.4.0), the AWS CLI 2.38.5, Java 21 (Temurin 21.0.13), Quarkus 3.40.1.1 and the AWS SDK for Java 2.46.7.\nSources: the Floci README (introduction, architecture, web console, compatibility tests, Testcontainers), floci-cli, LocalStack\u0026rsquo;s post about the delivery of its emulators, and the AWS guide on endpoints.\nFurther reading Java dev machine: 8, 17 and 25 side by side with mise Java in Docker and Kubernetes: avoid OOMKilled ","permalink":"https://aymenjamaaoui.dev/en/blog/aws-locally-floci/","summary":"\u003cp\u003eMy Java code stores files in S3 and data in DynamoDB. How do I develop\nand try it on my machine, without an AWS account, without a bill, and\nwithout changing the code that talks to AWS?\u003c/p\u003e\n\u003cp\u003eThis post introduces \u003ca href=\"https://github.com/floci-io/floci\"\u003eFloci\u003c/a\u003e, a\nlocal AWS emulator, with an end-to-end example: the AWS CLI creates the\nresources, a small Java application uses them, and the Floci web\nconsole shows them. I contribute to the Floci project.\u003c/p\u003e","title":"AWS locally with Floci: an emulator, not a mock"},{"content":"A Java developer often works on several stacks at the same time: a legacy project in Java 8, a service in 17, a new project in 25. Maven here, Gradle there. Node for docs, Python for MkDocs. And around them, tools: aws, gh, kubectl, helm.\nEach SDK and each tool comes with its own installer. Little by little, the machine fills up: global versions that mix, sudo, a .zshrc filled by installers, an AWS variable left global. My question: how to have the right version of each SDK and each tool in each project, without installing anything as root, with one tool, and a terminal that stays fast?\nHere is my dev machine on macOS, built on a clean OS that I want to keep clean.\nMy criteria for a dev machine Nothing as root. No sudo to install or update a dev tool. Everything lives in my user folder. One tool for SDKs and tools. One place, one command, one file to read. The right version on cd. I enter a project and I get its Java and Maven versions, without thinking about it. Isolation. Each project has its versions. A special environment (for example a local AWS emulator) stays in one tab. A terminal that stays fast, even with completions for all the tools. The rest of the post answers these criteria, in this order.\nmise in a few words mise describes itself as \u0026ldquo;dev tools, env vars, and tasks in one CLI\u0026rdquo;: it installs the tools of a project, sets its environment variables and runs its tasks. Here, I use it mostly for the tools. It is one binary, under the MIT license, and it also works on Linux and Windows.\nThe install is one command, with no sudo:\ncurl -fsSL https://mise.run | sh The script checks the SHA-256 sum of the binary and puts it in ~/.local/bin/mise. It does not change ~/.zshrc: it prints the line to add. I add it myself:\neval \u0026#34;$(\u0026#34;$HOME/.local/bin/mise\u0026#34; activate zsh)\u0026#34; In a new tab, mise doctor confirms it:\nactivated: yes ... No problems found mise keeps everything in my folder: the tools in ~/.local/share/mise/installs/\u0026lt;tool\u0026gt;/\u0026lt;version\u0026gt;, the configuration in ~/.config/mise/config.toml.\nOne detail for IntelliJ: an app started from the Dock reads the environment of the login shell, not the one of .zshrc. The mise docs recommend adding the \u0026ldquo;shims\u0026rdquo; to ~/.zprofile, so that these apps find the tools too:\neval \u0026#34;$(\u0026#34;$HOME/.local/bin/mise\u0026#34; activate zsh --shims)\u0026#34; All SDKs and tools in two commands First the languages, with their LTS version when there is one:\nmise use -g java@temurin-25 node@lts maven@3.10 gradle@9 python@3.15 Then the command-line tools:\nmise use -g uv starship gh aws-cli helm github:floci-io/floci-cli -g means \u0026ldquo;global\u0026rdquo;: these are the default versions of the machine. Most of them come from the mise registry (more than 1,000 tools). A tool that is not in the registry installs from its GitHub releases, like github:floci-io/floci-cli here.\nThe result is a readable file that describes the machine:\n# ~/.config/mise/config.toml [tools] java = \u0026#34;temurin-25\u0026#34; node = \u0026#34;lts\u0026#34; maven = \u0026#34;3.10\u0026#34; gradle = \u0026#34;9\u0026#34; python = \u0026#34;3.15\u0026#34; uv = \u0026#34;latest\u0026#34; starship = \u0026#34;latest\u0026#34; gh = \u0026#34;latest\u0026#34; aws-cli = { version = \u0026#34;latest\u0026#34;, symlink_bins = \u0026#34;true\u0026#34; } helm = \u0026#34;latest\u0026#34; \u0026#34;github:floci-io/floci-cli\u0026#34; = \u0026#34;latest\u0026#34; A version is easy to read: temurin-25 takes the latest Temurin 25, lts the latest Node LTS, 3.10 the latest patch of Maven 3.10. On a new machine, you copy this file, then run mise install.\nuv, on top of Python? mise installs Python itself. uv is for what you do with Python: uvx mkdocs serve runs MkDocs without a global install, and uv sync manages the dependencies of a project.\nOne version per project Each project can have its own file. For quickfixj, a project still on Java 8:\n# ~/devs/GitHub/quickfixj/mise.local.toml [tools] java = \u0026#34;corretto-8\u0026#34; maven = \u0026#34;3.10\u0026#34; mise.local.toml is a personal file: the mise docs describe it as \u0026ldquo;personal overrides, not committed\u0026rdquo;. I add it to the .git/info/exclude file of the project, so that Git does not see it. For a version shared with the team, you put a mise.toml in the repository instead. The file closest to the current folder wins, and what it does not set comes from the global configuration.\nThis is what the first screenshot shows. In floci, java -version gives Java 25; in quickfixj, Java 8. And mise ls --current tells where each version comes from: Java and Maven from the mise.local.toml of the project, Node from ~/.config/mise/config.toml. mise also updates JAVA_HOME.\nA project already has a version file, like a .nvmrc? mise can read it, once the option is on:\nmise settings add idiomatic_version_file_enable_tools node In a folder whose .nvmrc says 20, I get Node 20; anywhere else, Node 24 (the LTS). Nothing to change in the project.\nA comfortable terminal The prompt: Starship Starship shows what matters in the current folder: the Git branch, the project version, Java, Maven, the AWS profile. mise already installed it; you only turn it on at the end of ~/.zshrc:\neval \u0026#34;$(starship init zsh)\u0026#34; Its docs ask for a Nerd Font, a font with icons. I took JetBrains Mono Nerd Font, from the official Nerd Fonts releases (the SHA-256 sum of each archive is published there). To stay without root, the font goes to ~/Library/Fonts: it is installed for my user only. Then I choose it in the terminal settings.\nCompletions without slowing the shell The Tab key completes the commands of docker, kubectl, gh… Many guides load each completion at startup:\nsource \u0026lt;(kubectl completion zsh) The program then runs in each new tab. Docker and kubectl document another way: write the completion once to a file, in a folder that zsh knows (the fpath). zsh loads it only on the first Tab.\nmkdir -p ~/.zsh/completions kubectl completion zsh \u0026gt; ~/.zsh/completions/_kubectl docker completion zsh \u0026gt; ~/.zsh/completions/_docker gh completion -s zsh \u0026gt; ~/.zsh/completions/_gh And in ~/.zshrc, before compinit:\nfpath=(\u0026#34;$HOME/.zsh/completions\u0026#34; $fpath) autoload -Uz compinit compinit I measured how long a shell takes to open with the same 10 completions (docker, kubectl, mise, gh, glab, helm, uv, uvx, Starship, hvm), 10 runs for each case:\nMethod Shell startup No completion 93 ms Files in the fpath 92 ms source \u0026lt;(… completion zsh) 348 ms The files cost nothing at startup. Their only limit: they describe the tool at the time you created them. After an update (mise upgrade), I create them again with a small script, ~/.zsh/update-completions.zsh, with one line per tool.\nTwo special cases. The aws completion is not a file but a program; the AWS docs give these two lines for zsh:\nautoload -Uz bashcompinit \u0026amp;\u0026amp; bashcompinit complete -C aws_completer aws And an alias keeps the completion of its command: alias k=kubectl, then compdef k=kubectl.\nHistory and a few aliases macOS keeps 1,000 lines of history (SAVEHIST in /etc/zshrc). I keep more, with no duplicates, shared between tabs:\nHISTSIZE=50000 SAVEHIST=50000 setopt HIST_IGNORE_ALL_DUPS SHARE_HISTORY HIST_IGNORE_SPACE With HIST_IGNORE_SPACE, a command that starts with a space is not saved: useful for a command that contains a secret. And a few aliases:\nalias ls=\u0026#39;ls -G\u0026#39; # colors alias ll=\u0026#39;ls -lrth\u0026#39; alias lla=\u0026#39;ls -lrtha\u0026#39; alias reload=\u0026#39;exec zsh\u0026#39; # reload the shell alias ports=\u0026#39;lsof -nP -iTCP -sTCP:LISTEN\u0026#39; # who listens on which port Isolate an environment: the Floci example Floci is a local AWS emulator (I introduce it in AWS locally with Floci). To use it, the AWS client must call http://localhost:4566. The easy way is a global variable: export AWS_ENDPOINT_URL=http://localhost:4566 in .zshrc. But then every AWS command on the machine goes to the emulator, and you end up forgetting it.\nThe AWS docs let you put the address in a named profile of ~/.aws/config:\n[profile floci] region = us-east-1 endpoint_url = http://localhost:4566 The test credentials go to ~/.aws/credentials, under [floci]. I have no default profile. The result:\nin a normal tab, aws finds no credentials (\u0026ldquo;Unable to locate credentials\u0026rdquo;): the command goes neither to Floci nor to a real AWS account; with AWS_PROFILE=floci (or --profile floci), it goes to Floci. So that I do not type AWS_PROFILE each time, a file ~/.zsh/profiles/floci.zsh sets it and loads the floci completion:\nexport AWS_PROFILE=floci if command -v floci \u0026gt;/dev/null; then source \u0026lt;(floci completion zsh) fi The floci completion is a script that cannot go into the fpath. So it stays in this profile, and costs nothing to the other tabs.\nIn iTerm2, a \u0026ldquo;dynamic profile\u0026rdquo; creates a \u0026ldquo;Floci\u0026rdquo; profile from a JSON file, put in ~/Library/Application Support/iTerm2/DynamicProfiles/. It inherits from the default profile and shows a badge:\n{ \u0026#34;Profiles\u0026#34;: [ { \u0026#34;Name\u0026#34;: \u0026#34;Floci\u0026#34;, \u0026#34;Guid\u0026#34;: \u0026#34;647AE153-1576-4C43-A514-1581774C216B\u0026#34;, \u0026#34;Dynamic Profile Parent Name\u0026#34;: \u0026#34;Default\u0026#34;, \u0026#34;Badge Text\u0026#34;: \u0026#34;Floci\u0026#34; } ] } The iTerm2 docs ask for a unique Guid: I created it with uuidgen. iTerm2 puts the profile name in the $ITERM_PROFILE variable (I saw it, but its docs do not mention it). ~/.zshrc then loads the Floci profile in these tabs. In another terminal, like Ghostty or the one in IntelliJ, a profile floci function does the same.\nIf you already have tools installed as root A tool already installed as root? Take aws, often installed with the official package in /usr/local. First install the mise version (mise use -g aws-cli), check with which -a aws that it comes first, then uninstall the old one with the Uninstall the AWS CLI page from AWS. Same idea for the other tools: the mise version first, the old one after.\nSummary Criterion My answer Nothing as root mise and all the tools in ~/.local, the font in ~/Library/Fonts One tool mise use -g for SDKs and tools; one file, ~/.config/mise/config.toml The right version on cd a mise.local.toml or mise.toml per project, and the existing .nvmrc files Isolation one file per project; a named AWS profile with no default profile; a Floci tab A fast terminal completions in files: 92 ms instead of 348 ms The full ~/.zshrc:\n# PATH: no duplicates, user tools first typeset -U path PATH path=(\u0026#34;$HOME/.local/bin\u0026#34; $path) # History HISTSIZE=50000 SAVEHIST=50000 setopt HIST_IGNORE_ALL_DUPS SHARE_HISTORY HIST_IGNORE_SPACE # Completions: the files in ~/.zsh/completions fpath=(\u0026#34;$HOME/.zsh/completions\u0026#34; $fpath) autoload -Uz compinit compinit autoload -Uz bashcompinit \u0026amp;\u0026amp; bashcompinit complete -C aws_completer aws # Aliases alias ls=\u0026#39;ls -G\u0026#39; alias ll=\u0026#39;ls -lrth\u0026#39; alias lla=\u0026#39;ls -lrtha\u0026#39; alias reload=\u0026#39;exec zsh\u0026#39; alias ports=\u0026#39;lsof -nP -iTCP -sTCP:LISTEN\u0026#39; alias k=kubectl alias update-completions=\u0026#39;~/.zsh/update-completions.zsh\u0026#39; compdef k=kubectl # mise: the versions of the current folder eval \u0026#34;$(\u0026#34;$HOME/.local/bin/mise\u0026#34; activate zsh)\u0026#34; # Terminal profiles (~/.zsh/profiles), for example: profile floci profile() { local target=\u0026#34;$HOME/.zsh/profiles/${1:l}.zsh\u0026#34; if [[ -f \u0026#34;$target\u0026#34; ]]; then source \u0026#34;$target\u0026#34; else print -u2 \u0026#34;Unknown profile: $1 (see the files in ~/.zsh/profiles)\u0026#34; return 1 fi } case \u0026#34;$ITERM_PROFILE\u0026#34; in Floci) profile floci ;; esac # Prompt (keep it at the end) eval \u0026#34;$(starship init zsh)\u0026#34; Tested on macOS 27 (Apple Silicon) with zsh 5.9, mise 2026.10, Starship 1.27, iTerm2 and Ghostty, Floci 2.3.\nSources: the mise documentation (\u0026ldquo;Getting started\u0026rdquo;, \u0026ldquo;Configuration\u0026rdquo;, \u0026ldquo;Editors and IDEs\u0026rdquo;, Java, Node), the zsh manual, the completion pages of Docker, kubectl and the AWS CLI, the AWS guide on endpoints, the Starship guide and the docs of iTerm2 dynamic profiles.\nFurther reading One Chrome per Google account on macOS Java in Docker and Kubernetes: avoid OOMKilled ","permalink":"https://aymenjamaaoui.dev/en/blog/java-dev-machine-mise/","summary":"\u003cp\u003eA Java developer often works on several stacks at the same time: a\nlegacy project in Java 8, a service in 17, a new project in 25. Maven\nhere, Gradle there. Node for docs, Python for MkDocs. And around them,\ntools: \u003ccode\u003eaws\u003c/code\u003e, \u003ccode\u003egh\u003c/code\u003e, \u003ccode\u003ekubectl\u003c/code\u003e, helm.\u003c/p\u003e\n\u003cp\u003eEach SDK and each tool comes with its own installer. Little by little,\nthe machine fills up: global versions that mix, \u003ccode\u003esudo\u003c/code\u003e, a \u003ccode\u003e.zshrc\u003c/code\u003e\nfilled by installers, an AWS variable left global. My question: how to\nhave the right version of each SDK and each tool in each project,\nwithout installing anything as root, with one tool, and a terminal that\nstays fast?\u003c/p\u003e","title":"Java dev machine: 8, 17 and 25 side by side with mise"},{"content":"The pod restarts. kubectl get pod shows OOMKilled, and the exit code is 137. But the logs show no OutOfMemoryError, and the heap was far from full. The JVM did not run out of memory: the container did. The Linux kernel killed the process because it went over its limit.\nThis post shows what the JVM sees in a container, what adds to the heap, how to reproduce the kill, and how to set the memory so that everything fits. All the examples are in the containers/ folder of the public java-gc-lab repository. They use the official eclipse-temurin images: no image to build.\nThe Docker commands of this post If you do not use Docker much, here are the options used below.\nOption What it does docker run \u0026lt;image\u0026gt; \u0026lt;command\u0026gt; Starts a container from the image and runs the command in it. --rm Deletes the container when it stops. --name over-limit Gives the container a name, for the next commands. --memory=512m Memory limit of the container. This is the limit the JVM reads. --memory-swap=512m Limit for memory + swap. Same value as --memory: no swap. --cpus=2 CPU limit: the same as 2 CPUs of time. -v \u0026quot;$PWD/containers\u0026quot;:/app Mounts the local folder containers/ as /app in the container. -d Runs the container in the background. docker inspect \u0026lt;name\u0026gt; Shows the state of a stopped container: exit code, OOMKilled. docker exec \u0026lt;name\u0026gt; \u0026lt;command\u0026gt; Runs a command in a running container. docker logs \u0026lt;name\u0026gt; Shows what the container printed. Watch out for swap: with --memory alone, Docker also gives the same amount of swap. java -XshowSettings:system shows it: \u0026ldquo;Memory Limit: 1.00G\u0026rdquo;, \u0026ldquo;Memory \u0026amp; Swap Limit: 2.00G\u0026rdquo;. The container then uses swap instead of being killed at the limit. To get what happens in Kubernetes, I set --memory-swap to the same value.\nWhat the JVM sees in a container On Linux, the JVM reads the limits of the container (the cgroups): the memory and the number of CPUs. This is the option -XX:+UseContainerSupport, on by default. The JVM uses these limits to size the heap, choose the GC and set the number of threads.\nShowLimits.java prints what the JVM chose:\ndocker run --rm --memory=1g --cpus=2 -v \u0026#34;$PWD/containers\u0026#34;:/app eclipse-temurin:25 java /app/ShowLimits.java CPUs: 2 Max heap: 256 MiB GC: Copy, MarkSweepCompact \u0026ldquo;Copy, MarkSweepCompact\u0026rdquo; is the Serial GC. With other limits, the same results on Java 17, 21 and 25:\n--memory --cpus Max heap GC 256m 1 126 MiB Serial 1g 2 256 MiB Serial 1791m 2 448 MiB Serial 1792m 2 448 MiB G1 4g 2 1024 MiB G1 The JVM gives these values itself, with no program:\ndocker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XX:+PrintFlagsFinal -version | grep -E \u0026#34; MaxHeapSize | UseSerialGC | UseG1GC \u0026#34; docker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XshowSettings:system -version The first one prints the computed options (MaxHeapSize = 268435456, UseSerialGC = true). The second one shows the limits read from the container: \u0026ldquo;Provider: cgroupv2\u0026rdquo;, \u0026ldquo;Effective CPU Count: 2\u0026rdquo;, \u0026ldquo;Memory Limit: 1.00G\u0026rdquo;.\nThe default heap: 25 %, or 50 % for a small heap Without -Xmx, the maximum heap is 25 % of the memory the JVM sees: this is -XX:MaxRAMPercentage, 25 by default. In a 1 GB container, that is 256 MB. The other 75 % are not all free: the rest of the JVM needs them, as we will see below.\nFor a small heap, another option takes over: -XX:MinRAMPercentage, 50 by default. The documentation talks about a heap of \u0026ldquo;approximately 125 MB\u0026rdquo;. This is what the table shows: with 256 MB, the heap is 126 MB, not 64.\nYou set both options at startup:\njava -XX:MaxRAMPercentage=50 -jar app.jar A percentage follows the container limit: if the limit goes from 1 to 2 GB, the heap follows. A fixed -Xmx does not change (see the JVM options that matter).\nSerial or G1: a question of CPUs and memory The table has a surprise: with 2 CPUs and 1 GB, the JVM picks Serial, not G1. Oracle\u0026rsquo;s GC guide gives the rule: G1 on a \u0026ldquo;server-class\u0026rdquo; machine, Serial otherwise. A machine is \u0026ldquo;server-class\u0026rdquo; if the JVM detects \u0026ldquo;two or more processors and physical memory larger than or equal to 1792 MB\u0026rdquo;. The tests confirm it: Serial at 1791 MB, G1 at 1792 MB.\nSerial stops the application during each GC, with one thread. For a service with little memory, this is not always a problem. But you must know it, and choose it. To force G1: -XX:+UseG1GC.\nThe number of CPUs also changes the number of GC threads. With 2 CPUs, G1 has 2 parallel threads and 1 concurrent thread. The option -XX:ActiveProcessorCount=\u0026lt;n\u0026gt; replaces the number of CPUs that the JVM uses for these sizes. The documentation says it can be useful \u0026ldquo;when running multiple Java processes in docker containers\u0026rdquo;.\nWhat about the CPU request of Kubernetes? In a pod with requests.cpu: 1 and no CPU limit, the JVM sees all the CPUs of the machine: 12 on my machine. The same with the Docker option --cpu-shares (the cgroup parameter \u0026ldquo;cpu.shares\u0026rdquo;). A limit does reduce the number of CPUs the JVM sees: with limits.cpu: \u0026quot;2\u0026quot; (or --cpus=2), it sees 2.\nBefore Java 19: the JVM also used \u0026ldquo;cpu.shares\u0026rdquo; to count the CPUs, and could see fewer than available. Java 19 ignores it (JDK-8281181). In my tests, Temurin 17.0.20 also ignores it.\nWhat adds to the heap The container limit covers the whole process, not only the heap:\nthe Metaspace: the description of the loaded classes; the thread stacks: by default 1024 KB per thread on Linux/x64, and 2048 KB on Linux/Aarch64 (for example Docker on an Apple Silicon Mac); the code cache: the code compiled by the JIT; the direct memory: the ByteBuffer.allocateDirect buffers. Without -XX:MaxDirectMemorySize, the JVM chooses the limit itself; the GC data and other internal areas. A heap at 75 % of the limit leaves 25 % for all of this. If the rest needs more, the container is killed, even with a half-empty heap.\nReproduce the kill OverLimit.java keeps 250 MB in the heap, then takes direct memory by blocks of 10 MB, with no end. A 512 MB container with no swap, max heap at 75 % (384 MB):\ndocker run --name over-limit --memory=512m --memory-swap=512m -v \u0026#34;$PWD/containers\u0026#34;:/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 /app/OverLimit.java Heap: 250 MB used Direct memory: 10 MB ... Direct memory: 170 MB The output just stops. No OutOfMemoryError, no stack trace: the process was killed. docker inspect confirms it:\ndocker inspect over-limit --format \u0026#34;exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}\u0026#34; exit=137 OOMKilled=true 137 is 128 + 9: the process got signal 9 (SIGKILL), which cannot be caught. The JVM could not write anything, and -XX:+HeapDumpOnOutOfMemoryError does not help here: there was no OutOfMemoryError. Depending on the version, the kill comes between 170 and 190 MB of direct memory.\nSee where the memory goes with NMT Native Memory Tracking (NMT) counts the memory of the JVM by area. I run it again with NMT, and the argument 100 stops the direct memory at 100 MB; the program then waits 30 seconds:\ndocker run -d --name over-limit --memory=512m --memory-swap=512m -v \u0026#34;$PWD/containers\u0026#34;:/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:NativeMemoryTracking=summary /app/OverLimit.java 100 docker exec over-limit jcmd 1 VM.native_memory summary scale=MB Total: reserved=1875MB, committed=533MB - Java Heap (reserved=384MB, committed=384MB) - Class (reserved=1024MB, committed=2MB) - Thread (reserved=26MB, committed=1MB) - Code (reserved=245MB, committed=9MB) - Other (reserved=100MB, committed=100MB) - Metaspace (reserved=64MB, committed=9MB) (I removed some lines.) 1 is the process id of Java in the container. The heap takes 384 MB, and the 100 MB of direct memory are in \u0026ldquo;Other\u0026rdquo;. This small program has few classes and threads; a real application has more Metaspace, more threads and more compiled code. docker stats --no-stream over-limit shows the memory the container really uses: about 430 MiB in my runs, out of 512.\nBetter than a kill: a Java error With a limit on direct memory, the JVM stops before the container, with a clear message:\ndocker run --rm --memory=512m --memory-swap=512m -v \u0026#34;$PWD/containers\u0026#34;:/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:MaxDirectMemorySize=100m /app/OverLimit.java Exception in thread \u0026#34;main\u0026#34; java.lang.OutOfMemoryError: Cannot reserve 10485760 bytes of direct buffer memory (allocated: 104857600, limit: 104857600) An error with a stack trace tells what is full. A kill tells nothing. The other OutOfMemoryError messages are in the post about OutOfMemoryError.\nChoose MaxRAMPercentage There is no good percentage for all applications. The rule is simple: the maximum heap plus all the rest must fit in the limit. You measure the rest with NMT, on the application under load.\nIn the example, the program needs 250 MB of heap and 200 MB of direct memory. With 768 MB and 50 %:\ndocker run --name over-limit --memory=768m --memory-swap=768m -v \u0026#34;$PWD/containers\u0026#34;:/app eclipse-temurin:25 java -XX:MaxRAMPercentage=50 /app/OverLimit.java 200 Done: 250 MB heap, 200 MB direct, waiting 30 s docker inspect gives exit=0 OOMKilled=false. The heap can grow to 384 MB (50 % of 768), and 384 MB stay for the direct memory and the rest of the JVM. The more memory an application uses outside the heap (threads, direct memory, many classes), the lower the percentage must be.\nIn Kubernetes In a pod, each container has requests and limits. The Kubernetes documentation says that the request is used to choose the node for the pod; the limit is enforced by the kernel. For memory, a container that goes over its limit can be killed (OOM kill). For CPU, it is slowed down.\nSo the memory limit is the one the JVM reads. Without limits.memory, the JVM sees the memory of the node: in a pod with only requests.memory: 1Gi, the max heap is 1984 MiB on my machine (25 % of the 7.7 GB of the Docker VM), much more than the request.\nFor the example, the Java file goes into the pod with a ConfigMap: no image to build. The manifest pod-before.yaml is the killed case:\ncommand: [\u0026#34;java\u0026#34;, \u0026#34;-XX:MaxRAMPercentage=75\u0026#34;, \u0026#34;/app/OverLimit.java\u0026#34;] resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: \u0026#34;1\u0026#34; limits: memory: \u0026#34;512Mi\u0026#34; The kubectl commands used:\nCommand What it does kubectl create configmap over-limit-code --from-file=containers/OverLimit.java Stores the Java file in the cluster; the pod mounts it in /app. kubectl apply -f containers/pod-before.yaml Creates the pod described in the file. kubectl get pod over-limit Shows the status of the pod. kubectl describe pod over-limit Shows the details: state, reason, exit code, limits. kubectl logs over-limit Shows what the program printed. kubectl delete pod over-limit Deletes the pod. kubectl create configmap over-limit-code --from-file=containers/OverLimit.java kubectl apply -f containers/pod-before.yaml kubectl get pod over-limit NAME READY STATUS RESTARTS AGE over-limit 0/1 OOMKilled 0 7s kubectl describe pod over-limit State: Terminated Reason: OOMKilled Exit Code: 137 pod-after.yaml uses the size that fits: limits.memory: 768Mi, -XX:MaxRAMPercentage=50. After kubectl delete pod over-limit and kubectl apply -f containers/pod-after.yaml, the pod ends as Completed, exit code 0.\nWhat has changed What you still read What is true today Since -XX:+UseCGroupMemoryLimitForHeap so that Java sees the limit Nothing to add: the JVM reads the limits itself (UseContainerSupport); the old option is refused JDK 10, and Java 8u191; option removed in Java 11 -XX:MaxRAMFraction=4 -XX:MaxRAMPercentage=25 (and InitialRAMPercentage, MinRAMPercentage) JDK 10 \u0026ldquo;cpu.shares\u0026rdquo; changes the number of CPUs Java sees \u0026ldquo;cpu.shares\u0026rdquo; is ignored Java 19 (Temurin 17.0.20 too, in my tests) Tested with Docker Desktop 29.8 (Linux VM, aarch64, cgroup v2), its Kubernetes (v1.36), and the eclipse-temurin images 17.0.20, 21.0.12 and 25.0.4.\nSources: the documentation of the java command (JDK 25), Oracle\u0026rsquo;s GC guide, part \u0026ldquo;Ergonomics\u0026rdquo;, the release notes of JDK 10, Java 8u191 and JDK 19, and the Kubernetes documentation (Resource Management for Pods and Containers, Assign Memory Resources to Containers and Pods).\nFurther reading JVM options that matter, and the ones that no longer work Java OutOfMemoryError: every message, reproduced The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/java-docker-kubernetes-oomkilled/","summary":"\u003cp\u003eThe pod restarts. \u003ccode\u003ekubectl get pod\u003c/code\u003e shows \u003ccode\u003eOOMKilled\u003c/code\u003e, and the exit\ncode is 137. But the logs show no \u003ccode\u003eOutOfMemoryError\u003c/code\u003e, and the heap was\nfar from full. The JVM did not run out of memory: the container did.\nThe Linux kernel killed the process because it went over its limit.\u003c/p\u003e\n\u003cp\u003eThis post shows what the JVM sees in a container, what adds to the\nheap, how to reproduce the kill, and how to set the memory so that\neverything fits. All the examples are in the\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/tree/main/containers\"\u003e\u003ccode\u003econtainers/\u003c/code\u003e\u003c/a\u003e\nfolder of the public java-gc-lab repository. They use the official\n\u003ccode\u003eeclipse-temurin\u003c/code\u003e images: no image to build.\u003c/p\u003e","title":"Java in Docker and Kubernetes: avoid OOMKilled"},{"content":"In this series, we have already used about six tools: GC logs, jstat, several jcmd commands, Eclipse MAT. Each one answers a clear question, and each one has a cost. This post puts them in order. Then it shows JFR, the most complete tool of the JDK, on an example: find the method that creates millions of objects.\nWhich tool for which question Question Tool Command Cost Does the GC make long pauses? Does the heap after GC go up? GC logs -Xlog:gc*:file=gc.log keep it on in production Which area fills up, live? jstat jstat -gcutil \u0026lt;pid\u0026gt; 1s not given; its doc says \u0026ldquo;experimental and unsupported\u0026rdquo; How much heap is used? jcmd jcmd \u0026lt;pid\u0026gt; GC.heap_info medium Which class grows? jcmd jcmd \u0026lt;pid\u0026gt; GC.class_histogram high: runs a full GC What keeps the objects alive? heap dump + Eclipse MAT jcmd \u0026lt;pid\u0026gt; GC.heap_dump high: a big file Does the Metaspace grow? jcmd jcmd \u0026lt;pid\u0026gt; VM.metaspace medium What does native memory do? jcmd + NMT -XX:NativeMemoryTracking=summary, then jcmd \u0026lt;pid\u0026gt; VM.native_memory medium What do the threads do? jcmd jcmd \u0026lt;pid\u0026gt; Thread.print medium Which method allocates, where are the pauses, over time? JFR -XX:StartFlightRecording, then jfr view low with the default configuration The \u0026ldquo;medium\u0026rdquo; or \u0026ldquo;high\u0026rdquo; cost of the jcmd commands is the one given in their documentation. Each of these tools was already used in a post: GC logs, OutOfMemoryError, memory leaks and the heap dump with Eclipse MAT.\nJFR step by step Java Flight Recorder (JFR) is part of the JDK. It records events from the JVM and from the application: GC, allocations, threads, the most used methods…\nRecord. At startup:\njava -XX:StartFlightRecording:filename=hot.jfr,settings=profile -jar app.jar Or on an application that already runs: jcmd \u0026lt;pid\u0026gt; JFR.start, then jcmd \u0026lt;pid\u0026gt; JFR.dump filename=hot.jfr to write the recording.\nChoose the configuration. Oracle\u0026rsquo;s documentation recommends the default configuration for continuous recordings, with usually less than 1% overhead. The profile configuration records more events, such as allocation samples: use it to analyze a problem. The doc gives no number for its overhead.\nRead. The jfr command of the JDK shows tables, with no graphical interface:\njfr view allocation-by-site hot.jfr jfr view gc hot.jfr Before Java 21: jfr view does not exist. On Java 17, the command answers jfr: unknown command 'view': use jfr print or JDK Mission Control.\nExample: the method that creates millions of objects HotAllocation.java builds 10 million labels with String.format. We record it with JFR, then:\njfr view allocation-by-site hot.jfr Method Allocation Pressure --------------------------------------------------- ------------------- java.util.Formatter.parse(String) 24.70% java.lang.Long.toString(long) 18.83% java.lang.AbstractStringBuilder.\u0026lt;init\u0026gt;(int) 14.20% java.util.Arrays.copyOfRange(byte[], int, int) 10.61% java.util.ArrayList.grow(int) 10.35% Formatter.parse creates the most objects. These are samples: the percentages change a little from one run to the next, but the top of the table stays the same. Now we need the caller in our code: the stack of one sample gives it.\njfr print --events jdk.ObjectAllocationSample --stack-depth 6 hot.jfr stackTrace = [ java.util.Formatter.parse(String) line: 2810 java.util.Formatter.format(Locale, String, Object[]) line: 2744 java.util.Formatter.format(String, Object[]) line: 2698 java.lang.String.format(String, Object[]) line: 4455 HotAllocation.label(int) line: 16 HotAllocation.main(String[]) line: 24 The cause: String.format, called in label(). Each call parses the pattern \u0026quot;order-%08d\u0026quot; and creates several objects. In FixedAllocation.java, the same labels are built with a StringBuilder. Same result, with -Xmx256m and JFR on:\nHotAllocation FixedAllocation Java 25 32 GCs, 2.7 s 9 GCs, 0.7 s Java 21 44 GCs 12 GCs Java 17 94 GCs 10 GCs Fewer objects created, so fewer GCs: this is the \u0026ldquo;allocation rate\u0026rdquo; of the post about GC logs.\njcmd, the Swiss Army knife jcmd sends a command to a running JVM. jcmd \u0026lt;pid\u0026gt; help lists all the commands you can use. The ones of this series: GC.heap_info, GC.class_histogram, GC.heap_dump, GC.run, VM.metaspace, VM.native_memory, Thread.print, and JFR.start / JFR.dump. Its documentation gives the impact of each one: read it before you run a \u0026ldquo;High\u0026rdquo; command on a busy production system.\nVisualVM and JDK Mission Control If you prefer a window:\nVisualVM follows a JVM live (CPU, GC activity, heap, Metaspace), shows its threads, takes and opens heap dumps, and does simple profiling. JDK Mission Control opens .jfr files and shows the same data as jfr view, as charts. The same hot.jfr, opened in JMC. JMC counts allocations by class, jfr view allocation-by-site by method: both views come from the same recording.\nBefore Java 9: VisualVM came with the JDK (jvisualvm). It was removed in Java 9, and you now download it on its own.\nYou install both separately. The commands of this post are already in the JDK, and they work on a server with no screen.\nIn production Keep on all the time: GC logs in a file, and JFR with the default configuration. Now and then, when you know the cost: GC.class_histogram (a full GC), GC.heap_dump (a big file, with sensitive data), JFR with profile. Before you act: read again the \u0026ldquo;Impact\u0026rdquo; line of the command in the jcmd documentation. Tested with Java 17, 21 and 25 (Temurin 17.0.20, 21.0.12, 25.0.4), on macOS; the programs compile with Java 8.\nSources: the documentation of jcmd, jfr and jstat, the Flight Recorder guide on its configurations (JDK 25), JDK-8306703 (jfr view) and JDK-8180002 (VisualVM removed from the JDK).\nFurther reading Java heap dump: find a memory leak with Eclipse MAT Java memory leaks: 6 classic cases, reproduced Reading a Java GC log: every number explained The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/java-tools-jfr-jcmd-visualvm/","summary":"\u003cp\u003eIn this series, we have already used about six tools: GC logs, \u003ccode\u003ejstat\u003c/code\u003e,\nseveral \u003ccode\u003ejcmd\u003c/code\u003e commands, Eclipse MAT. Each one answers a clear question,\nand each one has a cost. This post puts them in order. Then it shows\nJFR, the most complete tool of the JDK, on an example: find the method\nthat creates millions of objects.\u003c/p\u003e\n\u003ch2 id=\"which-tool-for-which-question\"\u003eWhich tool for which question\u003c/h2\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eQuestion\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eTool\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eCommand\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eCost\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eDoes the GC make long pauses? Does the heap after GC go up?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eGC logs\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-Xlog:gc*:file=gc.log\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ekeep it on in production\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhich area fills up, live?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejstat\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejstat -gcutil \u0026lt;pid\u0026gt; 1s\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003enot given; its doc says \u0026ldquo;experimental and unsupported\u0026rdquo;\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eHow much heap is used?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd \u0026lt;pid\u0026gt; GC.heap_info\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003emedium\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhich class grows?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd \u0026lt;pid\u0026gt; GC.class_histogram\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ehigh: runs a full GC\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhat keeps the objects alive?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eheap dump + Eclipse MAT\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd \u0026lt;pid\u0026gt; GC.heap_dump\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ehigh: a big file\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eDoes the Metaspace grow?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd \u0026lt;pid\u0026gt; VM.metaspace\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003emedium\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhat does native memory do?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd\u003c/code\u003e + NMT\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:NativeMemoryTracking=summary\u003c/code\u003e, then \u003ccode\u003ejcmd \u0026lt;pid\u0026gt; VM.native_memory\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003emedium\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhat do the threads do?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003ejcmd \u0026lt;pid\u0026gt; Thread.print\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003emedium\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWhich method allocates, where are the pauses, over time?\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eJFR\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:StartFlightRecording\u003c/code\u003e, then \u003ccode\u003ejfr view\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003elow with the \u003ccode\u003edefault\u003c/code\u003e configuration\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThe \u0026ldquo;medium\u0026rdquo; or \u0026ldquo;high\u0026rdquo; cost of the \u003ccode\u003ejcmd\u003c/code\u003e commands is the one given\nin their documentation. Each of these tools was already used in a post:\n\u003ca href=\"/en/blog/read-java-gc-log/\"\u003eGC logs\u003c/a\u003e,\n\u003ca href=\"/en/blog/java-outofmemoryerror-messages/\"\u003eOutOfMemoryError\u003c/a\u003e,\n\u003ca href=\"/en/blog/java-memory-leaks/\"\u003ememory leaks\u003c/a\u003e and\n\u003ca href=\"/en/blog/java-heap-dump-eclipse-mat/\"\u003ethe heap dump with Eclipse MAT\u003c/a\u003e.\u003c/p\u003e","title":"JFR, jcmd, VisualVM: which tool for Java memory problems"},{"content":"In the post about memory leaks, two jcmd histograms showed which class grows. But not what keeps it alive. For that, you need a heap dump: a picture of the whole heap, with the objects and the links between them.\nWe take the same leak again: UnboundedCache.java, a static HashMap cache that gets a new key on each request. We analyze it with Eclipse Memory Analyzer (MAT), an open source tool that Oracle\u0026rsquo;s guide names for heap dump analysis. All the commands are in the heap-dump folder of the java-gc-lab repository.\nStep 1: take the heap dump javac -d out leaks/UnboundedCache.java java -Xmx256m -cp out UnboundedCache jcmd \u0026lt;pid\u0026gt; GC.heap_dump /tmp/cache.hprof After 20 seconds, the file is 22 MB. On a real application, it can be several GB: -gz=1 writes it gzipped. You can also let the JVM write it at the first OutOfMemoryError, with -XX:+HeapDumpOnOutOfMemoryError (see the post about OutOfMemoryError).\nTwo warnings:\nThe dump holds the values of the objects. MAT even shows the text of String objects. On a real application, it can hold passwords or personal data: do not share it, and delete it after use. Compile before you run. With java UnboundedCache.java (no javac), the class is loaded in memory by the source launcher, and MAT points to the main thread instead of the class that leaks. Run with -cp, like a normal application, the class is the one MAT points to. Step 2: Leak Suspects The \u0026ldquo;Leak Suspects\u0026rdquo; report is the first one to open. MAT lists the objects that keep the most memory alive:\nWhat to read: the class UnboundedCache keeps 93.87% of the heap, and the memory piles up in one single array, java.util.HashMap$Node[]: the internal array of a HashMap. The culprit is found at the first try.\nStep 3: shallow heap and retained heap MAT shows two sizes for each object:\nthe shallow heap: the memory of the object itself; the retained heap: the memory that the GC would free if this object were freed, that is everything it keeps alive. For a leak, the retained heap is what matters. The class UnboundedCache is the best example: its shallow heap is 8 bytes, its retained heap 17.9 MB.\nStep 4: the dominator tree For MAT, an object x dominates an object y if every path from a GC root to y goes through x. The dominator tree sorts the objects with this rule, from the biggest retained heap to the smallest:\nClass Name Shallow Heap Retained Heap Percentage class UnboundedCache @ 0x124213cd0 8 17,919,200 93.87% java.lang.Object[2708] @ 0x133e00000 JNI Global 10,848 105,528 0.55% java.lang.Object[8187] @ 0x133e71f40 JNI Global 32,768 48,424 0.25% The first line crushes all the others: this is where to dig.\nStep 5: the path to GC roots What is left is why a Report object stays alive. \u0026ldquo;Path to GC Roots\u0026rdquo; goes up from the object to a GC root, that is an object reachable from outside the heap, such as a loaded class (\u0026ldquo;System Class\u0026rdquo;). We exclude weak and soft references: they do not stop the GC from freeing an object.\nRead the path from the bottom up: the root keeps the class UnboundedCache, its static field CACHE keeps the HashMap, which keeps its array, which keeps each Report. It is exactly the chain of the diagram in the post about memory leaks, but this time found in a real dump.\nThe fix, and the heap curve The fix: give the cache a maximum size. In BoundedCache.java, a LinkedHashMap removes the oldest entry above 500 with removeEldestEntry. The heap after each GC goes up while the cache fills, then does not move anymore:\njava -Xmx256m -Xlog:gc heap-dump/BoundedCache.java [0.348s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 48M-\u0026gt;3M(256M) 1.559ms [16.656s][info][gc] GC(19) Pause Young (Normal) (G1 Evacuation Pause) 156M-\u0026gt;9M(256M) 0.828ms [35.667s][info][gc] GC(41) Pause Young (Normal) (G1 Evacuation Pause) 156M-\u0026gt;9M(256M) 0.984ms How to read these lines is explained in the post about GC logs.\nMAT without a window MAT also has a batch mode, useful on a server with no screen: it analyzes the dump and writes the reports as HTML. The screenshots of this post come from this mode:\nParseHeapDump.sh /tmp/cache.hprof org.eclipse.mat.api:suspects ParseHeapDump.sh /tmp/cache.hprof -command=dominator_tree org.eclipse.mat.api:query The first one writes cache_Leak_Suspects.zip, the second one cache_Query.zip. The full commands, with the path to GC roots, are in the README of the heap-dump folder. Remember to delete the dump and the index files that MAT writes next to it.\nTested with Java 25 (Temurin 25.0.4) and Eclipse MAT 1.17, on macOS; the fix also with Java 11, 17 and 21.\nSources: the Eclipse MAT help (Shallow vs. Retained Heap, Dominator Tree, Garbage Collection Roots, Batch mode), Oracle\u0026rsquo;s troubleshooting guide on memory leaks and the documentation of jcmd (JDK 25).\nFurther reading Java memory leaks: 6 classic cases, reproduced Java OutOfMemoryError: every message, reproduced Reading a Java GC log: every number explained The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/java-heap-dump-eclipse-mat/","summary":"\u003cp\u003eIn \u003ca href=\"/en/blog/java-memory-leaks/\"\u003ethe post about memory leaks\u003c/a\u003e, two\n\u003ccode\u003ejcmd\u003c/code\u003e histograms showed \u003cstrong\u003ewhich\u003c/strong\u003e class grows. But not \u003cstrong\u003ewhat\u003c/strong\u003e keeps\nit alive. For that, you need a heap dump: a picture of the whole heap,\nwith the objects and the links between them.\u003c/p\u003e\n\u003cp\u003eWe take the same leak again:\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/blob/level-2/leaks/UnboundedCache.java\"\u003eUnboundedCache.java\u003c/a\u003e,\na static \u003ccode\u003eHashMap\u003c/code\u003e cache that gets a new key on each request. We analyze\nit with Eclipse Memory Analyzer (MAT), an open source tool that Oracle\u0026rsquo;s\nguide names for heap dump analysis. All the commands are in the\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/tree/level-2/heap-dump\"\u003eheap-dump\u003c/a\u003e\nfolder of the \u003ccode\u003ejava-gc-lab\u003c/code\u003e repository.\u003c/p\u003e","title":"Java heap dump: find a memory leak with Eclipse MAT"},{"content":"In Java, the garbage collector frees the objects that nothing references anymore. So a memory leak is an object that is no longer useful but still referenced: the GC cannot free it. Oracle\u0026rsquo;s guide puts it this way: the application keeps references to objects \u0026ldquo;unintentionally\u0026rdquo;.\nHere are six classic leaks, each one reproduced by a small program of the public repository java-gc-lab, with what you see and the fix.\nWhat keeps an object alive The chain of references of a leak: a GC root, the static field AUDIT of the class StaticCollection, refers to an ArrayList, which refers to Order objects; they stay alive. A short-lived object that nothing refers to anymore is freed by the GC. What keeps an object alive every object reachable from a root stays alive GC root static AUDIT ArrayList the list of the field Order Order Order Order objects useless, but reachable stay alive: the leak short-lived object no reference left → freed by the GC The GC starts from \u0026ldquo;roots\u0026rdquo;: the static fields of the loaded classes, the local variables of the running threads… Every object that can be reached from a root stays alive. A leak is a path that someone forgot to cut.\nCase What keeps the object Fix Static collection a static field remove the elements, or do not keep them there Cache with no limit the map of the cache a maximum size or an expiry Listeners the list of listeners of the event bus remove the listener ThreadLocal in a pool the pool thread, which never stops remove() at the end of the task Class loaders one object of the old version release every reference to the old version Resources not closed nothing… but the file stays open try-with-resources How to see a leak The first sign: the heap after each GC goes up from one GC to the next. It is explained with real logs in the post about OutOfMemoryError.\nTo know which object leaks, take two class histograms, a few seconds or minutes apart, on the running application:\njcmd \u0026lt;pid\u0026gt; GC.class_histogram | head The class whose number of instances goes up is the leak. Be careful: this command runs a full GC first, so avoid it under heavy load on a sensitive production system. Each program of java-gc-lab leaks slowly (about 30 seconds) and prints its process id, to give you time to do it.\nThe six cases 1. A static collection private static final List\u0026lt;Order\u0026gt; AUDIT = new ArrayList\u0026lt;\u0026gt;(); // on each request: AUDIT.add(new Order()); A static field lives as long as its class, and so does the whole list. The histogram, after 5 s and then after 20 s:\n48: 368 5888 StaticCollection$Order 19: 1584 25344 StaticCollection$Order Fix: remove the elements when they are no longer needed, or do not keep them in a static collection.\n2. A cache with no limit private static final Map\u0026lt;String, Report\u0026gt; CACHE = new HashMap\u0026lt;\u0026gt;(); // a new key on each request: the cache only grows CACHE.put(requestId, new Report()); 48: 378 6048 UnboundedCache$Report 19: 1592 25472 UnboundedCache$Report Fix: a maximum size. With the JDK alone, a LinkedHashMap that overrides removeEldestEntry removes the oldest entry when the size goes over a maximum; its javadoc gives it as an example for a cache.\n3. Listeners never removed Screen screen = new Screen(); BUS.register(screen); // the screen is closed, but never removed from the bus 48: 375 6000 Listeners$Screen 19: 1592 25472 Listeners$Screen The event bus lives for a long time, and it keeps every screen. Fix: remove the listener when the screen is closed.\n4. A ThreadLocal in a thread pool pool.submit(() -\u0026gt; { BUFFER.get().add(new Record()); // no BUFFER.remove() }); 48: 376 6016 ThreadLocalInPool$Record 20: 1585 25360 ThreadLocalInPool$Record According to the javadoc, each thread keeps its copy of a ThreadLocal as long as it is alive. The threads of a pool never stop: their buffers grow without end. Fix: BUFFER.remove() at the end of each task, in a finally.\n5. Class loaders ClassLoader loader = new ClassLoader(parent) { }; PLUGINS.add(Proxy.newProxyInstance(loader, interfaces, handler)); Each \u0026ldquo;reload\u0026rdquo; of a plugin creates a class loader and a new class. The list keeps one object of each version, so no old class loader can be freed. Here the heap grows little: the Metaspace grows.\njcmd \u0026lt;pid\u0026gt; VM.metaspace | grep \u0026#34;Total Usage\u0026#34; Total Usage - 393 loaders, 3538 classes (1261 shared): Total Usage - 1515 loaders, 4661 classes (1261 shared): With a limit, it ends with OutOfMemoryError: Metaspace (see the post about OutOfMemoryError). Fix: release every reference to the old version (objects, classes, threads, listeners) before a reload.\n6. Resources not closed FileInputStream in = new FileInputStream(file); in.read(); // no close() This case is different. The stream is not kept anywhere, so it is not a heap leak: in my tests, when a GC runs, the JDK closes the files of the streams it frees (jcmd \u0026lt;pid\u0026gt; GC.run: from about 830 open files to about 90). But this program creates almost no garbage: no GC runs, and the open files pile up (lsof: about 400 after 5 s, 2,000 after 25 s). With a limit of 512 open files (in Docker):\njava.io.FileNotFoundException: /tmp/java-gc-lab8035917117579278064.txt (Too many open files) The heap is fine, and yet the application fails. Fix: try-with-resources. The javadoc of AutoCloseable says it: it ensures a prompt release and avoids running out of resources.\nAvoiding leaks In code reviews, look for static fields that hold collections, collections with no maximum size, register without unregister, ThreadLocal without remove(), streams opened outside a try-with-resources. Set a limit on everything that grows: caches, queues, buffers. Weak references: a WeakHashMap removes an entry when its key is no longer used elsewhere. A trap noted in the javadoc: if the value refers to its own key, the entry never goes away. Watch the heap after GC in production: a slow leak shows over hours, not over one minute. Tested with Java 8 (compiling, and the file case), 11, 17, 21 and 25 (Temurin 11.0.32, 17.0.20, 21.0.9, 25.0.4), on macOS; the file case also in Docker (eclipse-temurin:8 and :25).\nSources: Oracle\u0026rsquo;s troubleshooting guide on memory leaks, the documentation of jcmd, and the JDK 25 javadoc of ThreadLocal, LinkedHashMap, WeakHashMap and AutoCloseable.\nFurther reading Java heap dump: find a memory leak with Eclipse MAT Java OutOfMemoryError: every message, reproduced Reading a Java GC log: every number explained Common myths about the Java garbage collector The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/java-memory-leaks/","summary":"\u003cp\u003eIn Java, the garbage collector frees the objects that nothing\nreferences anymore. So a memory leak is an object that is no longer\nuseful but still referenced: the GC cannot free it. Oracle\u0026rsquo;s guide puts\nit this way: the application keeps references to objects\n\u0026ldquo;unintentionally\u0026rdquo;.\u003c/p\u003e\n\u003cp\u003eHere are six classic leaks, each one reproduced by a small program of\nthe public repository\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/tree/level-2/leaks\"\u003ejava-gc-lab\u003c/a\u003e,\nwith what you see and the fix.\u003c/p\u003e\n\u003ch2 id=\"what-keeps-an-object-alive\"\u003eWhat keeps an object alive\u003c/h2\u003e\n\n\u003cfigure class=\"aj-svg\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" class=\"gc-leak-chain\" viewBox=\"0 0 720 250\" width=\"720\" height=\"250\" role=\"img\" aria-labelledby=\"gc-leak-chain-title\"\u003e\n  \u003ctitle id=\"gc-leak-chain-title\"\u003eThe chain of references of a leak: a GC root, the static field AUDIT of the class StaticCollection, refers to an ArrayList, which refers to Order objects; they stay alive. A short-lived object that nothing refers to anymore is freed by the GC.\u003c/title\u003e\n  \u003cstyle\u003e\n    svg.gc-leak-chain .proc { fill: var(--entry, #ffffff); stroke: var(--tertiary, #d5dbe6); stroke-width: 1.5; }\n    svg.gc-leak-chain .box { fill: var(--code-bg, #f3f6fb); stroke: var(--tertiary, #d5dbe6); stroke-width: 1.5; }\n    svg.gc-leak-chain .root { fill: color-mix(in srgb, var(--accent, #2563eb) 6%, transparent); stroke: var(--accent, #2563eb); stroke-width: 2; }\n    svg.gc-leak-chain .obj { fill: var(--entry, #ffffff); stroke: var(--accent, #2563eb); stroke-width: 1.5; }\n    svg.gc-leak-chain .free { fill: none; stroke: var(--secondary, #6c6c6c); stroke-width: 1.5; stroke-dasharray: 5 4; }\n    svg.gc-leak-chain .arrow { stroke: var(--accent, #2563eb); stroke-width: 1.5; fill: none; marker-end: url(#gc-leak-chain-arrow); }\n    svg.gc-leak-chain .ah { fill: var(--accent, #2563eb); }\n    svg.gc-leak-chain text { font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, sans-serif; fill: var(--primary, #1e1e1e); }\n    svg.gc-leak-chain .h { font-size: 15px; font-weight: 700; }\n    svg.gc-leak-chain .t { font-size: 13px; font-weight: 600; }\n    svg.gc-leak-chain .s { font-size: 12px; fill: var(--secondary, #6c6c6c); }\n    svg.gc-leak-chain .m { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }\n  \u003c/style\u003e\n  \u003cdefs\u003e\n    \u003cmarker id=\"gc-leak-chain-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"\u003e\n      \u003cpath class=\"ah\" d=\"M0,0 L10,5 L0,10 z\"/\u003e\n    \u003c/marker\u003e\n  \u003c/defs\u003e\n  \u003crect class=\"proc\" x=\"1\" y=\"1\" width=\"718\" height=\"248\" rx=\"12\"/\u003e\n  \u003ctext class=\"h\" x=\"20\" y=\"28\"\u003eWhat keeps an object alive\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"226\" y=\"28\"\u003eevery object reachable from a root stays alive\u003c/text\u003e\n\n  \u003c!-- Root --\u003e\n  \u003crect class=\"root\" x=\"20\" y=\"64\" width=\"170\" height=\"64\" rx=\"8\"/\u003e\n  \u003ctext class=\"t\" x=\"34\" y=\"90\"\u003eGC root\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"34\" y=\"110\"\u003e\u003ctspan class=\"m\"\u003estatic AUDIT\u003c/tspan\u003e\u003c/text\u003e\n\n  \u003cpath class=\"arrow\" d=\"M190,96 L238,96\"/\u003e\n\n  \u003c!-- List --\u003e\n  \u003crect class=\"box\" x=\"240\" y=\"64\" width=\"150\" height=\"64\" rx=\"8\"/\u003e\n  \u003ctext class=\"t\" x=\"254\" y=\"90\"\u003e\u003ctspan class=\"m\"\u003eArrayList\u003c/tspan\u003e\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"254\" y=\"110\"\u003ethe list of the field\u003c/text\u003e\n\n  \u003cpath class=\"arrow\" d=\"M390,84 L478,58\"/\u003e\n  \u003cpath class=\"arrow\" d=\"M390,96 L478,96\"/\u003e\n  \u003cpath class=\"arrow\" d=\"M390,108 L478,134\"/\u003e\n\n  \u003c!-- Leaking objects --\u003e\n  \u003crect class=\"obj\" x=\"480\" y=\"44\" width=\"110\" height=\"28\" rx=\"14\"/\u003e\n  \u003ctext class=\"m\" x=\"508\" y=\"63\"\u003eOrder\u003c/text\u003e\n  \u003crect class=\"obj\" x=\"480\" y=\"82\" width=\"110\" height=\"28\" rx=\"14\"/\u003e\n  \u003ctext class=\"m\" x=\"508\" y=\"101\"\u003eOrder\u003c/text\u003e\n  \u003crect class=\"obj\" x=\"480\" y=\"120\" width=\"110\" height=\"28\" rx=\"14\"/\u003e\n  \u003ctext class=\"m\" x=\"508\" y=\"139\"\u003eOrder\u003c/text\u003e\n  \u003ctext class=\"t\" x=\"604\" y=\"90\"\u003eOrder\u003c/text\u003e\n  \u003ctext class=\"t\" x=\"604\" y=\"108\"\u003eobjects\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"480\" y=\"172\"\u003euseless, but reachable\u003c/text\u003e\n  \u003ctext class=\"t\" x=\"480\" y=\"190\"\u003estay alive: the leak\u003c/text\u003e\n\n  \u003c!-- Freed object --\u003e\n  \u003crect class=\"free\" x=\"240\" y=\"190\" width=\"150\" height=\"40\" rx=\"8\"/\u003e\n  \u003ctext class=\"s\" x=\"254\" y=\"208\"\u003eshort-lived object\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"254\" y=\"223\"\u003eno reference left\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"404\" y=\"215\"\u003e→ freed by the GC\u003c/text\u003e\n\u003c/svg\u003e\n\n\u003c/figure\u003e\n\n\u003cp\u003eThe GC starts from \u0026ldquo;roots\u0026rdquo;: the static fields of the loaded classes,\nthe local variables of the running threads… Every object that can be\nreached from a root stays alive. A leak is a path that someone forgot\nto cut.\u003c/p\u003e","title":"Java memory leaks: 6 classic cases, reproduced"},{"content":"Turning on the GC log is easy. Reading it is harder: one G1 line holds about ten pieces of information, and the words change from one Java version to the next.\nThis post reads real lines, written by three small programs of the public repository java-gc-lab: a normal load, very big objects, and a heap that is almost full.\nTurn on the log java -Xlog:gc*:file=gc.log:time,uptime,level,tags -jar app.jar gc*: all the GC messages (gc alone gives one line per GC); file=gc.log: in a file, with rotation built in (5 files of about 20 MB by default); time,uptime,level,tags: the \u0026ldquo;decorations\u0026rdquo; at the start of each line: the date and time, the time since the JVM started, the level and the tags. If you set nothing, the JVM uses uptime, level and tags. A pause line, with these decorations:\n[2026-10-22T23:23:46.842+0200][0.356s][info][gc ] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 117M-\u0026gt;24M(256M) 1.660ms The exact time (time) helps to link a pause to an event of the application, for example a slow request in another log.\nOne line, word by word One G1 log line on Java 25, cut into five parts: GC(1), the number of the GC; Pause Young (Normal), the type of pause; (G1 Evacuation Pause), the cause; 117M-\u0026gt;23M(256M), the heap before and after the GC and its size; 5.790ms, the length of the pause One G1 log line Java 25, -Xlog:gc GC(1) GC number Pause Young (Normal) type of pause (G1 Evacuation Pause) cause of the GC 117M-\u0026gt;23M(256M) heap before -\u0026gt; after (size) 5.790ms pause time Part Meaning [0.469s] the time since the JVM started (uptime) GC(1) the number of the GC: all the lines of one GC have the same one Pause Young (Normal) the type of pause: here a normal young pause. You also see Concurrent Start, Prepare Mixed, Mixed, Remark, Cleanup and Full (G1 Evacuation Pause) the cause of the GC, here the usual one. Other causes: G1 Humongous Allocation, System.gc(), G1 Compaction Pause (full GC) 117M-\u0026gt;23M(256M) the heap used before the GC, after the GC, and the heap size 5.790ms the length of the pause: the application is stopped during this time The words young, mixed and full are explained in the post about generations.\nThe 4 numbers to follow Here are the start and the end of the log of NormalLoad.java, an application that keeps about 16 MB of data and creates short-lived objects all the time:\njava -Xmx256m -Xlog:gc gc-logs/NormalLoad.java [0.341s][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 47M-\u0026gt;18M(256M) 3.090ms [0.469s][info][gc] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 117M-\u0026gt;23M(256M) 5.790ms [0.646s][info][gc] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 149M-\u0026gt;24M(256M) 4.129ms … [3.240s][info][gc] GC(15) Pause Young (Normal) (G1 Evacuation Pause) 171M-\u0026gt;24M(256M) 0.635ms [3.478s][info][gc] GC(16) Pause Young (Normal) (G1 Evacuation Pause) 171M-\u0026gt;24M(256M) 12.874ms The GC rate: 17 GCs in a little more than 3 seconds, about 5 per second. It goes up when the application creates more objects, or when the heap is too small. The pause time: from 0.4 to 12.9 ms here. Look at the longest ones first: they are the ones that users feel. The heap after the GC: 23 to 24 MB, stable. It is roughly the size of the live data. If it goes up from one GC to the next, it is a leak (see the post about OutOfMemoryError). The allocation rate: the heap before a GC, minus the heap after the previous GC, divided by the time between them. Between GC(0) and GC(1): (117 − 18) MB in 0.128 s, so about 770 MB/s. No line gives it directly: you have to compute it. Warning signs Humongous objects java -Xmx256m -Xlog:gc gc-logs/Humongous.java [0.306s][info][gc] GC(0) Pause Young (Concurrent Start) (G1 Humongous Allocation) 146M-\u0026gt;9M(256M) 1.583ms For G1, a \u0026ldquo;humongous\u0026rdquo; object is at least half a region. It does not go through eden: it gets its own regions, directly in the old generation. Here the program creates 2 MB arrays with 1 MB regions (Heap Region Size: 1M at startup with -Xlog:gc*), and the allocation itself starts the GC. Many G1 Humongous Allocation lines point to big arrays or big buffers created often.\nAn evacuation failure, then Pause Full java -Xmx64m -Xlog:gc gc-logs/NearTheLimit.java [0.295s][info][gc] GC(5) Pause Young (Prepare Mixed) (G1 Evacuation Pause) (Evacuation Failure: Allocation) 61M-\u0026gt;47M(64M) 0.250ms [1.348s][info][gc] GC(175) Pause Full (G1 Compaction Pause) 62M-\u0026gt;45M(64M) 5.691ms The program keeps about 30 MB of data in a 64 MB heap. G1 no longer finds enough room to copy the live objects: this is an evacuation failure. The reason is Allocation (no room) or Pinned (an object held in place by native code). If a GC frees nothing at all, G1 starts a full GC, which compacts the whole heap and can be very slow.\nBefore Java 18: the evacuation failure was called to-space exhausted. On Java 11 and 17, it was a separate line (GC(7) To-space exhausted); on Java 8, the end of the pause line. Java 21 writes (Evacuation Failure), and Java 25 adds the reason.\nA Pause Full line with G1, when it does not come from System.gc(), is always worth a look: the heap is too small for the live data, or there is a leak.\nPauses that get longer If the last number of the lines goes up over the hours, with the same type of pause, the GC has more and more work: more live data to copy, or a heap that fills up. Follow it closely.\nThe tools JFR (Java Flight Recorder), in the JDK: start it with -XX:StartFlightRecording, and read the recording with JDK Mission Control. It shows the GC pauses with the rest of the application. More details in the post about tools. Third-party tools also read the log files and draw charts, for example GCeasy (an online service) or GCViewer (open source). What has changed What you still read What is true today Since -XX:+PrintGCDetails, -Xloggc:gc.log -Xlog:gc*, -Xlog:gc*:file=gc.log (the old options are deprecated or refused, see the JVM options) Java 9 to-space exhausted Evacuation Failure, then Evacuation Failure: \u0026lt;reason\u0026gt; Java 18 (JDK-8269022), the reason on Java 25 in my tests -XX:+PrintGCDateStamps to get the time the time decoration of -Xlog Java 9 Tested with Java 8 (Corretto 8u504), 11, 17, 21 and 25 (Temurin 11.0.32, 17.0.20, 21.0.9, 25.0.4), on macOS, with G1.\nSources: the documentation of the java command (JDK 25, -Xlog), the G1 chapter of Oracle\u0026rsquo;s GC guide (humongous objects, evacuation failure, full GC), JDK-8269022 (the new name of the evacuation failure).\nFurther reading GC generations and cycles: young, old, minor, major, full Java OutOfMemoryError: every message, reproduced JVM options that matter, and the ones that no longer work The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/read-java-gc-log/","summary":"\u003cp\u003eTurning on the GC log is easy. Reading it is harder: one G1 line holds\nabout ten pieces of information, and the words change from one Java\nversion to the next.\u003c/p\u003e\n\u003cp\u003eThis post reads real lines, written by three small programs of the\npublic repository\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/tree/level-2/gc-logs\"\u003ejava-gc-lab\u003c/a\u003e:\na normal load, very big objects, and a heap that is almost full.\u003c/p\u003e\n\u003ch2 id=\"turn-on-the-log\"\u003eTurn on the log\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003ejava -Xlog:gc*:file=gc.log:time,uptime,level,tags -jar app.jar\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003e\u003ccode\u003egc*\u003c/code\u003e: all the GC messages (\u003ccode\u003egc\u003c/code\u003e alone gives one line per GC);\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003efile=gc.log\u003c/code\u003e: in a file, with rotation built in (5 files of about\n20 MB by default);\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003etime,uptime,level,tags\u003c/code\u003e: the \u0026ldquo;decorations\u0026rdquo; at the start of each\nline: the date and time, the time since the JVM started, the level and\nthe tags. If you set nothing, the JVM uses \u003ccode\u003euptime\u003c/code\u003e, \u003ccode\u003elevel\u003c/code\u003e and\n\u003ccode\u003etags\u003c/code\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eA pause line, with these decorations:\u003c/p\u003e","title":"Reading a Java GC log: every number explained"},{"content":"An OutOfMemoryError is not one single problem. The text after the colon tells you which memory area is full: the heap, the Metaspace, the direct memory or the threads. And each area is checked and fixed in a different way.\nFor each message, I wrote a small program that causes it. They are in the public repository java-gc-lab, one file per message, to run with java without compiling first (Java 11 or later).\nThe message tells you which area is full The memory areas of a Java process and the OutOfMemoryError of each one: unable to create native thread for the thread stacks, Java heap space and GC overhead limit exceeded for the heap, Metaspace for the Metaspace, Cannot reserve … bytes of direct buffer memory for the direct memory Java process each message tells you which area is full Thread stacks one system thread for each new Thread · -Xss limit: the system unable to create native thread Heap shared · managed by the GC -Xmx Java heap space GC overhead limit exceeded Parallel; G1 since 25.0.3 Metaspace native memory -XX:MaxMetaspaceSize Metaspace Direct memory ByteBuffer.allocateDirect() · -XX:MaxDirectMemorySize Cannot reserve … bytes of direct buffer memory (allocated: …, limit: …) on Java 17, 21 and 25 · outside the heap · Java 8 and 11: Direct buffer memory Message Full area First check Java heap space the heap the heap after each GC (-Xlog:gc*, jcmd \u0026lt;pid\u0026gt; GC.heap_info) GC overhead limit exceeded the heap, almost full all the time the same: the GC runs nonstop Metaspace the class metadata jcmd \u0026lt;pid\u0026gt; VM.metaspace Cannot reserve … bytes of direct buffer memory the direct memory -XX:MaxDirectMemorySize, jcmd \u0026lt;pid\u0026gt; VM.native_memory unable to create native thread the threads of the system jcmd \u0026lt;pid\u0026gt; Thread.print, the system limits Java heap space java -Xmx64m oom/HeapSpace.java java.lang.OutOfMemoryError: Java heap space The program keeps 1 MB arrays in a list: the GC cannot free anything, and the 64 MB heap is full in less than a second.\nOracle\u0026rsquo;s guide is clear: this message does not always mean a memory leak. The heap can simply be too small for the application. But in an application that runs for a long time, it is often the sign that it keeps references to objects without meaning to.\nTo know which one it is, look at the heap after each GC: with -Xlog:gc*, or with jcmd \u0026lt;pid\u0026gt; GC.heap_info on a running application. If it always goes back to the same level, the heap is just too small. If it goes up from one GC to the next, something keeps the objects.\nGC overhead limit exceeded java -Xmx32m -XX:+UseParallelGC oom/GcOverhead.java java.lang.OutOfMemoryError: GC overhead limit exceeded Here, the heap fills up slowly, with small objects. The GC runs nonstop and frees almost nothing. The JVM then prefers to stop rather than go on without making progress: according to the docs, it throws this error if more than 98% of the time goes to the GC and less than 2% of the heap is recovered, for five GCs in a row.\nThis message is not guaranteed. With the same program and the Parallel GC, here is what I get:\n-Xmx Java 8 Java 11 Java 17 Java 21 Java 25 16m GC overhead Java heap space Java heap space Java heap space Java heap space 32m no error after 60 s GC overhead GC overhead GC overhead no error after 60 s 128m no error after 60 s GC overhead GC overhead GC overhead Java heap space \u0026ldquo;No error after 60 s\u0026rdquo; is the worst case: the JVM spends its time in the GC and the application does nothing, with no error.\nFor a long time, only Parallel threw this error. G1, the default GC, has it too since Java 26, with a backport in Java 25.0.3 (JDK-8212084). But in my tests with G1 on Java 25.0.4, from -Xmx16m to -Xmx256m, I always get Java heap space. According to the guide, the usual cause is the same: live data that barely fits in the heap.\nMetaspace java -XX:MaxMetaspaceSize=32m oom/MetaspaceFull.java java.lang.OutOfMemoryError: Metaspace The Metaspace holds the class metadata, in native memory. The program creates a new class loader and a new class in it, again and again; since it keeps them, nothing can be unloaded.\nBy default, the Metaspace has no limit. If a limit is set with -XX:MaxMetaspaceSize, the guide advises to raise it. If the Metaspace keeps growing (jcmd \u0026lt;pid\u0026gt; VM.metaspace), the problem is more likely classes that cannot be unloaded, as in this program.\nDirect buffer memory java -XX:MaxDirectMemorySize=32m oom/DirectBuffer.java java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 32514048, limit: 33554432) Direct buffers (ByteBuffer.allocateDirect) live outside the heap. Their limit is -XX:MaxDirectMemorySize; without this option, the JVM chooses it.\nBefore Java 17: on Java 8 and 11, the message was just Direct buffer memory. Since Java 17, in my tests, it gives the size asked for, the memory already taken and the limit, which helps a lot.\nTo see this memory on a running application: -XX:NativeMemoryTracking=summary at startup, then jcmd \u0026lt;pid\u0026gt; VM.native_memory.\nUnable to create native thread This program starts threads until the system says no. Do not run it on your machine without a limit: I ran it in a container limited to 200 processes.\ndocker run --rm --pids-limit 200 -v \u0026#34;$PWD/oom\u0026#34;:/oom eclipse-temurin:25 java /oom/NativeThreads.java java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached Before Java 11: on Java 8, the message was unable to create new native thread.\nEach thread started with new Thread is a thread of the operating system, with its own stack outside the heap. So the cause is a limit of the system (number of processes or threads, memory), or an application that creates threads without end, for example a pool with no maximum size. jcmd \u0026lt;pid\u0026gt; Thread.print shows all the threads, with their stack traces. Raising -Xmx does not change a system limit.\nThe method, whatever the message Read the message: it tells you which area is full (table at the top). Get a heap dump: with -XX:+HeapDumpOnOutOfMemoryError, the JVM writes an .hprof file. In my tests, it is written for Java heap space, GC overhead limit exceeded and Metaspace, but not for direct buffers. Keep some disk space free: the file can be several GB. Look at the heap after each GC: stable, the heap is too small; going up, it is a leak (see below). Fix: give more room to the right area, or find what keeps the objects. Heap too small, or a leak? With -Xlog:gc, each GC writes one line. The number that matters is the one after the arrow: the heap after the GC, so roughly the data that is still alive.\nWith SteadyLoad.java, the live data does not change. After each GC, the heap goes back to the same level:\njava -Xmx64m -Xlog:gc oom/SteadyLoad.java GC(2) Pause Young (Normal) (G1 Evacuation Pause) 41M-\u0026gt;24M(64M) 0.379ms GC(5) Pause Young (Normal) (G1 Evacuation Pause) 47M-\u0026gt;24M(64M) 0.538ms GC(9) Pause Young (Normal) (G1 Evacuation Pause) 54M-\u0026gt;24M(64M) 0.624ms With SlowLeak.java, the program keeps 1 MB more in each round. After each GC, the heap goes up, until Java heap space:\njava -Xmx64m -Xlog:gc oom/SlowLeak.java GC(1) Pause Young (Normal) (G1 Evacuation Pause) 30M-\u0026gt;5M(64M) 2.292ms GC(4) Pause Young (Normal) (G1 Evacuation Pause) 42M-\u0026gt;11M(64M) 0.916ms GC(8) Pause Young (Normal) (G1 Evacuation Pause) 50M-\u0026gt;19M(64M) 0.938ms GC(12) Pause Young (Normal) (G1 Evacuation Pause) 54M-\u0026gt;29M(64M) 1.283ms On an application that is already running, without a restart, jstat -gcutil \u0026lt;pid\u0026gt; 1s prints the use of each area every second: in the second case, the O column (the old generation, in %) goes up the same way. Its docs say that it is experimental and unsupported.\nRaising -Xmx is not a mistake in itself: it is even the first action that Oracle\u0026rsquo;s guide gives for Java heap space. The mistake is to raise it again and again when the heap goes up after each GC: the leak is still there, it just takes longer to fill the heap.\nWhat has changed What you still read What is true today Since OutOfMemoryError: Direct buffer memory Cannot reserve … bytes of direct buffer memory (allocated: …, limit: …) Java 17 in my tests (11 still had the old one) unable to create new native thread unable to create native thread: possibly out of memory or process/resource limits reached Java 11 in my tests (8 had the old one) GC overhead limit exceeded comes only from Parallel G1 can throw it too Java 26, and Java 25.0.3 as a backport PermGen space Metaspace Java 8 Tested with Java 8 (Corretto 8u504), 11, 17, 21 and 25 (Temurin 11.0.32, 17.0.20, 21.0.9, 25.0.4), on macOS; the thread case in Docker (eclipse-temurin).\nSources: Oracle\u0026rsquo;s troubleshooting guide on memory leaks, the documentation of the java, jcmd and jstat commands (JDK 25), JDK-8212084 (G1 and the overhead limit).\nFurther reading JVM memory: heap, stack and Metaspace JVM options that matter, and the ones that no longer work The Java garbage collector: an up-to-date guide (Java 25) ","permalink":"https://aymenjamaaoui.dev/en/blog/java-outofmemoryerror-messages/","summary":"\u003cp\u003eAn \u003ccode\u003eOutOfMemoryError\u003c/code\u003e is not one single problem. The text after the\ncolon tells you which memory area is full: the heap, the Metaspace,\nthe direct memory or the threads. And each area is checked and fixed\nin a different way.\u003c/p\u003e\n\u003cp\u003eFor each message, I wrote a small program that causes it. They are in\nthe public repository\n\u003ca href=\"https://gitlab.com/aymenjamaaoui/java-gc-lab/-/tree/level-2/oom\"\u003ejava-gc-lab\u003c/a\u003e,\none file per message, to run with \u003ccode\u003ejava\u003c/code\u003e without compiling first (Java\n11 or later).\u003c/p\u003e","title":"Java OutOfMemoryError: every message, reproduced"},{"content":"There are many posts about the Java garbage collector, and many of them are old: PermGen, CMS, -XX:+PrintGCDetails… This guide sums up what is true in Java 25, in a few lines per topic. Each part links to a longer post, where every fact is checked in the official documentation or on a real JVM.\nI update this guide when the series moves on, or when a new Java version changes the rules.\nJVM memory The JVM keeps its data in several areas:\nthe stack: one per thread, with one frame per method call (parameters, local variables). It is freed by itself, without the GC. Too many nested calls give a StackOverflowError; the heap: the objects created with new, shared by all threads. This is the area of the garbage collector. Its maximum size is set with -Xmx; the Metaspace: the description of the classes, in native memory, outside the heap. It replaced PermGen in Java 8. The JVM also uses native memory for thread stacks, the code compiled by the JIT, direct buffers and its own internal data. -Xmx only limits the heap: the Java process always uses more.\n→ JVM memory: heap, stack and Metaspace\nGenerations and cycles Most objects die young. So the JVM keeps recent objects in the young generation (eden and two survivor spaces) and long-lived ones in the old generation:\nHow an object moves in the heap: it is created in eden, moves from one survivor space to the other at each minor collection, then it is promoted to the old generation Heap Young new objects Eden objects are created here object Survivor 0 the two survivor spaces Survivor 1 take turns; one is empty Old long-lived objects object 1 2 3 1 Minor collection: the live object is copied from eden to a survivor space. 2 At each minor collection, it moves to the other survivor space; its age goes up. 3 After enough copies (MaxTenuringThreshold, 15 by default): promoted to old. An object is born in eden, moves from one survivor space to the other at each GC, and is then promoted to the old generation after a number of copies (15 at most by default).\nIn Oracle\u0026rsquo;s guide, a minor collection cleans only the young generation, and a major collection cleans the whole heap. G1, the default GC, does not use these words in its logs: it talks about Young, Mixed and Full pauses. And since Java 9, it splits the heap into regions of the same size, instead of two big blocks.\n→ GC generations and cycles: young, old, minor, major, full\nThe garbage collectors Java 25 offers several GCs, each one for a need:\nG1, the default GC since Java 9: when response time matters more than throughput; ZGC and Shenandoah: very short pauses, for latency; Parallel: throughput first; Serial: a single thread, for small machines; Epsilon: never frees anything, for tests. CMS is gone since Java 14. Also watch the automatic choice: on a machine with a single processor or less than 1792 MB of memory, the JVM picks Serial instead of G1. So a small container can run with a different GC than the developer\u0026rsquo;s machine. To check it: -Xlog:gc writes the GC in use on its first line (Using G1).\n→ Java garbage collectors in 2026: which one is running yours?\nUseful options A few options are enough in most cases:\n-Xms and -Xmx: the initial and maximum heap size, often the same value on a server; -Xlog:gc*:file=gc.log: GC logs to a file, with rotation built in; -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath: a heap dump when the heap is full; -XX:MaxRAMPercentage: the maximum heap as a percentage of memory (25% by default), useful in a container. Many old options no longer work. Some are only deprecated (-XX:+PrintGCDetails, -Xloggc), but others stop the JVM at startup in Java 17, 21 and 25: -XX:MaxPermSize, -XX:+UseConcMarkSweepGC, -XX:+PrintGCDateStamps… To test a configuration, just run java \u0026lt;options\u0026gt; -version.\n→ JVM options that matter, and the ones that no longer work\nCommon myths System.gc() is only a request. But with G1, Parallel and Serial, it usually starts a full GC, which is expensive. finalize() is deprecated for removal: use try-with-resources or Cleaner instead. More heap often helps, as long as it fits in the memory of the machine or the container. Tuning the GC is not the first thing to do: G1 with its default settings, then the logs, then the application. The GC does not prevent memory leaks: an object that is still referenced is never freed. → Common myths about the Java garbage collector\nFAQ What is the difference between the heap and the stack? The stack belongs to one thread: it holds the local variables and is emptied when the method ends. The heap is shared by all threads: it holds the objects, and the GC cleans it. A local variable of an object type is on the stack, but the object itself is in the heap.\nWhat is the default GC in Java 25? G1, on a \u0026ldquo;server-class\u0026rdquo; machine: at least two processors and at least 1792 MB of memory. Otherwise, the JVM picks Serial.\nCan I force the GC? No. System.gc() is a request, with no guarantee. With G1, Parallel and Serial, it usually gives a full GC. With -XX:+DisableExplicitGC, the JVM ignores these calls.\nWhy does my application freeze for a moment? GC pauses are one possible cause, not the only one. To find out, look at the GC logs (-Xlog:gc*): each pause is written there with its duration. With G1, a Pause Full that does not come from System.gc() is a sign that the heap is too full, and the docs warn that it can be very slow. Full GCs that keep coming back point to a heap that is too small or a memory leak.\nWhich GC for a Spring Boot application? Spring Boot does not choose the GC: the JVM does, with the same rules as for any Java application. A Spring Boot 3.5 application on Java 25 starts with G1, and with Serial if the JVM sees only one processor. Start with G1 and its default settings, and check the GC in use with -Xlog:gc.\nWhat comes next in the series This guide covers the basics. The rest of the series will go further:\nunderstand and diagnose: the kinds of OutOfMemoryError, memory leaks, reading GC logs, the tools (JFR, jcmd, VisualVM) and heap dump analysis; measure and tune: Java in Docker and Kubernetes, off-heap memory, G1 or ZGC with measures, tuning a Spring Boot and Kafka service, and the GC with virtual threads. These posts will be added to this guide as they come out.\n","permalink":"https://aymenjamaaoui.dev/en/blog/java-garbage-collector-guide/","summary":"\u003cp\u003eThere are many posts about the Java garbage collector, and many of them\nare old: PermGen, CMS, \u003ccode\u003e-XX:+PrintGCDetails\u003c/code\u003e… This guide sums up what is\ntrue in \u003cstrong\u003eJava 25\u003c/strong\u003e, in a few lines per topic. Each part links to a\nlonger post, where every fact is checked in the official documentation\nor on a real JVM.\u003c/p\u003e\n\u003cp\u003eI update this guide when the series moves on, or when a new Java version\nchanges the rules.\u003c/p\u003e","title":"The Java garbage collector: an up-to-date guide (Java 25)"},{"content":"You still read a lot of advice about the garbage collector. Some of it was wrong from the start, and some of it is not true anymore.\nHere are five common myths, and what the official documentation and a real JVM say in Java 25.\n\u0026ldquo;System.gc() forces a GC\u0026rdquo; No: it is a request, not an order. The javadoc says that System.gc() \u0026ldquo;suggests\u0026rdquo; that the JVM frees memory, with no guarantee about how much is freed, or even that the work is done at all.\nIn practice, on Java 25, the request is often followed, and it is expensive:\nGC What System.gc() does G1 (default), Parallel, Serial A full GC: Pause Full (System.gc()) ZGC A major collection: Major Collection (System.gc()) G1 with -XX:+ExplicitGCInvokesConcurrent A young pause, then a concurrent cycle With -XX:+DisableExplicitGC Nothing: the call is ignored With -XX:+DisableExplicitGC, the JVM ignores calls to System.gc(), but it still runs a GC when it needs one.\n\u0026ldquo;finalize() is the place to free resources\u0026rdquo; No. finalize() is deprecated since Java 9, and deprecated for removal since Java 18 (JEP 421). The GC can call it very late, with no guaranteed delay. It runs on any thread, in any order. And if the finalizer thread cannot keep up, the objects that wait for it can fill the heap, up to an OutOfMemoryError.\nUse instead:\na close() method, with AutoCloseable and try-with-resources: the close is guaranteed, even when an exception is thrown; Cleaner (Java 9), to run a cleanup action when the object can no longer be reached. To check that an application no longer depends on finalizers, you can start it with --finalization=disabled (since Java 18).\n\u0026ldquo;More heap is always better\u0026rdquo; Often true, but not always. Oracle\u0026rsquo;s guide says to give the JVM as much memory as possible, unless you have problems with pauses. The default size is often too small.\nBut there are limits:\nthe maximum heap must stay below the memory of the machine, or the system starts paging; the JVM also uses memory outside the heap (Metaspace, thread stacks…): in a container, all of it must fit in the limit; with G1, a bigger heap usually makes the marking phase longer. The options to set the heap size are in the post about JVM options.\n\u0026ldquo;Tuning the GC is the first thing to do\u0026rdquo; No. Oracle\u0026rsquo;s guide says to start the application first and let the JVM choose the GC. For G1, its general advice is to keep the default settings, with at most a different pause time goal and a maximum heap size (-Xmx).\nBefore you change an option, look at what happens: the GC logs are a good start (-Xlog:gc*). And the guide reminds us: a change in the application can work better than tuning the JVM.\n\u0026ldquo;The GC prevents memory leaks\u0026rdquo; No. The GC frees the objects that nothing references anymore. But for Oracle, a memory leak in Java is exactly an application that keeps references to objects or classes without meaning to: a cache that only grows, a list that is never cleared… The GC cannot free them. The heap fills up, or the Metaspace for classes, up to an OutOfMemoryError.\nThere are also native memory leaks, outside the heap. For the ones inside the JVM itself, the Native Memory Tracking (NMT) tool helps to find them.\nWhat has changed What you still read What is true today Since Free resources in finalize() finalize() is deprecated for removal; use try-with-resources or Cleaner Java 9 (deprecated), Java 18 (for removal) A class leak fills the PermGen PermGen is gone: the Metaspace fills up Java 8 Tune the CMS options CMS is gone; G1 is the default GC, and its default settings are the advice Java 9 (G1 default), Java 14 (CMS removed) Sources: the javadoc of System.gc() and Object.finalize(), JEP 421 (finalization), the documentation of the java command, Oracle\u0026rsquo;s garbage collection guide and the troubleshooting guide about memory leaks (JDK 25).\nSummary System.gc() is an expensive request, finalize() is something to forget, a bigger heap helps as long as it fits in memory, you measure before you tune, and the GC does not protect you from references you forgot.\nFurther reading Java memory leaks: 6 classic cases, reproduced The Java garbage collector: an up-to-date guide (Java 25) JVM options that matter, and the ones that no longer work GC generations and cycles: young, old, minor, major, full ","permalink":"https://aymenjamaaoui.dev/en/blog/java-garbage-collector-myths/","summary":"\u003cp\u003eYou still read a lot of advice about the garbage collector. Some of it\nwas wrong from the start, and some of it is not true anymore.\u003c/p\u003e\n\u003cp\u003eHere are five common myths, and what the official documentation and a\nreal JVM say in Java 25.\u003c/p\u003e\n\u003ch2 id=\"systemgc-forces-a-gc\"\u003e\u0026ldquo;\u003ccode\u003eSystem.gc()\u003c/code\u003e forces a GC\u0026rdquo;\u003c/h2\u003e\n\u003cp\u003eNo: it is a request, not an order. The javadoc says that \u003ccode\u003eSystem.gc()\u003c/code\u003e\n\u0026ldquo;suggests\u0026rdquo; that the JVM frees memory, with no guarantee about how much\nis freed, or even that the work is done at all.\u003c/p\u003e","title":"Common myths about the Java garbage collector"},{"content":"You take an old startup script, you move to Java 25, and the JVM does not start anymore:\nUnrecognized VM option \u0026#39;PrintGCDateStamps\u0026#39; Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Many memory and GC options that you still find on the web have changed or are gone. Some are just ignored, others stop the JVM at startup. And the options that are useful today fit in a few lines.\nThe options that matter Option What it does When to change it When to leave it alone -Xms, -Xmx Initial and maximum heap size Almost always on a server: the default is often too small — -Xss Stack size of each thread After a StackOverflowError in a recursion that is really needed Otherwise: the default is enough -XX:+UseG1GC, -XX:+UseParallelGC, -XX:+UseZGC GC choice When you have a clear need (latency, throughput) By default: let the JVM choose first -XX:MaxGCPauseMillis Pause time goal (G1: 200 ms by default) If G1 pauses are too long for you To make the GC \u0026ldquo;faster\u0026rdquo;: shorter pauses can cost throughput -Xlog:gc* GC logs Always useful in production, to a file — -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath Heap dump when the heap is full In production, to understand an OutOfMemoryError — -XX:MaxRAMPercentage Maximum heap as a percentage of memory (25% by default) In a container, instead of a fixed -Xmx — Heap size: -Xms and -Xmx -Xms sets the initial heap size, -Xmx the maximum size. Without them, the JVM starts with 1/64 of the physical memory and can grow up to 1/4. On my machine (32 GB), that is 512 MB and 8 GB.\nOn a server, both often get the same value. Oracle\u0026rsquo;s guide explains why: it is more predictable, because the JVM no longer has to decide the heap size. But the other side is that if the value is wrong, the JVM cannot fix it.\n-Xss sets the stack size of each thread: 1 MB by default on Linux and macOS on x64, 2 MB on Aarch64 (Apple Silicon Macs, for example).\nPauses, logs and heap dumps -XX:MaxGCPauseMillis is a goal, not a promise. The documentation calls it a \u0026ldquo;soft goal\u0026rdquo;: the JVM does its best, but it may miss it. To reach a shorter goal, it can run the GC more often, and then the application does less useful work.\n-Xlog:gc* writes the GC logs. To send them to a file: -Xlog:gc*:file=gc.log. Rotation is built in: by default, 5 files of about 20 MB, which you can change with filecount and filesize.\n-XX:+HeapDumpOnOutOfMemoryError writes an .hprof file when the heap is full. It is off by default. Without -XX:HeapDumpPath, the file goes to the current directory, named java_pid\u0026lt;pid\u0026gt;.hprof. Be careful: it only works when the heap is full, not for the other kinds of OutOfMemoryError (for example when the JVM cannot create a new thread).\nIn a container On Linux, the JVM finds the container by itself: it sees its memory limit and its number of CPUs. -XX:MaxRAMPercentage gives the part of this memory used for the maximum heap, 25% by default. This percentage follows the container limit, but a fixed -Xmx does not move when the limit changes. The details, with measures: Java in Docker and Kubernetes.\nWhat changed: the options that no longer work Option Java 8 Java 11 Java 17, 21 and 25 Use instead -XX:PermSize, -XX:MaxPermSize Ignored, with a warning Ignored, with a warning Refused -XX:MetaspaceSize, -XX:MaxMetaspaceSize -XX:+UseConcMarkSweepGC Accepted Accepted, with a warning (deprecated) Refused Another GC (CMS is gone since Java 14) -XX:+UseParNewGC Accepted Refused Refused (removed in Java 10) — -XX:+UseCGroupMemoryLimitForHeap Experimental Refused Refused (removed in Java 11) Nothing: the JVM finds the container -XX:+PrintGCDateStamps, -XX:+PrintGCTimeStamps Accepted Refused Refused Nothing: -Xlog handles the dates -XX:+UseGCLogFileRotation Accepted Refused Refused Nothing: -Xlog handles rotation -XX:+PrintTenuringDistribution Accepted Refused Refused -Xlog:gc+age*=debug -XX:+PrintGCDetails Accepted Deprecated: a warning Deprecated: a warning, then -Xlog:gc* -Xlog:gc* -Xloggc:\u0026lt;file\u0026gt; Accepted Deprecated: a warning Deprecated: a warning, then -Xlog:gc:\u0026lt;file\u0026gt; -Xlog:gc:\u0026lt;file\u0026gt; So the old GC logging options were already refused in Java 11.\n\u0026ldquo;Refused\u0026rdquo; means \u0026ldquo;Unrecognized VM option\u0026rdquo;, and the JVM does not start. A deprecated option still works, with a warning:\n[0.001s][warning][gc] -XX:+PrintGCDetails is deprecated. Will use -Xlog:gc* instead. A recent case: -XX:+ZGenerational turned on the generational mode of ZGC in Java 21. Since Java 24, it is the only mode: in Java 25, the option is ignored with a warning, and -XX:+UseZGC is enough.\nTest an old option before you keep it The simplest way is to start the JVM with your options and -version:\njava -XX:+PrintGCDateStamps -Xmx2g -version If an option is refused, you see it at once, without starting the application. The documentation of the java command also has three lists: deprecated, obsolete and removed options. But they are short: an option removed a long time ago is not there anymore (-XX:PermSize, for example). It also has a table with the -Xlog equivalent of each old GC logging option.\nSources: the documentation of the java command and Oracle\u0026rsquo;s garbage collection guide (JDK 25), JEP 122 (PermGen), JEP 271 (GC logs with -Xlog), JEP 363 (CMS), JDK-8173421 (ParNew), JDK-8194086 (cgroups), JEP 490 (ZGC).\nKey points Keep few options, and know why each one is there: the heap size, GC logs to a file and the heap dump. Do not copy a list of options from a forum or an old script without understanding it: at best, part of it does nothing anymore; at worst, the JVM does not start.\nFurther reading The Java garbage collector: an up-to-date guide (Java 25) Common myths about the Java garbage collector Java garbage collectors in 2026: which one is running yours? GC generations and cycles: young, old, minor, major, full ","permalink":"https://aymenjamaaoui.dev/en/blog/jvm-options-that-matter/","summary":"\u003cp\u003eYou take an old startup script, you move to Java 25, and the JVM does\nnot start anymore:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eUnrecognized VM option \u0026#39;PrintGCDateStamps\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eError: Could not create the Java Virtual Machine.\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003eError: A fatal exception has occurred. Program will exit.\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eMany memory and GC options that you still find on the web have changed\nor are gone. Some are just ignored, others stop the JVM at startup. And\nthe options that are useful today fit in a few lines.\u003c/p\u003e","title":"JVM options that matter, and the ones that no longer work"},{"content":"Your application may be running with a different GC than you think.\nIn Java 25, G1 is the default garbage collector. But the JVM offers others, each one for a different need. And on a small machine, it picks another one on its own.\nThe GCs available in Java 25 GC What Oracle\u0026rsquo;s guide says Option Serial One thread does all the GC work. For a small data set (up to about 100 MB) or a single processor -XX:+UseSerialGC Parallel The \u0026ldquo;throughput collector\u0026rdquo;: like Serial, but with several threads. When raw performance comes first and pauses of a second or more are fine -XX:+UseParallelGC G1 A \u0026ldquo;mostly concurrent\u0026rdquo; collector: it does part of its work while the application runs. When response time matters more than throughput -XX:+UseG1GC ZGC Pauses under a millisecond, at the cost of some throughput. When low latency is the priority -XX:+UseZGC Shenandoah Another low-pause GC. Oracle\u0026rsquo;s guide does not cover it; it works on Temurin 25 -XX:+UseShenandoahGC And a special one: Epsilon. It allocates memory but never frees it. When the heap is full, the JVM stops. It is meant for tests (measuring an application without any GC, checking that a test does not create too many objects) or very short jobs. In Java 25, it is still experimental: you need -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC.\nThe default GC: G1… except on a small machine The JVM picks G1 on a \u0026ldquo;server-class\u0026rdquo; machine: at least two processors and at least 1792 MB of physical memory. Otherwise, it picks Serial.\nOn a developer machine, it is almost always G1. But in a small container limited to one CPU, the JVM silently switches to Serial. So your application can behave differently on your laptop and in production.\nOracle\u0026rsquo;s advice: unless you have strict pause-time requirements, run your application first and let the JVM choose.\nHow to check which GC is running At startup, with -Xlog:gc, the JVM tells you on the first line:\n[0.015s][info][gc] Using G1 Without starting your application, -XX:+PrintFlagsFinal shows what the JVM picks on this machine (ergonomic means \u0026ldquo;chosen by the JVM\u0026rdquo;):\nbool UseG1GC = true {product} {ergonomic} And for an application that is already running, jcmd \u0026lt;pid\u0026gt; VM.flags lists its options, for example -XX:+UseG1GC.\nWhat has changed Timeline of Java garbage collectors from Java 8 to Java 25: G1 by default in Java 9, CMS removed in Java 14, ZGC and Shenandoah ready for production in Java 15, then their generational modes Java GCs from Java 8 to Java 25 Java 8 Parallel by default PermGen → Metaspace Java 9 G1 by default CMS deprecated Java 11 ZGC, Epsilon (experimental) Java 12 Shenandoah (experimental) Java 14 CMS removed Java 15 ZGC, Shenandoah for production Java 21 Generational ZGC (option) Java 23 Generational ZGC by default Java 24 ZGC: generational only Java 25 Generational Shenandoah stable What you still read online What is true today Since Parallel is the default GC G1 is the default GC Java 9 CMS is the low-pause GC CMS is gone. On Java 17, 21 and 25, -XX:+UseConcMarkSweepGC even stops the JVM from starting Deprecated in Java 9, removed in Java 14 ZGC and Shenandoah are experimental They are ready for production Java 15 ZGC has no generations ZGC is generational, and that is the only mode since Java 24 Java 21 to 24 Shenandoah has no generations Its generational mode is stable in Java 25, but it is not the default (-XX:ShenandoahGCMode=generational) Java 25 Sources: Oracle\u0026rsquo;s garbage collection guide (JDK 25), JEP 248 (G1 by default), JEP 291 and JEP 363 (CMS), JEP 377 (ZGC), JEP 379 and JEP 521 (Shenandoah), JEP 318 (Epsilon).\nSummary G1 is a good place to start, and it is what the JVM picks on a normal machine. ZGC and Shenandoah aim for the shortest pauses, Parallel for throughput, Serial for small machines. Before you switch GCs, check which one is really running, especially in a container.\nFurther reading The Java garbage collector: an up-to-date guide (Java 25) JVM options that matter, and the ones that no longer work GC generations and cycles: young, old, minor, major, full JVM memory: heap, stack and Metaspace ","permalink":"https://aymenjamaaoui.dev/en/blog/java-garbage-collectors-2026/","summary":"\u003cp\u003eYour application may be running with a different GC than you think.\u003c/p\u003e\n\u003cp\u003eIn Java 25, G1 is the default garbage collector. But the JVM offers\nothers, each one for a different need. And on a small machine, it picks\nanother one on its own.\u003c/p\u003e\n\u003ch2 id=\"the-gcs-available-in-java-25\"\u003eThe GCs available in Java 25\u003c/h2\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eGC\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eWhat Oracle\u0026rsquo;s guide says\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eOption\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eSerial\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eOne thread does all the GC work. For a small data set (up to about 100 MB) or a single processor\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:+UseSerialGC\u003c/code\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eParallel\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eThe \u0026ldquo;throughput collector\u0026rdquo;: like Serial, but with several threads. When raw performance comes first and pauses of a second or more are fine\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:+UseParallelGC\u003c/code\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eG1\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eA \u0026ldquo;mostly concurrent\u0026rdquo; collector: it does part of its work while the application runs. When response time matters more than throughput\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:+UseG1GC\u003c/code\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eZGC\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ePauses under a millisecond, at the cost of some throughput. When low latency is the priority\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:+UseZGC\u003c/code\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eShenandoah\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eAnother low-pause GC. Oracle\u0026rsquo;s guide does not cover it; it works on Temurin 25\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003e-XX:+UseShenandoahGC\u003c/code\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eAnd a special one: \u003cstrong\u003eEpsilon\u003c/strong\u003e. It allocates memory but never frees it.\nWhen the heap is full, the JVM stops. It is meant for tests (measuring\nan application without any GC, checking that a test does not create too\nmany objects) or very short jobs. In Java 25, it is still experimental:\nyou need \u003ccode\u003e-XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC\u003c/code\u003e.\u003c/p\u003e","title":"Java garbage collectors in 2026: which one is running yours?"},{"content":"Minor GC, major GC, full GC: you see these words everywhere, but not always with the meaning the official documentation gives them.\nMost objects die young. So the JVM keeps new objects in the young generation and long-lived ones in the old generation. A GC that only cleans the young generation is fast, and it runs often.\nWhy generations? Oracle\u0026rsquo;s official guide to the garbage collector starts from a simple fact: most objects live for a very short time. This is called the generational hypothesis. An object created for a request, a loop or a calculation is useless almost right away.\nIf the JVM keeps new objects apart from long-lived ones, it can often clean the small area of new objects, where almost everything is dead, without scanning the whole heap every time.\nHow an object moves: eden, survivor, old How an object moves in the heap: it is created in eden, moves from one survivor space to the other at each minor collection, then it is promoted to the old generation Heap Young new objects Eden objects are created here object Survivor 0 the two survivor spaces Survivor 1 take turns; one is empty Old long-lived objects object 1 2 3 1 Minor collection: the live object is copied from eden to a survivor space. 2 At each minor collection, it moves to the other survivor space; its age goes up. 3 After enough copies (MaxTenuringThreshold, 15 by default): promoted to old. The young generation has three parts: eden and two survivor spaces.\nAn object is almost always created in eden. When the young generation is full, the GC cleans it. Objects that are still alive are copied to a survivor space, and eden is empty again. At the next GC, live objects move from one survivor space to the other. One of the two is always empty. After a number of copies, or when the survivor spaces are full, the object is copied to the old generation. We say it is promoted. You set the maximum number of copies with -XX:MaxTenuringThreshold. The default is 15.\nMinor, major, full: what the docs really say In Oracle\u0026rsquo;s guide:\na minor collection only cleans the young generation; a major collection cleans the whole heap, when the old generation is full. It usually takes much longer; the term full GC shows up, for example, for a complete clean-up asked for by System.gc(). Many articles say something else: \u0026ldquo;major GC = old generation only, full GC = the whole heap\u0026rdquo;. That is not how the official guide defines it.\nAnd in the logs, the words depend on the GC. G1, the default GC, never writes \u0026ldquo;minor\u0026rdquo; or \u0026ldquo;major\u0026rdquo;:\n[0.238s] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 35M-\u0026gt;3M(64M) 2.406ms [0.287s] GC(34) Pause Young (Mixed) (G1 Evacuation Pause) 60M-\u0026gt;35M(64M) 0.215ms [0.335s] GC(82) Pause Full (System.gc()) 53M-\u0026gt;27M(64M) 3.010ms A Young (Normal) pause cleans the young generation. A Young (Mixed) pause cleans the young generation plus part of the old one. ZGC, on the other hand, does write Minor Collection and Major Collection:\n[0.246s] GC(3) Minor Collection (Allocation Rate) 52M(81%)-\u0026gt;52M(81%) 0.008s [0.503s] GC(152) Major Collection (System.gc()) 38M(59%)-\u0026gt;30M(47%) 0.009s So when you read a GC log, first check which GC wrote it.\nWhen does G1 run a full GC? With G1, a full GC is a last resort: it compacts the whole heap, and the docs warn that it can be very slow. It mostly happens in two cases:\nthe heap is too full: the application creates objects faster than G1 can free space in the old generation; someone asks for it: a call to System.gc(), from the application or from a tool. A full GC now and then is not a problem. Full GCs that keep coming back are a warning sign: the heap is too small, or there is a memory leak.\nWhat has changed The G1 heap: regions of the same size; eden, survivor and old regions are mixed, between free regions G1 heap regions of the same size (4 MB here), mixed in memory E O S O E O O E O O E E O O S O E O O E O E O O E O O O E O S O E Eden S Survivor O Old free Young generation = eden and survivor regions; old generation = old regions. A mixed pause cleans the young generation plus some of the old regions. Since Java 9, G1 is the default GC. It no longer splits the heap into two big blocks: it divides it into regions of the same size (4 MB on a machine with 32 GB of memory). The young and old generations are sets of regions, and these regions are not always next to each other in memory.\nWhat you still read online What is true today Since The heap is split into two fixed blocks, young and old With G1, the heap is made of regions of the same size; the young and old generations are sets of regions Java 9 (G1 by default) ZGC has no generations ZGC is generational: an option in Java 21, the default in Java 23, the only mode since Java 24 (-XX:+ZGenerational is ignored, with a warning) Java 21 to 24 Major GC = cleaning the old generation In Oracle\u0026rsquo;s guide, a major collection cleans the whole heap — You tune the young generation with -Xmn or -XX:NewRatio With G1, the docs advise against it: it mostly turns off pause-time control — Sources: Oracle\u0026rsquo;s garbage collection guide (JDK 25), JEP 248 (G1 by default), JEP 439, JEP 474 and JEP 490 (generational ZGC).\nThe options in short -Xmn: the size of the young generation (initial and maximum); -XX:NewRatio: the size of the old generation compared to the young one (2 by default: the old generation is twice as big); -XX:SurvivorRatio: the size of eden compared to one survivor space (8 by default); -XX:MaxTenuringThreshold: the number of copies before promotion (15 by default). With G1, it is better not to fix the size of the young generation: G1 adjusts it by itself to meet its pause-time goal.\nSummary Objects are born in eden, survive a few GCs in the survivor spaces, and the long-lived ones end up in the old generation. \u0026ldquo;Minor\u0026rdquo; means the young generation; \u0026ldquo;major\u0026rdquo;, in Oracle\u0026rsquo;s docs, means the whole heap. With G1, you rather see young, mixed and full pauses, and the heap is made of regions.\nFurther reading The Java garbage collector: an up-to-date guide (Java 25) Java garbage collectors in 2026: which one is running yours? JVM memory: heap, stack and Metaspace ","permalink":"https://aymenjamaaoui.dev/en/blog/gc-generations-cycles-young-old-minor-major-full/","summary":"\u003cp\u003eMinor GC, major GC, full GC: you see these words everywhere, but not\nalways with the meaning the official documentation gives them.\u003c/p\u003e\n\u003cp\u003eMost objects die young. So the JVM keeps new objects in the \u003cstrong\u003eyoung\ngeneration\u003c/strong\u003e and long-lived ones in the \u003cstrong\u003eold generation\u003c/strong\u003e. A GC that\nonly cleans the young generation is fast, and it runs often.\u003c/p\u003e\n\u003ch2 id=\"why-generations\"\u003eWhy generations?\u003c/h2\u003e\n\u003cp\u003eOracle\u0026rsquo;s official guide to the garbage collector starts from a simple\nfact: most objects live for a very short time. This is called the\n\u003cstrong\u003egenerational hypothesis\u003c/strong\u003e. An object created for a request, a loop or\na calculation is useless almost right away.\u003c/p\u003e","title":"GC generations and cycles: young, old, minor, major, full"},{"content":"Does your old Java configuration fail to start on Java 17? The answer is in how the JVM uses memory.\nIn Java, each thread has its own stack for local variables. Objects live on the heap, the area the garbage collector cleans up. Class data lives in the Metaspace, outside the heap. The JVM also uses other memory areas that are easy to forget.\nThree areas you need to know Memory of a Java process: thread stacks, the heap, the Metaspace and other native memory areas Java process everything outside the heap is native memory Stacks one per thread · -Xss Thread main process() · client main() Thread 2 read() run() frames, local variables Heap shared · managed by the GC · -Xmx Client String List byte[] objects created with new interned strings static fields Metaspace native memory no limit by default Client fields, methods… String fields, methods… class descriptions -XX:MaxMetaspaceSize Also in native memory Code cache machine code from the JIT up to 240 MB by default Direct memory ByteBuffer.allocateDirect() -XX:MaxDirectMemorySize Internal data garbage collector, compiler and more The stack: one per thread Each thread gets its own stack. Every method call adds a frame to it, with the method\u0026rsquo;s parameters and local variables. When the method returns, its frame is gone. The garbage collector is not needed to free this memory.\nA local variable of an object type does not store the object itself. It stores a reference to an object on the heap.\nIf calls go too deep, for example with infinite recursion, the stack runs out of space and you get a StackOverflowError. You can change the stack size with -Xss. The default depends on the platform: 1 MB on Linux x64, 2 MB on Linux and macOS on ARM (AArch64).\nThe heap: objects, and where the GC works Every object you create with new goes on the heap, which all threads share. This is where the garbage collector works: it finds the objects that nothing points to anymore and frees their space.\nYou set the maximum heap size with -Xmx. Without it, the JVM usually sets the maximum to 25 % of the memory. On a machine with 32 GB of memory, that gives a heap of up to 8 GB.\nInterned strings (String.intern()) and the static fields of classes are on the heap too.\nThe Metaspace: classes, outside the heap For each class it loads, the JVM keeps a description of the class: its fields, its methods… This metadata lives in the Metaspace, in native memory, outside the heap.\nBy default, the Metaspace has no size limit: it grows as more classes are loaded. You can set a limit with -XX:MaxMetaspaceSize.\nWhat people often forget The heap and the Metaspace are not the whole story. The JVM also uses native memory for:\nthread stacks: one per thread, so more threads means more memory; the code cache: the machine code produced by the JIT compiler, up to 240 MB by default; direct memory: buffers allocated outside the heap with ByteBuffer.allocateDirect(). If you do not set -XX:MaxDirectMemorySize, the JVM picks the limit; internal data of the garbage collector, the compiler and more. What has changed Say a developer reuses an old JVM configuration, copied from an old project or a forum:\njava -Xms512m -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=256m -jar app.jar On Java 8, the application starts. The JVM just prints two warnings that are easy to miss in the logs:\nOpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0 OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0 On Java 17, 21 or 25, it does not start at all:\nUnrecognized VM option \u0026#39;PermSize=128m\u0026#39; Error: Could not create the Java Virtual Machine. And the JVM stops at the first unknown option: you remove PermSize, run it again, and now MaxPermSize fails.\nWhy? PermGen was removed in Java 8.\nWhat you still read online What is true today Since Classes live in PermGen, an area with a fixed size Their metadata lives in the Metaspace, in native memory, with no limit by default Java 8 You tune PermGen with -XX:PermSize and -XX:MaxPermSize These options were ignored with a warning, then rejected: the JVM does not start. Use -XX:MetaspaceSize and -XX:MaxMetaspaceSize instead Ignored up to Java 16, rejected since Java 17 OutOfMemoryError: PermGen space PermGen is gone, and so is this error Java 8 Interned strings and static fields live in PermGen They live on the heap Java 8 at the latest Sources: JEP 122, Remove the Permanent Generation, and the JDK 17 documentation for the java command (section \u0026ldquo;Removed Java Options\u0026rdquo;).\nTwo common mistakes \u0026ldquo;Everything is on the heap\u0026rdquo; The heap holds objects, plus interned strings and static fields. Classes, compiled code, thread stacks and direct buffers live elsewhere, in native memory. -Xmx does not limit them.\n\u0026ldquo;-Xmx is all the memory my application uses\u0026rdquo; -Xmx only limits the heap. The Java process always uses more than that. For example, with Java 25 and -Xmx512m, the JVM already reserves about 598 MB of usable memory (\u0026ldquo;committed\u0026rdquo; in Native Memory Tracking), just to print its version. When you size a machine or a container, count the whole process, not just the heap.\nSummary Three areas to remember: one stack per thread, the heap for objects and the Metaspace for classes. -Xmx only limits the heap, and the heap is only part of the process memory. If an old configuration still mentions PermGen, it is time to clean it up.\nFurther reading The Java garbage collector: an up-to-date guide (Java 25) GC generations and cycles: young, old, minor, major, full ","permalink":"https://aymenjamaaoui.dev/en/blog/jvm-memory-heap-stack-metaspace/","summary":"\u003cp\u003eDoes your old Java configuration fail to start on Java 17? The answer is\nin how the JVM uses memory.\u003c/p\u003e\n\u003cp\u003eIn Java, each thread has its own \u003cstrong\u003estack\u003c/strong\u003e for local variables. Objects\nlive on the \u003cstrong\u003eheap\u003c/strong\u003e, the area the garbage collector cleans up. Class data\nlives in the \u003cstrong\u003eMetaspace\u003c/strong\u003e, outside the heap. The JVM also uses other\nmemory areas that are easy to forget.\u003c/p\u003e\n\u003ch2 id=\"three-areas-you-need-to-know\"\u003eThree areas you need to know\u003c/h2\u003e\n\n\u003cfigure class=\"aj-svg\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" class=\"jvm-memory\" viewBox=\"0 0 720 440\" width=\"720\" height=\"440\" role=\"img\" aria-labelledby=\"jvm-memory-title\"\u003e\n  \u003ctitle id=\"jvm-memory-title\"\u003eMemory of a Java process: thread stacks, the heap, the Metaspace and other native memory areas\u003c/title\u003e\n  \u003cstyle\u003e\n    svg.jvm-memory .proc { fill: var(--entry, #ffffff); stroke: var(--tertiary, #d5dbe6); stroke-width: 1.5; }\n    svg.jvm-memory .zone { fill: var(--code-bg, #f3f6fb); stroke: var(--tertiary, #d5dbe6); stroke-width: 1.5; }\n    svg.jvm-memory .heap { fill: color-mix(in srgb, var(--accent, #2563eb) 6%, transparent); stroke: var(--accent, #2563eb); stroke-width: 2; }\n    svg.jvm-memory .item { fill: var(--entry, #ffffff); stroke: var(--tertiary, #d5dbe6); stroke-width: 1; }\n    svg.jvm-memory .obj { fill: var(--entry, #ffffff); stroke: var(--accent, #2563eb); stroke-width: 1.5; }\n    svg.jvm-memory text { font-family: -apple-system, BlinkMacSystemFont, \"Segoe UI\", Roboto, sans-serif; fill: var(--primary, #1e1e1e); }\n    svg.jvm-memory .h { font-size: 15px; font-weight: 700; }\n    svg.jvm-memory .t { font-size: 13px; }\n    svg.jvm-memory .s { font-size: 12px; fill: var(--secondary, #6c6c6c); }\n    svg.jvm-memory .m { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }\n    svg.jvm-memory .acc { fill: var(--accent, #2563eb); }\n    svg.jvm-memory .arrow { stroke: var(--accent, #2563eb); stroke-width: 1.5; fill: none; marker-end: url(#jvm-memory-arrow); }\n    svg.jvm-memory .ah { fill: var(--accent, #2563eb); }\n  \u003c/style\u003e\n  \u003cdefs\u003e\n    \u003cmarker id=\"jvm-memory-arrow\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\"\u003e\n      \u003cpath class=\"ah\" d=\"M0,0 L10,5 L0,10 z\"/\u003e\n    \u003c/marker\u003e\n  \u003c/defs\u003e\n\n  \u003c!-- The Java process --\u003e\n  \u003crect class=\"proc\" x=\"1\" y=\"1\" width=\"718\" height=\"438\" rx=\"12\"/\u003e\n  \u003ctext class=\"h\" x=\"20\" y=\"28\"\u003eJava process\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"148\" y=\"28\"\u003eeverything outside the heap is native memory\u003c/text\u003e\n\n  \u003c!-- Stacks --\u003e\n  \u003crect class=\"zone\" x=\"20\" y=\"44\" width=\"190\" height=\"250\" rx=\"8\"/\u003e\n  \u003ctext class=\"h\" x=\"34\" y=\"68\"\u003eStacks\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"34\" y=\"86\"\u003eone per thread · \u003ctspan class=\"m\"\u003e-Xss\u003c/tspan\u003e\u003c/text\u003e\n\n  \u003ctext class=\"t\" x=\"34\" y=\"110\"\u003eThread main\u003c/text\u003e\n  \u003crect class=\"item\" x=\"34\" y=\"118\" width=\"162\" height=\"26\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"44\" y=\"135\"\u003eprocess() · client\u003c/text\u003e\n  \u003crect class=\"item\" x=\"34\" y=\"148\" width=\"162\" height=\"26\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"44\" y=\"165\"\u003emain()\u003c/text\u003e\n\n  \u003ctext class=\"t\" x=\"34\" y=\"200\"\u003eThread 2\u003c/text\u003e\n  \u003crect class=\"item\" x=\"34\" y=\"208\" width=\"162\" height=\"26\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"44\" y=\"225\"\u003eread()\u003c/text\u003e\n  \u003crect class=\"item\" x=\"34\" y=\"238\" width=\"162\" height=\"26\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"44\" y=\"255\"\u003erun()\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"34\" y=\"284\"\u003eframes, local variables\u003c/text\u003e\n\n  \u003c!-- Heap --\u003e\n  \u003crect class=\"heap\" x=\"226\" y=\"44\" width=\"270\" height=\"250\" rx=\"8\"/\u003e\n  \u003ctext class=\"h acc\" x=\"240\" y=\"68\"\u003eHeap\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"240\" y=\"86\"\u003eshared · managed by the GC · \u003ctspan class=\"m\"\u003e-Xmx\u003c/tspan\u003e\u003c/text\u003e\n\n  \u003crect class=\"obj\" x=\"250\" y=\"110\" width=\"96\" height=\"34\" rx=\"17\"/\u003e\n  \u003ctext class=\"t\" x=\"272\" y=\"132\"\u003eClient\u003c/text\u003e\n  \u003crect class=\"obj\" x=\"376\" y=\"110\" width=\"96\" height=\"34\" rx=\"17\"/\u003e\n  \u003ctext class=\"t\" x=\"398\" y=\"132\"\u003eString\u003c/text\u003e\n  \u003crect class=\"obj\" x=\"250\" y=\"166\" width=\"96\" height=\"34\" rx=\"17\"/\u003e\n  \u003ctext class=\"t\" x=\"277\" y=\"188\"\u003eList\u003c/text\u003e\n  \u003crect class=\"obj\" x=\"376\" y=\"166\" width=\"96\" height=\"34\" rx=\"17\"/\u003e\n  \u003ctext class=\"t\" x=\"395\" y=\"188\"\u003ebyte[]\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"240\" y=\"232\"\u003eobjects created with \u003ctspan class=\"m\"\u003enew\u003c/tspan\u003e\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"240\" y=\"252\"\u003einterned strings\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"240\" y=\"272\"\u003estatic fields\u003c/text\u003e\n\n  \u003c!-- Reference from a local variable to an object --\u003e\n  \u003cpath class=\"arrow\" d=\"M196,131 C220,131 228,127 248,127\"/\u003e\n\n  \u003c!-- Metaspace --\u003e\n  \u003crect class=\"zone\" x=\"512\" y=\"44\" width=\"188\" height=\"250\" rx=\"8\"/\u003e\n  \u003ctext class=\"h\" x=\"526\" y=\"68\"\u003eMetaspace\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"526\" y=\"86\"\u003enative memory\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"526\" y=\"102\"\u003eno limit by default\u003c/text\u003e\n  \u003crect class=\"item\" x=\"526\" y=\"118\" width=\"160\" height=\"56\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"538\" y=\"140\"\u003eClient\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"538\" y=\"160\"\u003efields, methods…\u003c/text\u003e\n  \u003crect class=\"item\" x=\"526\" y=\"182\" width=\"160\" height=\"56\" rx=\"4\"/\u003e\n  \u003ctext class=\"m\" x=\"538\" y=\"204\"\u003eString\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"538\" y=\"224\"\u003efields, methods…\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"526\" y=\"268\"\u003eclass descriptions\u003c/text\u003e\n  \u003ctext class=\"m\" x=\"526\" y=\"284\"\u003e-XX:MaxMetaspaceSize\u003c/text\u003e\n\n  \u003c!-- Other native zones --\u003e\n  \u003ctext class=\"h\" x=\"20\" y=\"326\"\u003eAlso in native memory\u003c/text\u003e\n  \u003crect class=\"zone\" x=\"20\" y=\"338\" width=\"216\" height=\"82\" rx=\"8\"/\u003e\n  \u003ctext class=\"t\" x=\"34\" y=\"364\"\u003eCode cache\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"34\" y=\"384\"\u003emachine code from the JIT\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"34\" y=\"402\"\u003eup to 240 MB by default\u003c/text\u003e\n\n  \u003crect class=\"zone\" x=\"252\" y=\"338\" width=\"216\" height=\"82\" rx=\"8\"/\u003e\n  \u003ctext class=\"t\" x=\"266\" y=\"364\"\u003eDirect memory\u003c/text\u003e\n  \u003ctext class=\"m\" x=\"266\" y=\"384\"\u003eByteBuffer.allocateDirect()\u003c/text\u003e\n  \u003ctext class=\"m\" x=\"266\" y=\"402\"\u003e-XX:MaxDirectMemorySize\u003c/text\u003e\n\n  \u003crect class=\"zone\" x=\"484\" y=\"338\" width=\"216\" height=\"82\" rx=\"8\"/\u003e\n  \u003ctext class=\"t\" x=\"498\" y=\"364\"\u003eInternal data\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"498\" y=\"384\"\u003egarbage collector,\u003c/text\u003e\n  \u003ctext class=\"s\" x=\"498\" y=\"402\"\u003ecompiler and more\u003c/text\u003e\n\u003c/svg\u003e\n\n\u003c/figure\u003e\n\n\u003ch3 id=\"the-stack-one-per-thread\"\u003eThe stack: one per thread\u003c/h3\u003e\n\u003cp\u003eEach thread gets its own stack. Every method call adds a \u003cem\u003eframe\u003c/em\u003e to it,\nwith the method\u0026rsquo;s parameters and local variables. When the method\nreturns, its frame is gone. The garbage collector is not needed to free\nthis memory.\u003c/p\u003e","title":"JVM memory: heap, stack and Metaspace"},{"content":"All day long, I have two Chrome windows open: my personal profile (Gmail, my accounts) and my work profile (a Google Workspace account for the company). On macOS, this is annoying.\nThe problem On macOS, all Chrome profiles live in one app. The Dock and Cmd+Tab group windows by app. So:\nthe Dock shows one Chrome icon for all profiles; Cmd+Tab shows one Chrome. To go from the personal window to the work window, you need Cmd+`, the Window menu or Mission Control; in Mission Control, personal and work windows are mixed. I wanted something simple: one icon per account in the Dock, and one entry per account in Cmd+Tab.\nWhat I tried Chrome profiles. This is the normal solution, but it is also the cause of the problem: all profiles stay in the same app.\nAn Automator launcher. Many guides suggest a small app that starts Chrome with the right profile. It opens the right profile, but the window goes into the same Chrome app. The Dock and Cmd+Tab do not change.\nChrome web apps. \u0026ldquo;Install page as app\u0026rdquo; gives a real icon in the Dock and in Cmd+Tab. But it is one app per site: Gmail, Calendar, Drive… You cannot browse freely.\nExisting tools. Epichrome made apps based on Chromium. Its author stopped the project: the last version, 2.4.27, is from January 2022. chrome-schismator, a script on GitHub, starts several Chrome instances, each with its own icon. And I am not the only one who wants this: on Mozilla Connect, an idea asks for the same thing in Firefox. Its author writes that today you can only get it with \u0026ldquo;fragile third-party scripts\u0026rdquo;.\nThe idea: one app, one id macOS knows an app by its id: the CFBundleIdentifier in the Info.plist file. For Chrome, it is com.google.Chrome. All profiles have the same id, so macOS puts them together.\nSo the plan is easy: copy Google Chrome.app and change this id in the copy. For macOS, com.google.Chrome.Work is another app. It has its own Dock icon and its own Cmd+Tab entry.\nOne problem is left: the copy and the original Chrome must not share the same data folder (profiles, cookies, history). Here, Chromium already has the answer. In its source code (chrome/common/chrome_paths_mac.mm), a key of the Info.plist, CrProductDirName, sets the data folder in ~/Library/Application Support. Google uses it for Chrome Canary, which uses Google/Chrome Canary. So this is not a hack: it is a mechanism that Chromium already has.\nThe copy changes three things:\nInfo.plist key Value Effect CFBundleIdentifier com.google.Chrome.Work a separate app (Dock, Cmd+Tab) CFBundleName, CFBundleDisplayName Chrome Work the name you see CrProductDirName Google/ChromeWork a separate data folder One important detail: when you change the Info.plist, the Google signature is not valid anymore. You must sign the copy again on your Mac, with a local signature (\u0026ldquo;ad hoc\u0026rdquo;). This is the cause of almost all the limits. More on that below.\nThe script I put all this in a script, copy-chrome.sh. It copies Chrome, changes the Info.plist, adds a badge on the icon if you want, and signs the copy.\n# Create a copy, with the company logo as a badge ./copy-chrome.sh Work --badge ~/Pictures/company-logo.png # After an update of Chrome: make the copy again (the badge is kept) ./copy-chrome.sh Work # Remove the copy and all its data ./copy-chrome.sh Work --remove The badge is a round image, at the top left of the Chrome icon. A small Swift program, next to the script, draws it. With it, you see at once which Chrome is which in the Dock.\nOn the first launch, the copy starts empty, like a new Chrome. You sign in to your Google account, turn on sync, and your bookmarks, passwords and extensions come back. Cookies do not come back: you must sign in to websites again. The original Chrome and its profiles are not changed.\nWhat breaks This is not an official solution, and it has a cost. Everything comes from the same cause: Google does not sign the copy anymore.\nPasskeys saved on the Mac In a copy, passkeys saved on the Mac are not offered: not the ones from Apple Passwords, and not the ones from Google Password Manager. Normal passwords still work.\nWhat I saw on a sign-in page:\nOffered at sign-in Original Chrome Copy Password (Google Password Manager) yes yes Passkey from Google Password Manager yes no Passkey from Apple (Passwords, iCloud Keychain) yes no Passkey on the phone (QR code) yes yes Why? The Chrome from Google has special rights (\u0026ldquo;entitlements\u0026rdquo;) that Apple gives only to an app signed by its developer: access to passkeys and to some keychain groups. A local signature cannot have them. For example, TeamDev, who makes a browser based on Chromium, explains that Touch ID passkeys need the keychain-access-groups entitlement and a \u0026ldquo;provisioning profile\u0026rdquo; in the app. I did not check the exact cause in the Chromium code yet: it is the most likely reason, not a sure fact.\nThe workaround that works: \u0026ldquo;Use a passkey from another device\u0026rdquo;. Chrome shows a QR code, I scan it with my phone, and the passkey on the phone confirms the sign-in. You just need the phone near you, with Bluetooth on.\nThe keychain asks again After each new copy, macOS asks again for access to \u0026ldquo;Chrome Safe Storage\u0026rdquo;, the key that protects the passwords and cookies of Chrome. The keychain knows an app with a local signature by the exact hash of its code, and this hash changes with each signature. Just click Always Allow. Do not click Deny: Chrome could not read its data anymore.\nNo automatic update The copy does not update itself. After each update of the original Chrome, I run the script again. If the copy is already up to date, it does nothing.\nTwo traps during development For those who want to do it themselves:\nFinder attributes. Chrome files have the com.apple.FinderInfo attribute, and codesign refuses to sign: \u0026ldquo;resource fork, Finder information, or similar detritus not allowed\u0026rdquo;. Remove them first with xattr -cr. codesign --deep. Chrome sometimes keeps the old version of its framework next to the new one. --deep tries to sign everything and breaks this old version. The fix: sign only the main app. And one wall: macOS protects the data folder of the original Chrome (TCC). Reading it fails with \u0026ldquo;Operation not permitted\u0026rdquo;, even with sudo. So you cannot copy an existing profile: each copy starts from zero, and sync does the rest.\nWhich profile goes where? Because of passkeys, the split is quite clear:\nIn the original Chrome: the accounts that use passkeys or security keys. It is also the default browser, which gets the links you open from other apps. In a copy: the accounts with a password or SSO, for a well separated use. For me, this is the Google Workspace work account: Gmail, Calendar, Drive and the company tools. My personal profile uses many passkeys, so it stays in the original Chrome. One copy is enough, for work.\nSummary Solution Separate Dock and Cmd+Tab Mac passkeys Free browsing Chrome profiles no yes yes Chrome web apps yes, one per site likely no A copy with copy-chrome yes no (QR code) yes Another browser yes yes yes For me, the copy is the right trade-off: one icon per account, and all of Chrome for browsing. Losing passkeys is real, but the QR code works around it, and the profile that needs them most stays in the original Chrome.\nI tested it with Chrome 155 on macOS 27. Keep in mind that this is not official: Google does not support it, and a future version of Chrome or macOS can break it.\nThe script and the references The script is on GitHub, with its user guide: tips-and-scripts/macos/copy-chrome.\nTo go further:\nChromium source code: chrome_paths_mac.mm (the CrProductDirName key) TeamDev: passkeys and Touch ID in a Chromium browser Mozilla Connect: one Dock icon per profile Epichrome (project stopped) chrome-schismator ","permalink":"https://aymenjamaaoui.dev/en/blog/one-chrome-per-google-account-macos/","summary":"\u003cp\u003eAll day long, I have two Chrome windows open: my personal profile (Gmail, my accounts)\nand my work profile (a Google Workspace account for the company). On macOS, this is\nannoying.\u003c/p\u003e\n\u003ch2 id=\"the-problem\"\u003eThe problem\u003c/h2\u003e\n\u003cp\u003eOn macOS, all Chrome profiles live in \u003cstrong\u003eone app\u003c/strong\u003e. The Dock and Cmd+Tab group windows\nby app. So:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ethe Dock shows \u003cstrong\u003eone\u003c/strong\u003e Chrome icon for all profiles;\u003c/li\u003e\n\u003cli\u003eCmd+Tab shows \u003cstrong\u003eone\u003c/strong\u003e Chrome. To go from the personal window to the work window, you\nneed Cmd+`, the Window menu or Mission Control;\u003c/li\u003e\n\u003cli\u003ein Mission Control, personal and work windows are mixed.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI wanted something simple: one icon per account in the Dock, and one entry per account\nin Cmd+Tab.\u003c/p\u003e","title":"One Chrome per Google account on macOS"},{"content":" About\nAymen Jamaaoui Lead Java Developer · Backend \u0026amp; Full Stack\nI am based in Île-de-France. For more than 15 years, I have designed and improved Java applications, from finance to the public sector, where reliability is not negotiable. Distributed microservices, real-time data flows, secure applications: I like work where every detail matters. I care about software craftsmanship. I write clean, tested and readable code that another developer can take over easily, years later.\nWhat I do\nModernize, design, share Modernize existing systems: improve applications in production step by step, without breaking anything for users. Design distributed architectures: split a monolith into microservices, run in containers and deployed on Kubernetes. Automate delivery: set up CI/CD pipelines to deliver often and with no surprises. Secure applications: authentication, encrypted exchanges and fixing vulnerabilities. Help the team grow: review code, help with technical choices and support developers. Build with AI: use coding assistants and agents to go faster without losing quality. Skills\nMy toolbox Languages Java TypeScript Shell Python Groovy Backend Spring Boot Kafka REST SOAP GraphQL WebSockets Frontend Angular PrimeNG React Cloud \u0026amp; Infra Docker Cloud Kubernetes Jenkins Ansible AWS Data PostgreSQL Oracle MySQL MongoDB Redshift AI \u0026amp; agents Claude Code LLM RAG MCP function calling Where I worked\nDemanding institutions I have been lucky to work for institutions where mistakes are not an option. At BNP Paribas Securities Services, I took part in the migration to T2S, the European platform for securities settlement, up to the production cutovers. At the Bibliothèque nationale de France (the French national library), I designed from scratch the legal deposit portal for moving images. At Euronext, the leading stock exchange operator in Europe, I led the rebuild of a tool that runs thousands of tests every day, without stopping the teams\u0026rsquo; work.\nSee my full background → Open source\nGiving back to the community I contribute to open source. Here are some projects I have contributed to:\nQuickFIX/J: the open source Java implementation of the FIX protocol. GitHub → Simple Binary Encoding (SBE): a very fast binary encoding library, used in finance. GitHub → Floci: a local emulator of AWS services, written in Java with Quarkus. GitHub → Education \u0026amp; certifications\nQualifications Computer Science Engineer, graduated from ENSI in Tunisia in 2010.\nSpring Certified Professional Broadcom / VMware · 2025 AWS Solutions Architect Associate · Amazon Web Services · 2025 OCP Java SE 8 Oracle Certified Professional · Oracle · 2022 OCA Java SE 8 Oracle Certified Associate · Oracle · 2020 Languages\nHow to talk with me Arabic: native French: fluent English: intermediate Contact me A question, a project, an idea?\nContact me → ","permalink":"https://aymenjamaaoui.dev/en/about/","summary":"\u003cheader class=\"about-header about-header-photo\"\u003e\n  \u003cdiv class=\"about-header-text\"\u003e\n    \u003cp class=\"aj-overline\"\u003eAbout\u003c/p\u003e\n    \u003ch1 class=\"about-name\"\u003eAymen Jamaaoui\u003c/h1\u003e\n    \u003cp class=\"about-role\"\u003eLead Java Developer · Backend \u0026amp; Full Stack\u003c/p\u003e\n    \u003cdiv class=\"about-intro\"\u003e\n      \u003cp\u003eI am based in Île-de-France. For more than 15 years, I have designed and improved Java applications, from finance to the public sector, where reliability is not negotiable. Distributed microservices, real-time data flows, secure applications: I like work where every detail matters. I care about software craftsmanship. I write clean, tested and readable code that another developer can take over easily, years later.\u003c/p\u003e\n\n    \u003c/div\u003e\n    \u003cdiv class=\"social-icons\" \u003e\n    \u003ca href=\"https://github.com/aymenjamaaoui\" target=\"_blank\" rel=\"noopener noreferrer me\"\n        title=\"Github\"\u003e\n        \u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath\n        d=\"M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37 0 0 0-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44 0 0 0 20 4.77 5.07 5.07 0 0 0 19.91 1S18.73.65 16 2.48a13.38 13.38 0 0 0-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07 0 0 0 5 4.77a5.44 5.44 0 0 0-1.5 3.78c0 5.42 3.3 6.61 6.44 7A3.37 3.37 0 0 0 9 18.13V22\"\u003e\n    \u003c/path\u003e\n\u003c/svg\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://gitlab.com/aymenjamaaoui\" target=\"_blank\" rel=\"noopener noreferrer me\"\n        title=\"Gitlab\"\u003e\n        \u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath\n        d=\"M22.65 14.39L12 22.13 1.35 14.39a.84.84 0 0 1-.3-.94l1.22-3.78 2.44-7.51A.42.42 0 0 1 4.82 2a.43.43 0 0 1 .58 0 .42.42 0 0 1 .11.18l2.44 7.49h8.1l2.44-7.51A.42.42 0 0 1 18.6 2a.43.43 0 0 1 .58 0 .42.42 0 0 1 .11.18l2.44 7.51L23 13.45a.84.84 0 0 1-.35.94z\"\u003e\n    \u003c/path\u003e\n\u003c/svg\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://linkedin.com/in/aymen-jamaaoui/\" target=\"_blank\" rel=\"noopener noreferrer me\"\n        title=\"Linkedin\"\u003e\n        \u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath d=\"M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z\"\u003e\u003c/path\u003e\n    \u003crect x=\"2\" y=\"9\" width=\"4\" height=\"12\"\u003e\u003c/rect\u003e\n    \u003ccircle cx=\"4\" cy=\"4\" r=\"2\"\u003e\u003c/circle\u003e\n\u003c/svg\u003e\n    \u003c/a\u003e\n\u003c/div\u003e\n\n  \u003c/div\u003e\n  \u003cimg class=\"about-photo\" src=\"/images/portrait_hu_5ab6f3a5bffa9960.webp\" width=\"240\" height=\"240\" alt=\"Photo of Aymen Jamaaoui\"\u003e\n\u003c/header\u003e","title":"About"},{"content":" Background\nMore than 15 years of Java, from Tunisia to France It all started in Tunisia, with Java EE applications. It took me to critical systems in finance, in France. Here are the main steps.\nMore than 15 years of experience 2 countries 7 companies In France\nFreelance, finance and public sector I arrived in France in 2014. I moved from business applications to critical systems in finance.\n2023 – present\nFounder · J2RA IT Services\nMy software consulting and development company. I work with my clients through it.\n2022 – 2026\nLead Java Developer \u0026amp; DevOps · Euronext\nRebuild of a tool that runs thousands of tests every day. The monolith was split into containerized services, and Java, Spring Boot and Angular were moved to recent versions, with no downtime for the teams.\nSpring Boot Kafka Angular Kubernetes 2021 – 2022\nSenior Java Developer · Société Générale CIB\nMaintenance of Forex applications in the Kondor scope, for the front office. Development of a modern application that takes over some of their features.\nJava Spring Batch Sybase React 2018 – 2021\nSenior Java Developer · Bibliothèque nationale de France\nDesign from scratch of DELIA, the legal deposit portal for moving images. Work on two other digital legal deposit applications.\nSpring Security Spring Batch Angular PostgreSQL 2014 – 2018\nJava Backend Developer · BNP Paribas Securities Services\nSecurities back office: processing of Swift messages, the European T2S migration up to the production cutovers, and night production support.\nJava Hibernate Oracle Swift messages What I learned: Rigor first. Test, document, and never break production.\nIn Tunisia\nMy first steps Before France, there was Tunisia. There, I learned to work on everything: web, mobile, databases, automation.\n2012 – 2014\nJava \u0026amp; PHP Full Stack Developer · OpenBee\nAutomation of the software delivery platform of the OpenBee portal: MongoDB storage, OAuth2 gateway and single sign-on.\nPlay Framework Symfony MongoDB OAuth2 2010 – 2012\nJava EE Developer · Taggist\nReal-time geolocation and asset management applications, on the web and on Android. I also coached my first intern there.\nJava EE EJB JBoss Android What I learned: Versatility. Moving from web to mobile, from Java to PHP, and learning fast.\nEducation\nMy studies The basics: a lot of math, then computer science for good.\n2006 – 2010\nComputer Science Engineering Degree · ENSI, Tunis\nMajor: computer engineering and technology applications. Final project: an RFID solution to improve patient tracking in a clinic.\n2004 – 2006\nPreparatory Classes · IPEIN, Nabeul\nTwo intensive years of math and physics to prepare for the national entrance exam to engineering schools.\n2004\nBaccalauréat, Technical Track, With Honors · Lycée Ibn El Jazzar, Kairouan\nWhat I learned: Method. Breaking a hard problem down until it becomes simple.\n← Learn more about me Contact me A question, a project, an idea?\nContact me → ","permalink":"https://aymenjamaaoui.dev/en/background/","summary":"\u003cheader class=\"parcours-header\"\u003e\n  \u003cp class=\"aj-overline\"\u003eBackground\u003c/p\u003e\n  \u003ch1 class=\"parcours-title\"\u003eMore than 15 years of Java, from Tunisia to France\u003c/h1\u003e\n  \u003cdiv class=\"parcours-intro\"\u003e\n    \u003cp\u003eIt all started in Tunisia, with Java EE applications. It took me to critical systems in finance, in France. Here are the main steps.\u003c/p\u003e\n\n  \u003c/div\u003e\n  \u003cul class=\"parcours-numbers\"\u003e\n    \u003cli\u003eMore than 15 years of experience\u003c/li\u003e\n    \u003cli\u003e2 countries\u003c/li\u003e\n    \u003cli\u003e7 companies\u003c/li\u003e\n  \u003c/ul\u003e\n\u003c/header\u003e\n\n\n\u003csection class=\"parcours-block\"\u003e\n  \u003cp class=\"aj-overline\"\u003eIn France\u003c/p\u003e\n  \u003ch2 class=\"aj-section-title\" id=\"freelance-finance-and-public-sector\"\u003eFreelance, finance and public sector\u003c/h2\u003e\n  \u003cp class=\"parcours-transition\"\u003eI arrived in France in 2014. I moved from business applications to critical systems in finance.\u003c/p\u003e\n  \u003col class=\"parcours-timeline\"\u003e\n    \u003cli class=\"parcours-entry\"\u003e\n      \u003cp class=\"parcours-period\"\u003e2023 – present\u003c/p\u003e\n      \u003cp class=\"parcours-heading\"\u003e\u003cstrong\u003eFounder\u003c/strong\u003e · \u003cspan class=\"parcours-name\"\u003eJ2RA IT Services\u003c/span\u003e\u003c/p\u003e\n      \u003cp\u003eMy software consulting and development company. I work with my clients through it.\u003c/p\u003e\n    \u003c/li\u003e\n    \u003cli class=\"parcours-entry\"\u003e\n      \u003cp class=\"parcours-period\"\u003e2022 – 2026\u003c/p\u003e\n      \u003cp class=\"parcours-heading\"\u003e\u003cstrong\u003eLead Java Developer \u0026amp; DevOps\u003c/strong\u003e · \u003cspan class=\"parcours-name\"\u003eEuronext\u003c/span\u003e\u003c/p\u003e\n      \u003cp\u003eRebuild of a tool that runs thousands of tests every day. The monolith was split into containerized services, and Java, Spring Boot and Angular were moved to recent versions, with no downtime for the teams.\u003c/p\u003e\n      \u003cul class=\"aj-tags\"\u003e\n        \u003cli\u003eSpring Boot\u003c/li\u003e\n        \u003cli\u003eKafka\u003c/li\u003e\n        \u003cli\u003eAngular\u003c/li\u003e\n        \u003cli\u003eKubernetes\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli class=\"parcours-entry\"\u003e\n      \u003cp class=\"parcours-period\"\u003e2021 – 2022\u003c/p\u003e\n      \u003cp class=\"parcours-heading\"\u003e\u003cstrong\u003eSenior Java Developer\u003c/strong\u003e · \u003cspan class=\"parcours-name\"\u003eSociété Générale CIB\u003c/span\u003e\u003c/p\u003e\n      \u003cp\u003eMaintenance of Forex applications in the Kondor scope, for the front office. Development of a modern application that takes over some of their features.\u003c/p\u003e\n      \u003cul class=\"aj-tags\"\u003e\n        \u003cli\u003eJava\u003c/li\u003e\n        \u003cli\u003eSpring Batch\u003c/li\u003e\n        \u003cli\u003eSybase\u003c/li\u003e\n        \u003cli\u003eReact\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli class=\"parcours-entry\"\u003e\n      \u003cp class=\"parcours-period\"\u003e2018 – 2021\u003c/p\u003e\n      \u003cp class=\"parcours-heading\"\u003e\u003cstrong\u003eSenior Java Developer\u003c/strong\u003e · \u003cspan class=\"parcours-name\"\u003eBibliothèque nationale de France\u003c/span\u003e\u003c/p\u003e\n      \u003cp\u003eDesign from scratch of DELIA, the legal deposit portal for moving images. Work on two other digital legal deposit applications.\u003c/p\u003e\n      \u003cul class=\"aj-tags\"\u003e\n        \u003cli\u003eSpring Security\u003c/li\u003e\n        \u003cli\u003eSpring Batch\u003c/li\u003e\n        \u003cli\u003eAngular\u003c/li\u003e\n        \u003cli\u003ePostgreSQL\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli class=\"parcours-entry\"\u003e\n      \u003cp class=\"parcours-period\"\u003e2014 – 2018\u003c/p\u003e\n      \u003cp class=\"parcours-heading\"\u003e\u003cstrong\u003eJava Backend Developer\u003c/strong\u003e · \u003cspan class=\"parcours-name\"\u003eBNP Paribas Securities Services\u003c/span\u003e\u003c/p\u003e\n      \u003cp\u003eSecurities back office: processing of Swift messages, the European T2S migration up to the production cutovers, and night production support.\u003c/p\u003e\n      \u003cul class=\"aj-tags\"\u003e\n        \u003cli\u003eJava\u003c/li\u003e\n        \u003cli\u003eHibernate\u003c/li\u003e\n        \u003cli\u003eOracle\u003c/li\u003e\n        \u003cli\u003eSwift messages\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n  \u003caside class=\"parcours-takeaway\"\u003e\n    \u003cp\u003e\u003cstrong\u003eWhat I learned:\u003c/strong\u003e Rigor first. Test, document, and never break production.\u003c/p\u003e\n  \u003c/aside\u003e\n\u003c/section\u003e","title":"Background"},{"content":" Contact\nLet\u0026#39;s talk about it Do you work on Java applications, distributed architectures or the modernization of an existing system? Have a question, an idea, or just want to talk? Write to me.\nEmailFastest aymen.jamaaoui [at] gmail [dot] com Copy → LinkedInFastest in/aymen-jamaaoui → GitHub @aymenjamaaoui → GitLab @aymenjamaaoui → For a quick exchange, email or LinkedIn is the easiest way.\n","permalink":"https://aymenjamaaoui.dev/en/contact/","summary":"\u003cheader class=\"contact-header\"\u003e\n  \u003cp class=\"aj-overline\"\u003eContact\u003c/p\u003e\n  \u003ch1 class=\"contact-title\"\u003eLet\u0026#39;s talk about it\u003c/h1\u003e\n  \u003cdiv class=\"contact-intro\"\u003e\n    \u003cp\u003eDo you work on Java applications, distributed architectures or the modernization of an existing system? Have a question, an idea, or just want to talk? Write to me.\u003c/p\u003e\n\n  \u003c/div\u003e\n\u003c/header\u003e\n\n\n\u003cdiv class=\"contact-cards\"\u003e\n  \u003cdiv class=\"contact-card contact-email contact-card-strong\"\u003e\n    \u003cspan class=\"contact-icon\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 21\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath d=\"M4 4h16c1.1 0 2 .9 2 2v12c0 1.1-.9 2-2 2H4c-1.1 0-2-.9-2-2V6c0-1.1.9-2 2-2z\"\u003e\u003c/path\u003e\n    \u003cpolyline points=\"22,6 12,13 2,6\"\u003e\u003c/polyline\u003e\n\u003c/svg\u003e\u003c/span\u003e\n    \u003cspan class=\"contact-text\"\u003e\n      \u003cspan class=\"contact-name\"\u003e\n        \u003ca class=\"contact-card-link\" data-u=\"YXltZW4uamFtYWFvdWk=\" data-d=\"Z21haWwuY29t\"\u003e\u003cstrong\u003eEmail\u003c/strong\u003e\u003c/a\u003e\u003cspan class=\"contact-badge\"\u003eFastest\u003c/span\u003e\n      \u003c/span\u003e\n      \u003cspan class=\"contact-line-row\"\u003e\n        \u003cspan class=\"contact-line\"\u003eaymen.jamaaoui [at] gmail [dot] com\u003c/span\u003e\n        \u003cbutton type=\"button\" class=\"contact-copy\" data-copied=\"Copied\" aria-live=\"polite\" hidden\u003eCopy\u003c/button\u003e\n      \u003c/span\u003e\n    \u003c/span\u003e\n    \u003cspan class=\"contact-arrow\" aria-hidden=\"true\"\u003e→\u003c/span\u003e\n  \u003c/div\u003e\n  \u003ca class=\"contact-card contact-card-strong\" href=\"https://linkedin.com/in/aymen-jamaaoui/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cspan class=\"contact-icon\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath d=\"M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z\"\u003e\u003c/path\u003e\n    \u003crect x=\"2\" y=\"9\" width=\"4\" height=\"12\"\u003e\u003c/rect\u003e\n    \u003ccircle cx=\"4\" cy=\"4\" r=\"2\"\u003e\u003c/circle\u003e\n\u003c/svg\u003e\u003c/span\u003e\n    \u003cspan class=\"contact-text\"\u003e\n      \u003cspan class=\"contact-name\"\u003e\u003cstrong\u003eLinkedIn\u003c/strong\u003e\u003cspan class=\"contact-badge\"\u003eFastest\u003c/span\u003e\u003c/span\u003e\n      \u003cspan class=\"contact-line\"\u003ein/aymen-jamaaoui\u003c/span\u003e\n    \u003c/span\u003e\n    \u003cspan class=\"contact-arrow\" aria-hidden=\"true\"\u003e→\u003c/span\u003e\n  \u003c/a\u003e\n  \u003ca class=\"contact-card\" href=\"https://github.com/aymenjamaaoui\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cspan class=\"contact-icon\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath\n        d=\"M9 19c-5 1.5-5-2.5-7-3m14 6v-3.87a3.37 3.37 0 0 0-.94-2.61c3.14-.35 6.44-1.54 6.44-7A5.44 5.44 0 0 0 20 4.77 5.07 5.07 0 0 0 19.91 1S18.73.65 16 2.48a13.38 13.38 0 0 0-7 0C6.27.65 5.09 1 5.09 1A5.07 5.07 0 0 0 5 4.77a5.44 5.44 0 0 0-1.5 3.78c0 5.42 3.3 6.61 6.44 7A3.37 3.37 0 0 0 9 18.13V22\"\u003e\n    \u003c/path\u003e\n\u003c/svg\u003e\u003c/span\u003e\n    \u003cspan class=\"contact-text\"\u003e\n      \u003cspan class=\"contact-name\"\u003e\u003cstrong\u003eGitHub\u003c/strong\u003e\u003c/span\u003e\n      \u003cspan class=\"contact-line\"\u003e@aymenjamaaoui\u003c/span\u003e\n    \u003c/span\u003e\n    \u003cspan class=\"contact-arrow\" aria-hidden=\"true\"\u003e→\u003c/span\u003e\n  \u003c/a\u003e\n  \u003ca class=\"contact-card\" href=\"https://gitlab.com/aymenjamaaoui\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003cspan class=\"contact-icon\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 24 24\" fill=\"none\" stroke=\"currentColor\" stroke-width=\"2\"\n    stroke-linecap=\"round\" stroke-linejoin=\"round\"\u003e\n    \u003cpath\n        d=\"M22.65 14.39L12 22.13 1.35 14.39a.84.84 0 0 1-.3-.94l1.22-3.78 2.44-7.51A.42.42 0 0 1 4.82 2a.43.43 0 0 1 .58 0 .42.42 0 0 1 .11.18l2.44 7.49h8.1l2.44-7.51A.42.42 0 0 1 18.6 2a.43.43 0 0 1 .58 0 .42.42 0 0 1 .11.18l2.44 7.51L23 13.45a.84.84 0 0 1-.35.94z\"\u003e\n    \u003c/path\u003e\n\u003c/svg\u003e\u003c/span\u003e\n    \u003cspan class=\"contact-text\"\u003e\n      \u003cspan class=\"contact-name\"\u003e\u003cstrong\u003eGitLab\u003c/strong\u003e\u003c/span\u003e\n      \u003cspan class=\"contact-line\"\u003e@aymenjamaaoui\u003c/span\u003e\n    \u003c/span\u003e\n    \u003cspan class=\"contact-arrow\" aria-hidden=\"true\"\u003e→\u003c/span\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cp class=\"contact-closing\"\u003eFor a quick exchange, email or LinkedIn is the easiest way.\u003c/p\u003e","title":"Contact"}]