How Array References Work:
arr1 and arr2 point to the same memory location in the heap.arr2, the change will reflect in arr1 because both refer to the same object.When you create an object using new, the reference variable points to the object in heap memory.
How References Work with Objects:
p1 and p2 point to the same Person object. Changing the object via p2 affects p1.p1 and p2 point to the same Person object. Changing the object via p2 affects p1.Wrapper classes (Integer, Double, Boolean, etc.) wrap primitive types into objects. These are reference types, and Java performs autoboxing/unboxing to convert between primitive types and wrapper objects.
Wrapper classes (Integer, Double, Boolean, etc.) wrap primitive types into objects. These are reference types, and Java performs autoboxing/unboxing to convert between primitive types and wrapper objects.
Integer num1 = 100;
@@ -2876,12 +2915,11 @@ Wrapper Classes
Wrapper Caching
-- Java caches
Integer objects in the range -128 to 127 for performance.
-- Beyond this range, new objects are created.
+- Java caches
Integer objects in the range -128 to 127 for performance.
+- Beyond this range, new objects are created.
-
Reference and Deep Copy¶
-Shallow Copy: Copies only the reference, so both variables refer to the same object.
+Shallow Copy: Copies only the reference, so both variables refer to the same object.
Example
int[] original = {1, 2, 3};
@@ -2891,7 +2929,7 @@ Reference and Deep CopySystem.out.println(original[0]); // Output: 100
-Deep Copy: Creates a new object with the same data.
+Deep Copy: Creates a new object with the same data.
Example
int[] original = {1, 2, 3};
@@ -2901,34 +2939,31 @@ Reference and Deep CopySystem.out.println(original[0]); // Output: 1
-
Null/NullPointerException¶
-When a reference is not initialized, it holds the value null. Accessing a field or method on a null reference throws a NullPointerException.
+When a reference is not initialized, it holds the value null. Accessing a field or method on a null reference throws a NullPointerException.
-
Garbage Collection¶
-Java uses Garbage Collection to manage memory. When no references point to an object, it becomes eligible for garbage collection.
+Java uses Garbage Collection to manage memory. When no references point to an object, it becomes eligible for garbage collection.
Example
-
+We will learn about garbage collection more in depth in another article.
Summary¶
-- Strings: Immutable, stored in the String Pool if created with literals.
new String() creates a separate object.
+- Strings: Immutable, stored in the String Pool if created with literals.
new String() creates a separate object.
- Arrays: Reference types, so multiple variables can point to the same array object.
-- Classes: Objects are referenced in memory; multiple references can point to the same object.
+- Classes: Objects are referenced in memory multiple references can point to same object.
- Wrapper Classes: Use caching for certain ranges (e.g.,
Integer values from -128 to 127).
- Garbage Collection: Objects are eligible for garbage collection when no active references point to them.
-
String Pool In Depth¶
The String Pool (also called the intern pool) in Java is implemented using a Hash Table-like data structure internally. Let’s explore the design and behavior behind this structure:
Internals¶
@@ -3016,7 +3051,7 @@ String pool Summary
- December 10, 2024
+ December 31, 2024
@@ -3084,7 +3119,7 @@ String pool Summary
-
+
Welcome to Deep Dives by MG
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/#coming-soon","title":"Coming soon","text":""},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#under-the-hood-by-mrudhul-guda","title":"Under the Hood by Mrudhul Guda","text":""},{"location":"changelog/#0.4.0","title":"0.4.0 November 16, 2024","text":"The Abstract Factory Pattern is a creational design pattern that provides an interface for creating families of related or dependent objects without specifying their concrete classes. It promotes loose coupling between client code and the actual implementations, allowing the code to be more flexible and scalable.
The Abstract Factory pattern works as a super-factory that creates other factories. Each factory produced by the abstract factory is responsible for creating a family of related objects.
Key Characteristics
Class Diagram
AbstractFactory\n\u251c\u2500\u2500 createProductA()\n\u2514\u2500\u2500 createProductB()\n\nConcreteFactory1 \u2500\u2500\u2500\u2500> ProductA1, ProductB1\nConcreteFactory2 \u2500\u2500\u2500\u2500> ProductA2, ProductB2\n\nClient \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500> AbstractFactory, AbstractProduct\nLet\u2019s go with an example Imagine you are creating a UI component factory. Your application can switch between two themes: Dark Theme and Light Theme. Both themes provide the same types of components (buttons, text fields) but with different appearances.
Step-1: Define the Abstract Products// Abstract Product: Button\npublic interface Button {\n void render();\n}\n\n// Abstract Product: TextField\npublic interface TextField {\n void render();\n}\n// Concrete Product: Light Button\npublic class LightButton implements Button {\n @Override\n public void render() {\n System.out.println(\"Rendering a Light Button\");\n }\n}\n\n// Concrete Product: Dark Button\npublic class DarkButton implements Button {\n @Override\n public void render() {\n System.out.println(\"Rendering a Dark Button\");\n }\n}\n\n// Concrete Product: Light TextField\npublic class LightTextField implements TextField {\n @Override\n public void render() {\n System.out.println(\"Rendering a Light Text Field\");\n }\n}\n\n// Concrete Product: Dark TextField\npublic class DarkTextField implements TextField {\n @Override\n public void render() {\n System.out.println(\"Rendering a Dark Text Field\");\n }\npublic interface UIFactory {\n Button createButton();\n TextField createTextField();\n}\n// Concrete Factory for Light Theme\npublic class LightUIFactory implements UIFactory {\n @Override\n public Button createButton() {\n return new LightButton();\n }\n\n @Override\n public TextField createTextField() {\n return new LightTextField();\n }\n}\n\n// Concrete Factory for Dark Theme\npublic class DarkUIFactory implements UIFactory {\n @Override\n public Button createButton() {\n return new DarkButton();\n }\n\n @Override\n public TextField createTextField() {\n return new DarkTextField();\n }\n}\npublic class Application {\n private Button button;\n private TextField textField;\n\n public Application(UIFactory factory) {\n this.button = factory.createButton();\n this.textField = factory.createTextField();\n }\n\n public void renderUI() {\n button.render();\n textField.render();\n }\n\n public static void main(String[] args) {\n // Client can choose between different factories.\n UIFactory factory = new DarkUIFactory(); // Could be switched to LightUIFactory\n Application app = new Application(factory);\n app.renderUI();\n }\n}\nOutput:
Rendering a Dark Button\nRendering a Dark Text Field\nIn Spring Boot, the Abstract Factory pattern can complement dependency injection (DI) by delegating object creation logic to the factory. Here\u2019s how to implement it with Spring Boot.
Step-1: Define Factory Beans@Configuration\npublic class UIFactoryConfig {\n\n @Bean\n public UIFactory uiFactory(@Value(\"${app.theme}\") String theme) {\n if (\"dark\".equalsIgnoreCase(theme)) {\n return new DarkUIFactory();\n } else {\n return new LightUIFactory();\n }\n }\n}\n@RestController\n@RequestMapping(\"/ui\")\npublic class UIController {\n\n private final UIFactory uiFactory;\n\n @Autowired\n public UIController(UIFactory uiFactory) {\n this.uiFactory = uiFactory;\n }\n\n @GetMapping(\"/render\")\n public void renderUI() {\n Button button = uiFactory.createButton();\n TextField textField = uiFactory.createTextField();\n\n button.render();\n textField.render();\n }\n}\n# application.properties\napp.theme=dark\nIn this example, the theme is configured through the application.properties file, and the factory selection is handled by the Spring context.
The Abstract Factory Pattern is a powerful tool when designing systems that need to create multiple families of related objects. While it adds complexity, the benefits include extensibility, maintainability, and loose coupling. In a Spring Boot application, it works well alongside dependency injection, especially when configurations like themes or environments vary.
"},{"location":"fundamentaldives/DesignPatterns/Adapter/","title":"Adapter Design Pattern","text":""},{"location":"fundamentaldives/DesignPatterns/Adapter/#what","title":"What ?","text":"The Adapter Pattern is a structural design pattern in software development that allows objects with incompatible interfaces to work together. It acts as a bridge between two incompatible interfaces, providing a wrapper or a mediator to enable their interaction without changing their existing code.
This Pattern converts the interface of a class into another interface that a client expects. This helps integrate two systems with different interfaces so they can work together without altering their code. It is often used when a legacy system needs to be integrated with new components or when third-party APIs are integrated into an existing codebase.
Analogy
Think of a power plug adapter You have an appliance with a US plug (two flat pins), but you need to connect it to a European socket (two round holes). The adapter ensures that both the incompatible interfaces (US and European plugs) work together without modifying either.
"},{"location":"fundamentaldives/DesignPatterns/Adapter/#when-to-use","title":"When to Use ?","text":"There are two common ways to implement the Adapter Pattern:
In this approach, the adapter class extends the adaptee (the class that has the incompatible interface) and implements the interface that the client expects.
Class Adapter Java Example// Target Interface - The desired interface that client expects\ninterface MediaPlayer {\n void play(String audioType, String fileName);\n}\n\n// Adaptee - Incompatible interface that needs adaptation\nclass AdvancedMediaPlayer {\n void playMp3(String fileName) {\n System.out.println(\"Playing mp3 file: \" + fileName);\n }\n\n void playMp4(String fileName) {\n System.out.println(\"Playing mp4 file: \" + fileName);\n }\n}\n\n// Class Adapter - Adapts AdvancedMediaPlayer to MediaPlayer\nclass MediaAdapter extends AdvancedMediaPlayer implements MediaPlayer {\n @Override\n public void play(String audioType, String fileName) {\n if (audioType.equalsIgnoreCase(\"mp3\")) {\n playMp3(fileName);\n } else if (audioType.equalsIgnoreCase(\"mp4\")) {\n playMp4(fileName);\n }\n }\n}\n\n// Client Code\npublic class AudioPlayer {\n public static void main(String[] args) {\n MediaPlayer player = new MediaAdapter();\n player.play(\"mp3\", \"song.mp3\");\n player.play(\"mp4\", \"video.mp4\");\n }\n}\nMediaAdapter extends AdvancedMediaPlayer (inheriting the original functionality) and implements the MediaPlayer interface (adapting it to what the client expects).
In this approach, the adapter contains an instance of the adaptee class and delegates calls to the appropriate methods.
Object Adapter Java Example// Target Interface\ninterface MediaPlayer {\n void play(String audioType, String fileName);\n}\n\n// Adaptee\nclass AdvancedMediaPlayer {\n void playMp3(String fileName) {\n System.out.println(\"Playing mp3 file: \" + fileName);\n }\n\n void playMp4(String fileName) {\n System.out.println(\"Playing mp4 file: \" + fileName);\n }\n}\n\n// Object Adapter\nclass MediaAdapter implements MediaPlayer {\n private AdvancedMediaPlayer advancedPlayer;\n\n public MediaAdapter(AdvancedMediaPlayer advancedPlayer) {\n this.advancedPlayer = advancedPlayer;\n }\n\n @Override\n public void play(String audioType, String fileName) {\n if (audioType.equalsIgnoreCase(\"mp3\")) {\n advancedPlayer.playMp3(fileName);\n } else if (audioType.equalsIgnoreCase(\"mp4\")) {\n advancedPlayer.playMp4(fileName);\n }\n }\n}\n\n// Client Code\npublic class AudioPlayer {\n public static void main(String[] args) {\n AdvancedMediaPlayer advancedPlayer = new AdvancedMediaPlayer();\n MediaPlayer adapter = new MediaAdapter(advancedPlayer);\n adapter.play(\"mp3\", \"song.mp3\");\n adapter.play(\"mp4\", \"video.mp4\");\n }\n}\nIn this version, MediaAdapter holds a reference to the AdvancedMediaPlayer instance and delegates method calls instead of extending the class.
In a Spring Boot context, the Adapter Pattern can be used to integrate an external or legacy service with your application's service layer.
Integrating a Legacy Payment Service// Legacy Payment Service - Adaptee\nclass LegacyPaymentService {\n public void payWithCreditCard(String cardNumber) {\n System.out.println(\"Payment made using Legacy Credit Card: \" + cardNumber);\n }\n}\n\n// Target Interface\ninterface PaymentService {\n void processPayment(String cardNumber);\n}\n\n// Adapter Implementation - Integrating LegacyPaymentService with PaymentService\n@Component\nclass PaymentServiceAdapter implements PaymentService {\n private final LegacyPaymentService legacyService;\n\n // Constructor injection\n public PaymentServiceAdapter(LegacyPaymentService legacyService) {\n this.legacyService = legacyService;\n }\n\n @Override\n public void processPayment(String cardNumber) {\n legacyService.payWithCreditCard(cardNumber);\n }\n}\n\n// Spring Boot Controller\n@RestController\n@RequestMapping(\"/payments\")\npublic class PaymentController {\n\n private final PaymentService paymentService;\n\n @Autowired\n public PaymentController(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n\n @PostMapping\n public String makePayment(@RequestParam String cardNumber) {\n paymentService.processPayment(cardNumber);\n return \"Payment Successful\";\n }\n}\nLegacyPaymentService is an old service with an incompatible method.PaymentServiceAdapter acts as an adapter by implementing the PaymentService interface and internally calling the legacy service.PaymentController depends on PaymentService, which can now work seamlessly with the legacy system.The Adapter Pattern enhances flexibility by decoupling client code from specific implementations, promotes reusability by enabling compatibility between systems, improves maintainability by isolating legacy or third-party code, and simplifies testing through easy mock or stub usage.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/","title":"Bridge","text":""},{"location":"fundamentaldives/DesignPatterns/Bridge/#what","title":"What ?","text":"The Bridge Pattern is a structural design pattern that helps to decouple an abstraction from its implementation so that both can vary independently. This pattern is especially useful when you need to manage complex class hierarchies or have multiple dimensions of variations.
When both the abstraction (interface) and its implementation (how it works internally) need to evolve, the code becomes complex and hard to manage. Bridge helps to separate these concerns. The pattern separates the abstraction (interface) from the actual implementation and lets them evolve independently by delegating the concrete work to another interface.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/#structure","title":"Structure ?","text":"The structure involves two key parts:
The Abstraction contains a reference to the Implementor (interface or class). This lets the abstraction delegate the implementation details to the concrete implementations.
Class Diagram
Abstraction --> Implementor\n | |\nRefinedAbstraction ConcreteImplementor\nCircleRed, CircleBlue, SquareRed, SquareBlue this can be avoided with the Bridge pattern.Let\u2019s look at a real-world example rendering shapes on different platforms. The rendering logic could vary depending on the platform (Windows, Linux, etc.), but the shape (e.g., Circle, Rectangle) stays the same.
Spring Boot ExampleIn a Spring Boot application, you can use the Bridge pattern to switch between different implementations of a service dynamically, such as switching between multiple ways of sending notifications (e.g., Email, SMS). This is helpful when services have multiple implementations, and you need to inject them dynamically without changing the client code.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/#rendering-shapes-on-different-platforms","title":"Rendering Shapes on Different Platforms","text":"Step-1: Define the Implementor interfaceinterface Renderer {\n void render(String shape);\n}\nclass VectorRenderer implements Renderer {\n @Override\n public void render(String shape) {\n System.out.println(\"Rendering \" + shape + \" as vectors.\");\n }\n}\n\nclass RasterRenderer implements Renderer {\n @Override\n public void render(String shape) {\n System.out.println(\"Rendering \" + shape + \" as pixels.\");\n }\n}\nabstract class Shape {\n protected Renderer renderer;\n\n public Shape(Renderer renderer) {\n this.renderer = renderer;\n }\n\n public abstract void draw();\n}\nclass Circle extends Shape {\n public Circle(Renderer renderer) {\n super(renderer);\n }\n\n @Override\n public void draw() {\n renderer.render(\"Circle\");\n }\n}\n\nclass Rectangle extends Shape {\n public Rectangle(Renderer renderer) {\n super(renderer);\n }\n\n @Override\n public void draw() {\n renderer.render(\"Rectangle\");\n }\n}\npublic class BridgePatternDemo {\n public static void main(String[] args) {\n Shape circle = new Circle(new VectorRenderer());\n circle.draw(); // Output: Rendering Circle as vectors.\n\n Shape rectangle = new Rectangle(new RasterRenderer());\n rectangle.draw(); // Output: Rendering Rectangle as pixels.\n }\n}\npublic interface NotificationSender {\n void send(String message);\n}\n@Component\npublic class EmailSender implements NotificationSender {\n @Override\n public void send(String message) {\n System.out.println(\"Sending Email: \" + message);\n }\n}\n\n@Component\npublic class SmsSender implements NotificationSender {\n @Override\n public void send(String message) {\n System.out.println(\"Sending SMS: \" + message);\n }\n}\npublic abstract class Notification {\n protected NotificationSender sender;\n\n public Notification(NotificationSender sender) {\n this.sender = sender;\n }\n\n public abstract void notify(String message);\n}\n@Component\npublic class UrgentNotification extends Notification {\n\n @Autowired\n public UrgentNotification(NotificationSender sender) {\n super(sender);\n }\n\n @Override\n public void notify(String message) {\n System.out.println(\"Urgent Notification:\");\n sender.send(message);\n }\n}\n@RestController\n@RequestMapping(\"/notifications\")\npublic class NotificationController {\n\n private final UrgentNotification notification;\n\n @Autowired\n public NotificationController(UrgentNotification notification) {\n this.notification = notification;\n }\n\n @PostMapping(\"/send\")\n public String sendNotification(@RequestBody String message) {\n notification.notify(message);\n return \"Notification sent!\";\n }\n}\nIn this example, the Bridge Pattern allows you to switch between different ways of sending notifications (email or SMS) without changing the client code (the NotificationController).
The Bridge Pattern is an essential design pattern to consider when your class hierarchy is growing unmanageable due to multiple dimensions of variations. Use this pattern when you need to decouple abstraction from implementation and allow them to evolve independently, but avoid it when simpler solutions can suffice.
"},{"location":"fundamentaldives/DesignPatterns/Builder/","title":"Builder Design","text":""},{"location":"fundamentaldives/DesignPatterns/Builder/#what","title":"What ?","text":"The Builder Pattern is a creational design pattern that allows the construction of complex objects step by step. It separates the construction process from the actual object, giving more control over the construction process.
This Pattern simplifies the creation of complex objects with many optional fields by enabling incremental construction through method chaining, avoiding constructors with numerous parameters. It's ideal for objects requiring various configurations or optional parameters.
"},{"location":"fundamentaldives/DesignPatterns/Builder/#when-to-use","title":"When to Use ?","text":"Below is a basic Java example demonstrating the pattern. Assume we need to build a Car object with several optional fields.
public class Car {\n // Required fields\n private final String make;\n private final String model;\n\n // Optional fields\n private final String color;\n private final int year;\n private final boolean automatic;\n\n // Private constructor accessible only through Builder\n private Car(Builder builder) {\n this.make = builder.make;\n this.model = builder.model;\n this.color = builder.color;\n this.year = builder.year;\n this.automatic = builder.automatic;\n }\n\n // Getters (optional, based on your needs)\n public String getMake() { return make; }\n public String getModel() { return model; }\n public String getColor() { return color; }\n public int getYear() { return year; }\n public boolean isAutomatic() { return automatic; }\n\n // Static inner Builder class\n public static class Builder {\n // Required fields\n private final String make;\n private final String model;\n\n // Optional fields initialized to default values\n private String color = \"White\";\n private int year = 2020;\n private boolean automatic = true;\n\n // Builder constructor with required fields\n public Builder(String make, String model) {\n this.make = make;\n this.model = model;\n }\n\n // Setter-like methods for optional fields, returning the builder object\n public Builder color(String color) {\n this.color = color;\n return this;\n }\n\n public Builder year(int year) {\n this.year = year;\n return this;\n }\n\n public Builder automatic(boolean automatic) {\n this.automatic = automatic;\n return this;\n }\n\n // Build method to create the final Car object\n public Car build() {\n return new Car(this);\n }\n }\n\n @Override\n public String toString() {\n return \"Car [make=\" + make + \", model=\" + model + \n \", color=\" + color + \", year=\" + year + \n \", automatic=\" + automatic + \"]\";\n }\n}\npublic class Main {\n public static void main(String[] args) {\n // Using the builder to create a Car object\n Car car = new Car.Builder(\"Tesla\", \"Model S\")\n .color(\"Red\")\n .year(2023)\n .automatic(true)\n .build();\n\n System.out.println(car);\n }\n}\nOutput:
Car [make=Tesla, model=Model S, color=Red, year=2023, automatic=true]\nIn Spring Boot, you often need to build objects like DTOs, configurations, or entities with complex structures. Using the Builder Pattern can make object construction more manageable, especially when working with REST APIs.
Using Builder Pattern for DTOs in Spring Boot// Let's assume a UserDTO object for API responses.\npublic class UserDTO {\n private final String username;\n private final String email;\n private final String role;\n\n private UserDTO(Builder builder) {\n this.username = builder.username;\n this.email = builder.email;\n this.role = builder.role;\n }\n\n public static class Builder {\n private String username;\n private String email;\n private String role;\n\n public Builder username(String username) {\n this.username = username;\n return this;\n }\n\n public Builder email(String email) {\n this.email = email;\n return this;\n }\n\n public Builder role(String role) {\n this.role = role;\n return this;\n }\n\n public UserDTO build() {\n return new UserDTO(this);\n }\n }\n}\n@RestController\n@RequestMapping(\"/api/users\")\npublic class UserController {\n\n @GetMapping(\"/{id}\")\n public UserDTO getUserById(@PathVariable Long id) {\n // Simulate fetching user details from a database\n return new UserDTO.Builder()\n .username(\"johndoe\")\n .email(\"john.doe@example.com\")\n .role(\"ADMIN\")\n .build();\n }\n}\nThis approach ensures that the object returned from the API is constructed cleanly with only the necessary fields set.
Alternative WaysTelescoping Constructors Multiple overloaded constructors for different parameter combinations but Not ideal for readability and maintainability.
public Car(String make, String model) { ... }\npublic Car(String make, String model, String color) { ... }\npublic Car(String make, String model, String color, int year) { ... }\nSetter Methods Useful for mutable objects but doesn\u2019t guarantee immutability but less readable when constructing objects with many attributes.
Car car = new Car();\ncar.setMake(\"Tesla\");\ncar.setModel(\"Model S\");\ncar.setColor(\"Red\");\nThe Builder Pattern is an elegant way to handle object creation, especially when dealing with many fields or optional parameters. It ensures code readability, immutability, and flexibility while avoiding the need for numerous constructors. However, it should be used only when necessary, as simple objects may not benefit from it.
Note
In Spring Boot, the Builder Pattern can be effectively used for creating DTOs and other complex objects, improving both code readability and maintenance. This pattern fits well when dealing with REST API responses or configuration settings, ensuring your objects are built in a clear, consistent manner.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/","title":"Circuit Breakers","text":""},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#what","title":"What ?","text":"A circuit breaker is a design pattern used to prevent cascading failures and manage service availability. If a service call repeatedly fails, the circuit breaker \"trips\" and prevents further attempts, allowing the system to recover gracefully. This pattern mimics the behavior of electrical circuit breakers.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#why-is-it-needed","title":"Why is it Needed ?","text":"Real-world scenario
If Service A depends on Service B but Service B becomes unavailable, Service A will receive failures continuously. A circuit breaker prevents Service A from overloading itself and Service B by failing fast.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#types-of-circuit-breakers","title":"Types of Circuit Breakers","text":"There are multiple models of circuit breakers to choose from depending on use case:
Count-based Circuit Breaker - Trips if a predefined number of failures occur, eg: If there are 3 consecutive failed requests, the breaker opens.
Time-based Circuit Breaker - Monitors failures within a window of time and trips if the failure threshold is met, eg: If 5 requests out of 10 fail within 1 minute, it opens.
Sliding Window Circuit Breaker - A rolling window of requests over time determines if the circuit trips, Useful when failure patterns are sporadic.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#how-does-it-work","title":"How Does it Work?","text":"The basic mechanics of a circuit breaker involve three states:
Closed State
Open State
Half-Open State
State Transition Flow
Closed -> (failure threshold reached) -> Open -> (timeout) -> Half-Open -> (success) -> Closed\nSeveral popular libraries and frameworks make circuit breaker implementations simple. Below are code examples in Java and Python.
Java with Resilience4jPython with PyBreaker// Resilience4j is a library providing circuit breaker implementations.\nimport io.github.resilience4j.circuitbreaker.CircuitBreaker;\nimport io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;\nimport io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;\n\nimport java.time.Duration;\n\npublic class Example {\n public static void main(String[] args) {\n // Configuration\n CircuitBreakerConfig config = CircuitBreakerConfig.custom()\n .failureRateThreshold(50) // Open if 50% of requests fail\n .waitDurationInOpenState(Duration.ofSeconds(5)) // Wait 5 seconds before Half-Open\n .build();\n\n CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);\n CircuitBreaker circuitBreaker = registry.circuitBreaker(\"myService\");\n\n // Wrap a call in the circuit breaker\n String response = circuitBreaker.executeSupplier(() -> makeHttpRequest());\n\n System.out.println(response);\n }\n\n private static String makeHttpRequest() {\n // Simulate an HTTP request here\n return \"Success!\";\n }\n}\n# PyBreaker is a library implementing the Circuit Breaker pattern for Python applications.\n\nfrom pybreaker import CircuitBreaker, CircuitBreakerError\nimport requests\n\n# Define a circuit breaker\nbreaker = CircuitBreaker(fail_max=3, reset_timeout=5)\n\n@breaker\ndef fetch_data(url):\n response = requests.get(url)\n if response.status_code != 200:\n raise Exception(\"Service unavailable\")\n return response.json()\n\ntry:\n data = fetch_data('https://api.example.com/data')\n print(data)\nexcept CircuitBreakerError:\n print(\"Circuit is open. Service unavailable.\")\nCircuit breakers need to be monitored to ensure they perform correctly. You can integrate them with monitoring tools like Prometheus or Grafana. Many libraries offer hooks to capture metrics such as
Circuit breakers are crucial in modern, distributed systems, preventing unnecessary retries and protecting systems from cascading failures. As systems grow in complexity, using the circuit breaker pattern helps maintain high availability and resilience.
"},{"location":"fundamentaldives/DesignPatterns/Composite/","title":"Composite","text":""},{"location":"fundamentaldives/DesignPatterns/Composite/#what","title":"What ?","text":"The Composite Pattern is a structural design pattern used in software design to represent part whole hierarchies. It enables you to build complex object structures by treating both individual objects and compositions of objects uniformly.
This pattern allows you to treat a group of objects in the same way as a single object. This is especially useful when building tree structures (like directories or UI components).
Key Concepts
The idea is to define a common interface for all the objects, whether simple or complex, so they can be treated uniformly.
Component objects. It may contain both Leafs and other Composites.Basic Structure UML
Component (Interface or Abstract class)\n\u251c\u2500\u2500 Leaf (Concrete class)\n\u2514\u2500\u2500 Composite (Concrete class containing Components)\ndraw() should work for both individual shapes and a group of shapes.Component interface for both Leafs and Composites.// 1. Component Interface\ninterface Component {\n void showDetails(); // Common operation for both Leaf and Composite\n}\n\n// 2. Leaf Class (Single object)\nclass Employee implements Component {\n private String name;\n private String position;\n\n public Employee(String name, String position) {\n this.name = name;\n this.position = position;\n }\n\n @Override\n public void showDetails() {\n System.out.println(name + \" works as \" + position);\n }\n}\n\n// 3. Composite Class (Composite of Components)\nclass Department implements Component {\n private List<Component> employees = new ArrayList<>();\n\n public void addEmployee(Component employee) {\n employees.add(employee);\n }\n\n public void removeEmployee(Component employee) {\n employees.remove(employee);\n }\n\n @Override\n public void showDetails() {\n for (Component employee : employees) {\n employee.showDetails();\n }\n }\n}\n\n// 4. Client Code\npublic class CompositePatternDemo {\n public static void main(String[] args) {\n // Create individual employees\n Component emp1 = new Employee(\"John\", \"Developer\");\n Component emp2 = new Employee(\"Doe\", \"Tester\");\n\n // Create a department and add employees to it\n Department department = new Department();\n department.addEmployee(emp1);\n department.addEmployee(emp2);\n\n // Show details\n System.out.println(\"Department Details:\");\n department.showDetails();\n }\n}\nDepartment Details:\nJohn works as Developer\nDoe works as Tester\nIn Spring Boot, the Composite pattern can fit into cases where you model tree-based structures in your business logic, such as:
// 1. Component Interface\npublic interface ProductCategory {\n String getName();\n void showCategoryDetails();\n}\n\n// 2. Leaf Class\npublic class Product implements ProductCategory {\n private String name;\n\n public Product(String name) {\n this.name = name;\n }\n\n @Override\n public String getName() {\n return name;\n }\n\n @Override\n public void showCategoryDetails() {\n System.out.println(\"Product: \" + name);\n }\n}\n\n// 3. Composite Class\npublic class Category implements ProductCategory {\n private String name;\n private List<ProductCategory> children = new ArrayList<>();\n\n public Category(String name) {\n this.name = name;\n }\n\n public void add(ProductCategory category) {\n children.add(category);\n }\n\n public void remove(ProductCategory category) {\n children.remove(category);\n }\n\n @Override\n public String getName() {\n return name;\n }\n\n @Override\n public void showCategoryDetails() {\n System.out.println(\"Category: \" + name);\n for (ProductCategory child : children) {\n child.showCategoryDetails();\n }\n }\n}\n\n// 4. Controller in Spring Boot\n@RestController\n@RequestMapping(\"/categories\")\npublic class CategoryController {\n\n @GetMapping(\"/example\")\n public void example() {\n // Creating products\n ProductCategory p1 = new Product(\"Laptop\");\n ProductCategory p2 = new Product(\"Phone\");\n\n // Creating a category and adding products\n Category electronics = new Category(\"Electronics\");\n electronics.add(p1);\n electronics.add(p2);\n\n // Display details\n electronics.showCategoryDetails();\n }\n}\nCategory: Electronics\nProduct: Laptop\nProduct: Phone\n@OneToMany).The Composite Pattern is a powerful structural pattern for managing hierarchical, tree-like structures. It allows uniform handling of individual and composite objects, making it ideal for UI elements, filesystems, or business domains with nested elements. When integrating with Spring Boot, it works well in controllers, services, or JPA entities for modeling hierarchical data. However, avoid using it when there\u2019s no hierarchy or when performance is critical (deep recursion). Use it wisely, and it can help you reduce complexity and simplify your code.
"},{"location":"fundamentaldives/DesignPatterns/Decorator/","title":"Decorator","text":""},{"location":"fundamentaldives/DesignPatterns/Decorator/#what","title":"What ?","text":"The Decorator Pattern is a structural design pattern that allows behavior to be added to individual objects, either statically or dynamically, without affecting the behavior of other objects from the same class. This pattern is particularly useful when you need to add functionality to objects without subclassing and in scenarios where multiple combinations of behaviors are required.
This pattern is used to attach additional responsibilities or behaviors to an object dynamically. It wraps the original object, adding new behavior while keeping the object\u2019s interface intact. A decorator class has a reference to the original object and implements the same interface.
Key Concepts
Component interface.Component interface and holds a reference to a Component object.Decorator that add specific functionalities.Let's consider an example where we are building a coffee shop. Different types of coffees can be enhanced with add-ons like milk, sugar, etc. Using the decorator pattern, we can apply these add-ons dynamically without subclassing.
// Step 1: Component Interface\npublic interface Coffee {\n String getDescription();\n double getCost();\n}\n\n// Step 2: ConcreteComponent (Basic Coffee)\npublic class BasicCoffee implements Coffee {\n @Override\n public String getDescription() {\n return \"Basic Coffee\";\n }\n\n @Override\n public double getCost() {\n return 2.0;\n }\n}\n\n// Step 3: Decorator (Abstract)\npublic abstract class CoffeeDecorator implements Coffee {\n protected Coffee coffee; // The object being decorated\n\n public CoffeeDecorator(Coffee coffee) {\n this.coffee = coffee;\n }\n\n public String getDescription() {\n return coffee.getDescription();\n }\n\n public double getCost() {\n return coffee.getCost();\n }\n}\n\n// Step 4: Concrete Decorators (e.g., Milk, Sugar)\npublic class MilkDecorator extends CoffeeDecorator {\n public MilkDecorator(Coffee coffee) {\n super(coffee);\n }\n\n @Override\n public String getDescription() {\n return coffee.getDescription() + \", Milk\";\n }\n\n @Override\n public double getCost() {\n return coffee.getCost() + 0.5;\n }\n}\n\npublic class SugarDecorator extends CoffeeDecorator {\n public SugarDecorator(Coffee coffee) {\n super(coffee);\n }\n\n @Override\n public String getDescription() {\n return coffee.getDescription() + \", Sugar\";\n }\n\n @Override\n public double getCost() {\n return coffee.getCost() + 0.2;\n }\n}\n\n// Step 5: Usage\npublic class CoffeeShop {\n public static void main(String[] args) {\n Coffee coffee = new BasicCoffee();\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n\n coffee = new MilkDecorator(coffee);\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n\n coffee = new SugarDecorator(coffee);\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n }\n}\nBasic Coffee $2.0\nBasic Coffee, Milk $2.5\nBasic Coffee, Milk, Sugar $2.7\nIn Spring Boot, the decorator pattern can be used in scenarios such as logging, monitoring, or security checks. You can implement a decorator pattern to enhance service classes without changing their core logic. Here's an example where we decorate a service class to add logging functionality.
Component Interface (Service Layer)public interface UserService {\n String getUserDetails(String userId);\n}\n@Service\npublic class UserServiceImpl implements UserService {\n @Override\n public String getUserDetails(String userId) {\n return \"User details for \" + userId;\n }\n}\n@Service\npublic class LoggingUserService implements UserService {\n\n private final UserService userService;\n\n public LoggingUserService(UserService userService) {\n this.userService = userService;\n }\n\n @Override\n public String getUserDetails(String userId) {\n System.out.println(\"Fetching details for user: \" + userId);\n return userService.getUserDetails(userId);\n }\n}\n@Configuration\npublic class ServiceConfig {\n\n @Bean\n public UserService userService(UserServiceImpl userServiceImpl) {\n return new LoggingUserService(userServiceImpl);\n }\n}\nUserServiceImpl provides the basic functionality.LoggingUserService acts as a decorator, adding logging before calling the original method.The Decorator Pattern is a powerful and flexible way to enhance objects with additional behaviors dynamically without altering their structure. It shines in scenarios requiring combinations of behaviors and helps maintain clean, modular code.
In Spring Boot, it can be used for decorating services with additional features like logging, security, or metrics, allowing these aspects to remain separate from the core business logic.
This pattern should be used thoughtfully since excessive use can introduce complexity and make debugging difficult. However, when applied correctly, it ensures that code remains extensible and adheres to the Single Responsibility Principle and Open Closed Principle.
"},{"location":"fundamentaldives/DesignPatterns/Facade/","title":"Facade","text":"The Facade Pattern is a structural design pattern commonly used to provide a simple, unified interface to a complex subsystem of classes, libraries, or frameworks. This pattern makes a complex library or system easier to use by hiding the underlying complexities and exposing only the functionality that is relevant for the client.
"},{"location":"fundamentaldives/DesignPatterns/Facade/#what","title":"What ?","text":"Let's go through a simple Java example demonstrating a Facade pattern for a Home Theater System:
// Subsystem classes\nclass Amplifier {\n public void on() { System.out.println(\"Amplifier is ON.\"); }\n public void off() { System.out.println(\"Amplifier is OFF.\"); }\n}\n\nclass DVDPlayer {\n public void play() { System.out.println(\"Playing movie.\"); }\n public void stop() { System.out.println(\"Stopping movie.\"); }\n}\n\nclass Projector {\n public void on() { System.out.println(\"Projector is ON.\"); }\n public void off() { System.out.println(\"Projector is OFF.\"); }\n}\n\n// Facade class\nclass HomeTheaterFacade {\n private Amplifier amplifier;\n private DVDPlayer dvdPlayer;\n private Projector projector;\n\n public HomeTheaterFacade(Amplifier amp, DVDPlayer dvd, Projector proj) {\n this.amplifier = amp;\n this.dvdPlayer = dvd;\n this.projector = proj;\n }\n\n public void watchMovie() {\n System.out.println(\"Setting up movie...\");\n amplifier.on();\n projector.on();\n dvdPlayer.play();\n }\n\n public void endMovie() {\n System.out.println(\"Shutting down movie...\");\n dvdPlayer.stop();\n projector.off();\n amplifier.off();\n }\n}\n\n// Client code\npublic class FacadePatternDemo {\n public static void main(String[] args) {\n Amplifier amp = new Amplifier();\n DVDPlayer dvd = new DVDPlayer();\n Projector proj = new Projector();\n\n HomeTheaterFacade homeTheater = new HomeTheaterFacade(amp, dvd, proj);\n\n homeTheater.watchMovie();\n homeTheater.endMovie();\n }\n}\nSetting up movie...\nAmplifier is ON.\nProjector is ON.\nPlaying movie.\nShutting down movie...\nStopping movie.\nProjector is OFF.\nAmplifier is OFF.\nIn Spring Boot, the Facade pattern can be applied to services or controllers to hide the complexity of business logic or external systems. For example, a facade class can wrap multiple service calls or integrate external APIs to provide a simplified interface to the client (like a REST controller).
Let's go through a example of how to apply the Facade pattern in a Spring Boot application.
Example Scenario: A Payment System interacts with several services (like PaymentGatewayService, NotificationService, and OrderService). We create a PaymentFacade to simplify the interaction.
Step-1: Subsystem Services@Service\npublic class PaymentGatewayService {\n public void processPayment(String orderId) {\n System.out.println(\"Processing payment for order: \" + orderId);\n }\n}\n\n@Service\npublic class NotificationService {\n public void sendNotification(String message) {\n System.out.println(\"Sending notification: \" + message);\n }\n}\n\n@Service\npublic class OrderService {\n public void completeOrder(String orderId) {\n System.out.println(\"Completing order: \" + orderId);\n }\n}\n@Service\npublic class PaymentFacade {\n\n private final PaymentGatewayService paymentGatewayService;\n private final NotificationService notificationService;\n private final OrderService orderService;\n\n @Autowired\n public PaymentFacade(PaymentGatewayService paymentGatewayService,\n NotificationService notificationService,\n OrderService orderService) {\n this.paymentGatewayService = paymentGatewayService;\n this.notificationService = notificationService;\n this.orderService = orderService;\n }\n\n public void makePayment(String orderId) {\n System.out.println(\"Initiating payment process...\");\n paymentGatewayService.processPayment(orderId);\n orderService.completeOrder(orderId);\n notificationService.sendNotification(\"Payment completed for order: \" + orderId);\n }\n}\n@RestController\n@RequestMapping(\"/api/payment\")\npublic class PaymentController {\n\n private final PaymentFacade paymentFacade;\n\n @Autowired\n public PaymentController(PaymentFacade paymentFacade) {\n this.paymentFacade = paymentFacade;\n }\n\n @PostMapping(\"/pay/{orderId}\")\n public ResponseEntity<String> pay(@PathVariable String orderId) {\n paymentFacade.makePayment(orderId);\n return ResponseEntity.ok(\"Payment successful for order: \" + orderId);\n }\n}\nPaymentGatewayService, OrderService, and NotificationService.PaymentFacade to expose a single endpoint for making payments.The Facade Pattern is a powerful tool for simplifying interactions with complex systems. It is especially useful when working with large subsystems or external APIs, as it encapsulates the internal workings and provides a simple interface to clients. In a Spring Boot application, you can use it to manage complex business logic or interactions with multiple services within a single, cohesive facade. However, it should be used judiciously to avoid over-abstraction or unnecessary complexity.
Note
This makes the Facade pattern a valuable asset in both object-oriented design and modern frameworks like Spring Boot.
"},{"location":"fundamentaldives/DesignPatterns/Facade/#this-pattern-is-best-used-when","title":"This pattern is best used when:","text":"- You need to simplify client interactions.\n- You want to decouple the client from a complex subsystem.\n- You aim to improve maintainability and reduce dependencies.\n- The system is already simple.\n- Performance is a key concern.\nFactory Method is a creational design pattern that provides an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created (decide which class to instantiate). This pattern delegates the responsibility of object creation to subclasses rather than using a direct constructor call, It is one of the most widely used creational design patterns, It helps in the creation of objects without specifying the exact class of the object that will be created.
You provide a \"factory\" method that the client code calls to get the object, but the actual object that gets created is determined at runtime (based on some logic).
"},{"location":"fundamentaldives/DesignPatterns/FactoryMethod/#when-to-use","title":"When to Use ?","text":"public interface Notification {\n void notifyUser();\n}\npublic class SMSNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an SMS notification.\");\n }\n}\n\npublic class EmailNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an Email notification.\");\n }\n}\npublic abstract class NotificationFactory {\n public abstract Notification createNotification();\n}\npublic class SMSNotificationFactory extends NotificationFactory {\n @Override\n public Notification createNotification() {\n return new SMSNotification();\n }\n}\n\npublic class EmailNotificationFactory extends NotificationFactory {\n @Override\n public Notification createNotification() {\n return new EmailNotification();\n }\n}\npublic class Client {\n public static void main(String[] args) {\n NotificationFactory factory = new SMSNotificationFactory();\n Notification notification = factory.createNotification();\n notification.notifyUser();\n\n factory = new EmailNotificationFactory();\n notification = factory.createNotification();\n notification.notifyUser();\n }\n}\nSpring Boot relies heavily on dependency injection (DI) and Inversion of Control (IoC), which means Spring beans can act as factory classes to produce the desired objects.
"},{"location":"fundamentaldives/DesignPatterns/FactoryMethod/#example-notification-factory-with-spring-boot","title":"Example: Notification Factory with Spring Boot","text":"Define the Product Interface and Implementations (Same as Before)public interface Notification {\n void notifyUser();\n}\n\npublic class SMSNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an SMS notification.\");\n }\n}\n\npublic class EmailNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an Email notification.\");\n }\n}\nimport org.springframework.stereotype.Service;\n// This class will act as the **Factory**. You can make it a Spring **`@Service` or `@Component`** bean, so Spring manages it.\n@Service\npublic class NotificationFactory {\n public Notification createNotification(String type) {\n if (type.equalsIgnoreCase(\"SMS\")) {\n return new SMSNotification();\n } else if (type.equalsIgnoreCase(\"Email\")) {\n return new EmailNotification();\n }\n throw new IllegalArgumentException(\"Unknown notification type: \" + type);\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.PathVariable;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class NotificationController {\n\n @Autowired\n private NotificationFactory notificationFactory;\n\n @GetMapping(\"/notify/{type}\")\n public String sendNotification(@PathVariable String type) {\n Notification notification = notificationFactory.createNotification(type);\n notification.notifyUser();\n return \"Notification sent: \" + type;\n }\n}\nWhen you access /notify/SMS or /notify/Email, it will dynamically create and send the corresponding notification after running the spring boot application.
The Factory Pattern enables dynamic object creation without specifying exact classes, reducing coupling and improving maintainability. It uses a factory method to determine which class to instantiate. In Spring Boot, this pattern integrates seamlessly through factory beans and dependency injection, providing flexible and condition-based object creation.
Note
Factory Method is especially helpful in modular systems where new functionalities might be added frequently, and we want to minimize the impact of changes to existing code.
"},{"location":"fundamentaldives/DesignPatterns/Iterator/","title":"Iterator","text":"The Iterator pattern is a behavioral design pattern that allows sequential access to elements of a collection without exposing its underlying structure. The goal is to provide a way to access the elements of an aggregate object (such as an array, list, or set) one by one, without needing to understand how the collection is implemented.
"},{"location":"fundamentaldives/DesignPatterns/Iterator/#what","title":"What ?","text":"Iterator is a behavioral design pattern that lets you traverse elements of a collection without exposing its underlying representation (list, stack, tree, etc.).
Key Components
hasNext(), next()).Iterator interface for specific collections.iterator()).ConcurrentModificationException.Let's go through with a simple Java example of a custom iterator for a list of strings.
Defining the Iterator Interfaceinterface Iterator<T> {\n boolean hasNext();\n T next();\n}\nclass NameIterator implements Iterator<String> {\n private String[] names;\n private int index;\n\n public NameIterator(String[] names) {\n this.names = names;\n }\n\n @Override\n public boolean hasNext() {\n return index < names.length;\n }\n\n @Override\n public String next() {\n if (this.hasNext()) {\n return names[index++];\n }\n return null;\n }\n}\ninterface NameCollection {\n Iterator<String> getIterator();\n}\nclass NameRepository implements NameCollection {\n private String[] names = {\"John\", \"Alice\", \"Robert\", \"Michael\"};\n\n @Override\n public Iterator<String> getIterator() {\n return new NameIterator(names);\n }\n}\npublic class IteratorPatternDemo {\n public static void main(String[] args) {\n NameRepository namesRepository = new NameRepository();\n Iterator<String> iterator = namesRepository.getIterator();\n\n while (iterator.hasNext()) {\n String name = iterator.next();\n System.out.println(\"Name: \" + name);\n }\n }\n}\nJava already provides built-in iterators for its collection framework (Iterator, ListIterator, and Spliterator).
import java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\npublic class BuiltInIteratorExample {\n public static void main(String[] args) {\n List<String> names = new ArrayList<>();\n names.add(\"John\");\n names.add(\"Alice\");\n names.add(\"Robert\");\n\n Iterator<String> iterator = names.iterator();\n while (iterator.hasNext()) {\n System.out.println(\"Name: \" + iterator.next());\n }\n }\n}\nSpring Boot doesn\u2019t explicitly use the Iterator pattern as part of its core framework, but it does utilize Iterators internally (e.g., when dealing with ApplicationContext beans or data access layers). You can integrate the Iterator pattern within Spring Boot to handle collections of objects such as configurations, database entities, or APIs.
public class Book {\n private String title;\n\n public Book(String title) {\n this.title = title;\n }\n\n public String getTitle() {\n return title;\n }\n}\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class BookRepository {\n private List<Book> books = new ArrayList<>();\n\n public BookRepository() {\n books.add(new Book(\"Spring in Action\"));\n books.add(new Book(\"Java 8 in Action\"));\n books.add(new Book(\"Microservices Patterns\"));\n }\n\n public Iterator<Book> getIterator() {\n return books.iterator();\n }\n}\nimport org.springframework.stereotype.Service;\nimport java.util.Iterator;\n\n@Service\npublic class BookService {\n private final BookRepository bookRepository = new BookRepository();\n\n public void printBooks() {\n Iterator<Book> iterator = bookRepository.getIterator();\n while (iterator.hasNext()) {\n Book book = iterator.next();\n System.out.println(\"Book: \" + book.getTitle());\n }\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RequestMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\n@RequestMapping(\"/books\")\npublic class BookController {\n\n @Autowired\n private BookService bookService;\n\n @GetMapping\n public void listBooks() {\n bookService.printBooks();\n }\n}\nWhen you access http://localhost:8080/books, it will print the list of books using the iterator after running the application
The Iterator pattern is a powerful way to decouple iteration logic from collections, ensuring a clean separation of concerns. It\u2019s useful in Java projects where complex or multiple ways of traversal are required. When integrated into Spring Boot, the pattern can be applied to iterate over configurations, data models, or APIs efficiently.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/","title":"Prototype","text":""},{"location":"fundamentaldives/DesignPatterns/Prototype/#what","title":"What ?","text":"The Prototype Pattern is a creational design pattern used when the cost of creating a new object is expensive or complicated. Instead of creating new instances from scratch, this pattern suggests cloning existing objects to produce new ones.
The pattern allows cloning or copying existing instances to create new ones, ensuring that new objects are created without going through the expensive or complex instantiation process repeatedly.
Key Characteristics
In Java, the Prototype Pattern is implemented by making the class implement the Cloneable interface and overriding the clone() method from the Object class.
class Address {\n String street;\n String city;\n\n public Address(String street, String city) {\n this.street = street;\n this.city = city;\n }\n\n // Deep Copy\n public Address(Address address) {\n this.street = address.street;\n this.city = address.city;\n }\n\n @Override\n public String toString() {\n return street + \", \" + city;\n }\n}\n\nclass Employee implements Cloneable {\n String name;\n Address address;\n\n public Employee(String name, Address address) {\n this.name = name;\n this.address = address;\n }\n\n // Shallow Copy\n @Override\n protected Object clone() throws CloneNotSupportedException {\n return super.clone();\n }\n\n // Deep Copy\n public Employee deepClone() {\n return new Employee(this.name, new Address(this.address));\n }\n\n @Override\n public String toString() {\n return \"Employee: \" + name + \", Address: \" + address;\n }\n}\n\npublic class PrototypeExample {\n public static void main(String[] args) throws CloneNotSupportedException {\n Employee emp1 = new Employee(\"Alice\", new Address(\"123 Street\", \"New York\"));\n\n // Shallow Clone\n Employee emp2 = (Employee) emp1.clone();\n\n // Deep Clone\n Employee emp3 = emp1.deepClone();\n\n System.out.println(\"Original: \" + emp1);\n System.out.println(\"Shallow Copy: \" + emp2);\n System.out.println(\"Deep Copy: \" + emp3);\n\n // Modify the original object to see the effect on shallow vs deep copy\n emp1.address.street = \"456 Avenue\";\n\n System.out.println(\"After modifying the original object:\");\n System.out.println(\"Original: \" + emp1);\n System.out.println(\"Shallow Copy: \" + emp2);\n System.out.println(\"Deep Copy: \" + emp3);\n }\n}\nOriginal: Employee: Alice, Address: 123 Street, New York\nShallow Copy: Employee: Alice, Address: 123 Street, New York\nDeep Copy: Employee: Alice, Address: 123 Street, New York\n\nAfter modifying the original object:\nOriginal: Employee: Alice, Address: 456 Avenue, New York\nShallow Copy: Employee: Alice, Address: 456 Avenue, New York\nDeep Copy: Employee: Alice, Address: 123 Street, New York\nclone()): Modifying the original affects the copy (since both reference the same Address object).deepClone()): The copy remains unaffected because it contains a completely new Address object.Spring Framework allows defining prototype-scoped beans. Each time you request a bean with the prototype scope, Spring returns a new instance, effectively following the Prototype Pattern.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/#how-to-use-prototype-scope-in-spring-boot","title":"How to Use Prototype Scope in Spring Boot","text":"@Scope annotation to the bean definition.prototype scope to ensure each request gets a new object.import org.springframework.context.annotation.Scope;\nimport org.springframework.stereotype.Component;\n\n@Component\n@Scope(\"prototype\")\npublic class Employee {\n public Employee() {\n System.out.println(\"New Employee instance created.\");\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class EmployeeController {\n\n @Autowired\n private Employee employee1;\n\n @Autowired\n private Employee employee2;\n\n @GetMapping(\"/employees\")\n public String getEmployees() {\n return \"Employee 1: \" + employee1 + \" | Employee 2: \" + employee2;\n }\n}\nWhen you run this application and hit the /employees endpoint, you will see:
New Employee instance created.\nNew Employee instance created.\nThis shows that a new instance is created each time a prototype-scoped bean is injected.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/#summary","title":"Summary","text":"The Prototype Pattern offers an elegant way to clone existing objects, saving the overhead of complex object creation. It fits well when objects are expensive to create or share the same initial configuration. With Java's cloning mechanisms and Spring Boot's prototype scope, it is easy to implement. However, care must be taken when handling deep versus shallow copies, and the pattern should be avoided when objects are inexpensive to create.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/","title":"Singleton","text":""},{"location":"fundamentaldives/DesignPatterns/Singleton/#what","title":"What ?","text":"The Singleton Pattern is a creational design pattern that ensures a class has only one instance and provides a global access point to that instance.
Singleton is useful when exactly one instance of a class is needed across the system, like for logging, configuration, database connection pools, etc.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#when-to-use","title":"When to Use ?","text":"The instance is created when the class is loaded. This is the simplest way, but it doesn\u2019t support lazy loading.
Eager Initialization Implementationpublic class EagerSingleton {\n private static final EagerSingleton INSTANCE = new EagerSingleton();\n\n // Private constructor to prevent instantiation\n private EagerSingleton() {}\n\n public static EagerSingleton getInstance() {\n return INSTANCE;\n }\n}\nWhen to Use Eager
When the instance is required throughout the application, and we are okay with it being created at startup.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#lazy-initialization","title":"Lazy Initialization","text":"The instance is created only when needed (on first access). But this version is not thread-safe.
Lazy Initialization Implementationpublic class LazySingleton {\n private static LazySingleton instance;\n\n private LazySingleton() {}\n\n public static LazySingleton getInstance() {\n if (instance == null) {\n instance = new LazySingleton();\n }\n return instance;\n }\n}\nIssue with Lazy
Lazy Initialization is not suitable for multithreaded environments.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#using-synchronized","title":"Using Synchronized","text":"This solves the issue of thread safety by synchronizing the access method.
Synchronized Implementationpublic class ThreadSafeSingleton {\n private static ThreadSafeSingleton instance;\n\n private ThreadSafeSingleton() {}\n\n public static synchronized ThreadSafeSingleton getInstance() {\n if (instance == null) {\n instance = new ThreadSafeSingleton();\n }\n return instance;\n }\n}\nIssue with Synchronized
Performance overhead due to synchronization.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#double-checked-locking","title":"Double-Checked Locking","text":"This improves the performance by reducing the overhead of synchronized block.
Double-Checked Locking Implementationpublic class DoubleCheckedLockingSingleton {\n private static volatile DoubleCheckedLockingSingleton instance;\n\n private DoubleCheckedLockingSingleton() {}\n\n public static DoubleCheckedLockingSingleton getInstance() {\n if (instance == null) {\n synchronized (DoubleCheckedLockingSingleton.class) {\n if (instance == null) {\n instance = new DoubleCheckedLockingSingleton();\n }\n }\n }\n return instance;\n }\n}\nThis approach leverages static inner classes, which ensures thread safety and lazy loading without synchronization overhead.
Bill Pugh Singleton Implementationpublic class BillPughSingleton {\n private BillPughSingleton() {}\n\n // Static inner class responsible for holding the instance\n private static class SingletonHelper {\n private static final BillPughSingleton INSTANCE = new BillPughSingleton();\n }\n\n public static BillPughSingleton getInstance() {\n return SingletonHelper.INSTANCE;\n }\n}\nBest Practice
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#enum-singleton","title":"Enum Singleton","text":"This approach is the most concise and prevents issues with serialization and reflection attacks.
Enum Singleton Implementationpublic enum EnumSingleton {\n INSTANCE;\n\n public void someMethod() {\n System.out.println(\"Enum Singleton Instance\");\n }\n}\nRecommended
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#in-spring-boot","title":"In Spring Boot","text":"In Spring Boot, Spring\u2019s IoC container (Inversion of Control) makes singleton beans by default. Each bean in Spring is, by default, a singleton. So, you don\u2019t need to explicitly implement the Singleton pattern. Instead, you annotate the class with @Component or @Service, and Spring ensures that only one instance is created and managed.
import org.springframework.stereotype.Component;\n\n@Component\npublic class MySingletonService {\n public void doSomething() {\n System.out.println(\"Singleton service is working\");\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class MyController {\n\n private final MySingletonService singletonService;\n\n @Autowired\n public MyController(MySingletonService singletonService) {\n this.singletonService = singletonService;\n }\n\n @GetMapping(\"/test\")\n public String test() {\n singletonService.doSomething();\n return \"Check logs for Singleton Service\";\n }\n}\nNote
Spring manages lifecycle and thread safety for you, ensuring it behaves like a Singleton without extra code.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#comparison","title":"Comparison","text":"Implementation Thread Safety Lazy Initialization Serialization Safe Ease of Implementation Eager Initialization Yes No No Easy Lazy Initialization No Yes No Easy Thread-safe Singleton (Synchronized) Yes Yes No Moderate Double-Checked Locking Singleton Yes Yes No Moderate Bill Pugh Singleton Yes Yes No Best Practice Enum Singleton Yes Yes Yes Recommended"},{"location":"fundamentaldives/DesignPatterns/Singleton/#potential-issues","title":"Potential Issues","text":"The Singleton Pattern is a powerful tool when used appropriately. However, misuse can lead to tightly coupled code, concurrency issues, and testing difficulties.
Note
If you are working with Spring Boot, rely on Spring\u2019s built-in singleton beans instead of implementing your own singleton logic. Where thread safety, serialization, or distributed behavior is required, choose the appropriate Singleton implementation like Enum Singleton or Bill Pugh Singleton.
By default, a single instance of the bean is created and shared across the entire application (singleton scope). If two or more components use the same bean, they will refer to the same instance. However, if you need a new instance every time a bean is requested, you can change the scope to prototype. But be mindful Spring\u2019s singleton scope simplifies things like caching and state consistency, while prototype beans may introduce complexity.
The Strategy Pattern is a behavioral design pattern that allows you to define a family of algorithms, encapsulate each one, and make them interchangeable. It lets the algorithm vary independently from clients that use it, promoting flexibility, scalability, and separation of concerns.
In simpler terms, It enables selecting a specific algorithm at runtime based on the context, without modifying the client code.
"},{"location":"fundamentaldives/DesignPatterns/Strategy/#when-to-use","title":"When to Use?","text":"if-else or switch statements).Let\u2019s implement a payment system that allows different payment methods using the Strategy Pattern. We will define multiple payment strategies (like Credit Card and PayPal) and switch between them dynamically.
Step-1: Create the Strategy Interface// PaymentStrategy.java\npublic interface PaymentStrategy {\n void pay(int amount);\n}\n// CreditCardStrategy.java\npublic class CreditCardStrategy implements PaymentStrategy {\n private String cardNumber;\n private String name;\n\n public CreditCardStrategy(String cardNumber, String name) {\n this.cardNumber = cardNumber;\n this.name = name;\n }\n\n @Override\n public void pay(int amount) {\n System.out.println(amount + \" paid with credit card.\");\n }\n}\n\n// PayPalStrategy.java\npublic class PayPalStrategy implements PaymentStrategy {\n private String email;\n\n public PayPalStrategy(String email) {\n this.email = email;\n }\n\n @Override\n public void pay(int amount) {\n System.out.println(amount + \" paid using PayPal.\");\n }\n}\n// PaymentContext.java\npublic class PaymentContext {\n private PaymentStrategy strategy;\n\n public PaymentContext(PaymentStrategy strategy) {\n this.strategy = strategy;\n }\n\n public void setPaymentStrategy(PaymentStrategy strategy) {\n this.strategy = strategy;\n }\n\n public void pay(int amount) {\n strategy.pay(amount);\n }\n}\npublic class StrategyPatternDemo {\n public static void main(String[] args) {\n PaymentContext context = new PaymentContext(new CreditCardStrategy(\"1234-5678-9012\", \"John Doe\"));\n context.pay(100);\n\n // Switch strategy at runtime\n context.setPaymentStrategy(new PayPalStrategy(\"john.doe@example.com\"));\n context.pay(200);\n }\n}\nIn a Spring Boot application, the Strategy Pattern can be applied by injecting different strategy implementations using Spring\u2019s dependency injection.
Let's build a simple notification service where the user can choose between sending notifications via Email or SMS.
Create the Strategy Interface// NotificationStrategy.java\npublic interface NotificationStrategy {\n void sendNotification(String message);\n}\n// EmailNotification.java\nimport org.springframework.stereotype.Service;\n\n@Service(\"email\")\npublic class EmailNotification implements NotificationStrategy {\n @Override\n public void sendNotification(String message) {\n System.out.println(\"Sending Email: \" + message);\n }\n}\n\n// SMSNotification.java\nimport org.springframework.stereotype.Service;\n\n@Service(\"sms\")\npublic class SMSNotification implements NotificationStrategy {\n @Override\n public void sendNotification(String message) {\n System.out.println(\"Sending SMS: \" + message);\n }\n}\n// NotificationContext.java\nimport org.springframework.stereotype.Component;\n\n@Component\npublic class NotificationContext {\n\n private final Map<String, NotificationStrategy> strategies;\n\n public NotificationContext(Map<String, NotificationStrategy> strategies) {\n this.strategies = strategies;\n }\n\n public void send(String type, String message) {\n NotificationStrategy strategy = strategies.get(type);\n if (strategy == null) {\n throw new IllegalArgumentException(\"No such notification type\");\n }\n strategy.sendNotification(message);\n }\n}\n// NotificationController.java\nimport org.springframework.web.bind.annotation.*;\n\n@RestController\n@RequestMapping(\"/notify\")\npublic class NotificationController {\n\n private final NotificationContext context;\n\n public NotificationController(NotificationContext context) {\n this.context = context;\n }\n\n @PostMapping(\"/{type}\")\n public void sendNotification(@PathVariable String type, @RequestBody String message) {\n context.send(type, message);\n }\n}\n// Application.java\nimport org.springframework.boot.SpringApplication;\nimport org.springframework.boot.autoconfigure.SpringBootApplication;\n\n@SpringBootApplication\npublic class Application {\n public static void main(String[] args) {\n SpringApplication.run(Application.class, args);\n }\n}\nemail or sms).@Service annotation./notify/email or /notify/sms).Using Enum-based Strategies: If the algorithms are simple and limited, you can use an enum with methods for strategy logic.
public enum Operation {\n ADD {\n @Override\n public int execute(int a, int b) {\n return a + b;\n }\n },\n SUBTRACT {\n @Override\n public int execute(int a, int b) {\n return a - b;\n }\n };\n\n public abstract int execute(int a, int b);\n}\nUsing Java 8 Lambdas: Since Java 8, you can use lambdas to avoid creating multiple strategy classes.
import java.util.function.Consumer;\n\npublic class LambdaStrategyDemo {\n public static void main(String[] args) {\n Consumer<String> emailStrategy = message -> System.out.println(\"Email: \" + message);\n Consumer<String> smsStrategy = message -> System.out.println(\"SMS: \" + message);\n\n emailStrategy.accept(\"Hello via Email!\");\n smsStrategy.accept(\"Hello via SMS!\");\n }\n}\nThe Strategy Pattern is a powerful way to manage dynamic behavior selection in a clean and decoupled way. In a Spring Boot application, you can easily integrate it by using dependency injection. However, it\u2019s essential to use the pattern wisely to avoid unnecessary complexity or overhead. Use it when multiple behaviors or algorithms need to vary independently without modifying client code. Avoid using it if the added complexity is not justified.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/","title":"Concurrency and Parallelism","text":"Both concurrency and parallelism refer to ways a computer performs multiple tasks, but they differ in how the tasks are executed. Let's go through them, along with related concepts like threads, processes, and programs in this article.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#what-is-process","title":"What is Process ?","text":"A process is an instance of a running program (an executable code). Each process runs in isolation and gets its own memory space. eg: Opening a browser or text editor creates a process.
Characteristics
A thread is the smallest unit of execution within a process. Threads run within a process and share the same memory space. eg: A web browser might use multiple threads one for rendering pages, one for handling user input, and another for downloading files.
Characteristics
Concurrency is when multiple tasks make progress within the same time frame, but not necessarily at the same exact moment. Tasks switch back and forth, sharing resources like CPU time.
Analogy
It\u2019s like a chef preparing multiple dishes, working on one dish for a few minutes, switching to another, and then returning to the previous dish.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#concurrency-with-threads","title":"Concurrency with Threads","text":"Note
Concurrency focuses on dealing with multiple tasks by time-sharing resources.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#issues-in-concurrency","title":"Issues in Concurrency","text":"Below are key issues associated with concurrency.
Race ConditionsWhen two or more threads/processes try to access and modify shared data simultaneously, the final result may depend on the sequence of execution, leading to unpredictable outcomes.
Example: Two bank transactions updating the same account balance at the same time might result in lost updates.
How to Mitigate
Use synchronization mechanisms like locks or mutexes to control access to shared resources.
DeadlocksOccurs when two or more threads/processes block each other by holding resources and waiting for resources held by the other. Example: Process A holds Resource 1 and waits for Resource 2, which is held by Process B, and vice versa.
How to Mitigate
Use techniques like resource ordering, deadlock detection, or timeouts to avoid deadlocks.
LivelockIn livelock, threads are constantly changing states to respond to each other but never make actual progress. It\u2019s similar to two people trying to step aside but always stepping in each other\u2019s way. Example: Two threads repeatedly yield control to avoid conflict, but neither progresses.
How to Mitigate
Add randomness or back-off mechanisms to break the cycle.
StarvationA thread or process may be blocked indefinitely because other higher-priority tasks consume all the resources. Example: A low-priority thread never gets CPU time because high-priority threads always take precedence.
How to Mitigate
Use fair scheduling algorithms to ensure all tasks eventually get a chance to execute.
Context Switching OverheadSwitching between multiple threads or processes incurs a cost, as the CPU saves and restores the state of each thread. Excessive context switching can reduce performance. Example: An overloaded web server with too many threads may spend more time switching contexts than doing actual work.
How to Mitigate
Minimize the number of threads and optimize the task scheduling.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#what-is-parallelism","title":"What is Parallelism ?","text":"Parallelism is when multiple tasks are executed simultaneously, usually on different processors or cores.
Analogy
It\u2019s like having multiple chefs, each cooking one dish at the same time.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#parallelism-with-threads","title":"Parallelism with Threads","text":"Note
Parallelism requires multiple CPUs or cores for real simultaneous execution.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#issues-in-parallelism","title":"Issues in Parallelism","text":"Below are key issues associated with parallelism.
Load ImbalanceIf the workload is not evenly distributed across threads or processes, some cores might remain underutilized while others are overloaded. Example: In a matrix multiplication task, if one thread processes a large chunk and another a small chunk, the first thread might take longer, slowing down the whole task.
How to Mitigate
Use dynamic load balancing or work stealing techniques to distribute the workload effectively.
Scalability BottlenecksAs more threads or processes are added, the overhead of synchronization and communication increases, limiting performance improvements. Example: A program may scale well with 4 threads but show diminishing returns with 16 threads due to synchronization overhead.
How to Mitigate
Optimize algorithms for scalability and minimize shared resources to reduce synchronization costs.
False SharingOccurs when multiple threads on different cores modify variables that are close in memory, leading to unnecessary cache invalidations and reduced performance. Example: Two threads updating variables in the same cache line can cause frequent cache synchronization, slowing execution.
How to Mitigate
Align data properly in memory to avoid false sharing.
Communication OverheadIn parallel systems, threads or processes may need to communicate with each other, which adds overhead. Example: In distributed computing, passing messages between nodes can slow down the computation.
How to Mitigate
Reduce communication frequency or use message batching techniques to minimize overhead.
Debugging and Testing ComplexityDebugging concurrent or parallel programs is harder because issues like race conditions or deadlocks may only appear under specific conditions, making them difficult to reproduce. Example: A race condition might only occur when threads execute in a specific order, which is hard to detect in testing.
How to Mitigate
Use debugging tools like thread analyzers and log events to trace execution paths.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#example-scenarios","title":"Example Scenarios","text":"Concurrent Programming (e.g., in Java, Python)
async/await in Python), tasks switch between each other, sharing the same thread.Parallel Programming (e.g., using Python's multiprocessing or CUDA for GPU computation)
Ensuring that shared data remains consistent when accessed by multiple threads or processes is challenging. Example: If multiple threads increment the same counter, the final result may be incorrect without proper synchronization.
How to Mitigate
Use locks, semaphores, or atomic operations to ensure data consistency.
Performance Trade-offsParallel or concurrent execution does not always lead to better performance. In some cases, overhead from synchronization, communication, and context switching can negate performance gains. Example: A parallel algorithm may run slower on a small dataset due to the overhead of managing multiple threads.
How to Mitigate
Assess whether the overhead is justified and use profiling tools to analyze performance.
Non-Deterministic BehaviorIn concurrent and parallel systems, the order of execution is not guaranteed, leading to non-deterministic results. Example: Running the same multi-threaded program twice may produce different outcomes, making testing and debugging difficult.
How to Mitigate
Use locks and barriers carefully, and design programs to tolerate or avoid non-determinism where possible.
Resource ContentionThreads and processes compete for shared resources, such as memory, I/O, and network bandwidth, leading to bottlenecks. Example: Multiple processes writing to the same disk simultaneously may degrade performance.
How to Mitigate
Optimize resource usage and avoid unnecessary contention by reducing shared resources.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#summary","title":"Summary","text":"Concurrency deals with multiple tasks making progress within the same period (may or may not be simultaneous) whereas Parallelism deals with tasks running simultaneously on different cores or processors.
Processes and threads are core to both concurrency and parallelism, with threads sharing memory within a process and processes running independently with isolated memory.
While concurrency and parallelism offer significant benefits, they also come with substantial challenges. Managing issues such as race conditions, deadlocks, false sharing, and debugging complexity requires thoughtful design and appropriate use of synchronization techniques. Additionally, scalability bottlenecks and communication overhead can limit the effectiveness of parallel systems.
To mitigate these issues, some fixes are:
The DRY Principle stands for Don\u2019t Repeat Yourself. It is a fundamental software development principle aimed at reducing repetition of code and logic. The main idea is that duplication introduces potential risks, if you need to update logic in multiple places, you might forget some, leading to bugs and inconsistencies. When applied well, it improves maintainability, scalability, and clarity, yupp something that lot of codebases misses.
\"Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.\"
In other words, the DRY principle encourages developers to write modular, reusable code and avoid duplicating the same functionality in multiple places. It encourages us to minimize redundancy and write code that does one thing well, making our lives (and the lives of those who maintain our code) much easier.
"},{"location":"fundamentaldives/FundamentalPrinciples/DRY/#when-to-use","title":"When to Use ?","text":"Let's cover an example where we apply DRY principle to refactor duplicated logic into reusable.
Without DRY (Code Duplication) Examplepublic class OrderService {\n\n public void placeOrder(int productId, int quantity) {\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n // Logic to place order\n System.out.println(\"Order placed for product: \" + productId);\n }\n\n public void cancelOrder(int orderId) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n // Logic to cancel order\n System.out.println(\"Order cancelled: \" + orderId);\n }\n\n public void updateOrder(int orderId, int quantity) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n // Logic to update order\n System.out.println(\"Order updated with new quantity: \" + quantity);\n }\n}\nplaceOrder, cancelOrder, and updateOrder).We can extract the common logic into a reusable private method to apply the DRY principle.
With DRY (Refactored Code) Examplepublic class OrderService {\n\n public void placeOrder(int productId, int quantity) {\n validateQuantity(quantity);\n // Logic to place order\n System.out.println(\"Order placed for product: \" + productId);\n }\n\n public void cancelOrder(int orderId) {\n validateOrderId(orderId);\n // Logic to cancel order\n System.out.println(\"Order cancelled: \" + orderId);\n }\n\n public void updateOrder(int orderId, int quantity) {\n validateOrderId(orderId);\n validateQuantity(quantity);\n // Logic to update order\n System.out.println(\"Order updated with new quantity: \" + quantity);\n }\n\n // Reusable validation methods\n private void validateOrderId(int orderId) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n }\n\n private void validateQuantity(int quantity) {\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n }\n}\nvalidateOrderId and validateQuantity).The DRY principle ensures that code duplication is minimized for easier maintenance and improved consistency. In our example, we extracted common validation logic to private methods, adhering to DRY and making the code more maintainable. However, always be careful to avoid over-abstraction, as not all code repetition is bad. The goal is to achieve a balance between simplicity and reusability.
"},{"location":"fundamentaldives/FundamentalPrinciples/KISS/","title":"KISS Principle","text":""},{"location":"fundamentaldives/FundamentalPrinciples/KISS/#what","title":"What ?","text":"The KISS Principle stands for \"Keep It Simple, Stupid\". It\u2019s a design principle that emphasizes simplicity, stating that systems and code work best if they are kept simple rather than made unnecessarily complex. The main idea is to avoid over-engineering and unnecessary complications, which can introduce more bugs, make the code harder to maintain, and increase development time.
KISS encourages developers to create code or solutions that are easy to understand, maintain, and modify. The idea is not to use complicated approaches or unnecessary abstractions when a simpler, more straightforward approach will do.
\u201cSimple\u201d here doesn\u2019t mean incomplete or simplistic it means clear, focused, and straightforward.
"},{"location":"fundamentaldives/FundamentalPrinciples/KISS/#when-to-use","title":"When to Use ?","text":"While simplicity is valuable, there are situations where the KISS principle might not apply fully. Over-simplifying can sometimes lead to problems.
Let's Consider a example where we want to check if a number is even or odd.
Non-KISS Complex Codepublic class NumberUtils {\n public static boolean isEven(int number) {\n return isDivisibleByTwo(number);\n }\n\n private static boolean isDivisibleByTwo(int number) {\n if (number % 2 == 0) {\n return true;\n } else {\n return false;\n }\n }\n\n public static void main(String[] args) {\n System.out.println(\"Is 4 even? \" + isEven(4));\n }\n}\nisDivisibleByTwo.public class NumberUtils {\n public static boolean isEven(int number) {\n return number % 2 == 0;\n }\n\n public static void main(String[] args) {\n System.out.println(\"Is 4 even? \" + isEven(4));\n }\n}\nThe KISS principle encourages developers to create simple, maintainable, and readable code, it means avoiding unnecessary complexity. However, KISS must be balanced some projects or scenarios require complexity to meet performance, modularity, or security needs. In the end, applying KISS is about striking the right balance making the solution as simple as possible, but not simpler than necessary.
"},{"location":"fundamentaldives/FundamentalPrinciples/SOLID/","title":"SOLID Principles","text":""},{"location":"fundamentaldives/FundamentalPrinciples/SOLID/#single-responsibility-principle","title":"Single Responsibility Principle","text":"A class should have only one reason to change. This means that each class should focus on a single responsibility or feature.
Violation Example// Violates SRP: User class has multiple responsibilities.\npublic class User {\n private String name;\n private String email;\n\n public void sendEmail(String message) {\n // Sending email logic here...\n System.out.println(\"Email sent to \" + email);\n }\n\n public void saveUser() {\n // Save user to the database\n System.out.println(\"User saved to DB\");\n }\n}\n// Separate responsibilities into different classes.\npublic class User {\n private String name;\n private String email;\n\n // Getters and setters...\n}\n\npublic class UserRepository {\n public void save(User user) {\n System.out.println(\"User saved to DB\");\n }\n}\n\npublic class EmailService {\n public void sendEmail(User user, String message) {\n System.out.println(\"Email sent to \" + user.getEmail());\n }\n}\nSoftware components (classes, functions, etc.) should be open for extension but closed for modification. You shouldn\u2019t modify existing code to add new behavior instead, extend it.
Violation Example// Violates OCP: PaymentProcessor needs to be modified for new payment types.\npublic class PaymentProcessor {\n public void pay(String type) {\n if (type.equals(\"credit\")) {\n System.out.println(\"Processing credit card payment...\");\n } else if (type.equals(\"paypal\")) {\n System.out.println(\"Processing PayPal payment...\");\n }\n }\n}\n// Use an interface for extensibility.\ninterface PaymentMethod {\n void pay();\n}\n\npublic class CreditCardPayment implements PaymentMethod {\n public void pay() {\n System.out.println(\"Processing credit card payment...\");\n }\n}\n\npublic class PayPalPayment implements PaymentMethod {\n public void pay() {\n System.out.println(\"Processing PayPal payment...\");\n }\n}\n\npublic class PaymentProcessor {\n public void processPayment(PaymentMethod paymentMethod) {\n paymentMethod.pay();\n }\n}\nSubclasses should be substitutable for their base class without altering the correctness of the program.
Violation Example// Violates LSP: Square changes the behavior of Rectangle.\nclass Rectangle {\n protected int width, height;\n\n public void setWidth(int width) {\n this.width = width;\n }\n\n public void setHeight(int height) {\n this.height = height;\n }\n\n public int getArea() {\n return width * height;\n }\n}\n\nclass Square extends Rectangle {\n @Override\n public void setWidth(int width) {\n this.width = width;\n this.height = width; // Violates LSP: Unexpected behavior.\n }\n\n @Override\n public void setHeight(int height) {\n this.width = height;\n this.height = height;\n }\n}\n// Use separate classes to maintain correct behavior.\nclass Shape {\n public int getArea() {\n return 0;\n }\n}\n\nclass Rectangle extends Shape {\n protected int width, height;\n\n public Rectangle(int width, int height) {\n this.width = width;\n this.height = height;\n }\n\n @Override\n public int getArea() {\n return width * height;\n }\n}\n\nclass Square extends Shape {\n private int side;\n\n public Square(int side) {\n this.side = side;\n }\n\n @Override\n public int getArea() {\n return side * side;\n }\n}\nA client should not be forced to implement interfaces that it does not use. Instead, smaller, more specific interfaces should be preferred.
Violation Example// Violates ISP: Cat(r) needs to implement unnecessary methods.\ninterface Vehicle {\n void drive();\n void fly();\n}\n\nclass Car implements Vehicle {\n @Override\n public void drive() {\n System.out.println(\"Car is driving...\");\n }\n\n @Override\n public void fly() {\n // Car can't fly! This method is unnecessary.\n throw new UnsupportedOperationException(\"Car can't fly\");\n }\n}\n// Use separate interfaces for each capability.\ninterface Drivable {\n void drive();\n}\n\ninterface Flyable {\n void fly();\n}\n\nclass Car implements Drivable {\n @Override\n public void drive() {\n System.out.println(\"Car is driving...\");\n }\n}\n\nclass Plane implements Drivable, Flyable {\n @Override\n public void drive() {\n System.out.println(\"Plane is taxiing...\");\n }\n\n @Override\n public void fly() {\n System.out.println(\"Plane is flying...\");\n }\n}\nHigh-level modules should not depend on low-level modules. Both should depend on abstractions.
Violation Example// Violates DIP: High-level class depends on a specific implementation.\nclass SQLDatabase {\n public void connect() {\n System.out.println(\"Connected to SQL Database\");\n }\n}\n\nclass Application {\n private SQLDatabase database;\n\n public Application() {\n database = new SQLDatabase(); // Tight coupling to SQLDatabase.\n }\n\n public void start() {\n database.connect();\n }\n}\n// Depend on an abstraction instead of a specific implementation.\ninterface Database {\n void connect();\n}\n\nclass SQLDatabase implements Database {\n public void connect() {\n System.out.println(\"Connected to SQL Database\");\n }\n}\n\nclass NoSQLDatabase implements Database {\n public void connect() {\n System.out.println(\"Connected to NoSQL Database\");\n }\n}\n\nclass Application {\n private Database database;\n\n public Application(Database database) {\n this.database = database;\n }\n\n public void start() {\n database.connect();\n }\n}\nUser, EmailService, UserRepository. Open Closed Open for extension, closed for modification. Modify PaymentProcessor for new methods. Use PaymentMethod interface and extend classes. Liskov Substitution Subtypes should behave like their base type. Square modifies behavior of Rectangle. Separate Square and Rectangle classes. Interface Segregation Use small, specific interfaces. Car implements unnecessary fly() method. Split into Drivable and Flyable interfaces. Dependency Inversion Depend on abstractions, not implementations. App depends on SQLDatabase directly. Use Database interface for loose coupling."},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/","title":"YAGNI Principle","text":""},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/#what","title":"What ?","text":"YAGNI stands for \"You Aren\u2019t Gonna Need It.\" It is one of the core principles of Extreme Programming (XP) and Agile development. The principle advises developers not to add any functionality or code until it is truly needed. Essentially, YAGNI promotes simplicity and avoids speculative development.
Always implement things when you actually need them, never when you just foresee you might need them.
"},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/#why-use","title":"Why Use ?","text":"Lets go through a simple example to illustrate YAGNI in practice.
Without YAGNI Example Adding Unnecessary Functionalitypublic class UserService {\n\n // Current functionality: Retrieve a user by ID\n public User getUserById(int id) {\n // Logic to retrieve user\n return new User(id, \"John Doe\");\n }\n\n // Unnecessary feature: Speculating that we may need a \"deleteUser\" method in the future\n public void deleteUser(int id) {\n // Logic to delete user (unimplemented)\n System.out.println(\"User deleted: \" + id);\n }\n\n // Another unnecessary feature: Thinking we might need email notifications\n public void sendEmailNotification(User user) {\n // Logic to send email (unimplemented)\n System.out.println(\"Email sent to: \" + user.getEmail());\n }\n}\n\nclass User {\n private int id;\n private String name;\n\n public User(int id, String name) {\n this.id = id;\n this.name = name;\n }\n\n public String getEmail() {\n return name.toLowerCase() + \"@example.com\";\n }\n}\ndeleteUser() method.sendEmailNotification() method.public class UserService {\n\n // Only add the necessary method for now\n public User getUserById(int id) {\n // Logic to retrieve user\n return new User(id, \"John Doe\");\n }\n}\n\nclass User {\n private int id;\n private String name;\n\n public User(int id, String name) {\n this.id = id;\n this.name = name;\n }\n}\ndeleteUser() or sendEmailNotification().deleteUser() or notification feature, we can implement it when it becomes necessary.The YAGNI principle encourages developers to focus on delivering only the required features at a given point in time, avoiding speculative development that may never be used. This approach fosters simplicity, maintainability, and efficiency in the codebase. However, it should be applied carefully there are scenarios (like architecture or security) where anticipating needs is necessary, When used properly, YAGNI helps teams build better software, faster, and with fewer headaches down the line.
"},{"location":"langdives/Java/4Pillars/","title":"4 Pillars - What, How, and Why ?","text":""},{"location":"langdives/Java/4Pillars/#encapsulation","title":"Encapsulation","text":"What: Hiding the internal details of an object and only exposing necessary parts through public methods.
Why: It helps in data hiding and ensures controlled access to variables.
How: Use private variables to restrict direct access and provide getters and setters to access and modify the data.
Encapsulation Examplepublic class Person {\n private String name; // Encapsulated field\n\n // Getter\n public String getName() {\n return name;\n }\n\n // Setter\n public void setName(String name) {\n this.name = name;\n }\n}\nWhat: Allows a class (child/subclass) to acquire the properties and behaviors of another class (parent/superclass).
How: Use the extends keyword.
Why: Promotes code reusability and establishes a parent-child relationship.
Inheritance Exampleclass Animal {\n public void sound() {\n System.out.println(\"Animals make sound\");\n }\n}\n\nclass Dog extends Animal {\n @Override\n public void sound() {\n System.out.println(\"Dog barks\");\n }\n}\nWhat: Ability to process objects differently based on their data type or class.
Why: Increases flexibility and supports dynamic method invocation.
Polymorphism Example Method Overloadingclass Calculator {\n public int add(int a, int b) {\n return a + b;\n }\n\n public double add(double a, double b) {\n return a + b;\n }\n}\nclass Animal {\n public void sound() {\n System.out.println(\"Animal makes sound\");\n }\n}\n\nclass Cat extends Animal {\n @Override\n public void sound() {\n System.out.println(\"Cat meows\");\n }\n}\nWhat: Hiding the complex implementation details and only exposing the essential features.
Why: Helps in achieving modularity and loose coupling between components.
How: Use abstract classes and interfaces.
Abstraction Example Abstract Class Exampleabstract class Vehicle {\n abstract void start();\n}\n\nclass Car extends Vehicle {\n @Override\n void start() {\n System.out.println(\"Car starts with a key\");\n }\n}\ninterface Animal {\n void eat();\n}\n\nclass Dog implements Animal {\n @Override\n public void eat() {\n System.out.println(\"Dog eats bones\");\n }\n}\nextends keyword to derive subclasses. Overloading (compile-time) and overriding (runtime). Using abstract classes or interfaces. Key Benefit Data hiding and modular code. Reduces redundancy and promotes code reuse. Flexibility and extensibility of behavior. Promotes loose coupling and modularity. Access Modifiers Requires private, protected, or public. Involves all inheritance-accessible modifiers. Leverages method visibility across class hierarchies. Abstract methods can be protected or public (not private). Real-World Analogy A capsule with medicine inside it hides the internal components. A child inheriting traits from their parents. A shape object behaving differently as circle/square. A remote control exposing buttons without showing internal circuits. Code Dependency Independent within the class. Dependent on parent-child relationship. Involves multiple forms of a single method/class. Can work with unrelated classes sharing common behavior."},{"location":"langdives/Java/AccessModifPPPPP/","title":"Access modifiers","text":""},{"location":"langdives/Java/AccessModifPPPPP/#public","title":"Public","text":"Keyword: public Access: Accessible from anywhere (inside/outside the class, package, or project). Usage: Typically used for classes, methods, and variables that need global access.
public class MyClass {\n public int value = 10;\n public void display() {\n System.out.println(\"Public method\");\n }\n}\nKeyword: private Access: Accessible only within the same class. Usage: Used to hide class fields or methods, following the principle of encapsulation.
public class MyClass {\n private int value = 10; // Not accessible outside this class\n\n private void display() {\n System.out.println(\"Private method\");\n }\n}\nKeyword: protected Access: Accessible within the same package and by subclasses (even if outside the package). Usage: Useful when extending classes across packages.
public class MyClass {\n protected int value = 10;\n\n protected void display() {\n System.out.println(\"Protected method\");\n }\n}\nNote
If accessed by a subclass in a different package, it must be through inheritance (not directly via an instance).
"},{"location":"langdives/Java/AccessModifPPPPP/#package-private","title":"Package-Private","text":"Keyword: No keyword (Default Access) Access: Accessible only within the same package. Usage: Used for classes and members that don\u2019t need to be accessed outside their package.
Package-Private Exampleclass MyClass { // No access modifier, so it's package-private\n int value = 10;\n\n void display() {\n System.out.println(\"Package-private method\");\n }\n}\nNote
Package-private is the default if no modifier is specified.
"},{"location":"langdives/Java/AccessModifPPPPP/#access-summary","title":"Access Summary","text":"Modifier Same Class Same Package Subclass (Different Package) Other Packagespublic \u2705 \u2705 \u2705 \u2705 protected \u2705 \u2705 \u2705 (via inheritance) \u274c (default) \u2705 \u2705 \u274c \u274c private \u2705 \u274c \u274c \u274c"},{"location":"langdives/Java/Collections-JCF/","title":"Java Collections Framework","text":""},{"location":"langdives/Java/Collections-JCF/#categories-of-collections","title":"Categories of Collections","text":"Collections.synchronizedList().A resizable array, fast random access. It's backed by Array, When random access is needed and insertions are rare you can use this.
Operations & Complexities
O(1) O(1) (amortized) O(n) O(n) Thread Safety: Not synchronized, use Collections.synchronizedList() for thread safety.
List<String> list = new ArrayList<>();\nlist.add(\"Apple\");\nlist.get(0); // Fast access\nA Doubly linked list, better at frequent insertions and deletions. It's backed by Doubly Linked List, When insertion/deletion in the middle is frequent you can use this.
Operations & Complexities
O(n) O(1) (at head or tail) O(n)Thread Safety: Not synchronized.
ExampleList<String> list = new LinkedList<>();\nlist.add(\"Banana\");\nlist.addFirst(\"Apple\"); // O(1) insertion at head\nIt's Unordered, no duplicates, backed by Hash Table. You can use this When you need fast lookups and no duplicates.
Operations & Complexities
O(1) (on average) O(n)Thread Safety: Not synchronized, use Collections.synchronizedSet().
Set<String> set = new HashSet<>();\nset.add(\"Cat\");\nset.add(\"Dog\");\nThis maintains insertion order, backed by a Hash Table + Linked List. You can use this when you need order-preserving behavior.
Operations & Complexities: Same as HashSet (O(1) operations) but with slightly higher overhead due to linked list maintenance.
Set<String> set = new LinkedHashSet<>();\nset.add(\"Apple\");\nset.add(\"Banana\");\nA Sorted set, backed by Red-Black Tree, When you need sorted data.
Operations & Complexities
O(log n) O(n)Thread Safety: Not synchronized.
ExampleSet<Integer> set = new TreeSet<>();\nset.add(5);\nset.add(1); // Sorted automatically\nElements are ordered based on their natural order or a custom comparator. It's backed by Binary Heap, When priority-based retrieval is needed.
Operations & Complexities
O(log n) O(1) O(log n) Queue<Integer> queue = new PriorityQueue<>();\nqueue.add(10);\nqueue.add(5); // 5 will be at the top\nResizable-array-backed deque, allows adding/removing from both ends, When you need both stack and queue operations.
Operations & Complexities
O(1) O(n)Deque<String> deque = new ArrayDeque<>();\ndeque.addFirst(\"First\");\ndeque.addLast(\"Last\");\nStores key-value pairs, backed by Hash Table, Fast lookups for key-value pairs.
Operations & Complexities
O(1) (average) O(n)Thread Safety: Not synchronized, use ConcurrentHashMap for thread-safe operations.
Map<String, Integer> map = new HashMap<>();\nmap.put(\"Apple\", 1);\nmap.put(\"Banana\", 2);\nMaintains insertion order, backed by Hash Table + Linked List, When ordering of entries matters.
ExampleMap<String, Integer> map = new LinkedHashMap<>();\nmap.put(\"Apple\", 1);\nmap.put(\"Banana\", 2); // Maintains insertion order\nSorted map, backed by Red-Black Tree, When you need a sorted key-value store.
Operations & Complexities: Get/Put/Remove: O(log n)
Map<Integer, String> map = new TreeMap<>();\nmap.put(3, \"Three\");\nmap.put(1, \"One\"); // Sorted by key\nSynchronized Wrappers: Use Collections.synchronizedList() or Collections.synchronizedSet() to make collections thread-safe.
List<String> list = Collections.synchronizedList(new ArrayList<>());\nConcurrent Collections: Use ConcurrentHashMap, CopyOnWriteArrayList, or BlockingQueue for better thread-safe alternatives.
Garbage collection (GC) in Java is essential to automatic memory management, ensuring that objects no longer needed by an application are reclaimed, and the memory they occupied is freed. This allows Java developers to avoid memory leaks and other resource-management issues.
"},{"location":"langdives/Java/GarbageCollection/#basics","title":"Basics","text":"Heap Memory is divided into several areas, mainly
How GC Works ? Simply GC works by going through phases
Note
ZGC and Shenandoah use advanced algorithms that perform incremental marking, remapping, and concurrent sweeping. They avoid long pauses by offloading most GC work to concurrent threads.
"},{"location":"langdives/Java/GarbageCollection/#garbage-collection-concepts","title":"Garbage Collection Concepts","text":""},{"location":"langdives/Java/GarbageCollection/#generational-collection","title":"Generational Collection","text":"Java heap is divided into:
Young Generation: Divided into Eden Space and Survivor Spaces (S0, S1), New objects are created in Eden. After surviving a GC cycle, they move to a survivor space, and after multiple cycles, to the Old Generation.
Old Generation (Tenured): Contains long-lived objects.
Garbage collection types:
Dynamic Region Management:
ZGC uses regions independently to ensure ultra-low pause times, as each GC thread works on separate regions without blocking others.
Compressed Oops (Ordinary Object Pointers):
-XX:+UseCompressedOops\nJava threads stop only at specific safepoints for GC (or other JVM activities). A safepoint is a checkpoint where all threads must reach before GC can start for eg Executing bytecode that triggers allocation failure, method calls, or back branches in loops.
The JVM injects safepoint polls within running code to ensure threads hit these safepoints regularly, Too many safepoint pauses indicate GC tuning issues or excessive thread blocking.
JVM may delay triggering GC due to waiting for all threads to reach a safepoint, which introduces unpredictable latency. This is critical in low-latency systems like trading applications.
"},{"location":"langdives/Java/GarbageCollection/#stop-the-world-stw","title":"Stop the World (STW)","text":"A Stop-The-World (STW) event occurs when the garbage collector (GC) halts all application threads to perform critical tasks like marking live objects, reclaiming memory, and compacting the heap. These pauses, necessary to prevent heap inconsistencies, impact application performance, especially in latency-sensitive environments.
The duration of STW events depends on heap size, the number of live objects, and the GC algorithm. Traditional collectors like Serial GC and Parallel GC have longer STW pauses, while CMS reduces them with concurrent marking but still requires short pauses for initial and final marking. Modern GCs like G1 GC, ZGC, and Shenandoah GC minimize pauses by performing most work concurrently with application threads, achieving millisecond-range STW durations.
Optimizations include using low-latency collectors, tuning GC settings, reducing allocation pressure, and monitoring GC behavior with tools like JFR or VisualVM. For latency-critical applications, advanced collectors and careful memory management are essential to mitigate the impact of STW events.
"},{"location":"langdives/Java/GarbageCollection/#barriers-tables-fences","title":"Barriers, Tables & Fences","text":""},{"location":"langdives/Java/GarbageCollection/#write-barriers","title":"Write Barriers","text":"New Object Creation
Minor GC (Young Generation Collection)
Max Tenuring Threshold
Promotion Failures
When Old Generation Fills Up
Concurrent Collectors (CMS, G1, ZGC, Shenandoah)
What Causes Full GC?
What Happens During Full GC
Safepoints
Write Barriers
Soft, Weak, and Phantom References
GC Flow
Object Creation
Eden Full \u2192 Trigger Minor GC
Old Gen Full \u2192 Trigger Major GC
Concurrent Collections (G1, ZGC)
Full GC (Stop-the-World)
Fragmentation refers to the inefficient use of memory that occurs when free memory is split into small, non-contiguous blocks, making it difficult to allocate larger contiguous blocks even if the total free memory is sufficient. In Java, fragmentation can occur in both the young and old generations of the heap.
"},{"location":"langdives/Java/GarbageCollection/#types","title":"Types","text":"Internal Fragmentation: Occurs when a block of memory is larger than what is actually needed. For example, if an object requires 10 bytes but is allocated a 16-byte block, the remaining 6 bytes are wasted.
External Fragmentation: Happens when free memory is scattered in small chunks across the heap. This can lead to a situation where there isn\u2019t enough contiguous space available to fulfill a large allocation request, even if the total free memory is sufficient.
"},{"location":"langdives/Java/GarbageCollection/#causes","title":"Causes","text":"Object Lifetimes: Short-lived objects are frequently allocated and deallocated, especially in the young generation. This can create gaps in memory as these objects are collected, leading to external fragmentation.
Promotion of Objects: When objects in the young generation are promoted to the old generation, if the old generation is already fragmented, it may become difficult to allocate new objects.
Full GCs: In collectors like CMS (Concurrent Mark-Sweep), memory is reclaimed but not compacted, leaving fragmented free spaces.
"},{"location":"langdives/Java/GarbageCollection/#effects","title":"Effects","text":"OutOfMemoryError: Fragmentation can cause allocation failures, leading to OutOfMemoryError if there isn\u2019t enough contiguous memory available for new object allocations.
Increased GC Overhead: The JVM may spend more time during GC cycles trying to find suitable spaces for object allocation, which can degrade performance.
Heap Fragmentation: Some collectors (like CMS) suffer from heap fragmentation since they don\u2019t compact memory after reclaiming space.
Pinned Objects: Sometimes, objects cannot be moved during GC (e.g., JNI references or thread-local objects). This can lead to fragmentation.
"},{"location":"langdives/Java/GarbageCollection/#mitigating","title":"Mitigating","text":"Using G1 GC or ZGC: These collectors are designed to handle fragmentation better than older collectors. They manage memory in regions and perform compaction as part of their regular operations.
Heap Size Adjustments: Increasing the size of the old generation can help reduce the frequency of fragmentation issues.
Monitoring and Tuning: Regularly monitor memory usage and GC logs to identify fragmentation patterns. Tuning the JVM parameters can help alleviate fragmentation issues.
Object Pooling: Reusing objects instead of frequently allocating and deallocating them can help reduce fragmentation.
"},{"location":"langdives/Java/GarbageCollection/#configuring-garbage-collection","title":"Configuring garbage collection","text":"Configuring garbage collection and its parameters in Java is primarily done through JVM (Java Virtual Machine) options when starting your application.
"},{"location":"langdives/Java/GarbageCollection/#how-to-configure-params","title":"How to Configure Params","text":"Command-Line Options: You can specify GC options when you start your Java application using the java command.
java -Xms512m -Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -jar my-application.jar\n-Xms512m: Sets the initial heap size to 512 MB.-Xmx4g: Sets the maximum heap size to 4 GB.-XX:+UseG1GC: Uses the G1 Garbage Collector.-XX:MaxGCPauseMillis=100: Aims for a maximum pause time of 100 milliseconds.Environment Variables: For containerized applications (like those running in Docker or Kubernetes), you can set JVM options through environment variables or directly in the configuration file.
Example in DockerENV JAVA_OPTS=\"-Xms512m -Xmx4g -XX:+UseG1GC\"\nCMD java $JAVA_OPTS -jar your-application.jar\nConfiguration Files: Some applications allow you to specify JVM options in a configuration file, which can be helpful for managing multiple parameters in one place.
"},{"location":"langdives/Java/GarbageCollection/#common-gc-options","title":"Common GC Options","text":"Basic Heap Size Configuration
-Xms<size>: Sets the initial heap size.-Xmx<size>: Sets the maximum heap size.Choosing a Garbage Collector
G1 GC-XX:+UseG1GC\n-XX:+UseParallelGC\n-XX:+UseConcMarkSweepGC\n-XX:+UseZGC\n-XX:+UseShenandoahGC\nTuning G1 GC
-XX:MaxGCPauseMillis=<time>: Target maximum pause time.-XX:G1HeapRegionSize=<size>: Size of the regions in G1.-XX:G1ReservePercent=<percentage>: Percentage of heap reserved for G1 to avoid Full GC.Tuning Parallel GC
-XX:ParallelGCThreads=<number>: Number of threads to use for parallel GC.-XX:ConcGCThreads=<number>: Number of concurrent threads during GC.Monitoring and Logging
-Xlog:gc*:file=gc.log: Logs detailed GC information to gc.log.-XX:+PrintGCDetails: Outputs detailed information about each GC event.Controlling Object Promotion
-XX:MaxTenuringThreshold=<N>: Number of times an object can be copied in the young generation before being promoted to the old generation.Metaspace Configuration (for class metadata)
-XX:MetaspaceSize=<size>: Initial size of Metaspace.-XX:MaxMetaspaceSize=<size>: Maximum size of Metaspace.Here\u2019s an example of a command to start a Java application with G1 GC and some tuning parameters
Examplejava -Xms1g -Xmx8g -XX:+UseG1GC \\\n -XX:MaxGCPauseMillis=200 \\\n -XX:G1HeapRegionSize=16m \\\n -XX:InitiatingHeapOccupancyPercent=30 \\\n -XX:ConcGCThreads=2 \\\n -Xlog:gc*:file=gc.log \\\n -jar my-application.jar\nHeap Size and GC Frequency
GC Latency and Response Times
Application Throughput vs Latency
Survivor Space Tuning
-XX:SurvivorRatio=<N>\n-XX:MaxTenuringThreshold=<N>\nTuning G1 GC
-XX:MaxGCPauseMillis=100\n-XX:G1ReservePercent=10 // Extra heap reserved to avoid Full GC\n-XX:G1HeapRegionSize=<N> // Larger region size helps handle humongous objects efficiently\nYoung GC Tuning
-Xlog:gc*:file=gc.log\nTuning Java GC for High Performance
Choose the Right GC:
Analyze and Tune:
Handling Multi-Terabyte Heaps
GC in Cloud and Microservices
-XX:MaxRAMPercentage=75.0\nLatency Monitoring
-Xlog:gc*:file=gc.log\nGC Logs: Capture GC details by adding -Xlog:gc* or -XX:+PrintGCDetails JVM options.
[GC (Allocation Failure) [PSYoungGen: 2048K->512K(2560K)] 4096K->2048K(8192K), 0.0103451 secs]\nVisualVM: A monitoring tool bundled with the JDK for real-time JVM performance monitoring.
Java Flight Recorder (JFR): An advanced profiling tool that collects detailed JVM metrics, including GC data.
JConsole: Visualize JVM statistics and monitor heap usage.
"},{"location":"langdives/Java/GarbageCollection/#diagnosing-troubleshooting","title":"Diagnosing & Troubleshooting","text":"OutOfMemoryError (OOM): Common causes
Heap Dump Analysis:
jmap -dump:format=b,file=heapdump.hprof <pid>\nDetecting Leaks: Look for large, unreachable objects with static references or growing collections (e.g., large HashMap or ArrayList).
Java Flight Recorder (JFR): JFR provides detailed memory profiling without heavy overhead. Collect a recording and analyze it for object lifetimes, GC events, and thread behavior.
"},{"location":"langdives/Java/GarbageCollection/#summary","title":"Summary","text":"Choose the Right GC Collector
Monitor and Analyze
Avoid Full GC
Pre-tuning Advice
Gradle is a modern, powerful build automation tool used for building, testing, and deploying applications. It is particularly popular in Java and Android projects due to its flexibility and performance. Gradle uses a Groovy/Kotlin-based DSL to configure and manage builds, allowing for easy customization.
Gradle organizes builds using a Directed Acyclic Graph (DAG) of tasks, ensuring that tasks are executed only when necessary.
The Build process has Three phases
Initialization Phase
Configuration Phase
Execution Phase
Gradle build configuration is done in the build.gradle file. This file uses Groovy or Kotlin DSL to describe:
build.gradle Example
plugins {\n id 'java' // Apply Java plugin\n id 'application' // Allow running the app from CLI\n}\n\nrepositories {\n mavenCentral() // Use Maven Central for dependencies\n}\n\ndependencies {\n implementation 'org.apache.commons:commons-lang3:3.12.0' // Runtime dependency\n testImplementation 'junit:junit:4.13.2' // Test dependency\n}\n\napplication {\n mainClass = 'com.example.UnderTheHood' // Entry point of the app\n}\nbuild.gradle","text":"Plugins Section
plugins {\n id 'java'\n id 'application'\n}\njava: Adds support for Java projects.application: Allows running applications directly from Gradle with gradle run.maven-publish: Publishes artifacts to Maven repositories.android: Used for Android app development.java plugin sets up the project to be built as a Java application.Repositories Section
repositories {\n mavenCentral()\n}\nmavenCentral() is a popular repository that hosts many open-source libraries.Dependencies Section
dependencies {\n implementation 'org.apache.commons:commons-lang3:3.12.0'\n testImplementation 'junit:junit:4.13.2'\n}\nimplementation: Adds a runtime dependency.testImplementation: Adds a dependency for testing purposes.Application Configuration
application {\n mainClass = 'com.example.UnderTheHood'\n}\ngradle run command.Gradle allows automatic dependency management. Dependencies (like libraries and frameworks) are fetched from repositories such as:
Gradle resolves dependencies in the following order:
~/.gradle/caches/.mavenLocal() is specified (~/.m2/repository/).# Forces Gradle to use only the local cache \n# and does not try to access remote repositories.\ngradle build --offline\nGradle allows developers to create custom tasks to automate specific workflows.
Custom Tasks Example Create Custom Tasktask uth {\n doLast {\n println 'Hello, UnderTheHood ;)'\n }\n}\ngradle uth\nHello, UnderTheHood ;)\nCustom tasks can be chained and made dependent on other tasks:
task compileCode {\n dependsOn clean\n doLast {\n println 'Compiling code...'\n }\n}\nYou can publish your project\u2019s artifacts (e.g., JARs) to Maven Local or Remote repositories using the maven-publish plugin.
Local Maven Publish Example
Apply Maven Publish Pluginplugins {\n id 'maven-publish'\n}\npublishing {\n publications {\n mavenJava(MavenPublication) {\n from components.java\n }\n }\n repositories {\n mavenLocal() // Publish to local Maven repository (~/.m2/repository)\n }\n}\n# This will install the JAR into your local Maven repository.\ngradle publishToMavenLocal\n/my-project\n\u2502\n\u251c\u2500\u2500 build.gradle # Build configuration file\n\u251c\u2500\u2500 settings.gradle # Project settings file\n\u251c\u2500\u2500 src\n\u2502 \u2514\u2500\u2500 main\n\u2502 \u2514\u2500\u2500 java # Source code\n\u2502 \u2514\u2500\u2500 test\n\u2502 \u2514\u2500\u2500 java # Unit tests\n\u2514\u2500\u2500 build # Output directory (JAR, WAR)\nsrc/main/java: Contains the application source code.src/test/java: Contains unit test code.build/: Contains compiled artifacts (like JARs).Gradle supports multi-module projects where different modules are part of the same build.
Example Multi-Project Structure/root-project\n\u2502\n\u251c\u2500\u2500 build.gradle # Root project configuration\n\u251c\u2500\u2500 settings.gradle # Lists sub-projects\n\u251c\u2500\u2500 module-1/\n\u2502 \u2514\u2500\u2500 build.gradle # Configuration for module 1\n\u2514\u2500\u2500 module-2/\n \u2514\u2500\u2500 build.gradle # Configuration for module 2\nrootProject.name = 'multi-project-example'\ninclude 'module-1', 'module-2'\n# This will build all modules in the correct order.\ngradle build\nThe Gradle Wrapper is a feature that allows a project to include a specific Gradle version along with scripts to execute builds. This ensures that anyone working on the project uses the same Gradle version without requiring a manual installation.
The Gradle Wrapper consists of:
gradlew, Windows: gradlew.bat).gradle/wrapper/gradle-wrapper.properties) that specifies the Gradle version to use.gradle-wrapper.jar) that downloads Gradle if needed.Here are some essential Gradle commands for working with projects:
Command Descriptiongradle init Initializes a new Gradle project. gradle build Compiles, tests, and packages the project. gradle run Runs the application (if using the Application plugin). gradle clean Removes the build/ directory for a fresh build. gradle tasks Lists all available tasks. gradle test Runs all tests in the project. gradle publish Publishes artifacts to Maven repositories."},{"location":"langdives/Java/Gradle/#performance-benefits","title":"Performance Benefits","text":"Gradle is designed for speed and efficiency
Gradle provides several advantages for modern projects
Java is a high-level, object-oriented programming language designed for portability, security, and ease of use. It is known for its \"write once, run anywhere\" capability, allowing developers to create software that can run on any device with a Java Virtual Machine (JVM).
"},{"location":"langdives/Java/JDK-JRE-JVM/#architecture","title":"Architecture","text":"The Java architecture is composed of three main components:
"},{"location":"langdives/Java/JDK-JRE-JVM/#jdk","title":"JDK","text":"The JDK Java Development Kit is a comprehensive development environment for building Java applications. It provides all the tools necessary for Java developers to create, compile, and package Java applications.
Components
javac): Translates Java source code (.java extension) into bytecode (.class extension).javadoc, jdb, and jar.The JRE-Java Runtime Environment provides the libraries, Java Virtual Machine (JVM), and other components necessary for running Java applications. It does not include development tools, making it ideal for end-users who only need to run Java applications.
Components
The JVM Java Virtual Machine is an abstract computing machine that enables a computer to run Java programs. It is responsible for interpreting and executing the bytecode generated by the Java compiler.
Functions
.class files containing the bytecode into memory.Hierarchical Structure
JDK (includes javac, JRE, Tools)\n\u2514\u2500\u2500 JRE (includes JVM and libraries)\n \u2514\u2500\u2500 JVM (executes bytecode)\nThe execution of Java code involves several steps, transitioning through the JDK, JRE, and JVM before reaching the machine code that the computer's CPU executes.
Code Writing: Java developers write source code in plain text files using the .java extension. This code defines classes and methods that make up the Java application.
Code Compilation: The developer uses the Java compiler (javac), which is part of the JDK, to compile the .java file. This process translates the human-readable Java code into an intermediate form known as bytecode. The output of this step is a .class file containing the bytecode.
Running the Application: To run the Java application, the developer executes a command using the Java runtime environment (e.g., java ClassName), which triggers the JRE, The JRE includes the JVM, which performs the following steps:
Loading: The JVM locates the specified .class file and loads it into memory.
Linking: The JVM links the bytecode:
Execution: The JVM executes the bytecode:
Machine Code Execution: The machine code generated by the JVM is executed by the host operating system's CPU. This process allows Java applications to be platform-independent, as the same bytecode can run on any system that has a compatible JVM.
"},{"location":"langdives/Java/Java8vs11vs17vs21/","title":"Java 8 vs 11 vs 17 vs 21","text":"A detailed comparison of Java 8, Java 11, Java 17, and Java 21, summarizing the key differences, improvements, and deprecations introduced across these versions:
"},{"location":"langdives/Java/Java8vs11vs17vs21/#java-8-released-march-2014","title":"Java 8 (Released March 2014)","text":"Java 8 is a long-term support (LTS) release, bringing significant new features:
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features","title":"Major Features","text":"NullPointerException.java.time): Replaces the old java.util.Date.Java 11 is also LTS and a significant milestone since it removed many outdated APIs and modularized the runtime.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_1","title":"Major Features","text":"var in lambda).lines(), strip(), repeat(), isBlank() methods.readString(), writeString(), isSameFile() methods.Java 17 is an LTS release, refining many features introduced in Java 9-16 and stabilizing the platform.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_2","title":"Major Features","text":"instanceof: Simplifies type casting.Java 21 is a non-LTS release (though with unofficial support from some vendors). It introduces many experimental and innovative features.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_3","title":"Major Features","text":"List, Set, and Map with defined iteration order.instanceof) No No Yes Yes Virtual Threads (Loom) No No No Yes Garbage Collectors G1 GC ZGC ZGC, Shenandoah Improved ZGC, Shenandoah String Enhancements Basic strip(), repeat() Text Blocks String Templates TLS Version 1.2 1.3 1.3 1.3 Security Manager Available Deprecated Deprecated Removed Nashorn JavaScript Engine Yes Deprecated Removed Removed"},{"location":"langdives/Java/Java8vs11vs17vs21/#summary","title":"Summary","text":"For production systems, upgrading to Java 17 is generally recommended unless your project needs experimental features from Java 21.
"},{"location":"langdives/Java/JavaPassBy/","title":"Is Java Pass By Value or By Reference ?","text":"Java is stricly pass by value but lets go in depth.
"},{"location":"langdives/Java/JavaPassBy/#reference-vs-primitive","title":"Reference vs Primitive","text":"Primitive Types: These are the basic data types in Java (e.g., int, char, boolean). When you pass a primitive type to a method, a copy of the value is made.
Reference Types: These include objects, arrays, and instances of classes. When you pass a reference type to a method, you pass a reference (or pointer) to the actual object in memory, not the object itself.
"},{"location":"langdives/Java/JavaPassBy/#pass-by-value","title":"Pass by Value","text":"Java uses a mechanism called pass by value for method arguments, but it\u2019s important to clarify how this applies to primitive and reference types.
"},{"location":"langdives/Java/JavaPassBy/#primitive-types","title":"Primitive Types","text":"When you pass a primitive type to a method, the method receives a copy of the variable's value. Any changes made to this copy do not affect the original variable.
Examplepublic class PassByValueExample {\n public static void main(String[] args) {\n int num = 10;\n modifyValue(num); // Passing primitive\n System.out.println(num); // Output: 10\n }\n\n public static void modifyValue(int value) {\n value = 20; // Only modifies the copy\n }\n}\nWhen you pass a reference type to a method, the reference itself is passed by value. This means the method receives a copy of the reference to the object. While you can change the object's properties, you cannot change the reference to point to a different object.
Exampleclass MyClass {\n int value;\n\n MyClass(int value) {\n this.value = value;\n }\n}\n\npublic class PassByReferenceExample {\n public static void main(String[] args) {\n MyClass obj = new MyClass(10);\n modifyObject(obj); // Passing reference\n System.out.println(obj.value); // Output: 20\n }\n\n public static void modifyObject(MyClass myObject) {\n myObject.value = 20; // Modifies the object's property\n // myObject = new MyClass(30); // This would not affect the original object reference only changes local myObject.\n }\n}\nWhen a method is called, a new stack frame is created, and local variables (including method parameters) are stored in this stack frame. Objects are stored in the heap, and the reference to these objects is passed to methods. When you modify the object\u2019s state inside the method, it reflects outside the method because both the original reference and the parameter reference point to the same object in memory.
"},{"location":"langdives/Java/JavaPassBy/#scope-and-lifetime","title":"Scope and Lifetime","text":"Scope: Variables defined inside a method (local variables) can only be accessed within that method. Once the method completes, the local variables are destroyed.
Lifetime: Objects in the heap remain as long as they are referenced. When no references exist (e.g., after the method execution, and no references are held), they become eligible for garbage collection.
Java is pass-by-value:
Changes to the object through the reference affect the original object, but reassignment of the reference does not affect the original reference.
"},{"location":"langdives/Java/KeyWordsTerminolgies/","title":"Keywords and Terminolgies","text":""},{"location":"langdives/Java/KeyWordsTerminolgies/#class-object","title":"Class & Object","text":"class: Defines a class.interface: Declares an interface (a contract for classes).object: An instance of a class.new: Instantiates a new object.public / private / protected: Access control modifiers (already discussed). static: Indicates a field or method belongs to the class rather than instances, is shared across all instances of a class.final: Used to declare constants, prevent method overriding, or inheritance, A final method cannot be overridden by subclasses, A final class cannot be inherited.abstract: Used to declare incomplete methods or classes that must be extended. extends: A class inherits another class.implements: A class implements an interface.super: Refers to the parent class\u2019s members.this: Refers to the current instance of the class.if / else / switch: Conditional statements.for / while / do-while: Looping constructs.break / continue: Control loop execution.return: Exits from a method and optionally returns a value.try / catch / finally: Handle exceptions.throw / throws: Raise exceptions.assert: Check assumptions during development.new: Allocates memory for an object.null: A reference that points to no object.synchronized: Used to control access to methods or blocks in a multithreaded environment.volatile: Ensures visibility of changes to variables across threads.transient: Prevents serialization of a field.void: Specifies that a method does not return a value.primitive types: int, float, char, boolean, etc.instanceof: Tests if an object is of a particular type.enum: Declares an enumeration, a special type with predefined constant values.package: Defines a package (namespace) for classes.import: Brings other classes or packages into the current class.default: Provides default implementations in interfaces or switch statements.native: Declares methods implemented in native code (outside Java).strictfp: Ensures consistent floating-point calculations across platforms.const / goto: Reserved keywords (not used in Java).Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers synchronized blocks.
"},{"location":"langdives/Java/Locking-Intrinsic/#what-is-locking","title":"What is Locking?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock.
Java\u2019s synchronized key word is one of the primary ways to control access to shared resources in multithreaded programs. It ensures thread safety by providing mutual exclusion and visibility guarantees. Let's go further into every aspect of synchronized.
How synchronized Works ?
When a method or block is marked as synchronized, the JVM uses a monitor lock (intrinsic lock) for the associated object or class. The monitor is a synchronization construct provided by the JVM.
Two things happen when a thread enters a synchronized block or method:
Intrinsic Lock: Each Java object has an intrinsic lock (also called monitor lock) associated with it, The thread that enters the synchronized block acquires the intrinsic lock. When it leaves the block, it releases the lock, allowing other threads to acquire it.
"},{"location":"langdives/Java/Locking-Intrinsic/#synchronized-methods","title":"Synchronized Methods","text":""},{"location":"langdives/Java/Locking-Intrinsic/#instance-level-locking","title":"Instance-Level Locking","text":"When you synchronize a non-static method, the thread acquires the lock on the instance of the class (the this object).
public synchronized void increment() {\n // Lock acquired on the current instance (this)\n count++;\n}\nincrement() on the same object instance, only one thread will execute the method at a time. class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++;\n }\n\n public synchronized int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nWhy does this work ?
Since both threads are operating on the same Counter object, only one thread at a time can execute the increment() method due to instance-level locking.
A static synchronized method locks on the Class object (i.e., ClassName.class) rather than on an instance. This ensures that all threads calling static methods on the class are synchronized.
public synchronized static void staticIncrement() {\n // Lock acquired on the class object (Counter.class)\n}\nclass Counter {\n private static int count = 0;\n\n public synchronized static void increment() {\n count++;\n }\n\n public synchronized static int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) Counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) Counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + Counter.getCount()); // Output: 2000\n }\n}\nA synchronized block provides more control than a synchronized method. You can choose which object\u2019s intrinsic lock to use, instead of locking the entire method.
public void increment() {\n synchronized (this) { // Locking on the current instance\n count++;\n }\n}\nclass Counter {\n private int count = 0;\n private final Object lock = new Object();\n\n public void increment() {\n synchronized (lock) { // Locking on a custom object\n count++;\n }\n }\n}\npublic void updateBothCounters(Counter counter1, Counter counter2) {\n synchronized (counter1) { // Locking on the first Counter object\n counter1.increment();\n }\n synchronized (counter2) { // Locking on the second Counter object\n counter2.increment();\n }\n}\nEntry to the Monitor
Exit from the Monitor
Bias Locking and Lightweight Locking
Synchronizing unnecessary code slows down the application so use when necessary.
Minimize the scope of synchronization so use synchronized blocks rather than whole methods to reduce contention.
Ensure you synchronize on the same object across threads to avoid incorrect locking behavior.
Nested synchronized blocks can lead to deadlock. Use consistent lock ordering.
Deadlock, occurs if two or more threads are waiting for each other to release locks.
Performance Bottlenecks, overusing synchronization can lead to contention, where threads are constantly waiting to acquire locks.
Livelock, threads keep responding to each other without making progress (e.g., both threads keep yielding the lock to each other).
Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how deadlocks occur, how to prevent them, and practical examples of various techniques to detect and resolve deadlocks. A deadlock is a common concurrency issue in multithreaded programs and can severely impact performance.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#what-is-deadlock","title":"What is Deadlock ?","text":"A deadlock occurs when two or more threads are blocked indefinitely, Each thread is waiting for a lock held by the other, and neither can proceed.
This results in a circular wait, where no thread can release the locks it holds, leading to a deadlock condition.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#how-deadlock-occurs","title":"How Deadlock Occurs ?","text":"Let\u2019s revisit the classic deadlock example.
Deadlock Exampleclass A {\n public synchronized void methodA(B b) {\n System.out.println(Thread.currentThread().getName() + \": Locked A, waiting for B...\");\n try {\n Thread.sleep(50); // Simulate some work\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n b.last(); // Waiting for lock on object B\n }\n\n public synchronized void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside A.last()\");\n }\n}\n\nclass B {\n public synchronized void methodB(A a) {\n System.out.println(Thread.currentThread().getName() + \": Locked B, waiting for A...\");\n try {\n Thread.sleep(50); // Simulate some work\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n a.last(); // Waiting for lock on object A\n }\n\n public synchronized void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside B.last()\");\n }\n}\n\npublic class DeadlockDemo {\n public static void main(String[] args) {\n A a = new A();\n B b = new B();\n\n Thread t1 = new Thread(() -> a.methodA(b), \"Thread 1\");\n Thread t2 = new Thread(() -> b.methodB(a), \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nFlow Analysis
Thread 1 starts and calls a.methodA(b). It acquires the lock on object A and prints:
Thread 1: Locked A, waiting for B...\nThread 2 starts and calls b.methodB(a). It acquires the lock on object B and prints:
Thread 2: Locked B, waiting for A...\nNow:
A and waits for Thread 2 to release the lock on B.B and waits for Thread 1 to release the lock on A.Both threads are waiting indefinitely, resulting in a deadlock.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#how-to-avoid","title":"How to Avoid ?","text":"tryLock() with TimeoutIf all threads acquire locks in the same order, deadlock can be prevented.
Modified Example: Acquiring Locks in the Same Orderclass A {\n public void methodA(B b) {\n synchronized (this) {\n System.out.println(Thread.currentThread().getName() + \": Locked A, waiting for B...\");\n synchronized (b) {\n System.out.println(Thread.currentThread().getName() + \": Acquired lock on B\");\n b.last();\n }\n }\n }\n\n public void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside A.last()\");\n }\n}\n\nclass B {\n public void methodB(A a) {\n synchronized (this) {\n System.out.println(Thread.currentThread().getName() + \": Locked B, waiting for A...\");\n synchronized (a) {\n System.out.println(Thread.currentThread().getName() + \": Acquired lock on A\");\n a.last();\n }\n }\n }\n\n public void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside B.last()\");\n }\n}\n\npublic class DeadlockResolved {\n public static void main(String[] args) {\n A a = new A();\n B b = new B();\n\n Thread t1 = new Thread(() -> a.methodA(b), \"Thread 1\");\n Thread t2 = new Thread(() -> b.methodB(a), \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nExplanation
Both threads now acquire locks in the same order (A \u2192 B). This ensures that deadlock cannot occur.
tryLock() with Timeout","text":"The tryLock() method attempts to acquire a lock and fails gracefully if the lock is not available within a specified time.
tryLock() example import java.util.concurrent.TimeUnit;\nimport java.util.concurrent.locks.ReentrantLock;\n\npublic class TryLockDemo {\n private final ReentrantLock lockA = new ReentrantLock();\n private final ReentrantLock lockB = new ReentrantLock();\n\n public void methodA() {\n try {\n if (lockA.tryLock(1, TimeUnit.SECONDS)) {\n System.out.println(Thread.currentThread().getName() + \": Locked A\");\n Thread.sleep(50); // Simulate some work\n\n if (lockB.tryLock(1, TimeUnit.SECONDS)) {\n try {\n System.out.println(Thread.currentThread().getName() + \": Locked B\");\n } finally {\n lockB.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \": Could not acquire lock B, releasing A\");\n }\n\n lockA.unlock();\n }\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n public void methodB() {\n try {\n if (lockB.tryLock(1, TimeUnit.SECONDS)) {\n System.out.println(Thread.currentThread().getName() + \": Locked B\");\n Thread.sleep(50); // Simulate some work\n\n if (lockA.tryLock(1, TimeUnit.SECONDS)) {\n try {\n System.out.println(Thread.currentThread().getName() + \": Locked A\");\n } finally {\n lockA.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \": Could not acquire lock A, releasing B\");\n }\n\n lockB.unlock();\n }\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n public static void main(String[] args) {\n TryLockDemo demo = new TryLockDemo();\n\n Thread t1 = new Thread(demo::methodA, \"Thread 1\");\n Thread t2 = new Thread(demo::methodB, \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nExplanation
If a thread fails to acquire a lock within the timeout, it releases any locks it holds, avoiding a deadlock.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#detecting-using-monitoring-tools","title":"Detecting Using Monitoring Tools","text":"You can detect deadlocks using tools like:
tryLock() with timeout to avoid indefinite blocking.AtomicInteger or ConcurrentHashMap when possible.Deadlocks are one of the most common and dangerous issues in multithreaded programming.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/","title":"Issues with Locking - LiveLock","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how livelock occur, how to prevent them, and practical examples of various techniques to detect and resolve livelock.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#what-is-livelock","title":"What is Livelock ?","text":"In a livelock, multiple threads remain active but unable to make progress because they keep responding to each other\u2019s actions. Unlike deadlock, where threads are stuck waiting for locks indefinitely, threads in a livelock keep changing their states in response to each other, but they fail to reach a final state or make useful progress.
Key difference from deadlock
In deadlock, threads are blocked waiting for each other, while in livelock, threads are not blocked, but they keep releasing and reacquiring locks or changing states in a way that prevents progress.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#example-of-livelock","title":"Example of Livelock","text":"Consider two people trying to pick up a spoon to eat, but they keep yielding to each other in an attempt to be polite. Neither person can make progress because they\u2019re constantly checking and responding to each other\u2019s actions.
Livelock Exampleclass Spoon {\n private boolean isAvailable = true;\n\n public synchronized boolean pickUp() {\n if (isAvailable) {\n isAvailable = false;\n return true;\n }\n return false;\n }\n\n public synchronized void putDown() {\n isAvailable = true;\n }\n}\n\npublic class LivelockDemo {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n while (!spoon.pickUp()) {\n System.out.println(\"Person 1: Waiting for spoon...\");\n Thread.yield(); // Yield control to other threads\n }\n System.out.println(\"Person 1: Picked up spoon!\");\n });\n\n Thread person2 = new Thread(() -> {\n while (!spoon.pickUp()) {\n System.out.println(\"Person 2: Waiting for spoon...\");\n Thread.yield(); // Yield control to other threads\n }\n System.out.println(\"Person 2: Picked up spoon!\");\n });\n\n person1.start();\n person2.start();\n }\n}\nExplanation
Excessive Yielding or Giving Way: Threads are too polite and keep yielding to each other and In Java, using Thread.yield() repeatedly can lead to livelock.
Conflicting Retrying Logic: Both threads retry in response to each other\u2019s behavior, creating a circular dependency.
Improper Design of Locking Mechanisms, Threads repeatedly release and reacquire locks without making useful progress.
Poor Handling of Shared Resources, When shared resources are locked and unlocked too frequently, threads can repeatedly fail to acquire them.
Using timeouts helps threads avoid indefinite waiting. If a thread cannot acquire the lock within a certain time, it can stop trying or take an alternative path.
UsingtryLock() with Timeout import java.util.concurrent.TimeUnit;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private final ReentrantLock lock = new ReentrantLock();\n\n public boolean pickUp() throws InterruptedException {\n // Try to acquire the lock with a timeout\n return lock.tryLock(1, TimeUnit.SECONDS);\n }\n\n public void putDown() {\n lock.unlock();\n }\n}\n\npublic class LivelockFixed {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n try {\n if (spoon.pickUp()) {\n System.out.println(\"Person 1: Picked up spoon!\");\n spoon.putDown();\n } else {\n System.out.println(\"Person 1: Couldn't get the spoon in time.\");\n }\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread person2 = new Thread(() -> {\n try {\n if (spoon.pickUp()) {\n System.out.println(\"Person 2: Picked up spoon!\");\n spoon.putDown();\n } else {\n System.out.println(\"Person 2: Couldn't get the spoon in time.\");\n }\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n person1.start();\n person2.start();\n }\n}\nWhy it works ?
If a thread fails to acquire the lock within 1 second, it backs off instead of trying indefinitely.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#use-back-off-strategies","title":"Use Back-off Strategies","text":"A back-off strategy makes threads wait for a random amount of time before retrying. This avoids a situation where two threads keep checking the same lock in sync.
Back-off Strategy Exampleimport java.util.Random;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private final ReentrantLock lock = new ReentrantLock();\n private final Random random = new Random();\n\n public boolean tryPickUp() {\n return lock.tryLock();\n }\n\n public void putDown() {\n lock.unlock();\n }\n\n public void backOff() throws InterruptedException {\n Thread.sleep(random.nextInt(100)); // Wait for a random time\n }\n}\n\npublic class LivelockWithBackoff {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n try {\n while (!spoon.tryPickUp()) {\n System.out.println(\"Person 1: Waiting...\");\n spoon.backOff(); // Wait before retrying\n }\n System.out.println(\"Person 1: Picked up spoon!\");\n spoon.putDown();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread person2 = new Thread(() -> {\n try {\n while (!spoon.tryPickUp()) {\n System.out.println(\"Person 2: Waiting...\");\n spoon.backOff(); // Wait before retrying\n }\n System.out.println(\"Person 2: Picked up spoon!\");\n spoon.putDown();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n person1.start();\n person2.start();\n }\n}\nWhy it works ?
The random back-off time prevents threads from retrying in lockstep, avoiding livelock.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#avoid-excessive-yielding","title":"Avoid Excessive Yielding","text":"Frequent use of Thread.yield() can lead to livelock. Instead, use timeouts or back-off strategies to prevent threads from constantly giving way to each other.
Use Condition variables (available with ReentrantLock) to properly coordinate threads waiting on specific conditions.
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private boolean isAvailable = true;\n private final ReentrantLock lock = new ReentrantLock();\n private final Condition spoonAvailable = lock.newCondition();\n\n public void pickUp() throws InterruptedException {\n lock.lock();\n try {\n while (!isAvailable) {\n spoonAvailable.await(); // Wait until spoon is available\n }\n isAvailable = false;\n } finally {\n lock.unlock();\n }\n }\n\n public void putDown() {\n lock.lock();\n try {\n isAvailable = true;\n spoonAvailable.signal(); // Notify waiting thread\n } finally {\n lock.unlock();\n }\n }\n}\nWhy it works ?
Using condition variables ensures that only one thread proceeds when the spoon becomes available, avoiding busy-waiting and yielding.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#best-practices","title":"Best Practices","text":"yield() with smarter coordination mechanisms like conditions.Livelocks can be tricky to detect because threads remain active, but they fail to make meaningful progress. By using timeouts, back-off strategies, condition variables, and proper locking mechanisms.
"},{"location":"langdives/Java/Locking-Issues-Others/","title":"Issues with Locking - Other Issues","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
We will cover key locking issues in Java in this article like race conditions, thread contention, missed signals, nested locks, overuse of locks, and performance impact. Each section contains causes, examples, solutions, and best practices to avoid or mitigate these issues.
"},{"location":"langdives/Java/Locking-Issues-Others/#race-conditions-despite-locking","title":"Race Conditions Despite Locking","text":"CauseA race condition occurs when multiple threads access a shared resource without proper synchronization, leading to inconsistent results based on the timing of thread execution. Even with partial locks, a shared variable may still be accessed inconsistently if not protected properly.
Race Condition Example
class Counter {\n private int count = 0;\n\n public void increment() {\n synchronized (this) {\n count++;\n }\n }\n\n public int getCount() {\n // Not synchronized, potential race condition.\n return count;\n }\n}\nProblem
count to 1.count can be read by Thread 2, Thread 3 increments it again.getCount() is not synchronized, the returned value may skip increments due to improper timing.Always synchronize access to shared variables, even on read operations if other threads can modify the data.
class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++;\n }\n\n public synchronized int getCount() {\n return count;\n }\n}\nUse AtomicInteger if possible for thread-safe increments:
import java.util.concurrent.atomic.AtomicInteger;\n\nclass Counter {\n private final AtomicInteger count = new AtomicInteger(0);\n\n public void increment() {\n count.incrementAndGet();\n }\n\n public int getCount() {\n return count.get();\n }\n}\nWhen multiple threads compete for the same lock, they spend time waiting for the lock to become available, reducing throughput and performance.
Contention Example
class BankAccount {\n private int balance = 100;\n\n public synchronized void withdraw(int amount) {\n balance -= amount;\n }\n\n public synchronized int getBalance() {\n return balance;\n }\n}\nProblem
If multiple threads frequently access the withdraw() method, contention for the lock will occur, degrading performance.
Minimize the Scope of Synchronized Blocks:
public void withdraw(int amount) {\n if (amount <= 0) return;\n\n synchronized (this) {\n balance -= amount;\n }\n}\nUse ReadWriteLock if reads dominate writes:
import java.util.concurrent.locks.ReentrantReadWriteLock;\n\nclass BankAccount {\n private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();\n private int balance = 100;\n\n public void withdraw(int amount) {\n lock.writeLock().lock();\n try {\n balance -= amount;\n } finally {\n lock.writeLock().unlock();\n }\n }\n\n public int getBalance() {\n lock.readLock().lock();\n try {\n return balance;\n } finally {\n lock.readLock().unlock();\n }\n }\n}\nUse lock-free data structures like AtomicInteger or ConcurrentHashMap to reduce contention.
When a thread misses a notify() signal because it was not yet waiting on the lock, a lost wake-up occurs. This results in threads waiting indefinitely for a signal that has already been sent.
Lost Wake-Up Example
public synchronized void produce() throws InterruptedException {\n while (available) {\n wait(); // Missed if notify() was called before reaching here.\n }\n available = true;\n notify();\n}\nAlways Use a while Loop Instead of if, The while loop ensures the thread rechecks the condition after being notified (to avoid spurious wake-ups).
public synchronized void produce() throws InterruptedException {\n while (available) {\n wait();\n }\n available = true;\n notify();\n}\nUse Condition Variables with ReentrantLock for finer control:
private final ReentrantLock lock = new ReentrantLock();\nprivate final Condition condition = lock.newCondition();\nprivate boolean available = false;\n\npublic void produce() throws InterruptedException {\n lock.lock();\n try {\n while (available) {\n condition.await(); // Wait on condition.\n }\n available = true;\n condition.signal();\n } finally {\n lock.unlock();\n }\n}\nUsing multiple locks can cause deadlocks if threads acquire locks in different orders.
Deadlock Example
synchronized (lock1) {\n synchronized (lock2) {\n // Critical section\n }\n}\nProblem
If Thread 1 acquires lock1 and Thread 2 acquires lock2, both threads will wait indefinitely for each other\u2019s lock, resulting in a deadlock.
tryLock() with timeout to avoid blocking indefinitely: if (lock1.tryLock(1, TimeUnit.SECONDS)) {\n if (lock2.tryLock(1, TimeUnit.SECONDS)) {\n try {\n // Critical section\n } finally {\n lock2.unlock();\n }\n }\n lock1.unlock();\n}\nUsing too many locks or locking too frequently can reduce parallelism, resulting in poor scalability, If every method in a class is synchronized, threads will frequently block each other, reducing concurrency and efficiency.
Solution and Best PracticesConcurrentHashMap) instead of traditional synchronized collections: ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();\nLocking adds overhead in the form of: - Context switches between threads. - CPU cache invalidation. - JVM's monitor management for intrinsic locks.
Performance Issues with Synchronized Code
Excessive locking causes contention and frequent context switches, impacting throughput and latency.
Solution and Best PracticesAtomicInteger for simple counters.Use tryLock() for non-blocking operations:
if (lock.tryLock()) {\n try {\n // Critical section\n } finally {\n lock.unlock();\n }\n}\nMinimize the scope of synchronized blocks to reduce contention.
Use fair locks to avoid starvation:
ReentrantLock lock = new ReentrantLock(true); // Fair lock\nUse lock-free data structures when possible (e.g., ConcurrentHashMap).
Monitor and detect deadlocks using tools like VisualVM or JConsole.
Follow consistent lock ordering to prevent deadlocks.
Locking is essential to ensure thread safety, but improper use can lead to issues such as race conditions, deadlocks, livelocks, contention, and performance degradation. Understanding these issues and following best practices will help you write efficient, scalable, and thread-safe code. Using fine-grained locks and concurrent utilities wisely to maximize concurrency while minimizing risks.
"},{"location":"langdives/Java/Locking-Issues-Starvation/","title":"Issues with Locking - Starvation","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how starvation occur, how to prevent them, and practical examples of various techniques to detect and resolve starvation.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#what-is-starvation","title":"What is Starvation ?","text":"Starvation is a condition where low-priority threads are unable to gain access to resources because higher-priority threads or unfair scheduling policies monopolize CPU time or locks. As a result, the low-priority thread starves and never gets the chance to run, even though it is ready to execute.
This issue can manifest in multithreaded programs when locks or resources are continuously granted to specific threads, leaving others waiting indefinitely. It can occur not only due to CPU scheduling but also due to improper locking strategies, unfair algorithms, or resource starvation.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#causes-of-starvation","title":"Causes of Starvation","text":"Unfair Locking Mechanism, When using unfair locks, the lock may always favor threads that request it recently over those that have been waiting longer.
Priority Inversion, High-priority threads monopolize the CPU and low-priority threads rarely get scheduled. This is especially problematic in priority-based scheduling systems.
Thread Contention and Resource Starvation, Too many threads competing for a limited number of resources (like I/O or locks), some threads may always be able to acquire the resources, leaving others waiting indefinitely.
Busy-Waiting Loops, Threads that continuously request resources or repeatedly try to acquire locks without releasing the CPU can cause other threads to starve.
Thread Prioritization in Java, If threads have different priorities, the JVM might schedule higher-priority threads more frequently, leading to starvation of lower-priority threads.
import java.util.concurrent.locks.ReentrantLock;\n\npublic class StarvationDemo {\n private static final ReentrantLock lock = new ReentrantLock(); // Unfair lock\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n }\n }\n };\n\n Thread highPriority = new Thread(task, \"High-Priority\");\n highPriority.setPriority(Thread.MAX_PRIORITY);\n\n Thread lowPriority = new Thread(task, \"Low-Priority\");\n lowPriority.setPriority(Thread.MIN_PRIORITY);\n\n highPriority.start();\n lowPriority.start();\n }\n}\nExplanation
Using fair locks ensures that the longest-waiting thread gets the lock first. This prevents threads from skipping the queue and ensures all threads get a chance to execute.
Fair Lock Exampleimport java.util.concurrent.locks.ReentrantLock;\n\npublic class FairLockDemo {\n private static final ReentrantLock lock = new ReentrantLock(true); // Fair lock\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n }\n }\n };\n\n Thread highPriority = new Thread(task, \"High-Priority\");\n Thread lowPriority = new Thread(task, \"Low-Priority\");\n\n highPriority.setPriority(Thread.MAX_PRIORITY);\n lowPriority.setPriority(Thread.MIN_PRIORITY);\n\n highPriority.start();\n lowPriority.start();\n }\n}\nNote
Although Java allows you to assign priorities to threads, the JVM\u2019s thread scheduler may not always honor them consistently. It\u2019s generally recommended to avoid relying on thread priorities for critical tasks, If you need to control thread scheduling, use fair locks or condition variables instead of thread priorities.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#backoff-strategies","title":"Backoff Strategies","text":"Using backoff strategies (delays) between retries can help reduce contention for resources. This ensures that no thread monopolizes the CPU by continuously attempting to acquire a resource.
Backoff Strategy Exampleimport java.util.concurrent.locks.ReentrantLock;\n\npublic class BackoffDemo {\n private static final ReentrantLock lock = new ReentrantLock();\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n try {\n Thread.sleep((long) (Math.random() * 100)); // Random delay\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }\n }\n };\n\n Thread t1 = new Thread(task, \"Thread-1\");\n Thread t2 = new Thread(task, \"Thread-2\");\n\n t1.start();\n t2.start();\n }\n}\nNote
Random delays ensure that threads do not engage in busy-waiting loops, reducing contention and improving fairness.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#use-thread-pools","title":"Use Thread Pools","text":"When dealing with many concurrent tasks, using a thread pool ensures that threads are fairly scheduled and resources are shared efficiently.
Using ThreadPoolExecutor Exampleimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class ThreadPoolDemo {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(2);\n\n Runnable task = () -> {\n System.out.println(Thread.currentThread().getName() + \" is running\");\n };\n\n for (int i = 0; i < 5; i++) {\n executor.submit(task);\n }\n\n executor.shutdown();\n }\n}\nNote
Using thread pools avoids creating too many threads and ensures fair resource sharing.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#avoid-long-critical-sections","title":"Avoid Long Critical Sections","text":"Instead of relying on priorities or busy-waiting, use Condition objects with ReentrantLock to manage thread coordination efficiently.
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.Lock;\nimport java.util.concurrent.locks.ReentrantLock;\n\npublic class ConditionDemo {\n private static final Lock lock = new ReentrantLock();\n private static final Condition condition = lock.newCondition();\n\n public static void main(String[] args) {\n new Thread(() -> {\n lock.lock();\n try {\n System.out.println(\"Waiting...\");\n condition.await(); // Wait for a signal\n System.out.println(\"Resumed\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n } finally {\n lock.unlock();\n }\n }).start();\n\n new Thread(() -> {\n lock.lock();\n try {\n Thread.sleep(1000);\n condition.signal(); // Signal the waiting thread\n System.out.println(\"Signaled\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n } finally {\n lock.unlock();\n }\n }).start();\n }\n}\nNote
Using conditions helps avoid busy-waiting and ensures efficient thread signaling.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#best-practices","title":"Best Practices","text":"Starvation is a subtle but serious issue in multithreaded programs, particularly when some threads are prioritized over others or when resources are monopolized by specific threads. By using fair locks, thread pools, backoff strategies, and avoiding long critical sections.
"},{"location":"langdives/Java/Locking-Reentrant/","title":"Locking.","text":"Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers reentrant locks.
"},{"location":"langdives/Java/Locking-Reentrant/#what-is-locking","title":"What is Locking ?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock.
ReentrantLock ?","text":"The ReentrantLock class, introduced in Java 5, offers more control over thread synchronization than the synchronized keyword. It allows for advanced locking techniques such as fairness policies, tryLock, and interruptible locks. Let\u2019s explore everything about ReentrantLock, including its use cases, internal mechanisms, and best practices.
ReentrantLock is a concrete class in the java.util.concurrent.locks package that implements the Lock interface.
Note
synchronized keyword, which automatically releases the lock when the block exits.import java.util.concurrent.locks.ReentrantLock;\n\nclass Counter {\n private int count = 0;\n private final ReentrantLock lock = new ReentrantLock();\n\n public void increment() {\n lock.lock(); // Acquire the lock\n try {\n count++;\n } finally {\n lock.unlock(); // Release the lock\n }\n }\n\n public int getCount() {\n lock.lock();\n try {\n return count;\n } finally {\n lock.unlock();\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nLock Acquisition: When a thread calls lock(), it tries to acquire the lock. If the lock is available, the thread proceeds otherwise, it blocks until the lock becomes available.
Reentrancy: A thread that holds the lock can acquire the lock again without blocking. This is useful when a thread *nters a method that also calls another synchronized method or block that requires the same lock.
Fair vs Unfair Locking:
Fair Lock: Threads are granted access in the order they requested the lock. this lock ensures that the longest-waiting thread gets the lock first, the main advantage is this avoids thread starvation and the disadvantage is it may have lower performence due to increased overhead
ReentrantLock lock = new ReentrantLock(true); // Fair lock\nUnfair Lock: Threads may skip the queue if the lock is released, improving performance but reducing fairness the main advantage is better throughout because threads are allowed to \"Jump the queue\" but disadvantage is it can lead to thread starvation where some threads may not get chance to execute.
ReentrantLock lock = new ReentrantLock(); // Unfair lock (default)\nThe tryLock() method attempts to acquire the lock without blocking. It returns true if the lock is acquired, otherwise false.
if (lock.tryLock()) {\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} else {\n System.out.println(\"Could not acquire lock, doing something else...\");\n}\nWhen you want to avoid blocking indefinitely if the lock is not available.
"},{"location":"langdives/Java/Locking-Reentrant/#trylock-with-timeout","title":"tryLock with Timeout","text":"The tryLock(long timeout, TimeUnit unit) method waits for a specific amount of time to acquire the lock.
import java.util.concurrent.TimeUnit;\n\nif (lock.tryLock(1, TimeUnit.SECONDS)) {\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} else {\n System.out.println(\"Could not acquire lock within timeout.\");\n}\nWhen waiting indefinitely is not practical, such as network operations or I/O tasks.
"},{"location":"langdives/Java/Locking-Reentrant/#interruptible-lock-acquisition","title":"Interruptible Lock Acquisition","text":"The lockInterruptibly() method allows a thread to acquire the lock but respond to interrupts while waiting.
try {\n lock.lockInterruptibly(); // Wait for lock, but respond to interrupts\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} catch (InterruptedException e) {\n System.out.println(\"Thread was interrupted.\");\n}\nUse when a thread needs to be interrupted while waiting for a lock.
"},{"location":"langdives/Java/Locking-Reentrant/#behavior","title":"Behavior","text":"A reentrant lock means that the same thread can acquire the lock multiple times without blocking itself. However, the thread must release the lock the same number of times to fully unlock it.
Behavior Exampleclass ReentrantExample {\n private final ReentrantLock lock = new ReentrantLock();\n\n public void outerMethod() {\n lock.lock();\n try {\n System.out.println(\"In outer method\");\n innerMethod();\n } finally {\n lock.unlock();\n }\n }\n\n public void innerMethod() {\n lock.lock();\n try {\n System.out.println(\"In inner method\");\n } finally {\n lock.unlock();\n }\n }\n}\nIn this example, outerMethod calls innerMethod, and both methods acquire the same lock. This works without issues because ReentrantLock allows reentrant locking.
The Condition interface (associated with a ReentrantLock) allows a thread to wait for a condition to be met before proceeding. It provides better control than the traditional wait()/notify().
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass ConditionExample {\n private final ReentrantLock lock = new ReentrantLock();\n private final Condition condition = lock.newCondition();\n private boolean ready = false;\n\n public void awaitCondition() throws InterruptedException {\n lock.lock();\n try {\n while (!ready) {\n condition.await(); // Wait for signal\n }\n System.out.println(\"Condition met, proceeding...\");\n } finally {\n lock.unlock();\n }\n }\n\n public void signalCondition() {\n lock.lock();\n try {\n ready = true;\n condition.signal(); // Signal waiting thread\n } finally {\n lock.unlock();\n }\n }\n}\nReentrantLock has more overhead than synchronized due to fairness policies and explicit lock management, Use synchronized for simple scenarios, use reentrantLock for more complex locking requirements(eg: tryLock, fairness).
Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers read-write locks.
"},{"location":"langdives/Java/Locking-ReentrantReadWrite/#what-is-locking","title":"What is Locking ?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock and ReentrantReadWriteLock.
ReentrantReadWriteLock?","text":"The ReentrantReadWriteLock is a specialized lock from Java\u2019s java.util.concurrent.locks package, designed to improve performance in concurrent systems by separating read and write operations. It provides more efficient locking behavior when the majority of operations are read-only, allowing multiple readers to access the shared resource simultaneously but blocking writers until all reading operations are complete.
A ReentrantReadWriteLock maintains two types of locks:
This separation helps optimize performance for read-heavy workloads.
Exampleimport java.util.concurrent.locks.ReentrantReadWriteLock;\n\nclass SharedResource {\n private int data = 0;\n private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();\n\n public void write(int value) {\n lock.writeLock().lock(); // Acquire write lock\n try {\n data = value;\n System.out.println(\"Data written: \" + value);\n } finally {\n lock.writeLock().unlock(); // Release write lock\n }\n }\n\n public int read() {\n lock.readLock().lock(); // Acquire read lock\n try {\n System.out.println(\"Data read: \" + data);\n return data;\n } finally {\n lock.readLock().unlock(); // Release read lock\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n SharedResource resource = new SharedResource();\n\n // Writer thread\n Thread writer = new Thread(() -> resource.write(42));\n\n // Reader threads\n Thread reader1 = new Thread(() -> resource.read());\n Thread reader2 = new Thread(() -> resource.read());\n\n writer.start();\n reader1.start();\n reader2.start();\n\n writer.join();\n reader1.join();\n reader2.join();\n }\n}\nFair vs Unfair Locking: By default, ReentrantReadWriteLock uses unfair locking for better performance, but they may introduce performance overhead
ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true); // Fair lock\nReentrancy: A thread holding a write lock can acquire the read lock without blocking, This is useful for scenarios where a thread needs to read and modify shared data atomically.
Downgrading: A thread can downgrade a write lock to a read lock without releasing the lock entirely.
lock.writeLock().lock();\ntry {\n // Critical write section\n lock.readLock().lock(); // Downgrade to read lock\n} finally {\n lock.writeLock().unlock(); // Release write lock\n}\nIn scenarios with frequent readers, a writer may starve because readers keep acquiring the read lock, delaying the writer indefinitely.
Solution
Use a fair lock
ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true); // Enable fairness\nFair locks ensure that waiting writers get a chance to execute after the current readers finish.
DeadLockIf a read lock and write lock are acquired in an inconsistent order across multiple threads, it can lead to deadlock.
Deadlock example
Thread 1: Acquire write lock -> Attempt to acquire read lock (blocks)\nThread 2: Acquire read lock -> Attempt to acquire write lock (blocks)\nSolution
If there are frequent write operations, the system behaves similarly to using a normal ReentrantLock, as readers must wait for writers to release the lock.
Solution
Use lock-free data structures (like AtomicReference) or ReadWriteLock only when reads significantly outnumber writes.
If a thread holding the write lock tries to release it before acquiring the read lock, data inconsistencies can occur.
Correct Lock Downgrading Example
lock.writeLock().lock();\ntry {\n // Write critical section\n lock.readLock().lock(); // Downgrade to read lock\n} finally {\n lock.writeLock().unlock(); // Release write lock\n}\n// Perform read operations under the read lock.\nRead-heavy Workloads, when the majority of operations are reads and only a few writes occur.
Improved Performance over ReentrantLock, In scenarios where multiple threads need to read data simultaneously, ReentrantReadWriteLock improves performance by allowing concurrent reads.
When You Need to Separate Read and Write Operations, use it to prevent blocking readers unnecessarily during write operations.
Write-heavy Applications, If writes occur frequently, the write lock will block readers, negating the benefits of the read-write separation.
Simple Synchronization Requirements, If your application doesn\u2019t require complex locking behavior, use synchronized or ReentrantLock instead.
Lock-free Alternatives Available, consider atomic classes or concurrent data structures (e.g., ConcurrentHashMap) if they meet your needs.
Use Fair Locks Only When Necessary, Use unfair locks by default for better performance. Enable fairness only if starvation becomes an issue.
Minimize the Duration of Locks, Avoid holding read or write locks longer than necessary to reduce contention.
Use Lock Downgrading Cautiously, Always acquire the read lock before releasing the write lock to avoid inconsistencies.
Monitor Lock Usage, Use tools like VisualVM or JConsole to monitor thread activity and detect potential contention or deadlocks.
Use Lock-Free Data Structures When Possible, For simple counters or flags, use atomic classes (e.g., AtomicInteger) to avoid locking overhead.
ReentrantReadWriteLock is a powerful tool that allows multiple threads to read concurrently while ensuring exclusive access for writes. However, it is most effective in read-heavy scenarios. Understanding potential issues like write starvation, deadlocks, and performance degradation is essential for using this lock effectively, by following best practices like consistent lock ordering, minimizing lock duration, and monitoring lock usage, you can avoid common pitfalls and maximize the performance benefits of ReentrantReadWriteLock.
synchronized vs ReentrantLock","text":""},{"location":"langdives/Java/LockingIntrinsicReentrant/#differences","title":"Differences","text":"Feature synchronized ReentrantLock Basic Concept Uses intrinsic lock (monitor) on objects. Uses an explicit lock from java.util.concurrent.locks. Lock Acquisition Acquired implicitly when entering a synchronized block or method. Acquired explicitly via lock() method. Release of Lock Automatically released when the thread exits the synchronized block or method. Must be explicitly released via unlock(). Reentrancy Supports reentrancy (same thread can acquire the same lock multiple times). Supports reentrancy just like synchronized. Fairness Unfair by default (no control over thread access order). Can be fair or unfair (configurable with ReentrantLock(true)). Interruptibility Cannot respond to interrupts while waiting for the lock. Supports interruptible locking via lockInterruptibly(). Try Locking Not supported. A thread will block indefinitely if the lock is not available. Supports tryLock() to attempt locking without blocking or with timeout. Condition Variables Uses wait() / notify() / notifyAll() methods on the intrinsic lock. Supports multiple Condition objects for finer-grained wait/notify control. Timeout Support Not supported. If the lock is held by another thread, it will wait indefinitely. Supports timeout locking with tryLock(long timeout, TimeUnit unit). Performance Overhead Low for simple scenarios with little contention. Higher overhead but provides greater control over locking behavior. Fair Locking Option Not supported (always unfair). Fair locking can be enabled with ReentrantLock(true). Use in Non-blocking Operations Not possible. Possible with tryLock() (non-blocking). Flexibility and Control Limited to synchronized methods or blocks. Greater flexibility: lock multiple sections, lock only part of a method, or use multiple conditions. Suitability for Deadlock Avoidance Requires external logic to prevent deadlocks (acquire locks in the same order). Easier to prevent deadlocks using tryLock() and timeouts. Memory Usage No additional memory overhead. Uses the object\u2019s monitor. Requires additional memory for lock objects and lock metadata. Readability and Simplicity Easier to read and maintain (especially for small, simple use cases). More complex code with explicit lock management. Error Handling No need to manage lock release in a finally block. The lock is automatically released. Requires explicit unlock() in finally blocks to avoid deadlocks or memory leaks. Thread Starvation Prone to thread starvation in high contention scenarios. Can prevent starvation using fair lock mode. Recommended Use Case Best for simple synchronization needs where you don\u2019t need advanced control. Recommended for complex concurrency scenarios needing fine-grained locking, fairness, tryLock, or interruptibility."},{"location":"langdives/Java/LockingIntrinsicReentrant/#when-to-use","title":"When to Use ?","text":"Use synchronized Use ReentrantLock When you need simple, block-level or method-level synchronization. When you need advanced control over locking behavior (e.g., tryLock, fairness, or interruptibility). When you want automatic lock release (less error-prone). When you need multiple locks or condition variables. When performance matters in low-contention scenarios (lower overhead). When dealing with high contention and you need fair scheduling to prevent starvation. When you don't need non-blocking operations or timeouts. When you want non-blocking operations using tryLock() or timeout-based locking. When the code needs to be simple and easy to read. When code complexity is acceptable for greater flexibility."},{"location":"langdives/Java/LockingIntrinsicReentrant/#summary","title":"Summary","text":"Both synchronized and ReentrantLock have their own strengths and use cases. Use synchronized for simpler, lower-level concurrency needs, and ReentrantLock when you need more control, fairness, or advanced features like non-blocking locking and condition variables.
In general: - synchronized is easier to use and less error-prone. - ReentrantLock is more powerful and flexible, but with more overhead and complexity.
Apache Maven is a build automation and project management tool primarily for Java projects. It uses XML (pom.xml) to describe the project's structure, dependencies, and build lifecycle. Maven focuses on the \u201cconvention over configuration\u201d principle, meaning it provides a standard way to structure and build projects with minimal configuration.
"},{"location":"langdives/Java/Maven/#how-maven-works","title":"How Maven Works ?","text":"Maven operates using a build lifecycle consisting of pre-defined phases. When you execute a specific phase, all preceding phases are executed as well.
Maven Lifecycle Phases Phase Descriptionvalidate Validates the project structure. compile Compiles the source code. test Runs the unit tests. package Packages the compiled code into a JAR or WAR. verify Verifies the package meets specifications. install Installs the JAR into the local Maven repository. deploy Deploys the artifact to a remote repository. Maven revolves around the POM (Project Object Model), which defines:
We will understand pom.xml in next section more.
pom.xml","text":"The POM (Project Object Model) file is the heart of a Maven project. It defines dependencies, build plugins, and project metadata.
Basic Example of pom.xml
<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 \n http://maven.apache.org/xsd/maven-4.0.0.xsd\">\n <modelVersion>4.0.0</modelVersion>\n\n <groupId>com.example</groupId>\n <artifactId>my-app</artifactId>\n <version>1.0-SNAPSHOT</version>\n <packaging>jar</packaging>\n\n <dependencies>\n <dependency>\n <groupId>org.apache.commons</groupId>\n <artifactId>commons-lang3</artifactId>\n <version>3.12.0</version>\n </dependency>\n </dependencies>\n\n <build>\n <plugins>\n <plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.8.1</version>\n <configuration>\n <source>1.8</source>\n <target>1.8</target>\n </configuration>\n </plugin>\n </plugins>\n </build>\n</project>\nKey Components of pom.xml
com.example).my-app).1.0-SNAPSHOT).maven-compiler-plugin).Maven simplifies dependency management by automatically downloading required libraries from repositories.
Scopes of Dependencies
compile: Available at compile time and runtime (default scope).test: Only used for testing purposes.provided: Available at compile time but not packaged (e.g., Servlet API).runtime: Available at runtime only.Example Dependency Declaration
<dependency>\n <groupId>org.apache.commons</groupId>\n <artifactId>commons-lang3</artifactId>\n <version>3.12.0</version>\n</dependency>\nMaven resolves dependencies from repositories:
~/.m2/repository, stores downloaded artifacts.Adding a Custom Repository
<repositories>\n <repository>\n <id>my-repo</id>\n <url>https://my-repo-url</url>\n </repository>\n</repositories>\nMaven plugins extend its functionality. Plugins can handle tasks such as compiling, testing, or packaging.
Maven Compiler Plugin
<plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.8.1</version>\n <configuration>\n <source>1.8</source>\n <target>1.8</target>\n </configuration>\n</plugin>\n/my-project\n\u2502\n\u251c\u2500\u2500 pom.xml # Project Object Model configuration\n\u251c\u2500\u2500 src\n\u2502 \u2514\u2500\u2500 main\n\u2502 \u2514\u2500\u2500 java # Source code\n\u2502 \u2514\u2500\u2500 test\n\u2502 \u2514\u2500\u2500 java # Unit tests\n\u2514\u2500\u2500 target # Output directory (JAR, WAR)\nsrc/main/java: Contains the application source code.src/test/java: Contains unit test code.target/: Contains compiled artifacts (like JARs).Maven supports multi-module projects, allowing multiple related projects to be managed together.
Directory Structure/parent-project\n\u2502\n\u251c\u2500\u2500 pom.xml (parent)\n\u251c\u2500\u2500 module-1/\n\u2502 \u2514\u2500\u2500 pom.xml\n\u2514\u2500\u2500 module-2/\n \u2514\u2500\u2500 pom.xml\n<modules>\n <module>module-1</module>\n <module>module-2</module>\n</modules>\nmvn install\nSimilar to Gradle, Maven has a wrapper (mvnw) that ensures the project uses a specific Maven version.
mvn -N io.takari:maven:wrapper\n./mvnw clean installmvnw.cmd clean installHere are some common Maven commands
Command Descriptionmvn compile Compiles the source code. mvn test Runs unit tests. mvn package Packages the code into a JAR/WAR. mvn install Installs the artifact to the local repository. mvn deploy Deploys the artifact to a remote repository. mvn clean Cleans the target/ directory. mvn dependency:tree Displays the project's dependency tree."},{"location":"langdives/Java/Maven/#best-practices","title":"Best Practices","text":"Maven is a mature, stable tool that simplifies building and managing Java applications. Its focus on conventions reduces the need for complex configurations, making it ideal for enterprise projects. While Maven may lack some of the flexibility and speed of Gradle, it is widely used in large organizations for its reliability and standardization. For projects requiring strict conventions and extensive dependency management, Maven remains a popular choice.
"},{"location":"langdives/Java/MavenVsGradle/","title":"Maven vs Gradle","text":""},{"location":"langdives/Java/MavenVsGradle/#comparision","title":"Comparision","text":"Aspect Maven Gradle Configuration Style Uses XML (pom.xml). Uses Groovy/Kotlin DSL (build.gradle). Performance Slower, especially for large projects (no build caching). Faster with incremental builds and caching. Flexibility Follows convention over configuration, less customizable. Highly customizable, supports custom build logic. Dependency Management Maven Central and custom repositories. Supports Maven Central, JCenter, Ivy, and custom repositories. Plugin System Pre-built Maven plugins (strict lifecycle integration). More flexible plugins with multiple custom task types. Build Output Produces JARs, WARs, and other artifacts. Produces JARs, WARs, and custom artifacts more easily. Multi-Project Support Good for enterprise projects with structured multi-module builds. Excellent for multi-module projects, especially in complex setups. Integration with CI/CD Easily integrates with Jenkins, GitHub Actions, Bamboo. Same level of integration with Jenkins, CircleCI, GitHub Actions. Use in Android Development Not suitable. Preferred build tool for Android development. Incremental Builds Not supported. Supported, resulting in faster builds. Offline Mode Uses the local Maven repository (~/.m2/repository). Uses a local cache (~/.gradle/caches/) and has offline mode. Version Control of Build Tool Maven Wrapper (mvnw) ensures consistent versions. Gradle Wrapper (gradlew) ensures consistent versions. Preferred Projects Enterprise Java applications with well-defined standards. Android apps, complex and large projects with custom build requirements."},{"location":"langdives/Java/MavenVsGradle/#when-to-use-gradle","title":"When to Use Gradle ?","text":"Use Maven if:
Use Gradle if:
In conclusion, both Maven and Gradle are excellent tools, and the choice depends on the project requirements. For enterprise applications, Maven remains a solid choice. For Android apps, large multi-module projects, or performance-critical builds, Gradle stands out as the preferred option.
"},{"location":"langdives/Java/MemoryModel/","title":"Java Memory Model","text":"Java uses a memory model that divides memory into different areas, primarily the heap and stack.
"},{"location":"langdives/Java/MemoryModel/#heap-memory","title":"Heap Memory","text":"The heap is mainly used for dynamic memory allocation. Objects created using the new keyword are stored in the heap. Coming to it's life time objects in the heap remain in memory until they are no longer referenced and are garbage collected. This means the lifetime of an object is not tied to the scope of a method and Accessing memory in the heap is slower than in the stack due to its dynamic nature and the potential for fragmentation.
Note
-Xms for initial heap size and -Xmx for maximum heap size). new keyword.MyClass obj = new MyClass(); // Heap allocation\nThe stack is mainly used for static memory allocation. It stores method call frames, which contain local variables, method parameters, and return addresses. coming to the lifetime of a variable in the stack is limited to the duration of the method call. Once the method returns, the stack frame is popped off, and the memory is reclaimed and accessing stack memory is faster than heap memory because it follows a Last In, First Out (LIFO) order, allowing for quick allocation and deallocation.
Note
-Xss).Example
public class MemoryExample {\n public static void main(String[] args) {\n int localVar = 10; // Stack memory\n\n MemoryExample obj = new MemoryExample(); // Heap memory\n obj.display(localVar); // Passing parameter, stack memory\n }\n\n public void display(int param) { // Stack memory\n System.out.println(param);\n String message = \"Hello\"; // Heap memory (String object)\n }\n}\nJava has 8 primitive data types that store simple values directly in memory.
Type Size Default Value Range Examplebyte 1 byte (8 bits) 0 -128 to 127 byte b = 100; short 2 bytes (16 bits) 0 -32,768 to 32,767 short s = 30000; int 4 bytes (32 bits) 0 -2^31 to (2^31)-1 int i = 100000; long 8 bytes (64 bits) 0L -2^63 to (2^63)-1 long l = 100000L; float 4 bytes (32 bits) 0.0f ~\u00b13.4E38 (7 decimal digits precision) float f = 3.14f; double 8 bytes (64 bits) 0.0 ~\u00b11.8E308 (15 decimal digits precision) double d = 3.14159; char 2 bytes (16 bits) '\\u0000' Unicode characters (0 to 65,535) char c = 'A'; boolean 1 bit (virtual) false true or false boolean b = true;"},{"location":"langdives/Java/PrimitiveReferenceTypes/#reference-types","title":"Reference Types","text":"Reference types store references (addresses) to objects in memory, unlike primitive types that store values directly.
String: Represents a sequence of characters.
String str = \"Hello, World!\";\nArrays: Collections of elements of the same type.
int[] numbers = {1, 2, 3};\nClasses and Objects: Custom data types representing real-world entities.
class Person {\n String name;\n}\nPerson p = new Person();\nInterfaces: Contracts that classes can implement.
interface Animal {\n void sound();\n}\nEnums: Special classes that define a set of constants.
enum Day {\n MONDAY, TUESDAY, WEDNESDAY;\n}\nWrapper Classes: Used to convert primitive types into objects (auto-boxing/unboxing).
byte Byte short Short int Integer long Long float Float double Double char Character boolean Boolean"},{"location":"langdives/Java/PrimitiveReferenceTypes/#differences","title":"Differences","text":"Aspect Primitive Types Reference Types Storage Store actual values. Store references to objects in memory. Memory Allocation Stored in stack memory. Stored in heap memory. Default Values Zero/false equivalents. null for uninitialized references. Examples int, char, boolean. String, Arrays, Classes, Interfaces, etc."},{"location":"langdives/Java/ReferenceTypesInDepth/","title":"Reference Types In Depth.","text":"Let's deep dive to understand How memory management, object references, and behaviors work in Java, with a focus on String handling and other reference types like arrays, classes, and wrapper objects.
"},{"location":"langdives/Java/ReferenceTypesInDepth/#intro","title":"Intro","text":"String or an array), only the reference (address) is copied, not the actual data. This means multiple references can point to the same object.The String class in Java is a special reference type with some unique behaviors. Strings are immutable once a String object is created, it cannot be changed. Any modification on a String results in the creation of a new object in memory.
String Pool (Interned Strings): A special memory area inside the heap used to store string literals. If a string literal like \"Hello\" is created, Java first checks the string pool to see if it already exists. If it does, it returns the reference from the pool. If not, the string is added to the pool.
String s1 = \"Hello\"; // Stored in the String Pool\nString s2 = \"Hello\"; // s2 points to the same object as s1\n\nSystem.out.println(s1 == s2); // true (same reference)\nHeap Memory: When you use the new keyword, a new String object is always created in the heap memory. Even if the same string already exists in the string pool, the new keyword forces the creation of a separate instance in the heap.
String s1 = new String(\"Hello\"); // creates a new object outside the pool in the heap.\n\nString s2 = \"Hello\"; // Stored in the String Pool\n\nSystem.out.println(s1 == s2); // false (different references)\nnew String(), Java forces the creation of a new object in heap even if the same string exists in the pool."},{"location":"langdives/Java/ReferenceTypesInDepth/#arrays","title":"Arrays","text":"Arrays are reference types, meaning the array variable stores a reference to the memory location where the array data is stored.
Exampleint[] arr1 = {1, 2, 3};\nint[] arr2 = arr1; // arr2 now references the same array as arr1\n\narr2[0] = 10; // Modifies the original array\n\nSystem.out.println(arr1[0]); // Output: 10 (both arr1 and arr2 reference the same array)\nHow Array References Work:
arr1 and arr2 point to the same memory location in the heap.arr2, the change will reflect in arr1 because both refer to the same object.When you create an object using new, the reference variable points to the object in heap memory.
class Person {\n String name;\n}\n\nPerson p1 = new Person();\np1.name = \"Alice\";\n\nPerson p2 = p1; // p2 points to the same object as p1\np2.name = \"Bob\";\n\nSystem.out.println(p1.name); // Output: Bob (both references point to the same object)\nHow References Work with Objects:
p1 and p2 point to the same Person object. Changing the object via p2 affects p1.Wrapper classes (Integer, Double, Boolean, etc.) wrap primitive types into objects. These are reference types, and Java performs autoboxing/unboxing to convert between primitive types and wrapper objects.
Integer num1 = 100;\nInteger num2 = 100;\n\nSystem.out.println(num1 == num2); // true (for values within -128 to 127)\n\nInteger num3 = 200;\nInteger num4 = 200;\n\nSystem.out.println(num3 == num4); // false (new objects for values beyond 127)\nWrapper Caching
Integer objects in the range -128 to 127 for performance.Shallow Copy: Copies only the reference, so both variables refer to the same object.
Exampleint[] original = {1, 2, 3};\nint[] shallowCopy = original; // Points to the same array\n\nshallowCopy[0] = 100;\nSystem.out.println(original[0]); // Output: 100\nDeep Copy: Creates a new object with the same data.
Exampleint[] original = {1, 2, 3};\nint[] deepCopy = original.clone(); // Creates a new array\n\ndeepCopy[0] = 100;\nSystem.out.println(original[0]); // Output: 1\nNullPointerException","text":"When a reference is not initialized, it holds the value null. Accessing a field or method on a null reference throws a NullPointerException.
Person p = null;\nSystem.out.println(p.name); // Throws NullPointerException\nJava uses Garbage Collection to manage memory. When no references point to an object, it becomes eligible for garbage collection.
ExamplePerson p1 = new Person(); // Object created\np1 = null; // Now eligible for garbage collection\nnew String() creates a separate object.Integer values from -128 to 127).The String Pool (also called the intern pool) in Java is implemented using a Hash Table-like data structure internally. Let\u2019s explore the design and behavior behind this structure:
"},{"location":"langdives/Java/ReferenceTypesInDepth/#internals","title":"Internals","text":"Hash Table Concept: The String Pool uses a Hash Map-like structure internally, where the key is the string literal, and the value is a reference to the interned string in the pool. This ensures fast lookups and deduplication of identical strings because hash-based structures allow O(1) average-time complexity for lookup operations.
Behavior of String Pool: If a new string literal is created, the pool Checks if the string already exists (by computing its hash), If the string exists, it returns the existing reference. If not, the string is added to the pool.
Rehashing and Thread Safety: The interned pool is part of the JVM's heap and managed by the Java String class. Since Java 7, the pool is stored in the heap instead of the PermGen space to allow better memory management. The pool structure is thread-safe, meaning multiple threads can safely use the pool.
How the pool works internally
class StringPool {\n private static Map<String, String> pool = new HashMap<>();\n\n public static String intern(String str) {\n if (pool.containsKey(str)) {\n return pool.get(str); // Return existing reference\n } else {\n pool.put(str, str); // Add to the pool\n return str;\n }\n }\n}\nString.intern(), Java interns the string, meaning it adds the string to the pool if it's not already present. String Pool Usage Example public class Main {\n public static void main(String[] args) {\n String s1 = new String(\"Hello\");\n String s2 = s1.intern(); // Adds \"Hello\" to the pool, if not already present\n\n String s3 = \"Hello\"; // Uses the interned string from the pool\n\n System.out.println(s2 == s3); // true (same reference from the pool)\n }\n}\nKey Takeaways
OutOfMemoryError if too many strings were interned.intern() on every string can increase the overhead slightly (because of the hash lookup). It\u2019s only beneficial for reducing memory usage when you expect many repeated strings.The String Pool is implemented using a Hash Table-like data structure, This allows for efficient string reuse through fast lookups and ensures no duplicate literals are created. Strings added via literals or intern() are stored in the pool, with existing references returned on subsequent requests.
Enables functional programming by treating functions as first-class citizens.
(parameters) -> expression or (parameters) -> { statements }List<String> names = Arrays.asList(\"Alice\", \"Bob\", \"Charlie\");\nnames.forEach(name -> System.out.println(name));\nA functional interface is an interface with only one abstract method. This is important because lambda expressions can be used to provide the implementation for these interfaces.
Example Example functional interface@FunctionalInterface // Optional but ensures the interface has only one abstract method.\ninterface MyFunction {\n int apply(int a, int b); // Single abstract method\n}\nNow, when you want to use this interface, you don\u2019t need to create a class and provide an implementation like before. Instead, you can use a lambda expression to quickly provide the logic.
Using Lambda with MyFunctionMyFunction addition = (a, b) -> a + b; // Lambda expression for addition\nSystem.out.println(addition.apply(5, 3)); // Output: 8\n(a, b) -> a + b is the lambda expression.apply(int a, int b) method of the MyFunction interface.A method reference is a shorthand way of writing a lambda when a method already exists that matches the lambda\u2019s purpose. This makes the code more concise and readable.
Example withforEach and Method Reference Consider the following list of names:
List<String> names = Arrays.asList(\"Alice\", \"Bob\", \"Charlie\");\nYou want to print all names using forEach(). You could do it with a lambda like this:
names.forEach(name -> System.out.println(name)); // Lambda expression\nNow, Java provides a shorthand: Method Reference. Since System.out.println() already matches the structure (String) -> void, you can write:
names.forEach(System.out::println); // Method reference\nSystem.out::println is a method reference to the println() method of System.out.name -> System.out.println(name) but is cleaner.Use method references when:
// 1. Static method reference\nFunction<String, Integer> parse = Integer::parseInt;\nSystem.out.println(parse.apply(\"123\")); // Output: 123\n\n// 2. Instance method reference on an arbitrary object\nList<String> words = Arrays.asList(\"one\", \"two\", \"three\");\nwords.sort(String::compareToIgnoreCase); // Sorts case-insensitively\nIntroduced in Java 8 to process collections in a declarative way.
Core Stream Operations
"},{"location":"langdives/Java/StreamsLambdas/#creation","title":"Creation","text":"Stream<Integer> stream = Stream.of(1, 2, 3, 4);\nList<String> list = Arrays.asList(\"A\", \"B\", \"C\");\nStream<String> streamFromList = list.stream();\n// Filters elements based on a predicate.\nList<Integer> evenNumbers = stream.filter(n -> n % 2 == 0).toList();\n// Transforms elements.\nList<Integer> lengths = list.stream().map(String::length).toList();\n// Sorts elements.\nList<Integer> sortedList = stream.sorted().toList();\n// Iterates through elements.\nlist.stream().forEach(System.out::println);\n// Collects elements into a collection.\nList<String> newList = list.stream().filter(s -> s.startsWith(\"A\")).collect(Collectors.toList());\n// Reduces the elements to a single result.\nint sum = Stream.of(1, 2, 3, 4).reduce(0, Integer::sum);\n// Used to process elements in parallel for better performance.\nlist.parallelStream().forEach(System.out::println);\nint sumOfEvens = Stream.of(1, 2, 3, 4, 5, 6)\n .filter(n -> n % 2 == 0)\n .reduce(0, Integer::sum);\nSystem.out.println(sumOfEvens); // Output: 12\nList<String> upperCaseNames = list.stream()\n .map(String::toUpperCase)\n .collect(Collectors.toList());\nMap<Integer, List<String>> groupedByLength = list.stream()\n .collect(Collectors.groupingBy(String::length));\nmap()).Thread pool configuration is critical for optimizing the performance of your applications. Poorly configured thread pools can lead to problems such as CPU starvation, thread contention, memory exhaustion, or poor resource utilization. In this Article, we\u2019ll dive deep into CPU-bound vs I/O-bound tasks, explore how to determine optimal thread pool sizes, and discuss key considerations such as queue types and rejection policies.
"},{"location":"langdives/Java/ThreadPoolTuning/#cpu-vs-io-bound-tasks","title":"CPU vs I/O Bound Tasks","text":"When configuring thread pools, it is essential to classify your tasks as CPU-bound or I/O-bound, as this distinction guides the number of threads your pool should maintain.
"},{"location":"langdives/Java/ThreadPoolTuning/#cpu-bound-tasks","title":"CPU-Bound Tasks","text":"Tasks that perform intensive computations (e.g., mathematical calculations, data processing, encoding), and here limiting factor is the CPU core availability. So its better to avoid context switching overhead by keeping the number of threads close to the available CPU cores.
Optimal Thread Pool Size for CPU-Bound Tasksint coreCount = Runtime.getRuntime().availableProcessors();\nExecutorService cpuBoundPool = Executors.newFixedThreadPool(coreCount);\nNote
If more threads than CPU cores are running, threads will compete for CPU cycles, causing context switching, which adds overhead.
Optimal Threads = Number of Cores\nTasks that spend most of the time waiting for I/O operations (e.g., network, database, file I/O). and here the limiting factor is the time spent waiting on I/O. So it's better to use more threads than the number of cores to ensure that idle CPU cycles are used efficiently while waiting for I/O.
Optimal Thread Pool Size for I/O-Bound Tasksint coreCount = Runtime.getRuntime().availableProcessors();\nint optimalThreads = coreCount * 2 + 1;\nExecutorService ioBoundPool = Executors.newFixedThreadPool(optimalThreads);\nNote
Since the tasks spend significant time waiting for I/O, more threads can be created to make sure the CPU is not idle while other threads wait for input/output operations.
Optimal Threads = Number of Cores * (1 + Wait Time / Compute Time)\nChoosing the right work queue is crucial for memory management and task scheduling. The queue holds tasks waiting to be executed when all threads are busy.
"},{"location":"langdives/Java/ThreadPoolTuning/#unbounded-queue","title":"Unbounded Queue","text":"A queue with no size limit, but if too many tasks are submitted, it can lead to memory exhaustion (out-of-memory errors).
LinkedBlockingQueueBlockingQueue<Runnable> queue = new LinkedBlockingQueue<>();\nSuitable only if you expect tasks to complete quickly and the queue will not grow indefinitely.
"},{"location":"langdives/Java/ThreadPoolTuning/#bounded-queue","title":"Bounded Queue","text":"A queue with a fixed size limit, it prevents unbounded memory usage, and If the queue is full, tasks will be rejected or handled based on a rejection policy.
ArrayBlockingQueueBlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(10);\nIdeal for controlled environments where you want to cap the number of waiting tasks.
"},{"location":"langdives/Java/ThreadPoolTuning/#thread-pool-size-tuning","title":"Thread Pool Size Tuning","text":"For CPU-Bound TasksOptimal Threads = Number of Cores\nOptimal Threads = Number of Cores * (1 + Wait Time / Compute Time)\nIf a thread spends 70% of the time waiting on I/O, and only 30% performing work:
Optimal Threads = 4 * (1 + 0.7 / 0.3) = 12\nWhen the task queue is full and the pool is at its maximum size, the ThreadPoolExecutor must decide what to do with new tasks. You can configure rejection policies to handle these situations.
RejectedExecutionException.new ThreadPoolExecutor.AbortPolicy();\nnew ThreadPoolExecutor.CallerRunsPolicy();\nnew ThreadPoolExecutor.DiscardPolicy();\nnew ThreadPoolExecutor.DiscardOldestPolicy();\nMonitoring thread pools ensures that your configuration is correct and performing well. You can monitor the following metrics:
Key Metrics to Monitor
ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS,\n new ArrayBlockingQueue<>(2));\n\nSystem.out.println(\"Active Threads: \" + executor.getActiveCount());\nSystem.out.println(\"Task Count: \" + executor.getTaskCount());\nSystem.out.println(\"Completed Tasks: \" + executor.getCompletedTaskCount());\nSometimes, you may need to adjust the pool size at runtime to respond to changing workloads.
Example: Adjusting Thread Pool Size DynamicallyThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS,\n new ArrayBlockingQueue<>(10));\n\n// Adjust core and max pool size dynamically\nexecutor.setCorePoolSize(3);\nexecutor.setMaximumPoolSize(6);\nint poolSize = Runtime.getRuntime().availableProcessors();\nExecutorService executor = Executors.newFixedThreadPool(poolSize);\nFor I/O-bound tasks, use more threads than the number of cores.
Use bounded queues to prevent memory issues.
Monitor and tune Continuously monitor thread pools and adjust configuration as needed.
Gracefully shutdown thread pools to prevent resource leaks
executor.shutdown();\ntry {\n if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {\n executor.shutdownNow();\n }\n} catch (InterruptedException e) {\n executor.shutdownNow();\n}\nUse CallerRunsPolicy for tasks that must be executed but are non-critical (to prevent task loss).
A thread pool is a collection of worker threads that are created at the start and reused to perform multiple tasks. When tasks are submitted to the pool, a free thread picks up the task and executes it. If no threads are free, the tasks wait in a queue until one becomes available.
"},{"location":"langdives/Java/ThreadPools/#advantages-of-thread-pooling","title":"Advantages of Thread Pooling","text":"The Executors class provides convenient factory methods to create thread pools:
newFixedThreadPool()newCachedThreadPool()newSingleThreadExecutor()newScheduledThreadPool()For greater control, you can instantiate ThreadPoolExecutor directly.
Creates a pool with a fixed number of threads. When all threads are busy, tasks are placed in a queue and executed as soon as a thread becomes available.
newFixedThreadPool import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class FixedThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(3);\n\n for (int i = 1; i <= 6; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA dynamic thread pool where threads are created as needed. If threads are idle for 60 seconds, they are terminated. If a thread is available, it will be reused for a new task.
newCachedThreadPool import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class CachedThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newCachedThreadPool();\n\n for (int i = 1; i <= 5; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA single-threaded executor that ensures tasks are executed sequentially in the order they are submitted. If the thread dies due to an exception, a new thread is created to replace it.
newSingleThreadExecutor import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class SingleThreadExecutorExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newSingleThreadExecutor();\n\n for (int i = 1; i <= 3; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA scheduled thread pool allows you to schedule tasks to run after a delay or periodically at a fixed rate.
newScheduledThreadPool import java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.TimeUnit;\n\npublic class ScheduledThreadPoolExample {\n public static void main(String[] args) {\n ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);\n\n Runnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\n\n // Schedule task to run after 3 seconds\n scheduler.schedule(task, 3, TimeUnit.SECONDS);\n\n // Schedule task to run repeatedly every 2 seconds\n scheduler.scheduleAtFixedRate(task, 1, 2, TimeUnit.SECONDS);\n\n // Allow the tasks to complete after 10 seconds\n scheduler.schedule(() -> scheduler.shutdown(), 10, TimeUnit.SECONDS);\n }\n}\nThreadPoolExecutor is the core implementation of thread pools in Java. Using it allows you to fine-tune the thread pool\u2019s behavior with more control over the number of threads, queue type, and rejection policy.
ThreadPoolExecutor executor = new ThreadPoolExecutor(\n corePoolSize, // Minimum number of threads\n maximumPoolSize, // Maximum number of threads\n keepAliveTime, // Idle time before a thread is terminated\n timeUnit, // Time unit for keepAliveTime\n workQueue, // Queue to hold waiting tasks\n threadFactory, // Factory to create new threads\n handler // Rejection policy when the queue is full\n);\nimport java.util.concurrent.*;\n\npublic class CustomThreadPoolExecutorExample {\n public static void main(String[] args) {\n ThreadPoolExecutor executor = new ThreadPoolExecutor(\n 2, 4, 30, TimeUnit.SECONDS,\n new LinkedBlockingQueue<>(2), // Task queue with capacity 2\n Executors.defaultThreadFactory(),\n new ThreadPoolExecutor.CallerRunsPolicy() // Rejection policy\n );\n\n // Submit 6 tasks to the pool\n for (int i = 1; i <= 6; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nCommon Rejection Policies in ThreadPoolExecutor
RejectedExecutionException when a task is rejected.The Runnable interface represents a task that can run asynchronously in a thread but does not return any result or throw a checked exception.
@FunctionalInterface\npublic interface Runnable {\n void run();\n}\npublic class RunnableExample {\n public static void main(String[] args) {\n Runnable task = () -> {\n System.out.println(\"Executing task in: \" + Thread.currentThread().getName());\n };\n\n Thread thread = new Thread(task);\n thread.start();\n }\n}\nRunnable when no result is expected from the task.The Callable interface is similar to Runnable, but it can return a result and throw a checked exception.
@FunctionalInterface\npublic interface Callable<V> {\n V call() throws Exception;\n}\nimport java.util.concurrent.Callable;\n\npublic class CallableExample {\n public static void main(String[] args) throws Exception {\n Callable<Integer> task = () -> {\n System.out.println(\"Executing task in: \" + Thread.currentThread().getName());\n return 42;\n };\n\n // Direct call (for demonstration)\n Integer result = task.call();\n System.out.println(\"Task result: \" + result);\n }\n}\nCallable when a result or an exception is expected.A Future represents the result of an asynchronous computation. It provides methods to check if the computation is complete, wait for the result, and cancel the task if necessary.
public interface Future<V> {\n boolean cancel(boolean mayInterruptIfRunning);\n boolean isCancelled();\n boolean isDone();\n V get() throws InterruptedException, ExecutionException;\n V get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException;\n}\nimport java.util.concurrent.*;\n\npublic class FutureExample {\n public static void main(String[] args) throws ExecutionException, InterruptedException {\n ExecutorService executor = Executors.newSingleThreadExecutor();\n\n Callable<Integer> task = () -> {\n Thread.sleep(2000); // Simulate some work\n return 42;\n };\n\n Future<Integer> future = executor.submit(task);\n\n // Do something else while the task executes asynchronously\n System.out.println(\"Task is running...\");\n\n // Wait for the result\n Integer result = future.get();\n System.out.println(\"Task result: \" + result);\n\n executor.shutdown();\n }\n}\nFuture allows you to submit tasks to thread pools and retrieve their results once completed.get(): Blocks until the task completes and returns the result.isDone(): Checks if the task is completed.cancel(): Cancels the task if it's still running.BlockingQueue is a thread-safe queue that blocks the calling thread when:
public interface BlockingQueue<E> extends Queue<E> {\n void put(E e) throws InterruptedException;\n E take() throws InterruptedException;\n // Other methods for timed operations, size, etc.\n}\nimport java.util.concurrent.*;\n\npublic class BlockingQueueExample {\n public static void main(String[] args) {\n BlockingQueue<Integer> queue = new ArrayBlockingQueue<>(2);\n\n // Producer thread\n new Thread(() -> {\n try {\n queue.put(1);\n System.out.println(\"Added 1 to the queue\");\n queue.put(2);\n System.out.println(\"Added 2 to the queue\");\n queue.put(3); // This will block until space is available\n System.out.println(\"Added 3 to the queue\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }).start();\n\n // Consumer thread\n new Thread(() -> {\n try {\n Thread.sleep(1000); // Simulate some delay\n System.out.println(\"Removed from queue: \" + queue.take());\n System.out.println(\"Removed from queue: \" + queue.take());\n System.out.println(\"Removed from queue: \" + queue.take());\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }).start();\n }\n}\nBlockingQueue is commonly used for task queues in thread pools (ThreadPoolExecutor).Types of BlockingQueues
ArrayBlockingQueue: A fixed-size queue.LinkedBlockingQueue: A potentially unbounded queue.PriorityBlockingQueue: A priority-based queue.run() method) Yes (call() method) Use Case Simple background tasks Tasks that need to return a value or throw an exception"},{"location":"langdives/Java/ThreadPools/#how-these-work-together","title":"How These Work Together","text":"Using Runnable in a Thread PoolExecutorService executor = Executors.newFixedThreadPool(2);\nRunnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\nexecutor.execute(task);\nexecutor.shutdown();\nExecutorService executor = Executors.newFixedThreadPool(2);\nCallable<Integer> task = () -> 42;\nFuture<Integer> future = executor.submit(task);\nSystem.out.println(\"Result: \" + future.get());\nexecutor.shutdown();\nBlockingQueue<Runnable> queue = new LinkedBlockingQueue<>(2);\nThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS, queue);\nRunnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\nexecutor.execute(task);\nexecutor.shutdown();\nAtomicity is a fundamental concept in multithreading and concurrency that ensures operations are executed entirely or not at all, with no intermediate states visible to other threads. In Java, atomicity plays a crucial role in maintaining data consistency in concurrent environments.
This Article covers everything about atomic operations, issues with atomicity, atomic classes in Java, and best practices to ensure atomic behavior in your code.
"},{"location":"langdives/Java/Threads-Atomicity/#what-is-atomicity","title":"What is Atomicity ?","text":"In a multithreaded program, atomicity guarantees that operations are executed as a single, indivisible unit. When an operation is atomic, it ensures that:
Without atomic operations, multiple threads could interfere with each other, leading to race conditions and data inconsistencies. For example, if two threads try to increment a shared counter simultaneously, the result may not reflect both increments due to interleaving of operations.
"},{"location":"langdives/Java/Threads-Atomicity/#problems","title":"Problems ?","text":"Non-Atomic Operations on Primitive Data Types Counter Increment Exampleclass Counter {\n private int count = 0;\n\n public void increment() {\n count++; // Not atomic\n }\n\n public int getCount() {\n return count;\n }\n}\nProblem
The statement count++ is not atomic. It consists of three operations
count.count.If two threads execute count++ simultaneously, one increment might be lost due to race conditions.
Java provides several ways to ensure atomicity, including:
synchronized blocks and methods.ReentrantLock.java.util.concurrent.atomic package (recommended for simple atomic operations).Atomic Classes","text":"The java.util.concurrent.atomic package offers classes that support lock-free, thread-safe operations on single variables. These classes rely on low-level atomic operations (like CAS \u2014 Compare-And-Swap) provided by the underlying hardware.
AtomicInteger \u2013 Atomic operations on integers.AtomicLong \u2013 Atomic operations on long values.AtomicBoolean \u2013 Atomic operations on boolean values.AtomicReference<V> \u2013 Atomic operations on reference variables.AtomicStampedReference<V> \u2013 Supports versioned references to prevent ABA problems.AtomicInteger","text":"Example: Solving the Increment Problem import java.util.concurrent.atomic.AtomicInteger;\n\nclass AtomicCounter {\n private final AtomicInteger count = new AtomicInteger(0);\n\n public void increment() {\n count.incrementAndGet(); // Atomic increment\n }\n\n public int getCount() {\n return count.get();\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n AtomicCounter counter = new AtomicCounter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) {\n counter.increment();\n }\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) {\n counter.increment();\n }\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nincrementAndGet() method ensures atomicity without using locks.synchronized or ReentrantLock.AtomicBoolean","text":"Example: Managing Flags Safely import java.util.concurrent.atomic.AtomicBoolean;\n\nclass FlagManager {\n private final AtomicBoolean isActive = new AtomicBoolean(false);\n\n public void activate() {\n if (isActive.compareAndSet(false, true)) {\n System.out.println(\"Flag activated.\");\n } else {\n System.out.println(\"Flag already active.\");\n }\n }\n\n public void deactivate() {\n if (isActive.compareAndSet(true, false)) {\n System.out.println(\"Flag deactivated.\");\n } else {\n System.out.println(\"Flag already inactive.\");\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n FlagManager manager = new FlagManager();\n\n Thread t1 = new Thread(manager::activate);\n Thread t2 = new Thread(manager::activate);\n\n t1.start();\n t2.start();\n }\n}\ncompareAndSet() changes the flag only if it matches the expected value, ensuring thread safety.
AtomicReference","text":"Example: Atomic Operations on Objects import java.util.concurrent.atomic.AtomicReference;\n\nclass Person {\n String name;\n\n Person(String name) {\n this.name = name;\n }\n}\n\npublic class AtomicReferenceExample {\n public static void main(String[] args) {\n AtomicReference<Person> personRef = new AtomicReference<>(new Person(\"Alice\"));\n\n // Atomic update of the reference\n personRef.set(new Person(\"Bob\"));\n System.out.println(\"Updated Person: \" + personRef.get().name);\n }\n}\nWhen to Use ?
Use AtomicReference when you need atomic operations on object references.
AtomicStampedReference","text":"The ABA problem occurs when a value changes from A to B and then back to A. AtomicStampedReference solves this by associating a version (stamp) with the value.
import java.util.concurrent.atomic.AtomicStampedReference;\n\npublic class AtomicStampedReferenceExample {\n public static void main(String[] args) {\n AtomicStampedReference<Integer> ref = new AtomicStampedReference<>(1, 0);\n\n int[] stamp = new int[1];\n Integer value = ref.get(stamp);\n System.out.println(\"Initial Value: \" + value + \", Stamp: \" + stamp[0]);\n\n boolean success = ref.compareAndSet(1, 2, stamp[0], stamp[0] + 1);\n System.out.println(\"CAS Success: \" + success + \", New Value: \" + ref.get(stamp) + \", New Stamp: \" + stamp[0]);\n }\n}\nAtomicStampedReference ensures that the same value change does not go undetected by tracking the version.
Use AtomicInteger or AtomicBoolean instead of synchronized methods like simple counters or flags.
Use AtomicReference for lock-free algorithms (Non blocking Algos).
Use AtomicStampedReference for versioned updates to prevent ABA problems.
Atomic classes only work with single variables. For multiple variables, use synchronized or ReentrantLock.
For complex operations involving multiple state changes, atomic classes are insufficient.
Atomic operations avoid deadlocks but can still suffer from livelocks (threads continuously retrying without progress).
synchronized or ReentrantLock.The atomic classes in Java\u2019s java.util.concurrent.atomic package offer lock-free, thread-safe operations that are ideal for simple state management. By ensuring atomicity, these classes help avoid race conditions and improve the performance and scalability of multithreaded applications. However, they are best suited for single-variable updates for more complex operations, locks or transactional mechanisms may still be necessary.
Java offers multithreading to perform multiple tasks concurrently, improving performance and responsiveness. This deep dive covers every key concept of Java threading with detailed explanations and code examples.
Before that let's have a quick rewind of fundamental concept of concurrency and parallelism.
"},{"location":"langdives/Java/Threads/#concurrency-and-parallelism","title":"Concurrency and Parallelism","text":"Concurrency: Multiple tasks start, run, and complete in overlapping time periods (not necessarily simultaneously).
Parallelism: Multiple tasks run exactly at the same time (requires multi-core processors).
We have another article where we gone through fundamentals of concurrency and parallelism in depth though we cover some of the stuff here to but its recommeneded to go through this artice Concurrency and Parallelism
Java achieves both using threads, thread pools, and various libraries such as Executors, Fork/Join Framework, and Streams API, We will go through them one by one and in this article we mostly cover Threads.
"},{"location":"langdives/Java/Threads/#what-is-a-thread","title":"What is a Thread?","text":"A thread is a lightweight sub-process. A Java program has at least one thread \u2014 the main thread, which starts with the main() method. You can create additional threads to execute code concurrently. Each thread shares the same process memory, but has its own stack, registers, and program counter.
You can create a thread in two ways:
Extending the Thread classclass MyThread extends Thread {\n public void run() {\n System.out.println(\"Thread running: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n MyThread t1 = new MyThread();\n t1.start(); // Start the thread\n }\n}\nclass MyRunnable implements Runnable {\n public void run() {\n System.out.println(\"Runnable running: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n Thread t1 = new Thread(new MyRunnable());\n t1.start(); // Start the thread\n }\n}\nWhen to Use ?
Runnable: When you need to inherit from another class, use Runnable since Java does not support multiple inheritance.Thread: If your class does not need to extend any other class, extending Thread might be more intuitive.A thread in Java goes through the following states:
Thread Lifecycle
class MyThread extends Thread {\n public void run() {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n MyThread t1 = new MyThread(); // NEW\n t1.start(); // RUNNABLE\n\n // Join to wait for the thread to complete\n t1.join(); // Terminated once finished\n System.out.println(\"Thread has terminated.\");\n }\n}\nNote
wait() or waits indefinitely.sleep() or join(time) is invoked, meaning the thread will resume after a specific time.A daemon thread is a background thread that provides support services, like the garbage collector. It does not prevent the JVM from shutting down once all user threads are completed.
Example of Daemon Threadclass DaemonThread extends Thread {\n public void run() {\n if (Thread.currentThread().isDaemon()) {\n System.out.println(\"This is a daemon thread.\");\n } else {\n System.out.println(\"This is a user thread.\");\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n DaemonThread t1 = new DaemonThread();\n t1.setDaemon(true); // Set as daemon thread\n t1.start();\n\n DaemonThread t2 = new DaemonThread();\n t2.start();\n }\n}\nWhen to use Daemon Threads ?
For background tasks like logging, garbage collection, or monitoring services.
"},{"location":"langdives/Java/Threads/#thread-priority","title":"Thread Priority","text":"Java assigns a priority to each thread, ranging from 1 (MIN_PRIORITY) to 10 (MAX_PRIORITY). The default priority is 5 (NORM_PRIORITY). Thread priority affects scheduling, but it\u2019s platform-dependent \u2014 meaning it doesn\u2019t guarantee execution order.
Setting Thread Priority Exampleclass PriorityThread extends Thread {\n public void run() {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName() +\n \" with priority: \" + Thread.currentThread().getPriority());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n PriorityThread t1 = new PriorityThread();\n PriorityThread t2 = new PriorityThread();\n\n t1.setPriority(Thread.MIN_PRIORITY); // Priority 1\n t2.setPriority(Thread.MAX_PRIORITY); // Priority 10\n\n t1.start();\n t2.start();\n }\n}\nWhen to use Priority Threads
Only when certain tasks should have preferential scheduling over others. However, Java thread scheduling is not guaranteed, so don't rely solely on priority.
"},{"location":"langdives/Java/Threads/#thread-synchronization","title":"Thread Synchronization","text":"When multiple threads access shared resources (like variables), synchronization ensures that only one thread modifies the resource at a time. Use the synchronized keyword to prevent race conditions.
class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++; // Only one thread can increment at a time\n }\n\n public int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final count: \" + counter.getCount());\n }\n}\nWhen to use Synchronization ?
When multiple threads access critical sections of code to avoid inconsistent data.
"},{"location":"langdives/Java/Threads/#inter-thread-communication","title":"Inter-thread Communication","text":"Java allows threads to communicate using wait-notify methods, avoiding busy waiting.
wait(): Makes the thread wait until another thread notifies it.notify(): Wakes up a single waiting thread.notifyAll(): Wakes up all waiting threads.class SharedResource {\n private int value;\n private boolean available = false;\n\n public synchronized void produce(int val) throws InterruptedException {\n while (available) {\n wait(); // Wait if value is already available\n }\n value = val;\n available = true;\n System.out.println(\"Produced: \" + value);\n notify(); // Notify the consumer thread\n }\n\n public synchronized void consume() throws InterruptedException {\n while (!available) {\n wait(); // Wait if value is not available\n }\n System.out.println(\"Consumed: \" + value);\n available = false;\n notify(); // Notify the producer thread\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n SharedResource resource = new SharedResource();\n\n Thread producer = new Thread(() -> {\n try {\n for (int i = 1; i <= 5; i++) resource.produce(i);\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread consumer = new Thread(() -> {\n try {\n for (int i = 1; i <= 5; i++) resource.consume();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n producer.start();\n consumer.start();\n }\n}\nThreadLocal provides a way to create thread-isolated variables. Each thread gets its own copy of the variable, and changes made by one thread do not affect others. This is useful when you don\u2019t want threads to share a common state.
public class ThreadLocalExample {\n private static ThreadLocal<Integer> threadLocal = ThreadLocal.withInitial(() -> 1);\n\n public static void main(String[] args) {\n Thread t1 = new Thread(() -> {\n threadLocal.set(100);\n System.out.println(\"Thread 1: \" + threadLocal.get());\n });\n\n Thread t2 = new Thread(() -> {\n threadLocal.set(200);\n System.out.println(\"Thread 2: \" + threadLocal.get());\n });\n\n t1.start();\n t2.start();\n }\n}\nWhen to use ?
Useful in multi-threaded environments (like database transactions) where each thread needs its own context without interference from other threads.
"},{"location":"langdives/Java/Threads/#volatile-variables","title":"Volatile Variables","text":"The volatile keyword ensures visibility of changes to variables across threads. Without volatile, thread-local caches may not reflect the latest changes made by other threads.
public class VolatileExample {\n private static volatile boolean running = true;\n\n public static void main(String[] args) {\n Thread t = new Thread(() -> {\n while (running) {\n // Busy-wait\n }\n System.out.println(\"Thread stopped.\");\n });\n\n t.start();\n\n try { Thread.sleep(1000); } catch (InterruptedException e) { }\n running = false; // Change will be visible to other threads\n }\n}\nWhen to Use Volatile
Use volatile for variables accessed by multiple threads without needing synchronization (e.g., flags).
"},{"location":"langdives/Java/Threads/#when-to-use-volatile","title":"When to Usevolatile ?","text":"volatile is Necessary class VolatileExample {\n private volatile boolean running = true;\n\n public void stop() {\n running = false; // Change becomes immediately visible to other threads\n }\n\n public void run() {\n while (running) {\n // Do something\n }\n System.out.println(\"Thread stopped.\");\n }\n\n public static void main(String[] args) throws InterruptedException {\n VolatileExample example = new VolatileExample();\n\n Thread t = new Thread(example::run);\n t.start();\n\n Thread.sleep(1000);\n example.stop(); // Stop the thread\n }\n}\nHere, volatile ensures that the change to running made by the stop() method is immediately visible to the thread executing run(). Without volatile, the run() thread might never see the change and keep running indefinitely.
volatile ?","text":"count++).Atomic classes).class Counter {\n private volatile int count = 0;\n\n public void increment() {\n count++; // Not atomic! Two threads can still read the same value.\n }\n\n public int getCount() {\n return count;\n }\n}\nIssue
Even though count is marked volatile, count++ is not atomic. Two threads could read the same value and increment it, leading to lost updates. To fix this, use synchronized or AtomicInteger.
synchronized or volatile ?","text":"If you don\u2019t use volatile or synchronized, some dangerous scenarios can occur. Like this:
class SharedResource {\n private boolean available = false;\n\n public void produce() {\n available = true; // Change not guaranteed to be visible immediately\n }\n\n public void consume() {\n while (!available) {\n // Busy-waiting, might never see the change to `available`\n }\n System.out.println(\"Consumed!\");\n }\n}\nProblem
If available is not marked volatile, the change made by produce() might not be visible to the consume() thread immediately. The consumer thread might be stuck in an infinite loop because it doesn't see the latest value of available.
Note
volatile or synchronized is used.Synchronized over volatile ?","text":"Let's go through an example where its okay to use just synchroinized instead of volatile
Examplepublic synchronized void produce(int val) throws InterruptedException {\n while (available) {\n wait(); // Wait if value is already available\n }\n value = val;\n available = true;\n System.out.println(\"Produced: \" + value);\n notify(); // Notify the consumer thread\n}\nSynchronized Keyword:
synchronized block or method (like produce() or consume()), it acquires a lock on the SharedResource object. This ensures mutual exclusion \u2014 only one thread can access the critical section at a time.synchronized block. This ensures that the latest value of available is visible to all other threads.Wait-Notify Mechanism:
wait() and notify() methods release and acquire the lock, enforcing happens-before relationships (meaning operations before notify() or wait() are visible to other threads).available = true in produce(), another thread waiting in consume() will see the updated value when it wakes up.Because this code uses synchronized methods and wait-notify, the necessary memory visibility is achieved without needing volatile.
volatile synchronized Visibility Ensures visibility of changes. Ensures visibility and atomicity. Atomicity Not guaranteed. Guaranteed (only one thread at a time). Performance Faster (no locking). Slower (locking involved). Use Case For flags, simple state updates. For complex operations, critical sections. Overhead Low (no blocking). High (involves blocking and context switches)."},{"location":"langdives/Java/Threads/#thread-memory","title":"Thread Memory","text":"The memory consumption per thread and the maximum number of threads in Java depend on several factors, such as:
Each Java thread consumes two key areas of memory:
Thread Stack Memory: Each thread gets its own stack, which holds Local variables (primitives, references), Method call frames, Intermediate results during method execution.
Note
The default stack size depends on the JVM and platform:
You can change the stack size with the -Xss JVM option:
java -Xss512k YourProgram\nNative Thread Metadata: In addition to stack memory, the OS kernel allocates metadata per thread (for thread control structures). This varies by platform but is typically in the range of 8 KB to 16 KB per thread.
"},{"location":"langdives/Java/Threads/#memory-per-thread","title":"Memory per Thread ?","text":"The typical memory consumption per thread:
Thus, a single thread could use ~1 MB to 1.1 MB of memory.
"},{"location":"langdives/Java/Threads/#max-threads-you-can-create","title":"Max Threads you Can Create ?","text":"The number of threads you can create depends on:
-Xmx).-Xss).Let's say:
-Xmx2g).-Xss1m).Maximum threads = 6 GB / 1 MB per thread = ~6000 threads.
OS Limits on Threads
Even if memory allows for thousands of threads, the OS imposes limits:
/etc/security/limits.conf).OutOfMemoryError: unable to create new native thread Occurs when the OS can't allocate more native threads.
Performance Degradation as context switching between many threads becomes expensive, leading to CPU thrashing.
Rather than creating many threads manually, use thread pools to manage a fixed number of threads efficiently:
Thread Pools Exampleimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class ThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(10);\n for (int i = 0; i < 100; i++) {\n executor.submit(() -> {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName());\n });\n }\n executor.shutdown();\n }\n}\nThread pools reuse threads, reducing memory usage and improving performance.
"},{"location":"langdives/Java/Threads/#how-to-increase-max-thread","title":"How to Increase Max Thread ?","text":"On Linux, you can increase the maximum threads per process
Check current limitsulimit -u # Max user processes\nulimit -u 65535\nyour_user_name hard nproc 65535\nyour_user_name soft nproc 65535\nKey points
Spring is a popular, open-source Java-based framework used to create enterprise-level applications. It provides a comprehensive programming and configuration model that simplifies Java development. At its core, Spring focuses on dependency injection and inversion of control (IoC), providing an abstraction over Java's complexity.
"},{"location":"langdives/Java/Spring/#core-goals-of-spring","title":"Core goals of Spring","text":"The Spring ecosystem consists of various projects for different use cases:
A comprehensive list of Spring Boot annotations, covering core Spring Boot, configuration, web, data, testing, and more. I'll organize them by categories with keys (annotation names) and values (purpose/use cases) for easy reference.
"},{"location":"langdives/Java/Spring/SpringAnnotations/#core-annotations","title":"Core Annotations","text":"Annotation Purpose/Use Case@SpringBootApplication Main entry point for a Spring Boot application. Combines @Configuration, @ComponentScan, and @EnableAutoConfiguration. @EnableAutoConfiguration Enables automatic configuration of Spring beans based on the classpath and defined properties. @ComponentScan Scans the package and its sub-packages for Spring components (e.g., @Component, @Service). @Configuration Marks a class as a source of bean definitions. Used to define Spring beans programmatically. @Bean Declares a method as a Spring bean, registered in the application context. @Import Imports additional configuration classes. @ImportResource Loads bean definitions from external XML configuration files."},{"location":"langdives/Java/Spring/SpringAnnotations/#web-and-rest-annotations","title":"Web and REST Annotations","text":"Annotation Purpose/Use Case @RestController Marks a class as a REST API controller. Combines @Controller and @ResponseBody. @Controller Marks a class as a web controller. Works with view templates (like Thymeleaf). @RequestMapping Maps HTTP requests to specific handler methods or classes. Can be used on classes or methods. @GetMapping Maps HTTP GET requests to specific handler methods. @PostMapping Maps HTTP POST requests to specific handler methods. @PutMapping Maps HTTP PUT requests to specific handler methods. @DeleteMapping Maps HTTP DELETE requests to specific handler methods. @PatchMapping Maps HTTP PATCH requests to specific handler methods. @RequestBody Binds the HTTP request body to a Java object. Used in REST controllers. @ResponseBody Binds the return value of a method directly to the HTTP response body. @RequestParam Binds HTTP query parameters to method arguments. @PathVariable Binds URI template variables to method parameters. @RequestHeader Binds HTTP request headers to method parameters. @CookieValue Binds cookie values to method parameters. @ModelAttribute Binds form data to a model object. @SessionAttributes Declares session-scoped model attributes. @CrossOrigin Enables cross-origin requests (CORS) for specific endpoints."},{"location":"langdives/Java/Spring/SpringAnnotations/#jpa-jdbc-annotations","title":"JPA, JDBC Annotations","text":"Annotation Purpose/Use Case @Entity Marks a class as a JPA entity. @Table Specifies the database table for a JPA entity. @Id Marks a field as the primary key of a JPA entity. @GeneratedValue Specifies how the primary key value should be generated. @Column Specifies the mapping of a field to a database column. @OneToOne Establishes a one-to-one relationship between entities. @OneToMany Establishes a one-to-many relationship between entities. @ManyToOne Establishes a many-to-one relationship between entities. @ManyToMany Establishes a many-to-many relationship between entities. @JoinColumn Specifies the foreign key column for a relationship. @Query Defines a custom JPQL or SQL query on a repository method. @Transactional Marks a method or class as transactional. Ensures ACID properties in data operations. @EnableJpaRepositories Enables JPA repositories for data access. @Repository Marks a class as a data repository. @EnableTransactionManagement Enables declarative transaction management."},{"location":"langdives/Java/Spring/SpringAnnotations/#security-annotations","title":"Security Annotations","text":"Annotation Purpose/Use Case @EnableWebSecurity Enables Spring Security for web applications. @EnableGlobalMethodSecurity Enables method-level security annotations like @PreAuthorize and @PostAuthorize. @PreAuthorize Applies authorization logic before a method is invoked. @PostAuthorize Applies authorization logic after a method has executed. @Secured Secures a method by roles (deprecated in favor of @PreAuthorize). @RolesAllowed Specifies which roles are allowed to access a method. @WithMockUser Simulates a user for testing security."},{"location":"langdives/Java/Spring/SpringAnnotations/#testing-annotations","title":"Testing Annotations","text":"Annotation Purpose/Use Case @SpringBootTest Runs integration tests for a Spring Boot application. Loads the full application context. @WebMvcTest Tests only web layer components (e.g., controllers). @DataJpaTest Tests only JPA repositories. Configures an in-memory database. @MockBean Replaces a bean with a mock during tests. @SpyBean Replaces a bean with a spy during tests. @TestConfiguration Provides additional bean configurations for tests. @BeforeEach Runs before each test method in a test class. @AfterEach Runs after each test method in a test class."},{"location":"langdives/Java/Spring/SpringAnnotations/#profiles-annotations","title":"Profiles Annotations","text":"Annotation Purpose/Use Case @ConfigurationProperties Binds external configuration properties to a Java bean. @EnableConfigurationProperties Enables support for @ConfigurationProperties beans. @Profile Specifies the profile under which a bean is active (e.g., dev, prod). @Value Injects a value from the properties or environment. @PropertySource Loads properties from an external file. @Environment Provides access to the current environment settings."},{"location":"langdives/Java/Spring/SpringAnnotations/#actuator-metrics-annotations","title":"Actuator & Metrics Annotations","text":"Annotation Purpose/Use Case @Endpoint Defines a custom Actuator endpoint. @ReadOperation Marks a method as a read operation for an Actuator endpoint. @WriteOperation Marks a method as a write operation for an Actuator endpoint. @DeleteOperation Marks a method as a delete operation for an Actuator endpoint. @Timed Measures the execution time of a method. @Gauge Exposes a gauge metric to Actuator. @Metered Marks a method to be counted as a metric (deprecated in favor of @Timed)."},{"location":"langdives/Java/Spring/SpringAnnotations/#microservices-annotations","title":"Microservices Annotations","text":"Annotation Purpose/Use Case @EnableDiscoveryClient Enables service registration with Eureka, Consul, or Zookeeper. @EnableFeignClients Enables Feign clients for inter-service communication. @CircuitBreaker Implements circuit-breaking logic using Resilience4j. @Retryable Enables retry logic for a method. @LoadBalanced Enables load balancing for REST clients."},{"location":"langdives/Java/Spring/SpringAnnotations/#miscellaneous-annotations","title":"Miscellaneous Annotations","text":"Annotation Purpose/Use Case @Conditional Conditionally registers a bean based on custom logic. @Async Marks a method to run asynchronously. @Scheduled Schedules a method to run at fixed intervals or cron expressions. @EventListener Marks a method to listen for application events. @Cacheable Caches the result of a method. @CacheEvict Evicts entries from a cache."},{"location":"langdives/Java/Spring/SpringAnnotations/#summary","title":"Summary","text":"This is a comprehensive list of all major Spring Boot annotations, categorized by their functionality. With these annotations, Spring Boot makes it easier to develop applications by reducing boilerplate code, automating configuration, and offering powerful tools for testing, security, and microservices development.
"},{"location":"langdives/Java/Spring/SpringBoot/","title":"Spring Boot","text":"This article covers how Spring Boot automates configurations, deals with microservices, and manages monitoring, security, and performance.
"},{"location":"langdives/Java/Spring/SpringBoot/#what-is-spring-boot","title":"What is Spring Boot ?","text":"Spring Boot is an extension of the Spring Framework that simplifies the development of Java applications by offering:
The goal of Spring Boot is to help developers build stand-alone, production-grade applications quickly and with less fuss.
"},{"location":"langdives/Java/Spring/SpringBoot/#application-architecture","title":"Application Architecture","text":"A Spring Boot application consists of
spring-boot-starter-web for web apps).application.properties or application.yml.@SpringBootApplication: Combines three core annotations
@Configuration: Marks the class as a source of bean definitions.@EnableAutoConfiguration: Enables auto-configuration based on classpath contents.@ComponentScan: Scans for Spring components (beans, services, controllers).@RestController: Marks a class as a controller with REST endpoints.
@Autowired: Injects dependencies automatically by type.
@Value: Injects values from properties files.
@Profile: Activates beans only under specific profiles (e.g., dev, prod).
Starters are pre-configured dependency bundles for common functionalities
spring-boot-starter-web: For web applications.spring-boot-starter-data-jpa: For working with databases using JPA.spring-boot-starter-security: For authentication and authorization.spring-boot-starter-actuator: For monitoring and management.Spring Boot uses @EnableAutoConfiguration to detect dependencies and automatically configure beans for you.
For example: If spring-boot-starter-data-jpa is present, it will
DataSource.EntityManagerFactory to manage JPA entities.@EnableTransactionManagement.How to Debug Auto-Configuration:
Add --debug to the application start command to list all auto-configurations.
$ java -jar myapp.jar --debug\nIf you need to exclude an auto-configuration, use:
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})\nStartup: Spring Boot applications initialize with SpringApplication.run(), The lifecycle involves loading beans, initializing contexts, and wiring dependencies.
Embedded Server: By default, Spring Boot uses Tomcat as the embedded server. Others include Jetty and Undertow, The server listens on a configurable port (default: 8080).
Shutdown: Spring Boot provides graceful shutdown using @PreDestroy or hooks via SpringApplication.addShutdownHook().
application.properties","text":"Spring Boot applications are configured using either application.properties
application.properties server.port=8081\nspring.datasource.url=jdbc:mysql://localhost:3306/mydb\nspring.datasource.username=root\nspring.datasource.password=password\napplication.yml","text":"Using Profiles (e.g., Dev vs. Prod) in application.yml
application.yml server:\n port: 8080\nspring:\n profiles:\n active: dev\n\n---\nspring:\n profiles: dev\n datasource:\n url: jdbc:h2:mem:testdb\n\n---\nspring:\n profiles: prod\n datasource:\n url: jdbc:mysql://localhost:3306/proddb\nYou can activate profiles programmatically or through command-line options
$ java -Dspring.profiles.active=prod -jar myapp.jar\nYou can define your own custom properties and inject them into beans using @ConfigurationProperties.
@ConfigurationProperties(prefix = \"custom\")\npublic class CustomConfig {\n private String name;\n private int timeout;\n\n // Getters and Setters\n}\ncustom.name=SpringApp\ncustom.timeout=5000\n@Autowired\nprivate CustomConfig customConfig;\nYou can customize the embedded server by configuring the EmbeddedServletContainerFactory.
@Bean\npublic ConfigurableServletWebServerFactory webServerFactory() {\n TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();\n factory.setPort(9090);\n factory.addConnectorCustomizers(connector -> {\n connector.setAttribute(\"maxThreads\", 200);\n });\n return factory;\n}\nExposes application management and monitoring endpoints.
Some common endpoints:
/actuator/health: Health status./actuator/info: Application information./actuator/metrics: Metrics data (e.g., memory usage, request counts).management:\n endpoints:\n web:\n exposure:\n include: health, info\n security:\n enabled: true\nYou can customize or add your own metrics by using @Timed or MeterRegistry.
Basic Authentication Setup: Add spring-boot-starter-security dependency, by default, Spring Boot enables basic authentication with a generated password.
Custom Authentication
@Configuration\npublic class SecurityConfig extends WebSecurityConfigurerAdapter {\n @Override\n protected void configure(HttpSecurity http) throws Exception {\n http\n .authorizeRequests()\n .antMatchers(\"/public/\").permitAll()\n .anyRequest().authenticated()\n .and()\n .httpBasic();\n }\n}\nJWT Authentication: Use JSON Web Tokens (JWT) for token-based authentication in microservices architectures.
Unit Testing: Use @SpringBootTest for integration testing, Mock beans with @MockBean.
@SpringBootTest\npublic class MyServiceTest {\n @MockBean\n private MyRepository repository;\n\n @Autowired\n private MyService service;\n\n @Test\n public void testService() {\n when(repository.findById(1L)).thenReturn(Optional.of(new MyEntity()));\n assertTrue(service.findById(1L).isPresent());\n }\n}\nTest Containers: Use Testcontainers for running databases or services in Docker containers during tests.
Spring Boot streamlines the development process by providing auto-configuration, embedded servers, and a production-ready environment. It empowers developers to build and deploy microservices quickly, backed by powerful features like Spring Security, Spring Data, Actuator, and more. With its opinionated defaults and deep customizability, Spring Boot strikes a balance between simplicity and flexibility making it ideal for both beginners and advanced developers.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/","title":"Spring Core Framework","text":"the foundation of the entire Spring ecosystem. We'll explore each component and mechanism in detail, so by the end, you\u2019ll have a thorough understanding of how Spring Core works, including the IoC container, Dependency Injection (DI), Beans, ApplicationContext, Bean Lifecycle, AOP, and more.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#what-is-spring-core-framework","title":"What is Spring Core Framework","text":"The Spring Core Framework is the heart of the Spring ecosystem. It provides the essential features required to build Java applications, with a focus on dependency injection (DI) and inversion of control (IoC). At its core, Spring aims to eliminate the complexities of creating objects, managing dependencies, and wiring different components together.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#modules","title":"Modules","text":""},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-core","title":"Spring Core","text":"The foundational module that provides the IoC container and the basic tools for dependency injection (DI), It includes the core interfaces and classes like BeanFactory, ApplicationContext, BeanPostProcessor, BeanDefinition, and others.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-beans","title":"Spring Beans","text":"Manages the configuration, creation, and lifecycle of Spring beans.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-context","title":"Spring Context","text":"Provides a runtime environment for applications using the IoC container. It builds on the Spring Core and adds additional functionality like events and internationalization (i18n).
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-spel","title":"Spring SpEL","text":"SpEL(Spring Expression Language) A powerful expression language that can be used to dynamically query or manipulate bean properties.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#core-concepts","title":"Core Concepts","text":""},{"location":"langdives/Java/Spring/SpringCoreFramework/#inversion-of-control-ioc","title":"Inversion of Control (IoC)","text":"The Inversion of Control (IoC) principle is at the core of the Spring Framework. It shifts the control of object creation and management from the developer to the IoC container, promoting loose coupling and enhancing testability. Let\u2019s break it down conceptually and then dive into the Spring implementation.
In traditional programming, the application code creates and manages its dependencies directly.
Examplepublic class OrderService {\n private PaymentService paymentService;\n\n public OrderService() {\n this.paymentService = new PaymentService(); // Tight coupling\n }\n}\nExplanation
OrderService class directly instantiates PaymentService, meaning these two classes are tightly coupled. If the PaymentService class changes, you must modify the OrderService code as well.OrderService always uses a real PaymentService instead of a mock service.IoC Solution With Inversion of Control (IoC), the responsibility of creating the PaymentService is \"inverted\" and delegated to the Spring IoC container. Now, the IoC container injects the dependency into OrderService.
@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService; // Dependency injection\n }\n}\nExplanation
OrderService no longer creates the PaymentService itself.PaymentService and provides it to OrderService.BeanFactory is the basic IoC container in Spring. It provides basic dependency injection and bean management functionality.
Features of BeanFactory:
BeanFactory factory = new XmlBeanFactory(new FileSystemResource(\"beans.xml\"));\nOrderService service = (OrderService) factory.getBean(\"orderService\");\nHowever, BeanFactory is rarely used now because it lacks advanced features like event propagation, internationalization, and eager initialization, which are provided by ApplicationContext.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#applicationcontext","title":"ApplicationContext","text":"ApplicationContext is a more powerful IoC container that extends BeanFactory. It is widely used in modern Spring applications because of its rich features.
Features of ApplicationContext:
ContextRefreshedEvent).ApplicationContext context = new ClassPathXmlApplicationContext(\"beans.xml\");\nOrderService service = context.getBean(OrderService.class);\nIn most cases, developers use AnnotationConfigApplicationContext or Spring Boot to load configurations and manage beans.
Step-by-Step
Define Beans and Dependencies: Beans can be defined in XML, Java Configuration, or through Annotations like @Component.
<bean id=\"paymentService\" class=\"com.example.PaymentService\"/>\n<bean id=\"orderService\" class=\"com.example.OrderService\">\n <constructor-arg ref=\"paymentService\"/>\n</bean>\nSpring IoC Container Loads Configuration: The IoC container reads the configuration (XML, annotations, or Java-based) during startup.
Dependency Injection (DI): The IoC container identifies the dependencies and injects them using constructor, setter, or field injection.
Bean Initialization: The IoC container initializes all necessary beans (eagerly or lazily).
Bean Usage: The beans are now available for use by the application.
Dependency Injection (DI) is a pattern where objects are provided with their dependencies at runtime by the IoC container instead of creating them directly. Spring supports multiple types of dependency injection:
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#types-of-di","title":"Types of DI","text":"Constructor Injection: Dependencies are provided through the class constructor, Recommended for mandatory dependencies.
Constructor Injection Example@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\nSetter Injection: Dependencies are injected using setter methods, Useful for optional dependencies.
Setter Injection Example@Component\npublic class OrderService {\n private PaymentService paymentService;\n\n @Autowired\n public void setPaymentService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\nField Injection: Dependencies are injected directly into class fields, Not recommended since it makes unit testing harder.
Field Injection@Component\npublic class OrderService {\n @Autowired\n private PaymentService paymentService;\n}\nLoose Coupling: Objects don\u2019t manage their dependencies, making them easier to modify, extend, or replace.
Testability: Dependencies can be easily mocked during unit tests, enhancing test coverage.
Reusability: Components are easier to reuse across different parts of the application.
Configuration Management: Centralized configuration of beans ensures that all components behave consistently.
Configuration Overhead: Without frameworks like Spring, managing dependencies can lead to complex code. Spring solves this by centralizing configurations.
Circular Dependencies: Two beans dependent on each other can cause issues. Spring detects such circular dependencies and throws exceptions.
Managing Large Applications: With many components, configuration can get complicated. Spring Boot simplifies this with auto-configuration and starters.
Let\u2019s look at a complete example using Spring Core with constructor injection:
Full IoC Implementation Example Java Configuration Example@Configuration\npublic class AppConfig {\n\n @Bean\n public PaymentService paymentService() {\n return new PaymentService();\n }\n\n @Bean\n public OrderService orderService(PaymentService paymentService) {\n return new OrderService(paymentService);\n }\n}\n@Component\npublic class PaymentService {\n public void processPayment() {\n System.out.println(\"Payment processed.\");\n }\n}\n\n@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n\n public void placeOrder() {\n System.out.println(\"Order placed.\");\n paymentService.processPayment();\n }\n}\npublic class Main {\n public static void main(String[] args) {\n ApplicationContext context = new AnnotationConfigApplicationContext(AppConfig.class);\n OrderService orderService = context.getBean(OrderService.class);\n orderService.placeOrder();\n }\n}\nSpring beans are the building blocks of any Spring application. They represent the objects that the Spring IoC container manages throughout their lifecycle. Understanding beans and their lifecycle is critical for mastering the Spring Core framework. Let\u2019s explore everything about beans\u2014from creation, scopes, lifecycle, initialization, destruction, and more\u2014in detail.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#what-is-a-bean","title":"What is a Bean ?","text":"A bean in Spring is an object that is instantiated, assembled, and managed by the IoC container. The container creates, initializes, and wires these beans, ensuring that all dependencies are injected as needed. Beans are usually defined using:
@Component, @Service, @Repository, @Controller)Spring provides multiple ways to declare beans and register them with the IoC container:
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#xml-based-configuration","title":"XML-based Configuration","text":"Traditional way of defining beans using XML files.
XML Config Example<beans xmlns=\"http://www.springframework.org/schema/beans\" \n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n xsi:schemaLocation=\"http://www.springframework.org/schema/beans \n http://www.springframework.org/schema/beans/spring-beans.xsd\">\n\n <bean id=\"paymentService\" class=\"com.example.PaymentService\"/>\n <bean id=\"orderService\" class=\"com.example.OrderService\">\n <constructor-arg ref=\"paymentService\"/>\n </bean>\n</beans>\nSpring allows you to use Java classes to define beans. This is cleaner and avoids XML boilerplate.
Java Config Example@Configuration\npublic class AppConfig {\n\n @Bean\n public PaymentService paymentService() {\n return new PaymentService();\n }\n\n @Bean\n public OrderService orderService() {\n return new OrderService(paymentService());\n }\n}\nYou can annotate classes with @Component, @Service, @Repository, or @Controller. Spring automatically detects these beans if @ComponentScan is enabled.
Annotations Example@Component\npublic class PaymentService { }\n\n@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\n@Configuration\n@ComponentScan(basePackages = \"com.example\")\npublic class AppConfig { }\nThe scope of a bean defines the lifecycle and visibility of that bean within the Spring IoC container.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#types-of-bean-scopes","title":"Types of Bean Scopes","text":"singleton (default): A single instance of the bean is created and shared across the entire application, Used for stateless beans.
Example@Scope(\"singleton\")\n@Component\npublic class SingletonBean { }\nprototype: A new instance is created every time the bean is requested, Useful for stateful objects or temporary tasks.
Example@Scope(\"prototype\")\n@Component\npublic class PrototypeBean { }\nrequest: A new bean instance is created for each HTTP request, Used in web applications.
session: A single instance is created per HTTP session.
globalSession: A global session scope for applications using portlets.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#bean-lifecycle","title":"Bean Lifecycle","text":"Each Spring bean goes through several lifecycle phases, starting from instantiation to destruction. The Spring IoC container manages this lifecycle internally.
Bean Lifecycle Phases
Bean Instantiation: The IoC container instantiates the bean using the constructor or factory method.
Dependency Injection (DI): The container injects dependencies (via constructor, setter, or field injection).
Custom Initialization: If the bean implements InitializingBean or uses the @PostConstruct annotation, the initialization logic is executed here.
Bean Ready to Use: After initialization, the bean is ready for use by the application.
Custom Destruction: When the IoC container is shutting down, beans with @PreDestroy or DisposableBean are given a chance to release resources.
InitializingBean Interface: If a bean implements the InitializingBean interface, it must override the afterPropertiesSet() method, which is called after all properties are set.
public class MyService implements InitializingBean {\n @Override\n public void afterPropertiesSet() {\n System.out.println(\"MyService is initialized.\");\n }\n}\nDisposableBean Interface: If a bean implements the DisposableBean interface, it must override the destroy() method, which is called during the destruction phase.
public class MyService implements DisposableBean {\n @Override\n public void destroy() {\n System.out.println(\"MyService is being destroyed.\");\n }\n}\nUsing @PostConstruct and @PreDestroy: These annotations are the recommended way to manage initialization and destruction callbacks.
@Component\npublic class MyService {\n\n @PostConstruct\n public void init() {\n System.out.println(\"Initialization logic in @PostConstruct.\");\n }\n\n @PreDestroy\n public void cleanup() {\n System.out.println(\"Cleanup logic in @PreDestroy.\");\n }\n}\nCustom Initialization and Destruction Methods: You can also specify custom methods in the bean configuration.
Example (Java Config)@Bean(initMethod = \"init\", destroyMethod = \"cleanup\")\npublic MyService myService() {\n return new MyService();\n}\nAll singleton beans are created at the time of application startup (default behavior), Useful for performance since all dependencies are resolved upfront.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#lazy-initialization","title":"Lazy Initialization","text":"Beans are created only when they are first requested, You can enable lazy initialization at the bean level using @Lazy.
@Lazy\n@Component\npublic class LazyBean { }\nSpring IoC container resolves bean dependencies through constructor injection, setter injection, or field injection, You can specify bean dependencies explicitly using the depends-on attribute in XML.
<bean id=\"databaseConnection\" class=\"com.example.DatabaseConnection\"/>\n<bean id=\"orderService\" class=\"com.example.OrderService\" depends-on=\"databaseConnection\"/>\nThis ensures that the databaseConnection bean is initialized before the orderService bean.
A circular dependency occurs when two or more beans are mutually dependent on each other.
Example@Component\npublic class A {\n @Autowired\n private B b;\n}\n\n@Component\npublic class B {\n @Autowired\n private A a;\n}\nSpring tries to resolve circular dependencies using singleton beans by injecting proxies, but it fails for constructor injection. To avoid circular dependencies: - Refactor the code to reduce dependencies. - Use setter injection instead of constructor injection.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#bean-definition-inheritance","title":"Bean Definition Inheritance","text":"Spring allows bean definitions to inherit properties from a parent bean. This helps reduce configuration duplication.
Example (XML)<bean id=\"parentBean\" class=\"com.example.BaseService\">\n <property name=\"name\" value=\"Base Service\"/>\n</bean>\n\n<bean id=\"childBean\" class=\"com.example.ChildService\" parent=\"parentBean\">\n <property name=\"name\" value=\"Child Service\"/>\n</bean>\nAOP allows you to separate cross-cutting concerns (like logging, security, or transaction management) from the business logic. In Spring, AOP is implemented using aspects, advice, and pointcuts.
@Aspect\n@Component\npublic class LoggingAspect {\n @Before(\"execution(* com.example.*.*(..))\")\n public void logBefore(JoinPoint joinPoint) {\n System.out.println(\"Before method: \" + joinPoint.getSignature().getName());\n }\n}\n@Configuration\n@EnableAspectJAutoProxy\npublic class AppConfig { }\nSpring supports an event-driven model that allows you to build decoupled components. The ApplicationContext can publish events and allow listeners to respond to them.
Example of a Custom Eventpublic class CustomEvent extends ApplicationEvent {\n public CustomEvent(Object source) {\n super(source);\n }\n}\n@Component\npublic class CustomEventListener implements ApplicationListener<CustomEvent> {\n @Override\n public void onApplicationEvent(CustomEvent event) {\n System.out.println(\"Received custom event: \" + event);\n }\n}\nSpEL allows you to manipulate and query beans dynamically. It can be used inside XML or annotations.
Example of SpEL<bean id=\"myBean\" class=\"com.example.MyClass\">\n <property name=\"value\" value=\"#{2 + 3}\"/>\n</bean>\nA detailed comparison table that covers all possible differences between Spring Boot and Spring Framework. This comparison covers everything from setup, configuration, embedded servers, web applications, testing, microservices support, and much more.
"},{"location":"langdives/Java/Spring/SpringFrameworkVsSpringBoot/#differences","title":"Differences","text":"Category Spring Framework Spring Boot Purpose A comprehensive framework for building Java applications. An extension of Spring Framework to simplify configuration and create stand-alone applications. Setup Requires manual setup, including XML or Java-based configuration. Minimal setup with auto-configuration based on the classpath. Main Focus Provides flexibility and control over every aspect of the application. Focuses on rapid development with sensible defaults and opinions. Configuration Can use XML, Java-based, or annotation-based configuration. Uses annotations and properties/YAML files for configuration. Learning Curve Requires more learning time due to complexity. Easier to get started with for beginners due to auto-configuration and pre-built setups. Project Dependencies Requires managing multiple dependencies for each feature manually. Provides starters (e.g.,spring-boot-starter-web) to include required dependencies. Embedded Server No embedded server support; WAR files must be deployed to external servers (Tomcat, Jetty, etc.). Comes with embedded servers (Tomcat, Jetty, or Undertow) for running stand-alone applications. Deployment Deploy WAR/EAR files to external servers. Runs applications directly as JAR files with embedded servers. Auto-Configuration No auto-configuration; requires manual configuration of components. Auto-configures components based on available classpath dependencies. Application Entry Point Relies on external servlet containers to manage the lifecycle. Uses @SpringBootApplication as the entry point with SpringApplication.run(). Microservices Support Not specialized for microservices; requires additional tools. Built with microservices architecture in mind. Supports Spring Cloud, Eureka, Feign, etc. Performance Requires more configuration to optimize performance. Better suited for lightweight and high-performance microservices. Profiles Supports profiles for environment-specific configurations but requires more setup. Supports profiles easily through application.yml or application.properties. Testing Provides JUnit support but requires manual configuration for context loading. Provides easy testing with @SpringBootTest, @MockBean, @DataJpaTest, and others. Security Uses Spring Security but requires manual integration. Integrates Spring Security easily with spring-boot-starter-security. Database Access Provides JDBC, JPA, ORM support, but requires more configuration. Simplifies database access with Spring Data JPA and auto-configuration of DataSource. Starters and Dependency Management Requires manual management of dependencies and configurations. Provides Spring Boot Starters that bundle all required dependencies for specific use cases. Template Engines Supports Thymeleaf, JSP, and others with manual setup. Supports template engines with starters (e.g., spring-boot-starter-thymeleaf). Command-Line Interface (CLI) No built-in CLI support. Provides Spring Boot CLI to run Groovy scripts for quick development. Actuator and Monitoring Requires external monitoring tools or custom configurations. Comes with Spring Boot Actuator to monitor application health, metrics, and endpoints. DevTools for Hot Reload Requires manual setup for hot reloading of code changes. Provides Spring Boot DevTools for hot reloading during development. Support for Reactive Programming Supports Spring WebFlux and Project Reactor (from version 5.x). Fully supports Spring WebFlux for reactive, non-blocking programming. Circuit Breakers & Resilience Requires integration with third-party libraries like Hystrix. Seamlessly integrates with Resilience4j and Spring Cloud for resilience. Integration with Cloud Platforms Requires Spring Cloud or manual setup for cloud integration. Seamlessly integrates with Spring Cloud for cloud-native development. Logging Configuration Requires manual configuration of logging frameworks (e.g., Log4j, SLF4J). Provides auto-configured logging using Logback by default. Health Checks and Metrics Requires manual configuration to expose health metrics. Provides Actuator endpoints (/actuator/health, /actuator/metrics) out-of-the-box. Web Framework Uses Spring MVC for building web applications. Uses Spring MVC or Spring WebFlux with easy setup through starters. Restful API Development Requires manual setup of controllers and components. Provides easy development with @RestController and auto-configuration of REST endpoints. Command-Line Arguments Support Requires manual handling of command-line arguments. Easily reads command-line arguments with SpringApplication or @Value. Caching Support Requires setting up EhCache, Guava, or other caching solutions manually. Provides easy caching configuration with @EnableCaching and auto-configuration. Internationalization (i18n) Supports i18n but requires more setup. Supports i18n with minimal configuration through application.properties. Job Scheduling Requires integration with Quartz or other scheduling libraries. Supports scheduling with @Scheduled and Task Executors. Dependency Injection (DI) Provides dependency injection with IoC container. Same as Spring Framework but simplifies it with auto-wiring using @Autowired. Backward Compatibility Must manually update configurations when upgrading versions. Provides backward compatibility with most Spring projects. Community and Ecosystem Large community and extensive ecosystem. Built on top of Spring Framework with additional tools for modern development."},{"location":"langdives/Java/Spring/SpringFrameworkVsSpringBoot/#summary","title":"Summary","text":"Spring Boot simplifies Spring Framework by:
Spring Framework gives more control and flexibility but at the cost of manual setup and configuration.
Spring Boot is optimized for rapid development, especially for microservices and cloud-native applications.
Spring Framework is still relevant for legacy applications or when fine-grained control is necessary.
"},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/","title":"High Availability and Fault Tolerance","text":""},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/#high-availability-ha","title":"High Availability (HA)","text":"High Availability (HA) refers to a system or infrastructure's ability to remain operational and accessible for a very high percentage of time, minimizing downtime. In essence, it's a design principle used in IT to ensure that services, applications, or systems are continuously available, even in the event of hardware failures, software issues, or unexpected disruptions.
Key Aspects of High Availability
Redundancy: Multiple components (like servers, databases, or network paths) are set up so that if one fails, another can take over immediately without interrupting the service.
Failover: If a primary system component fails, the workload automatically switches to a backup or standby system to avoid service disruptions.
Load Balancing: Distribution of workloads across multiple servers or resources helps prevent any single system from becoming overwhelmed and failing. This also increases performance and resiliency.
Clustering: Multiple servers or resources act as a \"cluster,\" functioning together as a single system. If one server fails, the cluster can redistribute the tasks to the remaining ones.
Monitoring and Alerts: Constant health checks and monitoring tools are used to detect potential problems early. Alerts can notify administrators to address issues before they impact availability.
Data Replication: Critical data is often replicated across multiple locations or systems. This ensures that in case of a failure, there\u2019s no data loss and services can continue with minimal disruption.
Availability is often expressed as a percentage. For instance, an uptime of 99.99% means the service is expected to be down for only 52 minutes in a year.
Common availability standards
Data Centers: Cloud providers like AWS, Microsoft Azure, and Google Cloud offer high availability zones to ensure service uptime even during hardware failures or network outages.
Web Applications: E-commerce sites (like Amazon) employ HA architecture to ensure they remain operational at all times, as any downtime directly impacts revenue.
Banking Systems: Financial services need HA to ensure uninterrupted access to customer transactions, ATMs, and payment processing services.
Telecommunication Networks: Cellular networks and ISPs use HA to maintain voice, data, and internet services without interruptions.
Fault Tolerance refers to the ability of a system, network, or application to continue functioning correctly even when one or more of its components fail. It ensures continuous operation without loss of service, despite hardware, software, or other types of faults occurring in the system. Fault tolerance plays a crucial role in ensuring high reliability and availability of critical systems.
Key Principles of Fault Tolerance
Redundancy: Multiple components are deployed so that if one component fails, another takes over seamlessly. Redundant systems include extra servers, power supplies, storage drives, and networks.
Failover: When a primary component or service fails, the workload automatically \"fails over\" to a backup system with minimal disruption.
Error Detection and Correction: The system includes mechanisms to detect, isolate, and correct errors to maintain proper functioning. This is often implemented using parity bits, checksums, or watchdog timers.
Replication: Critical data is copied across multiple devices or systems (e.g., in distributed systems). This way, even if a fault occurs in one part, other replicas maintain the system\u2019s integrity.
Graceful Degradation: If a fault occurs, the system may reduce performance or disable non-essential services instead of shutting down entirely, thus ensuring the most critical functions remain operational.
Hardware Fault Tolerance:
Software Fault Tolerance:
Network Fault Tolerance:
Error Detection and Recovery:
Airplane Control Systems: Modern aircraft use redundant flight control systems, so if one control unit fails, another takes over to ensure passenger safety.
Nuclear Power Plants: Fault tolerance is critical to ensure safe operations. Redundant sensors, control systems, and cooling systems prevent catastrophic failures.
Financial Systems: Banks and payment processing systems must tolerate faults to ensure that transactions are processed without downtime or data corruption.
Cloud Computing Services: Cloud providers like AWS or Google Cloud implement fault tolerance using distributed systems, replication, and redundant storage across multiple locations.
Cost: Building redundant systems is expensive, as it requires additional hardware, software, and management resources.
Complexity: Managing fault-tolerant systems is complex due to the need for real-time monitoring, synchronization, and failover testing.
Performance Overhead: Some fault tolerance techniques, like replication or checkpointing, can introduce performance bottlenecks.
High Availability (HA) and Fault Tolerance (FT) are two strategies aimed at keeping systems operational, but they approach this goal differently. Here's a detailed comparison:
In other words, high availability aims to reduce downtime, whereas fault-tolerant systems aim for zero downtime, even during failures.
Aspect High Availability (HA) Fault Tolerance (FT) Definition Ensures minimal downtime by quickly switching to backup systems when a failure occurs. Ensures continuous operation even during failures, with no noticeable interruption. Goal Minimize downtime and ensure service is restored quickly. Eliminate downtime and maintain seamless operation during faults. Approach Uses redundant components and failover systems to switch operations when needed. Uses duplication of systems to ensure tasks are always mirrored on another system. Downtime Small amount of downtime during failover (milliseconds to minutes). No downtime; systems operate continuously, even during faults. Example Use Cases E-commerce websites (e.g., Amazon) that switch servers when one fails. Airplane control systems, which cannot afford any interruptions. Redundancy Type Active-Passive: Backup components are activated only when primary systems fail. Active-Active: All components are working simultaneously, and one continues if the other fails. Cost Less expensive since backup systems are not always active. More expensive due to constant replication and active systems running in parallel. Complexity Easier to implement and manage due to reliance on failover mechanisms. More complex, requiring real-time synchronization and parallel operation. Performance Impact Some performance hit during failover but minimal. Higher overhead, as multiple systems operate simultaneously. Use Case Example Cloud platforms (like AWS) use high availability to ensure that servers recover quickly after a failure. Nuclear power plants employ fault-tolerant systems to keep critical processes running with no interruptions. Failure Handling Handles component failures through redundancy and quick recovery mechanisms. Prevents failure from affecting the system by running identical processes or systems in parallel."},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/#summary","title":"Summary","text":"In summary, high availability ensures that critical systems are always accessible with minimal interruptions. Organizations rely on HA strategies to meet customer expectations, protect revenue, and ensure business continuity, especially in industries where even a small amount of downtime can have serious consequences and fault tolerance is the ability of a system to keep operating without interruption despite experiencing faults or failures. It is crucial for mission-critical systems in industries like aviation, finance, and healthcare, where downtime or errors could lead to catastrophic outcomes.
In essence, High Availability focuses on minimizing downtime by recovering quickly from failures, while Fault Tolerance eliminates downtime altogether by ensuring the system continues running seamlessly. HA is less costly and easier to implement, while FT is expensive and complex but essential for critical environments where even a few seconds of downtime are unacceptable.
"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/","title":"Docker and Kubernetes","text":""},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#overview","title":"Overview","text":""},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#docker","title":"Docker","text":"Docker is a platform that enables developers to build, package, and run applications in lightweight containers. It ensures applications are portable and can run consistently across different environments, from development to production.
Key Components
Dockerfile. Images are layered and reusable.Kubernetes (often abbreviated as K8s) is an open-source platform for automating the deployment, scaling, and management of containerized applications. It abstracts away the complexity of running containers at scale across multiple machines.
Key Components
Docker focuses on building, packaging, and running containers. It handles application-level concerns whereas Kubernetes focuses on orchestrating, scaling, and managing containers across distributed environments.
"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#how-they-work-together","title":"How they Work Together","text":"Create a Dockerfile for your application and build the image Step-1: Build Docker Images
docker build -t my-app:latest .\nStore the image in a registry (e.g., Docker Hub) Step-2: Push to Registry
docker push my-app:latest\nUse the built Docker image in a Kubernetes Deployment YAML file and apply it with: Deploy on Kubernetes
kubectl apply -f deployment.yaml\nInstall Docker: Follow Docker\u2019s official installation guide for your operating system.
Create a Dockerfile: Example:
FROM python:3.9\nWORKDIR /app\nCOPY . .\nRUN pip install -r requirements.txt\nCMD [\"python\", \"app.py\"]\nBuild and Run Docker Containers:
docker build -t my-app:latest .\nRun Container:
docker run -d -p 5000:5000 my-app:latest\nUse Docker Compose:
docker-compose.yml: version: '3.8'\nservices:\n web:\n image: my-app:latest\n ports:\n - \"5000:5000\"\ndocker-compose up\nInstall Kubernetes: Use Minikube for local development or create a production cluster using cloud providers like GKE, AKS, or EKS.
Define Kubernetes Resources: Create YAML manifests for Deployments and Services.
Deployment YAML:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: my-app\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: my-app\n template:\n metadata:\n labels:\n app: my-app\n spec:\n containers:\n - name: my-app\n image: my-app:latest\n ports:\n - containerPort: 5000\nService YAML:
apiVersion: v1\nkind: Service\nmetadata:\n name: my-app-service\nspec:\n type: NodePort\n ports:\n - port: 5000\n targetPort: 5000\n selector:\n app: my-app\nDeploy to Kubernetes:
kubectl apply -f deployment.yaml\nkubectl apply -f service.yaml\nWith Docker Compose: Define services in docker-compose.yml and run:
docker-compose up\nWith Kubernetes (Using Minikube):
kubectl.docker swarm init\nUse overlay networks for service communication across nodes.
Using Kubernetes:
Kubernetes: Pods use DNS names or Service endpoints to communicate.
Multiple Machines Communication:
docker-compose.yml file Multiple YAML files for resources Scaling Manual Automated scaling with kubectl scale High Availability Limited Built-in redundancy and self-healing Use Case Simple applications on one machine Complex workloads across clusters"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#8-additional-considerations","title":"8. Additional Considerations","text":"Kubernetes: Use Persistent Volumes (PV) and Persistent Volume Claims (PVC).
Monitoring and Logging:
Use Prometheus and Grafana in Kubernetes for comprehensive monitoring.
CI/CD Integration:
Docker Compose can also use the build context to pull code directly from GitHub when creating an image. Here\u2019s how:
docker-compose.yml with GitHub Repository as the Build Context:version: '3.8'\nservices:\n app:\n build:\n context: https://github.com/your-username/your-repo.git\n dockerfile: Dockerfile\n ports:\n - \"5000:5000\"\ndocker-compose up --build\nNote: - This method works only if the GitHub repository is public. For private repositories, you\u2019ll need to provide authentication (e.g., via SSH keys or GitHub tokens). - Docker will pull the latest version from the specified GitHub repository and build the image based on the Dockerfile in the repository.
Can I use a Docker Compose file with Kubernetes? Not directly. Kubernetes doesn\u2019t understand Docker Compose syntax, but there are tools like Kompose that can convert Docker Compose files into Kubernetes YAML files.
What happens if I try to run a Docker Compose file inside a Kubernetes cluster? It won\u2019t work. Kubernetes will look at you confused (figuratively), because it expects YAML manifests with its own syntax, not a docker-compose.yml.
Why do Kubernetes YAML files look scarier than Docker Compose files? Kubernetes YAML files are more complex because they handle more advanced scenarios like scaling, networking, and rolling updates, which Docker Compose doesn\u2019t attempt to address.
Do I need to uninstall Docker if I switch to Kubernetes? Nope! Docker is still useful for building and testing images locally, even if you\u2019re deploying to Kubernetes. In fact, Kubernetes can use Docker as a container runtime.
Will Docker containers fight with Kubernetes Pods if they run on the same machine? Nope, they\u2019ll coexist peacefully. Docker containers and Kubernetes Pods can run side by side without conflict. They\u2019re friends, not rivals!
Can I copy-paste my Docker Compose file into Kubernetes and hope it works? Sorry, no shortcuts here. You need to convert the Compose file into Kubernetes resources, either manually or using tools like Kompose.
Is Docker Compose faster than Kubernetes because it has fewer YAML files? Yes, Docker Compose is faster to set up for local development because it\u2019s simpler. But for production-scale orchestration, Kubernetes is much more powerful.
How do I know if my container is happy inside a Kubernetes Pod? Check with this command:
kubectl get pods\nRunning, your container is content. If you see CrashLoopBackOff, it\u2019s definitely not happy! Can I use Kubernetes without the cloud, or will it complain? You can use Minikube or kind (Kubernetes in Docker) to run Kubernetes locally on your machine. No cloud required.
What\u2019s the difference between docker-compose up and kubectl apply -f?
docker-compose up: Starts containers defined in a docker-compose.yml file. kubectl apply -f: Deploys resources (like Pods, Deployments) described in a Kubernetes YAML file to your cluster.Do I still need to learn Docker Swarm if I already know Kubernetes? Not really. Docker Swarm is simpler but not as widely used in production as Kubernetes. Kubernetes has become the de facto standard.
Can a single Pod run multiple Docker Compose services? Yes! A Pod can run multiple containers, similar to how Docker Compose runs multiple services. However, in Kubernetes, these containers should be tightly coupled (e.g., sharing resources).
If Docker Compose is easier, why do people torture themselves with Kubernetes? Kubernetes offers features like scaling, self-healing, and load balancing. It\u2019s overkill for simple projects but essential for large, distributed applications.
Is Kubernetes just a fancy way of saying, \u201cI don\u2019t want to use Docker Compose\u201d? Not exactly. Docker Compose is great for local setups, while Kubernetes is a powerful orchestration tool for running applications across multiple nodes at scale.
What\u2019s the difference between a Pod and a Container? Can I use the words interchangeably? Not quite. A Pod is a wrapper that can contain one or more containers. Pods are the smallest deployable unit in Kubernetes, but a container is just an isolated environment for running applications.
If a container crashes in Kubernetes, does Kubernetes get sad? Nope! Kubernetes will restart the container automatically. That\u2019s part of its self-healing magic.
Will my application break if I use a Docker image from 2015? It might! Older images could have compatibility issues or security vulnerabilities. Use them only if you\u2019re sure they still meet your needs.
Is Kubernetes allergic to Windows, or will it run happily there? Kubernetes supports Windows nodes, but the experience is smoother with Linux. Most people deploy Kubernetes on Linux-based clusters.
Can I use both Docker and Kubernetes at the same time? Or will it cause chaos? Yes, you can use both. Build your containers with Docker, push them to a registry, and deploy them with Kubernetes. No chaos \u2013 just smooth workflows.
Why can\u2019t Docker Compose just learn scaling and take over Kubernetes' job? Docker Compose is intentionally lightweight and simple. Adding Kubernetes-like features would complicate it and defeat its original purpose.
How much YAML is too much YAML? If you start dreaming in YAML, it\u2019s probably too much. But seriously, Kubernetes relies heavily on YAML, so learning to manage it effectively is key.
Can Kubernetes work without YAML files? (Please say yes!) Unfortunately, no. YAML files are essential for defining resources in Kubernetes. You can use Helm charts to simplify it, but YAML is unavoidable.
What happens if I forget to push my Docker image before deploying with Kubernetes? Your deployment will fail because Kubernetes won\u2019t find the image in the registry. Always remember to push!
Can I use kubectl commands on Docker containers? Nope. kubectl is specifically for managing Kubernetes resources. Use docker commands for Docker containers.
Is Kubernetes only for tech wizards, or can normal humans use it too? Normal humans can use it too! The learning curve is steep, but with practice, anyone can master it.
Do I need to sacrifice sleep to understand Kubernetes? Maybe at first. But once you get the hang of it, Kubernetes will become your friend, and sleep will return.
Can a Docker container tell the difference between running on Kubernetes and Docker Compose? Nope! The container itself doesn\u2019t care where it\u2019s running. As long as it gets its dependencies and configuration, it\u2019ll happily run anywhere.
Can I run two Docker Compose files on one machine? Yes, use the -p option to specify different project names for each Compose file.
Can services communicate across multiple machines? Yes, with Docker Swarm or Kubernetes, services can communicate across machines using overlay networks or Kubernetes networking.
Is Docker Compose suitable for production? Not recommended for large-scale production. Use Kubernetes or Docker Swarm instead.
How do I set up Kubernetes on a single machine? Use Minikube to run a local Kubernetes cluster.
What file formats are used by Docker and Kubernetes? Docker uses Dockerfile and docker-compose.yml. Kubernetes uses YAML files for resources like Deployments and Services.
Elasticsearch is a search engine based on Apache Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#elasticsearch-basics-and-fundamentals","title":"ElasticSearch Basics and Fundamentals","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-core-concepts","title":"1. Core Concepts:","text":"Diving into Elasticsearch's core concepts is essential for understanding its architecture and functionality.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#11-documents-and-indices","title":"1.1 Documents and Indices","text":"{\n \"title\": \"Learning Elasticsearch\",\n \"author\": \"John Doe\",\n \"published_date\": \"2023-01-15\"\n}\nPUT /books\nPUT /books/_mapping\n{\n \"properties\": {\n \"title\": { \"type\": \"text\" },\n \"author\": { \"type\": \"keyword\" },\n \"published_date\": { \"type\": \"date\" }\n }\n}\nPUT /books\n{\n \"settings\": {\n \"number_of_shards\": 3,\n \"number_of_replicas\": 1\n }\n}\nPOST /books/_doc/1\n{\n \"title\": \"Learning Elasticsearch\",\n \"author\": \"John Doe\",\n \"published_date\": \"2023-01-15\"\n}\nGET /books/_doc/1\nPOST /books/_doc/1/_update\n{\n \"doc\": { \"author\": \"Jane Doe\" }\n}\nDELETE /books/_doc/1\nAn inverted index is a fundamental data structure in Elasticsearch and other search engines. It optimizes search efficiency by storing a mapping from terms (words) to their locations within documents. Let\u2019s break it down into key components and processes:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#21-core-structure-of-inverted-index","title":"2.1. Core Structure of Inverted Index","text":"For instance, if a dataset contains the documents: - Doc 1: \u201cElasticsearch powers search\u201d - Doc 2: \u201cSearch powers insights\u201d
The inverted index would look like this:
\"Elasticsearch\" -> [Doc 1]\n\"powers\" -> [Doc 1, Doc 2]\n\"search\" -> [Doc 1, Doc 2]\n\"insights\" -> [Doc 2]\nThe process involves several stages: - Tokenization: Splitting text into words or tokens. - Normalization: Making tokens consistent, like converting to lowercase. - Stemming/Lemmatization (optional): Reducing words to their base or root forms. - Indexing: Populating the index with terms and the corresponding document references.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#23-how-searches-work","title":"2.3. How Searches Work","text":"When a user searches for a term, Elasticsearch retrieves the postings list from the inverted index, quickly locating documents containing that term. For multi-term queries, Elasticsearch can intersect postings lists, using logical operations (e.g., AND, OR) to combine or filter results.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#24-optimizations","title":"2.4. Optimizations","text":"Inverted indices are the foundation of Elasticsearch\u2019s speed and relevance in text search. This structure is tailored for high performance in full-text search scenarios, especially when complex queries and filtering are involved.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#3-analyzers","title":"3. Analyzers","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#31-what-is-an-analyzer","title":"3.1. What is an Analyzer?","text":"Creating a custom analyzer involves defining: - A tokenizer (e.g., edge-ngram tokenizer for partial word matches). - A list of token filters to process the tokens (e.g., synonym filters, ASCII folding for diacritical marks).
Example configuration:
{\n \"analysis\": {\n \"analyzer\": {\n \"custom_analyzer\": {\n \"type\": \"custom\",\n \"tokenizer\": \"whitespace\",\n \"filter\": [\"lowercase\", \"stop\", \"synonym\"]\n }\n }\n }\n}\nenglish, french) handle language-specific tokenization and filtering.Analyzers transform raw text into optimized, searchable data, playing a critical role in making Elasticsearch searches accurate and efficient.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#4-elasticsearch-queries","title":"4. ElasticSearch Queries","text":"In Elasticsearch, queries are central to retrieving data. They\u2019re categorized as leaf queries (operating on specific fields) and compound queries (combining multiple queries). Here's a deep dive into each type, with examples to illustrate their functionality:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#41-leaf-queries","title":"4.1 Leaf Queries","text":"These are standalone, field-specific queries (like term and match above) that don\u2019t depend on other queries to function.
Match Query: Used for analyzing full-text fields, capable of handling fuzziness, synonyms, and relevance scoring.
{\n \"query\": {\n \"match\": {\n \"description\": \"elastic search\"\n }\n }\n}\nMatch Phrase Query: Requires terms to appear in the specified order, useful for exact phrase searches.
{\n \"query\": {\n \"match_phrase\": {\n \"description\": \"elastic search\"\n }\n }\n}\n{\n \"query\": {\n \"term\": {\n \"status\": \"active\"\n }\n }\n}\n{\n \"query\": {\n \"terms\": {\n \"status\": [\"active\", \"pending\"]\n }\n }\n}\nCompound queries allow for complex logic by combining multiple queries, enabling fine-grained control over query conditions and relevance.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#421-bool-query","title":"4.2.1. Bool Query","text":"The most flexible compound query, allowing logic-based combinations:
Must Clause: The must clause ensures that all enclosed queries must match. Think of it as an \"AND\" operation.
\"title\" contains \"Elasticsearch\" and \"content\" contains \"query\". {\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"match\": { \"title\": \"Elasticsearch\" } },\n { \"match\": { \"content\": \"query\" } }\n ]\n }\n }\n}\nShould Clause: The should clause is a flexible OR operation. If any query in the should clause matches, the document is considered a match. Adding multiple should clauses increases relevance based on how many match.
tags field are included. {\n \"query\": {\n \"bool\": {\n \"should\": [\n { \"match\": { \"tags\": \"search\" } },\n { \"match\": { \"tags\": \"indexing\" } }\n ],\n \"minimum_should_match\": 1\n }\n }\n}\nMust Not Clause: The must_not clause excludes documents that match any query within it, similar to a \"NOT\" operation.
{\n \"query\": {\n \"bool\": {\n \"must\": { \"match\": { \"title\": \"Elasticsearch\" } },\n \"must_not\": { \"match\": { \"status\": \"archived\" } }\n }\n }\n}\nFilter Clause: The filter clause narrows down the result set without affecting relevance scores. It\u2019s optimized for structured queries and often used to improve query performance.
{\n \"query\": {\n \"bool\": {\n \"must\": { \"match\": { \"title\": \"Elasticsearch\" } },\n \"filter\": { \"term\": { \"status\": \"published\" } }\n }\n }\n}\nCombining multiple clauses:
{\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"match\": { \"title\": \"Elasticsearch\" } }\n ],\n \"should\": [\n { \"match\": { \"category\": \"tutorial\" } },\n { \"match\": { \"category\": \"guide\" } }\n ],\n \"must_not\": [\n { \"term\": { \"status\": \"archived\" } }\n ],\n \"filter\": [\n { \"range\": { \"publish_date\": { \"gte\": \"2023-01-01\" } } }\n ]\n }\n }\n}\nOptimizes for the highest relevance score among multiple queries, often used when querying across similar fields with varied wording. - Example: Searching for the most relevant match between \u201ctitle\u201d and \u201cdescription\u201d fields:
{\n \"query\": {\n \"dis_max\": {\n \"queries\": [\n { \"match\": { \"title\": \"elastic search\" } },\n { \"match\": { \"description\": \"elastic search\" } }\n ]\n }\n }\n}\nElasticsearch provides several geo-specific queries for filtering and scoring documents based on geographic location:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#431-geo-bounding-box-query","title":"4.3.1 Geo Bounding Box Query","text":"Defines a rectangular area by specifying two corner points (top-left and bottom-right). Documents with locations inside this box are matched.
{\n \"query\": {\n \"geo_bounding_box\": {\n \"location\": {\n \"top_left\": {\n \"lat\": 40.73,\n \"lon\": -74.1\n },\n \"bottom_right\": {\n \"lat\": 40.01,\n \"lon\": -71.12\n }\n }\n }\n }\n}\nFinds documents within a certain distance from a point. Useful for proximity searches, like \"find stores within 10 miles.\"
{\n \"query\": {\n \"geo_distance\": {\n \"distance\": \"50km\",\n \"location\": {\n \"lat\": 40.7128,\n \"lon\": -74.0060\n }\n }\n }\n}\nSearches within a polygon defined by a series of latitude and longitude points, allowing for irregular area shapes.
{\n \"query\": {\n \"geo_polygon\": {\n \"location\": {\n \"points\": [\n { \"lat\": 40.73, \"lon\": -74.1 },\n { \"lat\": 40.01, \"lon\": -71.12 },\n { \"lat\": 39.73, \"lon\": -73.1 }\n ]\n }\n }\n }\n}\nThe geo_shape query allows for more complex spatial filtering, using pre-defined shapes like circles, polygons, or lines. This is often used with indexed geometries.
{\n \"query\": {\n \"geo_shape\": {\n \"location\": {\n \"shape\": {\n \"type\": \"circle\",\n \"coordinates\": [-74.1, 40.73],\n \"radius\": \"1000m\"\n },\n \"relation\": \"within\"\n }\n }\n }\n}\nGeo filters are often used in combination with other query types within bool queries, allowing flexible, location-based filtering along with other criteria.
Finds published documents within a specific area and filters out archived content.
{\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"term\": { \"status\": \"published\" } }\n ],\n \"filter\": {\n \"geo_distance\": {\n \"distance\": \"50km\",\n \"location\": {\n \"lat\": 40.7128,\n \"lon\": -74.0060\n }\n }\n },\n \"must_not\": [\n { \"term\": { \"status\": \"archived\" } }\n ]\n }\n }\n}\nThese queries, combined thoughtfully, make Elasticsearch highly adaptable to various search needs.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#5-aggregations","title":"5. Aggregations","text":"Elasticsearch\u2019s aggregation framework is divided into metrics and bucket aggregations. Here\u2019s a deep dive into each, with subtypes and examples.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#51-metrics-aggregations","title":"5.1. Metrics Aggregations","text":"These calculate values from field data, like sums or averages.
Example: Calculate average price across documents.
{\n \"aggs\": {\n \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n }\n}\nSum Aggregation: Totals numeric field values.
Example: Sum of all sales amounts.
{\n \"aggs\": {\n \"total_sales\": { \"sum\": { \"field\": \"amount\" } }\n }\n}\nMin/Max Aggregations: Find minimum or maximum field values.
Example: Find the highest score.
{\n \"aggs\": {\n \"max_score\": { \"max\": { \"field\": \"score\" } }\n }\n}\nStats Aggregation: Generates summary stats (min, max, avg, sum, count).
Example: Summarize views across articles.
{\n \"aggs\": {\n \"article_views\": { \"stats\": { \"field\": \"views\" } }\n }\n}\nCardinality Aggregation: Counts unique values.
Example: Count unique users by ID.
{\n \"aggs\": {\n \"unique_users\": { \"cardinality\": { \"field\": \"user_id\" } }\n }\n}\nPercentiles/Percentile Ranks Aggregations: Show value distribution.
{\n \"aggs\": {\n \"response_percentiles\": {\n \"percentiles\": { \"field\": \"response_time\", \"percents\": [50, 90, 99] }\n }\n }\n}\nThese create groups (buckets) of documents based on field values or criteria. Each bucket can contain documents matching conditions and may contain further sub-aggregations.
Example: Group documents by category.
{\n \"aggs\": {\n \"by_category\": { \"terms\": { \"field\": \"category\" } }\n }\n}\nRange Aggregation: Group documents within defined ranges.
Example: Bucket users by age ranges.
{\n \"aggs\": {\n \"age_ranges\": {\n \"range\": { \"field\": \"age\", \"ranges\": [{ \"to\": 20 }, { \"from\": 20, \"to\": 30 }, { \"from\": 30 }] }\n }\n }\n}\nDate Histogram: Buckets documents based on date intervals (e.g., daily, monthly).
Example: Group events by month.
{\n \"aggs\": {\n \"monthly_events\": { \"date_histogram\": { \"field\": \"date\", \"calendar_interval\": \"month\" } }\n }\n}\nHistogram Aggregation: Creates buckets based on custom numeric ranges.
Example: Histogram of item prices.
{\n \"aggs\": {\n \"price_histogram\": { \"histogram\": { \"field\": \"price\", \"interval\": 10 } }\n }\n}\nFilter Aggregation: Buckets documents matching a specific filter.
Example: Filter for high-value orders.
{\n \"aggs\": {\n \"high_value_orders\": { \"filter\": { \"range\": { \"price\": { \"gt\": 100 } } } }\n }\n}\nFilters Aggregation: Allows multiple filter buckets in a single query.
Example: Separate buckets for high and low prices.
{\n \"aggs\": {\n \"price_buckets\": {\n \"filters\": {\n \"filters\": {\n \"expensive\": { \"range\": { \"price\": { \"gt\": 100 } } },\n \"cheap\": { \"range\": { \"price\": { \"lt\": 50 } } }\n }\n }\n }\n }\n}\nGeohash Grid Aggregation: For geolocation-based bucketing, especially with maps.
{\n \"aggs\": {\n \"geo_buckets\": { \"geohash_grid\": { \"field\": \"location\", \"precision\": 5 } }\n }\n}\nEach aggregation can nest other aggregations, allowing complex analysis structures.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#example-of-a-combined-aggregation","title":"Example of a Combined Aggregation","text":"Calculate the average order amount by city and age range:
{\n \"aggs\": {\n \"by_city\": {\n \"terms\": { \"field\": \"city\" },\n \"aggs\": {\n \"age_ranges\": {\n \"range\": { \"field\": \"age\", \"ranges\": [{ \"to\": 20 }, { \"from\": 20, \"to\": 30 }, { \"from\": 30 }] },\n \"aggs\": {\n \"avg_order_amount\": { \"avg\": { \"field\": \"order_amount\" } }\n }\n }\n }\n }\n }\n}\nWith these aggregations, Elasticsearch becomes a powerful analytics engine, enabling sophisticated data analysis directly within the index.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#6-sorting","title":"6. Sorting","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#61-basic-sorting","title":"6.1. Basic Sorting","text":"_score (relevance-based sorting).sort in the query.{\n \"sort\": [\n { \"price\": { \"order\": \"asc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\nkeyword fields).{\n \"sort\": [\n { \"release_date\": { \"order\": \"desc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\n{\n \"sort\": [\n { \"price\": { \"order\": \"asc\" } },\n { \"rating\": { \"order\": \"desc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\nnested with a path to sort by nested fields.{\n \"sort\": [\n {\n \"products.price\": {\n \"order\": \"asc\",\n \"nested\": {\n \"path\": \"products\",\n \"filter\": { \"range\": { \"products.price\": { \"gt\": 10 } } }\n }\n }\n }\n ],\n \"query\": { \"match_all\": {} }\n}\ngeo_distance to sort by proximity for geo_point fields.{\n \"sort\": [\n {\n \"_geo_distance\": {\n \"location\": \"40.715, -73.988\",\n \"order\": \"asc\",\n \"unit\": \"km\"\n }\n }\n ],\n \"query\": { \"match_all\": {} }\n}\npainless scripting language.{\n \"sort\": {\n \"_script\": {\n \"type\": \"number\",\n \"script\": {\n \"source\": \"doc['price'].value * params.factor\",\n \"params\": { \"factor\": 1.2 }\n },\n \"order\": \"desc\"\n }\n },\n \"query\": { \"match_all\": {} }\n}\nmissing, setting them as the highest or lowest values.{\n \"sort\": [\n { \"price\": { \"order\": \"asc\", \"missing\": \"_last\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\ncount or custom metrics.{\n \"aggs\": {\n \"top_brands\": {\n \"terms\": {\n \"field\": \"brand.keyword\",\n \"order\": { \"_count\": \"desc\" }\n }\n }\n }\n}\nElasticsearch's relevance scoring is crucial for ranking documents based on their similarity to a query. Here\u2019s an in-depth look:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#71-scoring-mechanism-and-bm25-algorithm","title":"7.1. Scoring Mechanism and BM25 Algorithm","text":"The BM25 (Best Matching 25) algorithm is Elasticsearch\u2019s default relevance scoring algorithm. BM25 improves upon traditional TF-IDF (Term Frequency-Inverse Document Frequency) by adjusting term frequency saturation and document length normalization, providing more nuanced relevance.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#core-components-of-bm25","title":"Core Components of BM25:","text":"The BM25 formula combines these components, with two main parameters: - k1: Controls term frequency saturation (default around 1.2). Higher values give more influence to term frequency. - b: Controls length normalization (default around 0.75). Higher values penalize longer documents more strongly. - BM25 Alogirthm
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#73-understanding-scoring-in-elasticsearch-queries","title":"7.3. Understanding Scoring in Elasticsearch Queries","text":"In Elasticsearch, relevance scores are generated by the \"match\" or \"multi_match\" queries. Each document receives a relevance score, and results are ranked based on these scores. You can inspect scores using the \"explain\": true parameter, which details each document\u2019s score and shows how BM25 factors contribute.
{\n \"query\": {\n \"match\": {\n \"content\": {\n \"query\": \"Elasticsearch relevance scoring\",\n \"boost\": 1.5\n }\n }\n },\n \"explain\": true\n}\ncontent field. The \"boost\" parameter can emphasize this field for relevance, while \"explain\": true helps analyze the scoring breakdown."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#74-improving-relevance-with-advanced-techniques","title":"7.4. Improving Relevance with Advanced Techniques","text":"Relevance scoring with BM25 is foundational to Elasticsearch\u2019s search quality, offering powerful controls for tuning results to your specific needs.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#8-pagination-and-cursors","title":"8. Pagination and Cursors","text":"Pagination in Elasticsearch is essential for handling large result sets efficiently, as it prevents overwhelming the client and server. Elasticsearch offers different methods for pagination, each with specific use cases. Let's break down each method:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#81-basic-pagination-with-from-and-size","title":"8.1. Basic Pagination withfrom and size","text":"from parameter skips a set number of results, while size controls the number of results returned.from increases (since Elasticsearch has to load and sort through all documents before the specified offset).Example:
{\n \"from\": 20,\n \"size\": 10,\n \"query\": { \"match_all\": {} }\n}\nfrom and size by using a cursor.search_after parameter to specify a unique field value (like a timestamp or an ID) from the last document of the previous page.Example:
{\n \"sort\": [ { \"timestamp\": \"asc\" }, { \"id\": \"asc\" } ],\n \"size\": 10,\n \"query\": { \"match_all\": {} },\n \"search_after\": [1627489200, \"XYZ123\"] \n}\nsearch_after takes the values from the timestamp and id fields of the last document on the previous page, ensuring seamless navigation."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#83-scroll-api-for-bulk-data-retrieval","title":"8.3. Scroll API for Bulk Data Retrieval","text":"Example Workflow: - First, initiate a scroll session:
{\n \"size\": 100,\n \"query\": { \"match_all\": {} },\n \"scroll\": \"1m\" \n}\n_scroll_id returned by the initial request to retrieve subsequent pages: {\n \"scroll\": \"1m\",\n \"scroll_id\": \"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAA...\"\n}\nAfter each scroll request, repeat until the returned results are empty, which indicates that all documents have been retrieved.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#84-point-in-time-pit-for-real-time-pagination-with-consistency","title":"8.4. Point-in-Time (PIT) for Real-time Pagination with Consistency","text":"Example Workflow: - First, initiate a Point-in-Time session:
POST /index_name/_pit?keep_alive=1m\npit_id with search_after for paginated queries: {\n \"size\": 10,\n \"query\": { \"match_all\": {} },\n \"pit\": { \"id\": \"PIT_ID\", \"keep_alive\": \"1m\" },\n \"sort\": [ { \"timestamp\": \"asc\" }, { \"id\": \"asc\" } ],\n \"search_after\": [1627489200, \"XYZ123\"]\n}\nDELETE /_pit\nsearch_after or PIT for deeper, efficient pagination.search_after and PIT handle real-time updates better.search_after and PIT require sorted fields to maintain proper order across paginated results.from & size Simple pagination for small datasets Performance drop for large from values Basic search pages, small datasets search_after Deep pagination without from overhead Requires sorted fields, can\u2019t skip pages Infinite scrolling, data tables with lots of records Scroll API Bulk data export/processing High memory usage, no real-time consistency Data migration, report generation Point-in-Time Consistent real-time pagination Needs frequent re-creation to avoid memory issues Dashboards, applications requiring consistent views Each method serves specific needs, balancing consistency, performance, and real-time capabilities. This setup allows Elasticsearch to handle vast and dynamic datasets while supporting efficient data retrieval.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#9-facets-and-filters","title":"9. Facets and Filters","text":"Faceting creates summaries of data, useful for search result filtering, like categorizing search results by price or brand. Filters, on the other hand, optimize performance by narrowing down documents without affecting scoring.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#91-facets-aggregations","title":"9.1. Facets (Aggregations)","text":"Faceting is a process in Elasticsearch that aggregates search results, providing structured summaries for complex queries. For example, if searching for \"laptops,\" facets can aggregate results by price range, brand, or processor type, allowing users to filter search results dynamically.
Filters enable the narrowing of search results by criteria (e.g., price < $500), improving query efficiency and bypassing relevance scoring. They\u2019re often used to pre-process data before a full-text search and work well with caches, resulting in faster query performance.
bool query, filters can be applied to queries under must or filter clauses. Filters in filter clauses are cached and reusable, significantly boosting speed for repetitive filters (e.g., \"available items only\").In complex search interfaces, facets allow users to drill down through categories, while filters further refine their selections without recalculating scores, ensuring responsive user experiences.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#94-sample-implementations","title":"9.4. Sample Implementations","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#941-creating-facets-aggregations","title":"9.4.1. Creating Facets (Aggregations)","text":"To implement facets, you\u2019ll define bucket aggregations to group and categorize data. For instance, creating a facet for \"price range\" and \"brand\" in a search for laptops:
GET /products/_search\n{\n \"query\": {\n \"match\": { \"description\": \"laptop\" }\n },\n \"aggs\": {\n \"price_ranges\": {\n \"range\": {\n \"field\": \"price\",\n \"ranges\": [\n { \"to\": 500 },\n { \"from\": 500, \"to\": 1000 },\n { \"from\": 1000 }\n ]\n }\n },\n \"brands\": {\n \"terms\": { \"field\": \"brand.keyword\" }\n }\n }\n}\nThis example provides a breakdown of price ranges and a count of each brand, creating flexible filters users can click on to refine results.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#942-using-filters-for-optimized-performance","title":"9.4.2. Using Filters for Optimized Performance","text":"Filters improve performance by narrowing results without scoring. Here\u2019s an example of using a bool query with a filter clause:
GET /products/_search\n{\n \"query\": {\n \"bool\": {\n \"must\": {\n \"match\": { \"description\": \"laptop\" }\n },\n \"filter\": [\n { \"term\": { \"in_stock\": true } },\n { \"range\": { \"price\": { \"lt\": 1000 } } }\n ]\n }\n }\n}\nIn this query, in_stock and price filters optimize search results without affecting scoring.
match should only appear in scoring-related queries (e.g., must). Using term or range filters for precise values (like IDs, prices) reduces computational load.Each node type in an Elasticsearch cluster has specialized roles that allow it to handle different aspects of indexing, searching, and managing data. Let's explore the node types in detail.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#11-master-node","title":"1.1. Master Node","text":"The master node is the cluster\u2019s brain, responsible for the overall management and health of the cluster.
Health Checks: Constantly checks the health of nodes, removing unresponsive nodes to maintain a stable cluster state.
Consensus and Election Process:
Data nodes are the primary nodes for storing data, processing indexing operations, and executing search and aggregation requests.
Also known as the client node, the coordinating node acts as a router for client requests, managing query distribution and response aggregation.
Ingest nodes preprocess data before it is indexed, often using ingest pipelines to transform and enrich data.
Each level in the cluster architecture plays a role in organizing and distributing data efficiently across nodes.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#21-cluster","title":"2.1. Cluster","text":"To deeply understand how request flow and cluster operations work in Elasticsearch, let\u2019s walk through each stage of the process in detail:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#31-receiving-a-request","title":"3.1. Receiving a Request","text":"When a client sends a request to Elasticsearch, it can be either a query (search request) or an indexing (write) request. Here\u2019s how this begins:
Identifying Relevant Shards and Nodes:
Selecting Shards (Primary or Replica):
At this stage, each shard executes the request locally. This process differs slightly between a search request and an indexing request.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#331-query-execution-search-requests","title":"3.3.1. Query Execution (Search Requests)","text":"Query Phase:
Fetch Phase:
Primary Shard Indexing:
Replication to Replica Shards:
After the query or indexing operation completes, the coordinating node consolidates the response:
Sorting and Ranking:
Returning the Response:
The indexing flow includes several key mechanisms that ensure data consistency and durability:
Primary-Replica Synchronization:
Distributed Write-Ahead Log (WAL):
Commit Process:
To visualize this, here\u2019s a simplified flow of the entire process:
Node Failure Handling: If a node fails, Elasticsearch promotes a replica shard to primary, maintaining data consistency and availability.
Cluster State Management:
To thoroughly understand Elasticsearch's storage and retrieval mechanisms, let\u2019s go deep into Lucene's segments, inverted index, and advanced data structures like BKD trees. Lucene, at its core, powers Elasticsearch, giving it the ability to handle and query massive datasets with impressive speed and efficiency.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#41-lucene-and-segments","title":"4.1. Lucene and Segments","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#411-what-is-a-segment","title":"4.1.1. What is a Segment?","text":"A segment in Lucene is a self-contained, immutable collection of documents that forms a subset of a shard. Each segment is essentially a mini-index with its own data structures, including inverted indexes, stored fields, and other data structures to facilitate efficient searching and retrieval.
Example: - Imagine a shard with 100 small segments. Lucene might merge them into fewer, larger segments (say, 10 segments), consolidating their data and removing any \"marked as deleted\" documents.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#413-advantages-of-segments","title":"4.1.3. Advantages of Segments","text":"The inverted index is Lucene\u2019s most fundamental data structure and is the backbone of Elasticsearch\u2019s fast full-text search capabilities.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#421-structure-of-an-inverted-index","title":"4.2.1. Structure of an Inverted Index","text":"The inverted index allows quick lookups by mapping terms to postings lists (lists of documents containing each term).
Example: - Suppose you index the text \"Elasticsearch is scalable search\". The inverted index might look like this:
Term Documents\n------------------------\n\"Elasticsearch\" [1]\n\"is\" [1, 2]\n\"scalable\" [1, 3]\n\"search\" [1]\nApart from the inverted index, Lucene also uses specialized data structures to handle numeric and spatial data.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#431-bkd-trees","title":"4.3.1. BKD Trees","text":"BKD Trees are used in Elasticsearch for indexing and querying numeric, date, and geospatial data, especially for high-cardinality fields (fields with a large number of unique values).
Splitting Mechanism: The tree splits data points along axes at each level, grouping data into smaller ranges or \"blocks.\" Each block holds a set of points within a defined range.
Hierarchical Storage: At each level of the tree, the data is divided along a specific dimension, allowing efficient narrowing of the search space during queries.
geo_point fields, Elasticsearch calculates distances within BKD trees, efficiently handling \"find near\" queries.Example: - Suppose you have a field geo_point representing user locations. A BKD tree indexes these coordinates, allowing Elasticsearch to quickly retrieve points within a bounding box or radius without scanning all documents.
BKD trees are essentially a form of a k-d tree (k-dimensional tree), optimized for indexing and searching over multiple dimensions. Each dimension can represent a distinct numeric field (e.g., latitude, longitude, or timestamp).
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#using-bkd-trees-for-queries","title":"Using BKD Trees for Queries","text":"BKD trees efficiently handle:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-bkd-trees","title":"Advantages of BKD Trees","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#432-doc-values","title":"4.3.2. Doc Values","text":"Doc values enable efficient retrieval of field values for sorting, aggregation, and faceting. Instead of retrieving data from inverted indexes (which are optimized for search), doc values provide a columnar storage format that is ideal for analytical tasks.
On-Disk Storage: Doc values are stored on disk rather than in memory, allowing efficient memory usage and enabling large datasets to be queried without excessive RAM usage.
sum, avg).Geo-point Doc Values: Specifically for geographic points, enabling geo-distance calculations.
Example: - Sorting results by price in a large index of products: - Doc values store price in a single column, which Elasticsearch reads to quickly sort documents without scanning each document individually.
Efficient Sorting and Aggregation: Since doc values are structured columnar data, they allow fast and memory-efficient operations.
Doc values are defined by the field type:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#using-doc-values-in-queries","title":"Using Doc Values in Queries","text":"Doc values are essential for operations such as:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-doc-values","title":"Advantages of Doc Values","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#433-summary-of-lucene-data-structures-in-elasticsearch","title":"4.3.3. Summary of Lucene Data Structures in Elasticsearch","text":"Data Structure Purpose Use Cases Benefits Segments Immutable sub-indices within a shard All document storage Concurrent searches, immutability Inverted Index Maps terms to documents Full-text search Fast term lookups BKD Trees Indexes numeric and multidimensional data Geospatial, timestamp queries Efficient range queries Doc Values Columnar storage for fields Sorting, aggregations Optimized memory usage Point Data Types Indexes geographic points (latitude-longitude) Proximity, bounding box queries Fast geospatial indexing"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#44-advanced-data-structures-in-lucene","title":"4.4. Advanced Data Structures in Lucene","text":"Let's dive into the point data types and spatial indexes used in Elasticsearch, especially focusing on how it handles geospatial data with geo_point fields. We\u2019ll look at how Quadtrees and R-trees work, their role in spatial indexing, and how they support geospatial queries such as bounding box and proximity searches.
Elasticsearch supports geospatial data using the geo_point and geo_shape data types: - geo_point: Stores latitude-longitude pairs for points on a map and is primarily used for proximity searches (e.g., \u201cfind locations within 10km\u201d). - geo_shape: Used for more complex shapes, such as polygons or multipoints, and is suitable for defining geographical areas like cities or lakes.
Geospatial queries include: - Bounding Box Queries: Searches for documents within a specific rectangle defined by coordinates. - Distance Queries: Searches for documents within a specified radius from a point. - Polygon Queries: Searches for documents within or intersecting with a complex polygonal area.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#442-quadtrees-and-r-trees-in-spatial-indexing","title":"4.4.2. Quadtrees and R-trees in Spatial Indexing","text":"Quadtrees and R-trees are tree-based data structures that organize spatial data by dividing the space into hierarchical grids or regions, allowing efficient geospatial query processing.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#quadtrees","title":"Quadtrees","text":"Quadtrees are hierarchical, 2-dimensional spatial indexes that recursively partition space into four quadrants or nodes, making them highly suitable for spatial data like latitude and longitude pairs.
The depth of the tree (how many times it splits) depends on data density, as each quadrant only further divides if it contains more than a specified number of points.
How it Works:
Querying: For a bounding box or radius query, the algorithm quickly eliminates entire quadrants that fall outside the search area, only scanning relevant sub-quadrants, drastically reducing search space.
Use Cases:
Example: Imagine we have a city map with thousands of restaurants, each represented as a point (latitude, longitude). - A quadtree organizes the map into quadrants based on restaurant density. Denser regions are divided further to create sub-quadrants. - To find restaurants within a specific neighborhood, the quadtree quickly filters out distant quadrants, only scanning nearby ones, significantly speeding up search.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-quadtrees","title":"Advantages of Quadtrees","text":"R-trees are another popular spatial data structure used to index multi-dimensional data (e.g., geographic shapes) by grouping nearby objects in bounding rectangles.
Each leaf node in an R-tree contains actual data points or bounding rectangles for objects (e.g., specific areas on a map).
How it Works:
Querying: For a search (e.g., finding locations within a radius), the R-tree filters bounding rectangles that don\u2019t intersect the search area, allowing the query to narrow down results without exhaustive scans.
Use Cases:
Example: Consider a map with various regions, like parks, lakes, and neighborhoods, each represented as a polygon. - An R-tree groups these polygons in bounding rectangles based on location. Polygons that are close to each other fall under the same rectangle. - When searching for parks within a 5km radius, the R-tree discards rectangles outside this range, only exploring relevant areas to find matching polygons.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-r-trees","title":"Advantages of R-trees","text":"While Elasticsearch doesn\u2019t directly expose Quadtrees and R-trees as configurations, Lucene, its underlying search library, utilizes versions of these structures to handle spatial indexing efficiently.
Points (like those in geo_point fields) benefit from this structure, as it speeds up bounding box queries and basic distance queries.
R-trees in Lucene:
geo_shape fields and other complex shapes where simple 2D point indexing isn\u2019t enough.Lucene optimizes these data structures to fit within its segment-based storage, allowing them to scale across multiple indices and segments, handling both large-scale geospatial queries and basic point-based distance queries.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#444-spatial-query-processing-in-elasticsearch","title":"4.4.4. Spatial Query Processing in Elasticsearch","text":"Using these data structures, Elasticsearch processes spatial queries as follows:
For a rectangular region, Elasticsearch leverages Quadtrees to restrict the search space to quadrants that intersect with the bounding box. Points or shapes within these quadrants are retrieved.
Distance Query:
For a proximity search (e.g., finding locations within 5km of a point), the Geo Distance Filter calculates distances from a central point and retrieves points from quadrants or nodes that fall within this radius.
Polygon Query:
These data structures optimize spatial queries in Elasticsearch, allowing it to handle diverse geospatial data efficiently. For example: - Bounding Box Queries are accelerated by Quadtrees, making them ideal for finding all points in a geographic area. - Distance Queries are optimized by both Quadtrees and BKD trees, allowing real-time retrieval of nearby points. - Polygon Queries are handled by R-trees, which efficiently manage irregular shapes and large polygons for accurate intersection checks.
By integrating these structures into Lucene, Elasticsearch supports powerful geospatial capabilities across various applications, including mapping services, logistics, and location-based searches.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#45-how-lucenes-structures-fit-into-elasticsearchs-query-flow","title":"4.5. How Lucene's Structures Fit into Elasticsearch\u2019s Query Flow","text":"As documents are indexed, Lucene tokenizes text fields, stores terms in the inverted index, and creates doc values for fields that require sorting or aggregation.
Segment Creation:
Documents are grouped into segments, with each segment containing its own inverted index, BKD trees, and doc values.
Query Execution:
Lucene\u2019s well-designed structures\u2014segments, inverted indexes, and multidimensional BKD trees\u2014create the foundation for Elasticsearch\u2019s speed and scalability, enabling it to support complex queries and large datasets efficiently.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#5-visual-representation-of-cluster-architecture-hierarchy-flow","title":"5. Visual Representation of Cluster Architecture (Hierarchy Flow)","text":"Elasticsearch Cluster\n\u2514\u2500\u2500 Nodes\n \u251c\u2500\u2500 Master Node\n \u2502 \u251c\u2500\u2500 Manages cluster state\n \u2502 \u2514\u2500\u2500 Handles shard allocation and rebalancing\n \u251c\u2500\u2500 Data Node\n \u2502 \u251c\u2500\u2500 Stores data and handles indexing/searching\n \u2502 \u251c\u2500\u2500 Manages primary and replica shards\n \u2502 \u2514\u2500\u2500 Processes local queries and aggregations\n \u251c\u2500\u2500 Coordinating Node\n \u2502 \u251c\u2500\u2500 Routes client requests to data nodes\n \u2502 \u251c\u2500\u2500 Aggregates responses from data nodes\n \u2502 \u2514\u2500\u2500 Sends final response to the client\n \u2514\u2500\u2500 Ingest Node\n \u251c\u2500\u2500 Processes and transforms data before indexing\n \u2514\u2500\u2500 Enriches data with pipelines\n\nIndex\n\u2514\u2500\u2500 Shards (Primary and Replica)\n \u2514\u2500\u2500 Lucene Index (Each shard is a Lucene index)\n \u251c\u2500\u2500 Segments (Immutable data units in a shard)\n \u2502 \u251c\u2500\u2500 Inverted Index\n \u2502 \u251c\u2500\u2500 Doc Values\n \u2502 \u2514\u2500\u2500 BKD Trees (for numeric & geo fields)\n \u2514\u2500\u2500 Documents (JSON objects representing data records)(within segments)\nDiving into Elasticsearch\u2019s search and other thread pools is crucial to understanding its performance and scalability. These pools are essential for managing various tasks like indexing, searching, and handling incoming requests. Let\u2019s go through these pools from end to end, covering configurations, management, and performance metrics to monitor.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-overview","title":"1. Overview","text":"GET requests for individual documents).Here\u2019s a closer look at each pool, along with common configurations and considerations:
Search Pool
threads: Generally calculated based on available CPU cores. It can be configured as # of processors * 3.queue_size: Defines the maximum number of requests that can be queued before rejecting new ones. Default is 1000.queue_size if you experience high query load, but monitor memory use carefully to avoid OutOfMemory errors.search_pool.active: Shows active threads handling search requests.search_pool.queue: Tracks pending search requests.search_pool.rejected: Indicates rejected requests due to an overloaded pool.Index Pool
threads: Typically set to the number of processors.queue_size: Default is 200. If your indexing rate is high, you may need to increase this.index_pool.active: Active threads indexing data.index_pool.queue: Number of indexing requests waiting in the queue.index_pool.rejected: Tracks rejected indexing requests.Get Pool
threads: Tied to the number of processors, generally around # of processors * 2.queue_size: Default is 1000.get_pool.active: Active threads processing GET requests.get_pool.queue: Queued retrieval requests.get_pool.rejected: Indicates if any requests are being rejected.Bulk Pool
threads: Usually set as # of processors * 2.queue_size: Often set at 50 to limit memory usage.bulk_pool.active: Active threads handling bulk operations.bulk_pool.queue: Queued bulk requests.bulk_pool.rejected: Shows if the bulk pool is rejecting requests due to overload.Management Pool
1 to 5 threads as these tasks are not frequent.management_pool.active: Active management threads.management_pool.queue: Queued management tasks.Snapshot Pool
snapshot_pool.active: Active threads handling snapshots.snapshot_pool.queue: Queued snapshot operations.node_stats.thread_pool.* via monitoring tools (like Kibana or Prometheus) helps assess each pool's workload.active: Indicates active threads. High values suggest a busy node.queue: Shows queued tasks. A consistent rise may indicate that the pool size needs tuning.rejected: Rejected tasks are a clear sign of bottlenecks.# of processors * 3 1000 search_pool.active, search_pool.queue, search_pool.rejected Increase queue_size if many queries are queued; monitor memory usage to prevent OutOfMemory issues. Index Pool Handles indexing requests for documents # of processors 200 index_pool.active, index_pool.queue, index_pool.rejected For high indexing rates, increase queue size and thread count as necessary. Get Pool Retrieves individual documents # of processors * 2 1000 get_pool.active, get_pool.queue, get_pool.rejected Increase queue_size if retrieval requests are high; monitor latency and resource usage. Bulk Pool Processes bulk indexing operations # of processors * 2 50 bulk_pool.active, bulk_pool.queue, bulk_pool.rejected Keep queue_size modest to limit memory use; monitor latency during high bulk loads. Management Pool Manages maintenance tasks like merges 1\u20135 5 management_pool.active, management_pool.queue Generally low usage; monitor only if queue is frequently non-empty, indicating background task delays. Snapshot Pool Handles snapshot creation and restoration 1\u20132 5 snapshot_pool.active, snapshot_pool.queue Schedule snapshots during low-activity periods; adjust resources if snapshots interfere with other tasks. These pools are essential to optimizing Elasticsearch\u2019s handling of diverse workloads. Monitoring and adjusting each pool based on your workload ensures better performance and resource management across the Elasticsearch cluster.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#caching","title":"Caching","text":"In Elasticsearch, caching plays a vital role in speeding up queries by storing frequently accessed data at various levels, minimizing I/O operations and improving response times. Here\u2019s a detailed, end-to-end look at caching in Elasticsearch, from the lowest level to the highest, covering each caching mechanism and its role.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-types-of-caches-in-elasticsearch","title":"1. Types of Caches in Elasticsearch","text":"There are several caching mechanisms in Elasticsearch, each working at a different level:
Each of these caches serves a specific purpose and optimizes a different aspect of query processing.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#2-filesystem-cache-os-level","title":"2. Filesystem Cache (OS-level)","text":"keyword or numeric types.indices.fielddata.cache.size setting. Default eviction strategy is based on the Least Recently Used (LRU) policy.fielddata.memory_size: Shows the amount of memory used by field data.fielddata.evictions: Tracks the number of evictions, which can indicate if your cache size is too small.term queries and range queries, but does not cache full-text queries as they are more complex and vary widely.indices.queries.cache.size setting (default is 10% of the heap).query_cache.memory_size: Memory occupied by the query cache.query_cache.evictions: Counts the number of evictions due to cache full.query_cache.hit_count and query_cache.miss_count: Useful for determining cache effectiveness.request_cache=true and doesn\u2019t cache when track_total_hits is enabled.index.requests.cache.enable (enabled by default). Also LRU-based.request_cache.memory_size: Shows the memory consumed by the request cache.request_cache.evictions: Tracks evictions in the request cache.request_cache.hit_count and request_cache.miss_count: Indicates request cache hit ratio.segment.memory_in_bytes: Memory used for cached segments.segments.count: Number of segments in the index.indices.memory and indices.evictions to track the memory usage and evictions for index metadata.query_cache and request_cache. For frequent aggregations, prioritize field data cache.fscache: The filesystem cache is essential, so configure sufficient memory for the OS to hold index files and segments, avoiding excessive I/O operations.request_cache=true in your queries or adjusting cache policies at the index level.fielddata.memory_size, fielddata.evictions Increase size for high aggregation requirements Query Cache Shard-level Caches individual filter query results LRU-based, configurable query_cache.memory_size, query_cache.evictions, query_cache.hit_count, query_cache.miss_count Monitor hit/miss ratios to determine effectiveness Request Cache Shard-level Caches entire search request results LRU-based, per-index request_cache.memory_size, request_cache.evictions, request_cache.hit_count, request_cache.miss_count Best for aggregations on static data Segment Cache Node-level Caches Lucene index segments and postings Managed by Lucene segment.memory_in_bytes, segments.count Larger heap size improves cache efficiency Indices Cache Node-level Caches index metadata (mappings, settings) LRU-based indices.memory, indices.evictions Adjust based on frequency of metadata updates Each caching layer works in tandem to optimize query speed and efficiency, and monitoring these caches is essential for fine-tuning Elasticsearch performance to meet your specific use cases.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#index-life-management-ilm","title":"Index Life Management - ILM","text":"Elasticsearch\u2019s Hot-Warm-Cold (Hot-Warm-Cold-Frozen) architecture is part of its Index Lifecycle Management (ILM) system, designed to optimize storage and cost-efficiency for managing data that has different usage patterns over time. This architecture allows you to store data on different types of nodes (hot, warm, cold, and frozen) based on data retention needs and access frequency. Here\u2019s an in-depth look at each phase and how to effectively manage data with Elasticsearch\u2019s ILM:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-purpose-of-hot-warm-cold-architecture","title":"1. Purpose of Hot-Warm-Cold Architecture","text":"The ILM policy defines actions to transition data through different stages based on time or data access patterns:
Here\u2019s an example ILM policy to illustrate phase-based transitions for a log analytics use case:
{\n \"policy\": {\n \"phases\": {\n \"hot\": {\n \"actions\": {\n \"rollover\": {\n \"max_age\": \"7d\",\n \"max_size\": \"50gb\"\n },\n \"set_priority\": {\n \"priority\": 100\n }\n }\n },\n \"warm\": {\n \"min_age\": \"30d\",\n \"actions\": {\n \"allocate\": {\n \"number_of_replicas\": 1\n },\n \"forcemerge\": {\n \"max_num_segments\": 1\n },\n \"set_priority\": {\n \"priority\": 50\n }\n }\n },\n \"cold\": {\n \"min_age\": \"90d\",\n \"actions\": {\n \"allocate\": {\n \"require\": {\n \"data\": \"cold\"\n }\n },\n \"freeze\": {},\n \"set_priority\": {\n \"priority\": 0\n }\n }\n },\n \"delete\": {\n \"min_age\": \"365d\",\n \"actions\": {\n \"delete\": {}\n }\n }\n }\n }\n}\nThis Hot-Warm-Cold architecture in Elasticsearch enables you to balance cost, performance, and data accessibility across various hardware configurations, ensuring that data is always stored cost-effectively without compromising on necessary access patterns.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#list-of-rest-apis","title":"List of REST APIs","text":"API Endpoint Purpose Description Typical Latency Use Cases Index APIsPOST /{index}/_doc Index a Document Adds or updates a document in the specified index. Low (~5-10ms for small docs) Real-time data ingestion, document updates. POST /{index}/_bulk Bulk Indexing Allows indexing, updating, or deleting multiple documents in a single request. Low to Medium (~10-50ms) High-volume data ingestion, ETL processes. POST /{index}/_update/{id} Update a Document Partially updates a document by ID in the specified index. Low to Medium (~10-30ms) Updating specific fields in documents. DELETE /{index}/_doc/{id} Delete a Document Deletes a document by ID from the specified index. Low (~5-10ms) Document removal based on unique IDs. Search APIs GET /{index}/_search Search Documents Executes a search query with optional filters, aggregations, and pagination. Medium to High (~10-100ms, based on complexity) Full-text search, structured queries, analytics. POST /{index}/_search/scroll Scroll Search Enables retrieving large datasets by scrolling through search results. Medium to High (~50-200ms) Pagination for large datasets, data exports. DELETE /_search/scroll Clear Scroll Context Clears scroll contexts to free up resources. Low (~5ms) Resource management after scroll search. POST /_msearch Multi-Search Allows execution of multiple search queries in a single request. Medium to High (~20-150ms) Batch querying, dashboard visualizations. POST /{index}/_count Count Documents Counts the documents matching a query without returning full results. Low (~5-20ms) Quick counts of filtered datasets. Aggregation APIs POST /{index}/_search Aggregation Queries Used with the search API to retrieve aggregated data (e.g., histograms, averages). Medium to High (~20-150ms) Analytics, reporting, data summarization. Cluster and Node APIs GET /_cluster/health Cluster Health Returns health information on the cluster, nodes, and indices. Low (~5ms) Monitoring cluster health and node status. GET /_cluster/stats Cluster Statistics Provides statistics on cluster status, node usage, and storage. Low to Medium (~5-20ms) Cluster-wide monitoring and performance analysis. POST /_cluster/reroute Cluster Reroute Manually reroutes shards in the cluster. Medium (~20-50ms) Shard management and rebalancing. GET /_nodes/stats Node Statistics Returns stats for nodes, including CPU, memory, and thread pools. Low to Medium (~5-20ms) Node health monitoring, resource usage analysis. GET /_cat/nodes List Nodes Provides a list of all nodes in the cluster in a human-readable format. Low (~5ms) Node overview, node status. GET /_cat/indices List Indices Lists all indices with metadata on size, health, and document count. Low (~5ms) Index management and monitoring. Index Management APIs PUT /{index} Create Index Creates a new index with specific settings and mappings. Low (~10-20ms) Index setup and schema definition. DELETE /{index} Delete Index Deletes the specified index. Low (~5-10ms) Index removal, data management. PUT /{index}/_settings Update Index Settings Updates settings (e.g., refresh interval) of an index. Low (~10ms) Dynamic adjustments of index settings. POST /{index}/_refresh Refresh Index Refreshes an index to make recent changes searchable. Medium (~10-30ms) Ensures data is available for search in near-real-time. POST /{index}/_forcemerge Force Merge Index Reduces the number of segments in an index to optimize storage. High (~100ms - 1s+) Optimize index storage, improve search speed. Cache and Memory Management POST /{index}/_cache/clear Clear Index Cache Clears the cache for the specified index. Low (~5ms) Cache management for performance tuning. POST /_flush Flush Index Writes all buffered changes to disk. Medium (~10-30ms) Data durability, cache clearing. POST /{index}/_refresh Refresh Makes recent changes to an index visible to search. Medium (~10-30ms) Near real-time updates in the search index. Snapshot and Backup APIs PUT /_snapshot/{repo}/{snapshot} Create Snapshot Creates a snapshot of indices for backup. High (dependent on index size) Data backup, disaster recovery. GET /_snapshot/{repo}/{snapshot} Get Snapshot Status Checks the status of an existing snapshot. Low (~5ms) Monitor snapshot progress, status checks. DELETE /_snapshot/{repo}/{snapshot} Delete Snapshot Deletes an existing snapshot. Medium (~10-20ms) Snapshot lifecycle management, freeing storage. Security and Role Management POST /_security/role/{role} Create/Update Role Creates or updates a security role with specific permissions. Low (~5-10ms) Access control, role-based permissions. POST /_security/user/{user} Create/Update User Creates or updates a user in Elasticsearch. Low (~5-10ms) User management, access permissions. GET /_security/_authenticate Authenticate User Authenticates the current user. Low (~5ms) Session management, authentication checks."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#additional-notes","title":"Additional Notes","text":"indices.fielddata.cache.size based on available heap Field data cache size impacts sorting and aggregation speed, stored in memory. Set to 20-30% of available heap for high-aggregation workloads Query Cache Size 10% of heap by default (adjustable) Caches filter queries to speed up repeated search operations. Increase for clusters with frequent repetitive queries Request Cache Size Enabled per index, LRU-based Cache complete search requests, especially useful for aggregation-heavy queries on static data. Enable on indices with frequent aggregation queries Cluster Health GET /_cluster/health (monitor green, yellow, red status) Regularly monitor cluster health to identify potential issues in shard allocation and node status. Alerts for yellow or red statuses to detect unallocated shards Pending Tasks GET /_cluster/pending_tasks Track the number of pending tasks; delays may signal node or cluster overload. Monitor to ensure tasks are processed promptly CPU Usage per Node Track via node.cpu.percent Monitor CPU load on nodes, especially data and coordinating nodes handling heavy queries. Keep below 80% for balanced performance Search Latency Monitor search.fetch_time and search.query_time Search latency metrics indicate performance bottlenecks; high values can suggest tuning is needed. Target <100ms for interactive queries, <500ms for aggregations Indexing Latency Monitor indexing.index_time Tracks indexing speed; high values indicate indexing bottlenecks. Optimize disk I/O if consistently high GC Pause Time Track jvm.gc.collection_time Excessive GC pause time (>100ms) can degrade performance, especially on data nodes. Keep heap usage <75% to avoid frequent GC pauses Disk Utilization disk.used_percent Ensure disk usage remains within optimal limits to prevent resource contention. Monitor for high usage, keep below 75% Heap Usage per Node jvm.heap_used_percent Monitor heap usage across nodes; values near 100% can trigger frequent GC and degrade performance. Keep below 75% for stable performance Shard Count per Node Shards should not exceed 50-75 per data node Optimal shard count balances memory usage and search latency. Distribute shards evenly to avoid bottlenecks Index Rollover Frequency Based on data ingestion and retention policy Use index rollover for high-ingestion use cases to manage shard size and count. Time-based or size-based rollover (e.g., daily, 10GB) Snapshot Frequency Schedule during off-peak hours Regular snapshots for backup without affecting active workloads. Daily or weekly snapshots for disaster recovery Ingest Node CPU Requirement Optimize for transformation-heavy workloads Ingest nodes need higher CPU for ETL tasks before indexing. ~8 CPUs per ingest node for transformation-heavy clusters Write Thread Pool Size Controlled via thread_pool.write.size Configure thread pool size for write-heavy workloads. Default based on available processors; increase for high-write loads Bulk Thread Pool Size Set via thread_pool.bulk.size Bulk operations often have separate thread pools, useful for high-ingestion clusters. Default based on processors; increase if high bulk ingestion Query Throughput Measure search.thread_pool.queue The search queue size indicates if the search load is too high, leading to delays. Keep queue size low to avoid bottlenecks Bulk Queue Size Monitor bulk.thread_pool.queue Large bulk queue size indicates ingestion pressure; tune for high-ingestion needs. Increase queue size or add ingest nodes for bulk-heavy workloads Network Throughput Monitor network interface utilization High network usage can impact inter-node communication, especially during replication. Ensure sufficient network bandwidth for large clusters Network Latency Track round-trip time between nodes Low-latency network is critical for distributed search and replication. <1ms recommended within data centers, 1-5ms for cross-regions"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#questions","title":"Questions","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-sql-or-nosql","title":"1. SQL or NoSQL:","text":"Kafka is a distributed event streaming platform that allows applications to publish, store, and consume streams of data in real-time or batch mode. It is designed to handle continuous streams of events or records by functioning as a distributed commit log, where data is written sequentially and can be read independently by multiple consumers.
Kafka follows a publish-subscribe model where producers write data to topics divided into partitions, and consumers pull data from those partitions at their own pace. Kafka ensures that data flows reliably between producers and consumers, with fault-tolerant replication and durable storage to prevent data loss.
At its heart, Kafka provides three core functionalities: 1. Message Streaming: Enabling systems to send and receive continuous streams of data asynchronously. 2. Durable Storage: Persisting messages on disk, ensuring data is not lost even in case of failures. 3. Distributed Processing: Allowing data to be partitioned and processed across multiple servers for scalability and fault tolerance.
"},{"location":"techdives/DistrubutedSystems/Kafka/#core-components-and-keywords","title":"Core Components and Keywords","text":"Component / Keyword Description Topic A logical channel for data, used to categorize messages. Topics are divided into partitions for parallelism. Partition A segment of a topic that stores messages in a log structure. It ensures parallel processing. Each partition contains messages with offsets to track their position. Producer A client or application that publishes messages to a Kafka topic. Producers can distribute messages across partitions. Consumer A client or application that subscribes to Kafka topics and reads messages from partitions at their own pace. Broker A Kafka server that stores messages, handles requests, and coordinates with other brokers in a Kafka cluster. Kafka Cluster A group of multiple Kafka brokers working together to provide scalability and fault tolerance. Zookeeper / KRaft Zookeeper is used for metadata management and leader election in older Kafka versions. Newer versions replace Zookeeper with KRaft (Kafka Raft) for native metadata management. Offset A unique identifier for each message within a partition, representing its position in the log. Consumers use offsets to track the messages they have processed. Replication Kafka duplicates partitions across multiple brokers to ensure fault tolerance and data availability. Leader Partition The primary replica of a partition that handles all reads and writes for that partition. Other replicas act as followers. Follower Partition A copy of the leader partition that replicates its data. If the leader fails, a follower can take over. Consumer Group A group of consumers sharing the same group ID, ensuring that each partition is consumed by only one member of the group at any given time. In-Sync Replica (ISR) A replica that is up-to-date with the leader partition. Kafka promotes an ISR as the new leader if the current leader fails. Acknowledgments (ACKs) A producer configuration that defines when a message is considered successfully sent (e.g., only after being replicated to all followers). Retention Policy A configuration that determines how long Kafka retains messages before deleting or compacting them. Messages can be removed based on time or size limits. Log Compaction A process that keeps only the latest version of a key within a topic, useful for data clean-up and long-term storage. Controller A designated broker responsible for managing partition leadership and cluster rebalancing. Kafka Streams A lightweight client library used for processing and analyzing data streams directly within Kafka. Kafka Connect A framework for integrating Kafka with external systems by providing connectors for data ingestion and extraction."},{"location":"techdives/DistrubutedSystems/Kafka/#how-kafka-achieves-high-scalability","title":"How Kafka Achieves High Scalability ?","text":"Kafka\u2019s design is fundamentally scalable, allowing it to handle millions of events per second efficiently. It achieves this by leveraging distributed architecture, partitioning, horizontal scaling, and several load balancing strategies. Let\u2019s explore the mechanisms, architecture, and workflows that enable Kafka to scale end-to-end.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-partitioning-the-foundation-of-scalability","title":"1. Partitioning: The Foundation of Scalability","text":"Consumers in a consumer group split the partitions among themselves for parallel processing.
When Kafka detects new brokers, partition leadership rebalances automatically to evenly distribute the load across brokers.
Key-based partitioning: Ensuring messages with the same key always go to the same partition for order guarantees.
Kafka producers also use batching to group multiple messages into a single request, reducing network overhead and increasing throughput.
Compression: Messages can be compressed (e.g., with Gzip or Snappy) to reduce the amount of data transferred over the network.
Kafka also makes use of page cache to keep frequently accessed data in memory, further optimizing performance.
Metadata updates: Ensuring all brokers have consistent metadata.
Dynamic partition leadership reassignment ensures that even when new brokers are added or partitions are migrated, Kafka distributes workload evenly.
Kafka\u2019s design for high availability and fault tolerance centers on replication, leader election, distributed brokers, and self-healing mechanisms. Together, these mechanisms ensure Kafka can handle hardware, software, and network failures, while maintaining data integrity, durability, and continuity.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-partition-replication-foundation-of-ha-and-ft","title":"1. Partition Replication \u2013 Foundation of HA and FT","text":"This automated and rapid leader election ensures Kafka remains operational with minimal interruption.
"},{"location":"techdives/DistrubutedSystems/Kafka/#3-in-sync-replicas-isr-ensuring-data-integrity","title":"3. In-Sync Replicas (ISR) \u2013 Ensuring Data Integrity","text":"acks=all).Kafka provides acknowledgment modes for tuning data durability against performance. These settings control when a message is considered \u201csuccessfully written\u201d:
Kafka employs log compaction and retention to maintain long-term data availability:
A split-brain scenario happens when a broker loses connectivity with others, risking data inconsistency.
"},{"location":"techdives/DistrubutedSystems/Kafka/#kafkas-approach-to-preventing-split-brain","title":"Kafka\u2019s Approach to Preventing Split-Brain:","text":"This quorum-based replication prevents data corruption during network partitions.
"},{"location":"techdives/DistrubutedSystems/Kafka/#9-self-healing-and-automated-recovery","title":"9. Self-Healing and Automated Recovery","text":"Kafka\u2019s self-healing mechanisms enable it to quickly recover from broker or replica failures:
These self-healing features maintain availability and data consistency without requiring manual intervention.
"},{"location":"techdives/DistrubutedSystems/Kafka/#10-multi-datacenter-replication-for-disaster-recovery","title":"10. Multi-Datacenter Replication for Disaster Recovery","text":"Kafka supports multi-datacenter replication for cross-region fault tolerance using tools like Kafka MirrorMaker.
"},{"location":"techdives/DistrubutedSystems/Kafka/#how-multi-cluster-replication-ensures-availability-and-fault-tolerance","title":"How Multi-Cluster Replication Ensures Availability and Fault Tolerance:","text":"Kafka\u2019s producers and consumers have built-in resilience to handle failures gracefully:
3 or 5) High High Ensures multiple copies of data across brokers; if the leader fails, a follower can be promoted, maintaining data availability and durability. In-Sync Replicas (ISR) Use acks=all to ensure ISR sync before acknowledgments Moderate High Guarantees data consistency by ensuring messages are replicated to all ISR replicas before confirming writes, reducing data loss. Leader Election Mechanism Managed by Zookeeper or KRaft for automatic failover High Moderate Automatically promotes a follower to leader when the current leader fails, minimizing downtime. Controller Broker Redundancy provided by re-election if the current controller fails High Moderate Ensures the orchestrator of metadata and rebalancing has a backup, maintaining consistent cluster operations. Distributed Broker Placement Spread partitions across brokers; no two replicas on the same broker High High Reduces the risk of data unavailability and loss by preventing single points of failure. Rebalancing Strategy Configure partition reassignment for balanced broker load High Low Prevents overload on individual brokers, enhancing availability; this has less impact on data durability. Acknowledgment Policy (ACKs) Set acks=all for highest data durability Low High Ensures writes are only confirmed after replication to all ISR replicas, reducing the risk of data loss. Log Compaction Enable for compacted topics to retain latest state Moderate Moderate Retains the latest state of each key, useful for stateful applications; supports recovery but doesn\u2019t guarantee availability. Retention Policies Configure time-based or size-based retention High Low Maintains historical data for consumer recovery, contributing to high availability if consumers fall behind. Client Retry Mechanisms Configure producer and consumer retries High Low Enables producers and consumers to handle temporary broker unavailability, ensuring continuous operation. Consumer Group Rebalancing Set rebalancing policies to avoid bottlenecks High Low Ensures efficient load distribution among consumers, enhancing availability but minimally impacting data durability. Multi-Datacenter Replication Enable with Kafka MirrorMaker or similar tools High High Provides cross-region redundancy for both availability and fault tolerance, especially critical for disaster recovery. Backpressure Handling Use offset tracking and monitoring Moderate High Allows consumers to fall behind producers without causing data loss, enhancing fault tolerance by protecting against backpressure failures. Split-Brain Handling Managed by Zookeeper/KRaft to avoid conflicting writes Low High Prevents data inconsistency by ensuring only one leader exists per partition, critical for consistency in partitioned network conditions. Log Recovery Enable brokers to rebuild from log segments on restart Moderate High Ensures brokers can recover their state after a crash, minimizing data loss and ensuring continuity post-restart. Kafka\u2019s architecture for high availability and fault tolerance ensures the system remains resilient under various failure scenarios. Through partition replication, leader election, dynamic rebalancing, and multi-datacenter replication, Kafka provides a robust infrastructure with no single points of failure and near-zero downtime, making it reliable for critical real-time data streaming and processing applications.
"},{"location":"techdives/DistrubutedSystems/Kafka/#what-makes-kafka-unique","title":"What Makes Kafka Unique ?","text":"Append only Log-Based Architecture and High-Throughput with Low-Latency Design two of Kafka\u2019s core features that make it unique.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-log-based-architecture-the-foundation-of-kafkas-data-model","title":"1. Log-Based Architecture: The Foundation of Kafka\u2019s Data Model","text":"Kafka\u2019s log-based architecture is what makes it fundamentally different from traditional messaging systems. It\u2019s built on the concept of a distributed, partitioned, and immutable log, allowing Kafka to scale, preserve data ordering, and enable consumers to replay data as needed. Here\u2019s a deep dive into what this architecture entails and why it\u2019s special.
"},{"location":"techdives/DistrubutedSystems/Kafka/#how-log-based-architecture-works","title":"How Log-Based Architecture Works","text":"Offsets serve as pointers to each message\u2019s position within the log, making it easy for consumers to track their progress.
Immutable Log Storage:
This immutability provides consistency and durability, as every message has a fixed position within the partition log.
Replayable and Persistent Data:
Efficient Ordering: Kafka maintains strict ordering within each partition since messages are written sequentially. This guarantees that messages within a partition are processed in the order they were produced, which is essential for applications like transaction logs and real-time event processing.
Parallelism and Scalability: By dividing topics into multiple partitions, Kafka enables horizontal scalability. Each partition can be processed independently by separate consumers, allowing for parallel data processing and high throughput across large datasets.
Fault Tolerance and Durability: The log-based structure, combined with replication (having multiple copies of each partition on different brokers), ensures that data remains durable and available even in the event of broker failures.
Kafka\u2019s design is optimized for handling millions of events per second with minimal delay, even under heavy load. This high throughput and low latency are achieved through a combination of disk I/O optimizations, data compression, and efficient network handling. Let\u2019s explore these in detail.
"},{"location":"techdives/DistrubutedSystems/Kafka/#key-components-of-kafkas-high-throughput-low-latency-design","title":"Key Components of Kafka\u2019s High-Throughput, Low-Latency Design","text":"Sequential writes take advantage of modern disks\u2019 ability to handle high-speed sequential I/O, especially in SSDs, allowing Kafka to process large volumes of data quickly.
Page Cache Usage:
For producers writing data, Kafka batches messages in memory before flushing them to disk, improving throughput by reducing the number of disk write operations.
Zero-Copy Data Transfer:
This allows Kafka to handle network I/O with minimal CPU usage, reducing latency and increasing throughput, especially for large messages.
Batching and Compression:
Batching not only increases efficiency but also improves compression rates by reducing network overhead. Kafka supports Gzip, Snappy, and LZ4 compression, reducing data size and thus speeding up data transfer.
Network Optimization for Low Latency:
Here are the organized into individual tables based on their sections. Each table provides a summary of the key settings along with recommended values and explanations.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-broker-level-settings","title":"1. Broker-Level Settings","text":"Setting Recommended Value Purposereplication.factor 3 Ensures redundancy and fault tolerance, allowing partition recovery if a broker fails. min.insync.replicas 2 Reduces data loss risk by ensuring at least two replicas acknowledge writes. num.partitions 6 (or based on load) Balances throughput and scalability; more partitions allow greater parallel processing. log.retention.hours 168 (7 days) Controls how long messages are retained; suitable for standard processing and replay needs. log.segment.bytes 1 GB Manages the segment size for optimal disk usage and performance in log rolling. log.cleanup.policy delete or compact delete for default; compact for retaining only the latest version of each key. compression.type producer, snappy, or lz4 Saves bandwidth and improves throughput, especially under high data volume conditions."},{"location":"techdives/DistrubutedSystems/Kafka/#2-producer-level-settings","title":"2. Producer-Level Settings","text":"Setting Recommended Value Purpose acks all Ensures that all in-sync replicas acknowledge writes, increasing reliability. retries Integer.MAX_VALUE Handles transient network issues by allowing indefinite retries, preventing message loss. retry.backoff.ms 100 Introduces a pause between retries, avoiding retry flooding and improving stability. enable.idempotence true Prevents duplicate messages by ensuring exactly-once semantics in data delivery. batch.size 32 KB Enhances throughput by accumulating records into batches before sending them. linger.ms 5 Small linger time allows batches to fill, reducing network overhead without delaying sends. compression.type snappy or lz4 Compresses data to reduce payload size, saving bandwidth and reducing transfer time."},{"location":"techdives/DistrubutedSystems/Kafka/#3-consumer-level-settings","title":"3. Consumer-Level Settings","text":"Setting Recommended Value Purpose auto.offset.reset earliest Ensures consumers start reading from the beginning if no offset is committed. max.poll.records 500 Controls the batch size per poll, balancing throughput and processing time. session.timeout.ms 30,000 Provides enough time for consumers to process data without triggering unnecessary rebalances. heartbeat.interval.ms 10,000 Sets the interval for heartbeat checks within the session timeout, reducing rebalance triggers. fetch.min.bytes 1 MB Improves fetch efficiency by waiting for a minimum data size before retrieving. fetch.max.bytes 50 MB Enables large batch sizes for high-throughput consumers, reducing network calls. enable.auto.commit false Disables automatic offset commits, allowing applications to commit only after processing."},{"location":"techdives/DistrubutedSystems/Kafka/#4-cluster-level-settings-for-high-availability-and-fault-tolerance","title":"4. Cluster-Level Settings for High Availability and Fault Tolerance","text":"Setting Recommended Value Purpose min.insync.replicas 2 Ensures at least two replicas must be in sync, providing better durability. controlled.shutdown.enable true Enables controlled shutdown, allowing brokers to gracefully transition leadership. unclean.leader.election.enable false Prevents out-of-sync replicas from being elected as leaders, protecting data consistency. num.network.threads 8 Increases concurrency for network traffic, supporting high-throughput applications. num.io.threads 8 Increases I/O concurrency, allowing efficient data transfer under heavy load. num.replica.fetchers 4 Enhances replication speed by allowing multiple fetcher threads for synchronizing replicas."},{"location":"techdives/DistrubutedSystems/Kafka/#5-zookeeper-settings","title":"5. Zookeeper Settings","text":"Setting Recommended Value Purpose zookeeper.session.timeout.ms 18,000 Prevents frequent Zookeeper disconnections during high loads, stabilizing metadata handling. zookeeper.connection.timeout.ms 6,000 Ensures reliable connections to Zookeeper, reducing the likelihood of leader election issues."},{"location":"techdives/DistrubutedSystems/Kafka/#6-kraft-kafka-raft-settings","title":"6. KRaft (Kafka Raft) Settings","text":"Setting Recommended Value Purpose process.roles broker,controller Defines the roles of Kafka nodes. A KRaft cluster typically has combined broker and controller roles, but can be split if desired. controller.quorum.voters List of controllers Specifies the list of controller nodes in the form nodeID@hostname:port, where each entry represents a voter in the Raft consensus group. controller.listener.names CONTROLLER Designates the listener name for inter-controller communication in the Raft quorum. controller.heartbeat.interval.ms 2,000 Sets the interval between heartbeats for controller nodes, ensuring they stay connected and responsive within the Raft quorum. controller.metrics.sample.window.ms 30,000 Configures the window size for collecting metrics, helping to monitor Raft performance over time. controller.log.dirs /path/to/controller/logs Specifies the directory where controller logs are stored. It\u2019s best to use a dedicated disk for controller logs to avoid I/O contention with brokers. metadata.log.segment.bytes 1 GB Controls the segment size for metadata logs, managing disk usage and log rolling frequency for metadata in KRaft mode. metadata.log.retention.bytes -1 (unlimited) Configures metadata log retention based on disk space, allowing infinite retention by default. Adjust based on available storage. metadata.log.retention.ms 604,800,000 (7 days) Retains metadata for a set duration; typically configured for a week to enable rollback in case of issues. controller.socket.timeout.ms 30,000 Sets the timeout for controller-to-controller connections, ensuring stability during network issues. leader.imbalance.check.interval.seconds 300 (5 minutes) Defines the interval at which the controller checks for leader imbalance, helping to maintain even load distribution across brokers. Reference Links:
https://www.hellointerview.com/learn/system-design/deep-dives/kafka
"},{"location":"techdives/DistrubutedSystems/Redis/","title":"Redis","text":"Redis is an open-source, in-memory data structure store that serves as a database, cache, message broker, and streaming engine. Its versatility and high performance make it a popular choice for various applications. This comprehensive guide delves into Redis's architecture, data structures, commands, deployment strategies, and best practices.
"},{"location":"techdives/DistrubutedSystems/Redis/#1-introduction-to-redis","title":"1. Introduction to Redis","text":"Redis, short for Remote Dictionary Server, is renowned for its speed and flexibility. Operating primarily in memory, it supports various data structures and offers features like replication, persistence, and high availability.
"},{"location":"techdives/DistrubutedSystems/Redis/#2-core-data-structures","title":"2. Core Data Structures","text":""},{"location":"techdives/DistrubutedSystems/Redis/#21-strings","title":"2.1. Strings","text":"Definition: A string in Redis is a binary-safe sequence of bytes. It's the most basic data type in Redis, with a maximum size of 512 MB.
Examples: - Set and get a simple string:
SET key \"Hello, World!\"\nGET key\nSET counter 10\nINCR counter // Result: 11\nINCRBY counter 5 // Result: 16\nUnderlying Data Structure: Dynamic String (SDS - Simple Dynamic String)
Time Complexity: - SET key value: O(1) - GET key: O(1) - INCR key: O(1)
Best Use Cases: - Caching: Store frequently accessed values. - Counters: Track counts for metrics or events. - Session Data: Store serialized JSON for user sessions.
"},{"location":"techdives/DistrubutedSystems/Redis/#22-hashes","title":"2.2. Hashes","text":"Definition: A hash in Redis is a collection of field-value pairs, ideal for representing objects (e.g., user profiles).
Examples: - Creating and managing user data in a hash:
HSET user:1001 name \"Alice\" age 30 city \"New York\"\nHGET user:1001 name // Returns \"Alice\"\nHGETALL user:1001 // Returns all key-value pairs in hash\nUnderlying Data Structure: Hash Table or ZipList (for small hashes)
Time Complexity: - HSET key field value: O(1) (amortized) - HGET key field: O(1) - HGETALL key: O(N) (N being the number of fields in the hash)
Best Use Cases: - Storing Objects: Represent complex entities. - Configuration Settings: Store configurations as a set of key-value pairs.
"},{"location":"techdives/DistrubutedSystems/Redis/#23-lists","title":"2.3. Lists","text":"Definition: A list is an ordered collection of strings, allowing elements to be added at either end, functioning much like a linked list.
Examples: - Using a list to store recent activity:
LPUSH recent_activity \"login\" \"view_profile\" \"logout\"\nLRANGE recent_activity 0 -1 // Fetches all elements in the list\nUnderlying Data Structure: Linked List or QuickList (optimized for performance and memory usage)
Time Complexity: - LPUSH key value: O(1) - LRANGE key start stop: O(S+N) (S being the starting offset and N the number of elements retrieved)
Best Use Cases: - Activity Streams: Store recent actions or logs. - Task Queues: Implement FIFO or LIFO queues.
"},{"location":"techdives/DistrubutedSystems/Redis/#24-sets","title":"2.4. Sets","text":"Definition: Sets are unordered collections of unique strings, ideal for performing set operations.
Examples: - Managing unique tags:
SADD tags \"redis\" \"database\" \"in-memory\"\nSMEMBERS tags // Returns all unique tags\nUnderlying Data Structure: Hash Table or IntSet (for small sets of integers)
Time Complexity: - SADD key value: O(1) - SMEMBERS key: O(N) (N being the number of elements in the set) - SINTER key1 key2 ... keyN: O(N*M) (N being the number of sets and M the smallest set)
Best Use Cases: - Unique Values: Track unique items like IP addresses. - Social Networks: Represent social relationships (e.g., friends, followers).
"},{"location":"techdives/DistrubutedSystems/Redis/#25-sorted-sets","title":"2.5. Sorted Sets","text":"Definition: Similar to sets, but with an associated score that allows elements to be sorted by score.
Examples: - Storing leaderboard data:
ZADD leaderboard 100 \"Alice\" 200 \"Bob\"\nZRANGE leaderboard 0 -1 WITHSCORES\nUnderlying Data Structure: Skip List and Hash Table (for fast access and sorted ordering)
Time Complexity: - ZADD key score member: O(log(N)) - ZRANGE key start stop: O(log(N)+M) (M being the number of elements returned)
Best Use Cases: - Leaderboards: Rank users based on scores. - Event Prioritization: Sort items by priority or timestamp.
"},{"location":"techdives/DistrubutedSystems/Redis/#26-bitmaps","title":"2.6. Bitmaps","text":"Definition: Bitmaps use strings to store and manipulate individual bits, offering efficient binary storage.
Examples: - Tracking user flags:
SETBIT user_flags 5 1 // Sets the 6th bit to 1\nGETBIT user_flags 5 // Returns 1\nUnderlying Data Structure: String (each bit is set or retrieved from the byte representation)
Time Complexity: - SETBIT key offset value: O(1) - GETBIT key offset: O(1) - BITCOUNT key: O(N) (N being the length of the string)
Best Use Cases: - Feature Flags: Toggle features for users. - Activity Tracking: Record binary states like presence or attendance.
"},{"location":"techdives/DistrubutedSystems/Redis/#27-hyperloglogs","title":"2.7. HyperLogLogs","text":"Definition: HyperLogLog is a probabilistic structure for approximating unique element counts.
Examples: - Counting unique visitors:
PFADD visitors \"user1\" \"user2\"\nPFCOUNT visitors // Returns approximate unique count\nUnderlying Data Structure: Sparse and Dense Data Representations (optimized for low memory usage)
Time Complexity: - PFADD key element: O(1) - PFCOUNT key: O(1)
Best Use Cases: - Unique Counting: Approximate counts of unique views or visitors. - Low-Memory Use: Ideal for large datasets with memory constraints.
"},{"location":"techdives/DistrubutedSystems/Redis/#28-streams","title":"2.8. Streams","text":"Definition: A stream is a log-like data structure for managing continuous data flows, supporting consumer groups.
Examples: - Tracking event streams:
XADD mystream * name \"Alice\" action \"login\"\nXREAD COUNT 2 STREAMS mystream 0\nUnderlying Data Structure: Radix Tree (used for efficient storage and traversal of stream entries)
Time Complexity: - XADD key * field value: O(log(N)) (N being the number of items in the stream) - XREAD key start stop: O(log(N)+M) (M being the number of items returned)
Best Use Cases: - Event Sourcing: Track ordered events or logs. - Message Queues: Reliable message distribution with consumer groups.
"},{"location":"techdives/DistrubutedSystems/Redis/#29-geospatial-indexes","title":"2.9. Geospatial Indexes","text":"Definition: Redis provides commands for storing and querying location data with latitude and longitude.
Examples: - Adding and querying locations:
GEOADD cities 13.361389 38.115556 \"Palermo\"\nGEORADIUS cities 15 37 200 km\nUnderlying Data Structure: Geohash with Sorted Sets (uses sorted sets for indexing)
Time Complexity: - GEOADD key longitude latitude member: O(log(N)) - GEORADIUS key longitude latitude radius: O(log(N)+M) (M being the number of results)
Best Use Cases: - Location-Based Services: Search and display nearby locations. - Geofencing: Detect whether users enter specific geographic zones.
"},{"location":"techdives/DistrubutedSystems/Redis/#3-commands-table","title":"3. Commands Table","text":"Data Structure Definition Example Commands Best Use Cases Time Complexity Strings Binary-safe sequences of bytes for text or binary data.SET key value, GET key, INCR key, DECR key, APPEND key value, STRLEN key Caching, counters, session data SET: O(1), GET: O(1), INCR: O(1), APPEND: O(N) Hashes Collection of key-value pairs, suitable for objects. HSET key field value, HGET key field, HGETALL key, HDEL key field, HLEN key Storing objects, configuration settings HSET: O(1), HGET: O(1), HGETALL: O(N), HLEN: O(1) Lists Ordered collection of strings, acts like a linked list. LPUSH key value, RPUSH key value, LPOP key, RPOP key, LRANGE key start stop, LLEN key Activity streams, task queues LPUSH: O(1), LRANGE: O(S+N), LLEN: O(1) Sets Unordered collections of unique strings, optimized for sets. SADD key value, SREM key value, SMEMBERS key, SISMEMBER key value, SUNION key1 key2, SINTER key1 key2 Unique values, social relationships SADD: O(1), SMEMBERS: O(N), SINTER: O(N*M) Sorted Sets Sets with scores, allowing elements to be sorted by score. ZADD key score member, ZRANGE key start stop WITHSCORES, ZREM key member, ZSCORE key member, ZREVRANGE key start stop, ZCOUNT key min max Leaderboards, event prioritization ZADD: O(log(N)), ZRANGE: O(log(N)+M), ZSCORE: O(1) Bitmaps Stores and manipulates bits in a binary-safe string. SETBIT key offset value, GETBIT key offset, BITCOUNT key, BITOP operation destkey key1 key2 Feature flags, activity tracking SETBIT: O(1), GETBIT: O(1), BITCOUNT: O(N) HyperLogLogs Probabilistic structure for approximate unique counts. PFADD key element, PFCOUNT key, PFMERGE destkey sourcekey1 sourcekey2 Unique counting, low-memory usage PFADD: O(1), PFCOUNT: O(1), PFMERGE: O(N) Streams Log-like structure for managing continuous data flows. XADD key * field value, XREAD COUNT n STREAMS key, XGROUP CREATE key group consumer_id, XACK key group message_id, XDEL key message_id, XINFO key, XLEN key, XTRIM key MAXLEN ~ count Event sourcing, message queues XADD: O(log(N)), XREAD: O(log(N)+M), XGROUP: O(1), XACK: O(1) Geospatial Indexes Stores and queries location data with latitude and longitude. GEOADD key longitude latitude member, GEODIST key member1 member2, GEORADIUS key longitude latitude radius m km, GEORADIUSBYMEMBER key member radius m km, GEOHASH key member Location-based services, geofencing GEOADD: O(log(N)), GEORADIUS: O(log(N)+M)"},{"location":"techdives/DistrubutedSystems/Redis/#4-persistence-and-durability","title":"4. Persistence and Durability","text":"Redis operates primarily as an in-memory database, prioritizing speed and low-latency operations. However, it provides two main persistence mechanisms to ensure data durability:
Advantages: Lower I/O overhead, compact file size.
Disadvantages: Risk of data loss between snapshots if Redis crashes.
SET or INCR) received by the server, appending each to a file. This approach provides a more durable form of persistence because every operation is logged in real-time or semi-real-time.Advantages: Better durability, logs every operation for more frequent data persistence.
Disadvantages: Larger file sizes, higher I/O usage.
Choosing Between RDB and AOF: You can use either or both of these methods in combination based on your application needs. For example, using both allows for rapid recovery (RDB) with high durability (AOF).
"},{"location":"techdives/DistrubutedSystems/Redis/#5-replication-and-high-availability","title":"5. Replication and High Availability","text":"Redis supports replication to create replicas of the primary (master) instance, enabling multiple read replicas and providing redundancy.
Master-Replica Replication: A master Redis instance can replicate data to any number of replica instances, which can then serve read-only queries. This setup enhances scalability by spreading read requests across replicas and ensuring data availability in case the primary instance fails.
Redis Sentinel: Redis Sentinel is a high-availability solution that monitors Redis instances. It automatically promotes one of the replicas to the master role if the current master goes down, providing automatic failover and notifications. Sentinel also provides monitoring and notifications.
Best for: - Applications requiring high availability with automatic failover. - Scenarios where read-heavy workloads benefit from scaling reads across multiple replicas.
"},{"location":"techdives/DistrubutedSystems/Redis/#6-clustering-and-scalability","title":"6. Clustering and Scalability","text":"Redis supports sharding through Redis Cluster, which enables data partitioning across multiple nodes, allowing horizontal scalability and distributed storage. Redis Cluster uses hash slots to determine data distribution across nodes, ensuring no single node contains the entire dataset.
Hash Slot Mechanism: Redis Cluster divides the keyspace into 16,384 hash slots, assigning a subset of these slots to each node. Keys are then hashed into these slots, which determines their storage node.
Hash Slot Mapping: Redis Cluster uses 16,384 slots as a way to divide the keyspace. Every key is assigned to a specific slot by applying a hashing function (called CRC16) that maps the key to one of these 16,384 slots.
Multiple Keys per Slot: Each hash slot can contain many keys. This means you can store millions of keys, not just 16,384.
Distribution Across Nodes: In a Redis Cluster, each node is responsible for a subset of the 16,384 slots. For example:
Redis assigns keys to slots, and the slot location determines which node will store the key.
Dynamic Scaling: When more nodes are added to the cluster, the slots are rebalanced, and some slots are reassigned to the new nodes. Redis automatically adjusts the key distribution to accommodate the additional nodes, enabling the cluster to scale horizontally without needing to rewrite keys.
Automatic Failover: Redis Cluster supports failover, ensuring high availability by electing a new primary if a node fails.
Considerations: - Redis Cluster supports most single-key commands, but multi-key operations are restricted unless all keys map to the same slot. - Clustering can increase complexity in handling data operations across nodes but is essential for large datasets needing horizontal scalability.
"},{"location":"techdives/DistrubutedSystems/Redis/#7-security-considerations","title":"7. Security Considerations","text":"While Redis is often deployed in secure, private networks, security remains essential, especially for production environments:
AUTH command. Although not enabled by default, setting up AUTH is recommended for production instances.bind 127.0.0.1) and accessed through trusted networks or VPNs, especially if running on public cloud services. Redis supports setting bind and requirepass configuration directives.Redis has a robust ecosystem with libraries for popular programming languages, making it easy to integrate Redis across platforms:
redis-pyJedis, Lettuceioredis, node-redisgo-redisAdditionally, Redis CLI and Redis Insight are commonly used tools for managing and monitoring Redis instances.
"},{"location":"techdives/DistrubutedSystems/Redis/#9-rediss-single-threaded-nature-and-atomic-operations","title":"9. Redis\u2019s Single-Threaded Nature and Atomic Operations","text":"Redis uses a single-threaded event loop to handle client requests, which makes operations simpler and efficient. This single-threaded model has specific implications:
Atomic Operations: All Redis commands are atomic because Redis processes each command sequentially. Even if multiple clients issue commands simultaneously, Redis processes one command at a time. This ensures that operations like INCR or multi-step transactions are inherently thread-safe and don't require locks.
Efficiency: Since Redis is single-threaded, it avoids the complexity of thread management and the overhead of context switching, which helps maintain its high-speed performance. Redis\u2019s speed is due to efficient data structures, in-memory storage, and minimal CPU usage.
Impact on Use Cases: - In scenarios where atomicity is essential, such as counters or distributed locks, Redis's single-threaded nature provides strong consistency guarantees. - For CPU-bound workloads, Redis may be limited by its single-threaded design. However, since Redis is primarily I/O-bound, it scales well for read-heavy or network-intensive applications.
Benefits of Single-Threaded Execution: - Simplicity in design and implementation, eliminating race conditions. - Predictable performance with guaranteed atomicity.
Drawbacks: - Limited to single-core processing for request handling. For CPU-bound tasks, Redis's single-threading may become a bottleneck, but horizontal scaling (e.g., Redis Cluster) can help distribute the load.
"},{"location":"techdives/DistrubutedSystems/Redis/#10-approximate-requests-per-second-rps","title":"10. Approximate Requests Per Second (RPS)","text":"Operation Type Description Approximate RPS per Instance Notes Simple Reads (GET) Basic read operation for retrieving a single value 100,000 - 150,000 RPS Higher performance achievable on optimized hardware Simple Writes (SET) Basic write operation for setting a single key-value pair 100,000 - 150,000 RPS Slightly reduced if using AOF with always persistence Complex Reads (e.g., ZRANGE) Reads on complex data structures like sorted sets 50,000 - 80,000 RPS Lower due to additional computation and memory access Complex Writes (e.g., ZADD) Writes on complex data structures like sorted sets 50,000 - 80,000 RPS Additional processing to maintain sorted order impacts performance With AOF (Append-Only File) Writes with always mode persistence (AOF) 60,000 - 80,000 RPS Slightly reduced due to disk I/O overhead Snapshotting (RDB) Writes with periodic snapshots (RDB) 80,000 - 100,000 RPS Minimal impact on RPS except during snapshotting periods when CPU/I/O load is higher With Redis Cluster Distributed across multiple nodes Millions of RPS (scales with nodes) Redis Cluster allows horizontal scaling, increasing RPS proportionally with additional nodes"},{"location":"techdives/DistrubutedSystems/Redis/#notes","title":"Notes:","text":"Redis is highly effective as a caching layer, providing extremely low-latency data retrieval that reduces the load on backend databases and improves application performance.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works","title":"How It Works","text":"Configuration settings
Expiration and Eviction: Redis supports configurable expiration for keys, which allows cached data to expire after a specific time. It also supports eviction policies (like LRU or LFU) to automatically remove older or less-used items when memory limits are reached.
Set Up Caching Logic: In your application, implement caching logic where data is first checked in Redis. If the key doesn\u2019t exist in Redis (cache miss), retrieve it from the database, store it in Redis, and serve it to the client.
if not redis.exists(\"user:123\"):\n data = fetch_from_database(\"user:123\")\n redis.setex(\"user:123\", 3600, data) # Cache for 1 hour\nExample Commands:
SETEX key value TTL # Sets a value with an expiration time (TTL)\nGET key # Retrieves value\nDEL key # Deletes a cached key\nRedis is commonly used as a session store for web applications, especially in distributed environments where sharing session data across multiple servers is critical.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_1","title":"How It Works","text":"Set Up Session Key: For each user session, generate a unique session ID and store session data in Redis.
HSET session:abc123 user_id 456 name \"Alice\"\nEXPIRE session:abc123 1800 # Session expires after 30 minutes\nExample Commands:
HSET session:<session_id> field value # Sets session data as a hash\nHGETALL session:<session_id> # Retrieves all session data\nEXPIRE session:<session_id> ttl # Sets expiration for session\nRedis\u2019s support for data structures like HyperLogLogs, sorted sets, and streams enables it to handle real-time analytics, tracking metrics, counts, and trends without requiring a traditional database.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_2","title":"How It Works","text":"HyperLogLog for Unique Visitors:
PFADD visitors:today user123 # Adds a unique user\nPFCOUNT visitors:today # Gets approximate unique visitor count\nSorted Set for Leaderboards:
ZADD leaderboard 1000 user123 # Adds user with score\nZREVRANGE leaderboard 0 10 WITHSCORES # Retrieves top 10 users\nStream for Event Tracking:
XADD events * type \"click\" user \"user123\" # Adds event to stream\nXREAD COUNT 10 STREAMS events 0 # Reads 10 most recent events\nRedis\u2019s publish/subscribe (pub/sub) feature allows it to act as a lightweight message broker, facilitating real-time communication between distributed applications.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_3","title":"How It Works","text":"Set Up Publisher:
PUBLISH channel_name \"New user registered\" # Publishes a message to a channel\nSet Up Subscriber:
SUBSCRIBE channel_name # Subscribes to a channel\nPublish a message:
PUBLISH notifications \"User123 has logged in\"\nSubscribe to channel:
SUBSCRIBE notifications\nRedis provides geospatial commands that make it suitable for applications requiring location-based searches and geofencing, such as ride-sharing or delivery tracking.
Redis uses a geohashing-like approach to handle geospatial data, but it combines it with sorted sets to enable efficient location-based queries, Redis converts latitude and longitude coordinates into a geohash-like value, which is then stored as a score in a sorted set. This encoding allows Redis to store location data compactly and enables efficient proximity queries.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_4","title":"How It Works","text":"GEOADD to add locations with latitude, longitude, and an associated member (e.g., user ID or landmark).Store Locations:
GEOADD cities 13.361389 38.115556 \"Palermo\"\nGEOADD cities 15.087269 37.502669 \"Catania\"\nRetrieve Locations in Radius:
GEORADIUS cities 15 37 100 km # Finds cities within 100 km of coordinates (15, 37)\nGEORADIUSBYMEMBER cities \"Palermo\" 200 km # Finds locations within 200 km of \"Palermo\"\nAdd a Location:
GEOADD locations 40.7128 -74.0060 \"New York\"\nFind Nearby Locations:
GEORADIUS locations -74.0060 40.7128 50 km\nSETEX, GET, DEL Reduces latency, lowers database load Session Management Store user sessions for distributed web applications. HSET, HGETALL, EXPIRE Fast, centralized session access across servers Real-Time Analytics Track metrics, counts, and trends in real time. PFADD, PFCOUNT, ZADD, XADD, XREAD Provides instant insights, reduces need for dedicated platforms Message Brokering Facilitate real-time communication between services. PUBLISH, SUBSCRIBE Real-time updates, lightweight message broker Geospatial Apps Perform location-based searches and calculations. GEOADD, GEORADIUS, GEORADIUSBYMEMBER Efficient geospatial operations for location-based services"},{"location":"techdives/DistrubutedSystems/Redis/#12-redis-issues","title":"12. Redis Issues","text":"Let's dive deep into some key challenges, such as hot key issues, cache avalanche, cache penetration, cache stampede, and their corresponding solutions.
"},{"location":"techdives/DistrubutedSystems/Redis/#121-hot-key-issue","title":"12.1. Hot Key Issue","text":""},{"location":"techdives/DistrubutedSystems/Redis/#description","title":"Description","text":"A hot key issue occurs when a single key in Redis is accessed extremely frequently, causing uneven load distribution. This can happen in applications where certain data (e.g., a trending topic or popular product) is heavily requested. A hot key can overwhelm the Redis server or specific nodes in a Redis Cluster, leading to latency spikes and reduced performance.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes","title":"Causes","text":"Store multiple copies of the hot key in Redis (e.g., hotkey_1, hotkey_2, hotkey_3). Then, use application logic to randomly pick a replica each time the key is accessed. This distributes the load across multiple keys.
Use Redis Cluster:
In a Redis Cluster, distribute the load by sharding hot keys across nodes. This may not completely eliminate the issue, but it can help mitigate its impact by spreading access across the cluster.
Client-Side Caching:
Implement a local cache on the client side or within the application servers to reduce the frequency of requests to Redis. This technique works well when the data is static or changes infrequently.
Use a Load-Balancing Proxy:
A cache avalanche occurs when many cache keys expire at once, leading to a sudden flood of requests to the backend database as the cache misses accumulate. This can overwhelm the database, causing latency spikes or even outages.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_1","title":"Causes","text":"Set expiration times with a randomized offset (e.g., add a few seconds or minutes randomly) to avoid simultaneous expiry. For example:
ttl = 3600 + random.randint(-300, 300) # 3600 seconds +/- 5 minutes\nCache Pre-Warming:
Preload critical data into Redis before it expires. You can use background jobs to check key expiration and refresh data periodically.
Lazy Loading with Synchronized Locking:
Use a distributed locking mechanism to ensure that only one thread refreshes the data in Redis, while others wait. This can prevent multiple processes from overloading the backend database.
Fallback Graceful Degradation:
Cache penetration happens when requests for non-existent keys repeatedly bypass the cache and go to the backend database. Since these keys don\u2019t exist, they are never cached, resulting in continuous database requests, increasing the load on the database.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_2","title":"Causes","text":"When a request results in a database miss, store a null value in Redis with a short TTL (e.g., 5 minutes). Future requests for the same key will hit Redis instead of the database. Example:
if not redis.exists(\"non_existent_key\"):\n data = fetch_from_database(\"non_existent_key\")\n if data is None:\n redis.setex(\"non_existent_key\", 300, None) # Cache null for 5 minutes\nInput Validation:
Filter out clearly invalid requests before querying Redis or the backend. For instance, if certain key patterns are obviously invalid, ignore them early in the request flow.
Bloom Filter:
A cache stampede occurs when multiple threads or clients attempt to update an expired cache key simultaneously, causing a burst of requests to the backend database. This is similar to a cache avalanche but occurs at the key level rather than across all keys.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_3","title":"Causes","text":"Use a distributed lock (e.g., Redlock) to ensure that only one client refreshes the cache while others wait. This reduces the load on the database:
# Pseudocode for acquiring a lock\nif redis.setnx(\"lock:key\", 1):\n try:\n # Fetch and cache the data\n data = fetch_from_database(\"key\")\n redis.setex(\"key\", 3600, data)\n finally:\n redis.delete(\"lock:key\") # Release the lock\nEarly Re-Caching (Soft Expiration):
Implement soft expiration by setting a short expiration on frequently requested keys and refreshing them asynchronously before they expire. This keeps the data fresh in Redis and avoids a stampede.
Leverage Stale Data:
Redis operates in memory, so it has a limited capacity. When Redis reaches its memory limit, it must evict keys to free up space, potentially removing critical data. Improper eviction policies can lead to cache churn and data inconsistency.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_4","title":"Causes","text":"Redis offers multiple eviction policies (noeviction, allkeys-lru, volatile-lru, allkeys-lfu, etc.). Choose one that matches your data access patterns. For instance:
Optimize Data Size:
Reduce the memory footprint by optimizing data storage, such as using shorter key names or serializing data efficiently (e.g., storing integers directly rather than as strings).
Monitor and Scale:
Continuously monitor Redis memory usage with tools like Redis CLI or Redis Insights. If memory usage grows, consider horizontal scaling with Redis Cluster.
Use Redis as a Pure Cache:
Redis is designed for fast access, but certain operations can cause high latency, especially when handling large datasets or complex commands like ZRANGE on large sorted sets.
Avoid commands that can block or are computationally expensive. For example, break large list processing into smaller ranges instead of processing the entire list.
Monitor Slow Queries:
Use the Redis Slow Log to identify and optimize slow commands. Redis provides insights into commands that exceed a specified execution time threshold.
Use Sharding or Clustering:
allkeys-lru, allkeys-lfu, etc.). - Compress data (e.g., shorter key names). - Use Redis MEMORY USAGE to check memory footprint of specific keys. INFO memory, MEMORY USAGE CPU Utilization CPU load on the Redis server, indicating overall processing load. - Reduce CPU-intensive commands (ZRANGE on large sets, large LRANGE operations). - Offload tasks to background or batch processing if possible. - Use pipelining for batch operations. System tools (e.g., top), INFO cpu Cache Hit Ratio The ratio of cache hits to total cache requests (ideally close to 1). - Identify hot keys and cache them effectively. - Increase Redis memory if hit ratio is low due to evictions. - Ensure sufficient TTL to avoid frequent cache misses. INFO stats Evicted Keys Number of keys evicted due to memory limits. - Increase available memory if eviction is high. - Choose an appropriate eviction policy (allkeys-lru, volatile-ttl, etc.). - Adjust key TTLs to prevent frequent eviction of important data. INFO memory Connected Clients Number of clients connected to Redis at any given time. - Increase the maxclients configuration if reaching limits. - Use client-side caching to reduce load on Redis. INFO clients, CLIENT LIST Latency (Command Time) Measures the average response time per command in milliseconds. - Avoid using blocking or heavy commands on large data sets. - Distribute large data across a Redis Cluster. - Monitor slow log for commands that exceed expected time. SLOWLOG GET, INFO commandstats Command Rate Rate of commands per second, which affects overall performance. - Spread load across multiple Redis instances if command rate is high. - Use pipelining to reduce round-trips. - Optimize or reduce the frequency of unnecessary commands. INFO stats Key Expirations Number of keys that expire per second. - Add randomized TTLs to prevent cache avalanches. - Pre-warm critical keys to avoid sudden cache misses. - Monitor TTL settings to ensure balanced expiration. INFO stats Replication Lag Delay in data synchronization between master and replica nodes. - Tune repl-backlog-size for better sync reliability. - Monitor network latency and throughput between master and replica. - Use Redis Sentinel for reliable failover. INFO replication, REPLCONF Data Persistence Durability How frequently Redis saves data to disk (AOF/RDB). - Use RDB for infrequent snapshots; use AOF for higher durability. - Tune AOF rewrite frequency (auto-aof-rewrite-percentage). - Adjust RDB save intervals based on data criticality. CONFIG SET save, CONFIG SET appendonly Keyspace Misses Number of attempts to access non-existent keys. - Cache null values temporarily for non-existent keys to reduce misses. - Add input validation to filter invalid requests. - Use Bloom filters for non-existent keys in high-traffic systems. INFO stats, MEMORY USAGE Redis Slow Log Logs slow-running commands that exceed a threshold. - Use SLOWLOG to monitor commands that exceed time limits. - Adjust commands and optimize keys based on slow log findings. - Tune slowlog-log-slower-than to track performance bottlenecks. SLOWLOG GET, SLOWLOG RESET Network Bandwidth Measures bandwidth usage, impacting latency and speed. - Use Redis clustering to reduce network load on a single instance. - Enable pipelining and compression where possible. - Monitor and minimize network latency for high-frequency queries. System tools (e.g., ifconfig), INFO Eviction Policy Determines which keys Redis evicts first when memory limit is reached. - Choose policies based on use case (allkeys-lru, allkeys-lfu for caching, volatile-ttl for expiring keys first). - Regularly review and adjust TTLs for key eviction optimization. CONFIG SET maxmemory-policy, INFO memory Persistence Overhead Memory and CPU impact due to persistence settings (RDB or AOF). - Adjust save intervals or AOF rewriting to reduce persistence load. - Use a combination of AOF and RDB if the application requires high durability with performance. INFO persistence, CONFIG SET save Cluster Slot Utilization Measures how well data is balanced across Redis Cluster slots. - Rebalance slots if certain nodes handle disproportionate load. - Use Redis Cluster sharding to ensure balanced key distribution. - Regularly monitor slots and reshard as needed. CLUSTER INFO, CLUSTER NODES, CLUSTER REBALANCE"},{"location":"techdives/DistrubutedSystems/Redis/#14-best-practices","title":"14. Best Practices","text":"To make the most of Redis:
LRU, LFU, etc.) to manage memory limits. Redis provides a range of policies for automatically removing old keys.Here\u2019s a structured Q&A-style deep dive into Redis to address all of these technical aspects.
"},{"location":"techdives/DistrubutedSystems/Redis/#1-sql-or-nosql-if-nosql-what-type-of-nosql","title":"1. SQL or NoSQL? If NoSQL, what type of NoSQL?","text":"Q: Is Redis an SQL or NoSQL database?
A: Redis is a NoSQL database. Specifically, it is a key-value store that supports various data structures (e.g., strings, hashes, lists, sets, sorted sets, streams, bitmaps, and geospatial indexes).
"},{"location":"techdives/DistrubutedSystems/Redis/#2-type-of-db-supports-polymorphic","title":"2. Type of DB \u2026 Supports Polymorphic?","text":"Q: What type of NoSQL database is Redis, and does it support polymorphism?
A: Redis is a key-value in-memory data store with support for a variety of data structures. Redis does not natively support polymorphic types in the way that document-based NoSQL databases do, but you can achieve some level of polymorphism by encoding data in a structured way (e.g., JSON or hash maps).
"},{"location":"techdives/DistrubutedSystems/Redis/#3-main-feature-db-built-for-and-who-built-it-and-on-what","title":"3. Main Feature, DB Built For, and Who Built It and on What","text":"Q: What was Redis built for, who built it, and what are its main features?
A: Redis was initially created by Salvatore Sanfilippo as a high-performance in-memory database for use cases requiring low-latency, real-time data processing. Redis is built on C, and its main features include in-memory storage, data persistence, flexible data structures, and capabilities for caching, messaging, and real-time analytics.
"},{"location":"techdives/DistrubutedSystems/Redis/#4-olap-or-oltp-does-it-support-acid-or-base","title":"4. OLAP or OLTP? Does it support ACID or BASE?","text":"Q: Is Redis OLAP or OLTP, and does it adhere to ACID or BASE properties?
A: Redis is generally used in OLTP (Online Transaction Processing) scenarios due to its low-latency and high-throughput design. Redis does not natively support full ACID properties but can achieve atomic operations within individual commands due to its single-threaded nature. It follows the BASE (Basically Available, Soft state, Eventual consistency) model.
"},{"location":"techdives/DistrubutedSystems/Redis/#5-cap-theorem-where-does-redis-fall","title":"5. CAP Theorem \u2013 Where does Redis fall?","text":"Q: How does Redis align with the CAP theorem, and what does each part (Consistency, Availability, Partition Tolerance) mean?
A: Redis, especially in a clustered setup, adheres to the CP (Consistency and Partition Tolerance) model of the CAP theorem. In a non-clustered single-instance setup, Redis is highly consistent. However, in a clustered setup, it sacrifices some availability for consistency.
Time Consistency in Redis can be achieved with strict persistence settings and synchronous replication.
"},{"location":"techdives/DistrubutedSystems/Redis/#6-cluster-structure-from-cluster-to-records-the-whole-path","title":"6. Cluster Structure \u2013 From Cluster to Records, the Whole Path","text":"Q: What is the structure of a Redis cluster from clusters down to individual records?
A: A Redis cluster is organized as follows: - Cluster: Composed of multiple nodes. - Nodes: Each node is responsible for a subset of the keyspace, organized into hash slots (16,384 in total). - Shards: Each node represents a shard of the data and can replicate across replicas. - Keys/Records: Each key is hashed to a specific slot, determining the node responsible for storing it.
"},{"location":"techdives/DistrubutedSystems/Redis/#7-the-fundamentals-of-a-cluster-all-building-blocks-from-cluster-to-records","title":"7. The Fundamentals of a Cluster \u2013 All Building Blocks from Cluster to Records","text":"Q: What are the core building blocks of a Redis cluster?
A: Core components include: - Nodes: Independent Redis instances in a cluster. - Hash Slots: Redis divides keys into 16,384 slots for distribution across nodes. - Replication: Each primary node can have replicas to ensure data redundancy. - Partitions (Shards): Each node holds a partition of data for horizontal scalability.
"},{"location":"techdives/DistrubutedSystems/Redis/#8-multi-master-support","title":"8. Multi-Master Support","text":"Q: Does Redis support multi-master configurations?
A: Redis does not support multi-master configurations in its native setup. It uses a single-master architecture per shard to ensure consistency.
"},{"location":"techdives/DistrubutedSystems/Redis/#9-master-slave-relationship-in-data-nodes","title":"9. Master-Slave Relationship in Data Nodes","text":"Q: Does Redis follow a master-slave structure between data nodes?
A: Yes, in a Redis cluster, each data shard has a single master with one or more replicas (slaves) for redundancy. The slaves serve as read-only replicas unless promoted during failover.
"},{"location":"techdives/DistrubutedSystems/Redis/#10-node-structures-in-cluster","title":"10. Node Structures in Cluster","text":"Q: What are the structures of nodes in a Redis cluster?
A: In a Redis cluster, each node is responsible for a subset of hash slots, with a master node serving write requests and one or more replicas serving as failover or read-only instances.
"},{"location":"techdives/DistrubutedSystems/Redis/#11-cluster-scaling-horizontal-and-vertical","title":"11. Cluster Scaling \u2013 Horizontal and Vertical","text":"Q: Does Redis support horizontal and vertical scaling, and which is preferred?
A: Redis supports horizontal scaling (adding more nodes) via sharding in a cluster, which is generally preferred. Vertical scaling (adding more memory/CPU) is also possible but limited by hardware.
"},{"location":"techdives/DistrubutedSystems/Redis/#12-high-availability-explanation","title":"12. High Availability \u2013 Explanation","text":"Q: How does Redis provide high availability?
A: Redis achieves high availability through replication and Redis Sentinel for monitoring and automatic failover. Redis Cluster further enhances availability by automatically promoting replicas if a primary node fails.
"},{"location":"techdives/DistrubutedSystems/Redis/#13-fault-tolerance-explanation","title":"13. Fault Tolerance \u2013 Explanation","text":"Q: What mechanisms does Redis have for fault tolerance?
A: Redis ensures fault tolerance through data replication across replicas, and Sentinel monitors the master nodes to trigger failover in case of node failure.
"},{"location":"techdives/DistrubutedSystems/Redis/#14-replication","title":"14. Replication","text":"Q: How does replication work in Redis?
A: Redis replication is asynchronous by default, with each master node replicating data to one or more replicas. In the event of a master failure, a replica is promoted to master status.
"},{"location":"techdives/DistrubutedSystems/Redis/#15-partitioning-and-sharding","title":"15. Partitioning and Sharding","text":"Q: How does Redis handle partitioning and sharding?
A: Redis uses hash-based partitioning with 16,384 hash slots to distribute data across nodes. Each key is assigned a hash slot, which maps it to a specific node.
"},{"location":"techdives/DistrubutedSystems/Redis/#16-caching-in-depth","title":"16. Caching in Depth","text":"Q: How does Redis perform caching?
A: Redis is an in-memory cache, providing low-latency access with various caching strategies (e.g., TTL, eviction policies like LRU and LFU). It supports key expiration and eviction for memory management.
"},{"location":"techdives/DistrubutedSystems/Redis/#17-storage-type-trees-used-for-storage","title":"17. Storage Type \u2013 Trees Used for Storage","text":"Q: What storage type and structures does Redis use?
A: Redis stores data in memory using simple data structures and does not use B-trees or similar structures. Data is kept in-memory and optionally persisted to disk (AOF/RDB).
"},{"location":"techdives/DistrubutedSystems/Redis/#18-segments-or-page-approach","title":"18. Segments or Page Approach?","text":"Q: Does Redis use a segments approach, page approach, or something else?
A: Redis does not use segments or page-based storage. Data is stored in-memory and is managed directly by the Redis process.
"},{"location":"techdives/DistrubutedSystems/Redis/#19-indexing-how-does-it-work","title":"19. Indexing \u2013 How Does It Work?","text":"Q: How does Redis handle indexing?
A: Redis does not use traditional indexing. Instead, it directly maps keys to hash slots in the cluster, providing O(1) access time to each key.
"},{"location":"techdives/DistrubutedSystems/Redis/#20-routing","title":"20. Routing","text":"Q: How does Redis route requests to the correct node in a cluster?
A: Redis routes requests based on key hashing. The key is hashed to determine its slot, which maps it to a specific node.
"},{"location":"techdives/DistrubutedSystems/Redis/#21-latency-including-write-read-indexing-and-replication-latency","title":"21. Latency \u2013 Including Write, Read, Indexing, and Replication Latency","text":"Q: What are Redis\u2019s latency characteristics?
A: Redis provides sub-millisecond read/write latency under normal conditions. Replication latency is generally low, though network overhead may add some delay.
"},{"location":"techdives/DistrubutedSystems/Redis/#22-versioning","title":"22. Versioning","text":"Q: Does Redis support versioning?
A: Redis does not natively support versioning. Application logic may be required to manage version control if needed.
"},{"location":"techdives/DistrubutedSystems/Redis/#23-locking-and-concurrency","title":"23. Locking and Concurrency","text":"Q: How does Redis handle locking and concurrency?
A: Redis supports distributed locking through the Redlock algorithm for ensuring safe concurrent access across clients.
"},{"location":"techdives/DistrubutedSystems/Redis/#24-write-ahead-logging-wal","title":"24. Write-Ahead Logging (WAL)","text":"Q: Does Redis support WAL?
A: Redis does not use WAL directly. However, the Append-Only File (AOF) is similar, logging each write operation to ensure persistence.
"},{"location":"techdives/DistrubutedSystems/Redis/#25-change-data-capture-cdc-support","title":"25. Change Data Capture (CDC) Support","text":"Q: Does Redis support CDC?
A: Redis does not natively support Change Data Capture. External tools may be needed for real-time data change tracking.
"},{"location":"techdives/DistrubutedSystems/Redis/#26-query-type-and-query-in-depth","title":"26. Query Type and Query in Depth","text":"Q: What types of queries does Redis support?
A: Redis is key-based and supports simple read/write commands without complex query languages. Operations include GET, SET, HGET, ZADD, etc.
Q: Does Redis have query optimizers?
A: Redis does not have traditional query optimizers, as it operates in O(1) for most key-based lookups.
"},{"location":"techdives/DistrubutedSystems/Redis/#28-sql-support","title":"28. SQL Support","text":"Q: Does Redis support SQL?
A: Redis does not natively support SQL. However, RedisJSON or other libraries can provide SQL-like querying capabilities.
"},{"location":"techdives/DistrubutedSystems/Redis/#29-circuit-breakers","title":"29. Circuit Breakers","text":"Q:
Does Redis have built-in circuit breaker support?
A: Redis itself does not implement circuit breakers. This is typically handled at the application or middleware layer.
"},{"location":"techdives/DistrubutedSystems/Redis/#30-data-retention-and-lifecycle-management","title":"30. Data Retention and Lifecycle Management","text":"Q: How does Redis handle data lifecycle and retention?
A: Redis supports TTL on keys, and policies like Least Recently Used (LRU) enable retention management. Redis doesn\u2019t support multi-tier storage.
"},{"location":"techdives/DistrubutedSystems/Redis/#31-other-features","title":"31. Other Features","text":"Q: What other features does Redis offer?
A: Redis supports data structures like streams for event logging, pub/sub for messaging, and geospatial indexing for location-based queries.
"},{"location":"techdives/DistrubutedSystems/Redis/#32-additional-modules","title":"32. Additional Modules","text":"Q: What modules or libraries can be added to Redis?
A: Redis offers modules like RedisJSON (for JSON handling), RedisGraph (for graph data), and RedisBloom (for probabilistic data structures).
"},{"location":"techdives/DistrubutedSystems/Redis/#33-optimization-and-tuning-of-clusters","title":"33. Optimization and Tuning of Clusters","text":"Q: How do you optimize and tune Redis clusters?
A: Key optimizations include appropriate partitioning, replication settings, eviction policies, and monitoring memory/CPU usage.
"},{"location":"techdives/DistrubutedSystems/Redis/#34-backup-and-recovery","title":"34. Backup and Recovery","text":"Q: How does Redis handle backup and recovery?
A: Redis supports RDB snapshots and AOF for persistence. Backups are easily managed via AOF or manual RDB dumps.
"},{"location":"techdives/DistrubutedSystems/Redis/#35-security","title":"35. Security","text":"Q: What are Redis\u2019s security features?
A: Redis supports authentication (AUTH command), SSL/TLS encryption, IP whitelisting, and role-based access control.
"},{"location":"techdives/DistrubutedSystems/Redis/#36-migration","title":"36. Migration","text":"Q: Does Redis support migration tools?
A: Redis offers tools like redis-cli for basic migration, and Redis Enterprise provides more advanced migration capabilities.
"},{"location":"techdives/DistrubutedSystems/Redis/#37-recommended-cluster-setup","title":"37. Recommended Cluster Setup","text":"Q: What\u2019s the recommended Redis cluster setup?
A: Typically, a Redis Cluster setup starts with 3 master nodes (for redundancy) and 3 replicas for high availability, totaling 6 nodes.
"},{"location":"techdives/DistrubutedSystems/Redis/#38-basic-cluster-setup-with-node-numbers-in-distributed-mode","title":"38. Basic Cluster Setup with Node Numbers in Distributed Mode","text":"Q: How does a basic Redis cluster setup look in distributed mode?
A: A minimal Redis Cluster in distributed mode consists of 3 master nodes (handling 5,461 slots each) with 1 replica per master for redundancy.
"},{"location":"techdives/DistrubutedSystems/Redis/#39-segments-approach-or-page-approach-or-others","title":"39. Segments Approach or Page Approach or others","text":"Q: Does Redis use a segments approach, page approach, or another storage approach?
A: Redis does not use a segments or page-based approach as it is an in-memory database. Data is stored directly in memory with no fixed segment or page structure, allowing for rapid access to keys. Redis is optimized for speed, relying on data structures like hash tables and direct in-memory allocation rather than traditional on-disk segment or page methods common in disk-based databases.
"},{"location":"techdives/DistrubutedSystems/S3/","title":"Amazon S3 (Simple Storage Service)","text":""},{"location":"techdives/DistrubutedSystems/S3/#1-introduction","title":"1. Introduction","text":"Amazon S3 is a scalable object storage service offered by Amazon Web Services (AWS). It is designed to store and retrieve any amount of data from anywhere on the web, making it suitable for various use cases, including data backup, archiving, big data analytics, and hosting static websites.
"},{"location":"techdives/DistrubutedSystems/S3/#2-architecture-and-fundamentals","title":"2. Architecture and Fundamentals","text":"S3 offers a variety of storage classes optimized for different use cases, balancing cost and performance:
S3 Standard: For frequently accessed data, providing high durability and availability.
S3 Intelligent-Tiering: Automatically moves data between two access tiers based on changing access patterns, optimizing costs.
S3 Standard-IA (Infrequent Access): Lower-cost storage for infrequently accessed data with retrieval fees.
S3 One Zone-IA: For infrequently accessed data that does not require multiple availability zone redundancy.
S3 Glacier: For archival data, offering retrieval times from minutes to hours.
S3 Glacier Deep Archive: The lowest-cost storage option for rarely accessed data, with retrieval times of 12 hours or more.
Durability: S3 is designed for 99.999999999% (11 nines) durability, ensuring high protection against data loss by automatically storing data across multiple devices in multiple facilities within an AWS Region.
Availability: S3 offers 99.99% availability over a given year, ensuring data accessibility when needed.
Redundancy: Data is redundantly stored across multiple availability zones, providing resilience against failures.
Identity and Access Management (IAM): Provides fine-grained control over who can access S3 resources and what actions they can perform.
Bucket Policies and ACLs: Define permissions at the bucket and object level, allowing public or private access as needed.
Encryption: Supports both server-side and client-side encryption to protect data at rest. SSE options include:
Data Transfer Security: Supports SSL/TLS for data in transit to protect data from interception.
Access Logs: Enable server access logging to track requests for access to your S3 resources.
Versioning: Keeps multiple versions of an object, allowing recovery from accidental deletions or overwrites.
Lifecycle Management: Automatically transitions objects between storage classes or deletes objects based on defined rules.
Event Notifications: Configure notifications for events like object creation or deletion, which can trigger AWS Lambda functions or send messages to Amazon SNS.
Object Tags: Use object tagging for better organization and management, including cost allocation for various projects.
Transfer Acceleration: Speeds up uploads and downloads using Amazon CloudFront's globally distributed edge locations.
Multipart Upload: Enables the uploading of large objects in parts, improving performance and allowing resuming of uploads in case of network failures.
Caching with CloudFront: Integrate with Amazon CloudFront for content delivery, reducing latency and improving performance for global users.
Data Consistency: S3 provides strong read-after-write consistency for PUT and DELETE operations, ensuring that data is immediately available after a successful write.
Backup and Disaster Recovery: Store backups of critical data and applications to ensure business continuity.
Data Lakes: Centralize diverse data sources for analytics and machine learning - OLAP
Media Hosting: Store and serve images, videos, and other media files for web and mobile applications.
Big Data Analytics: Use S3 to store large datasets for processing with services like Amazon Athena, Amazon EMR, or Amazon Redshift.
Static Website Hosting: Serve static websites directly from S3, leveraging its high availability and durability.
Organize Data: Use a consistent naming convention for buckets and objects to improve manageability.
Monitor Usage: Utilize AWS CloudTrail and AWS CloudWatch to monitor S3 usage and access patterns.
Optimize Costs: Regularly review storage classes and utilize lifecycle policies to manage data costs effectively.
Implement Security Best Practices: Regularly audit permissions, enable logging, and implement encryption to secure data.
Data Governance: Implement data governance strategies to comply with legal and regulatory requirements.
Version control is the cornerstone of modern software development, and Git stands as the most widely used version control system in the world. Whether you're a beginner or an experienced if you're a developer understanding Git is crucial for collaborative and individual projects alike. In this article, we'll take a deep dive into Git, covering everything from its basics to its advanced features, to equip you with the knowledge to master it and with a cheat sheet at the end.
"},{"location":"techdives/GeneralConcepts/git/#what-is-git","title":"What is Git ?","text":"Git is a distributed version control system that tracks changes in files, enabling multiple developers to collaborate on a project effectively. Created in 2005 by Linus Torvalds, Git was initially designed for managing the Linux kernel's development. Today, it powers everything from small personal projects to massive enterprise software systems.
Key features
Getting started with Git begins with installing it on your system. Here's how you can set it up based on your operating system:
MacwindowsLinux Use Homebrew to install Gitbrew install git\nDownload Git for Windows from git-scm.com and follow the installer instructions.
Install Git using your distribution's package managersudo apt install git # For Debian/Ubuntu\nsudo yum install git # For CentOS/Red Hat\ngit --version\ngit config --global user.name \"Your Name\"\ngit config --global user.email \"your.email@example.com\"\ngit config --global core.editor \"code\" # Use VSCode or any editor\nTo start tracking changes in a project, initialize a repository
git init\nTo work on an existing project, clone its repository
git clone <repository-url>\nStage Changes: Add files to the staging area
git add <file>\nCommit Changes: Save changes to the repository
git commit -m \"<Write a proper commit message>\"\nView the status of your working directory and staged files
git status\nReview the project's history with
git log\ngit log --oneline # Concise view\nGit's branching system is one of its most powerful features. Branches allow you to work on different features or bug fixes without affecting the main codebase.
"},{"location":"techdives/GeneralConcepts/git/#creating-switching-branches","title":"Creating Switching Branches","text":"Create a new branch
git branch <branch-name>\ngit checkout <branch-name>\ngit switch <branch-name> # New alternative\ngit checkout -b <branch-name>\nTo integrate changes from one branch into another
git checkout main # replace main with custom branch\ngit merge <branch-name>\nIf Git detects conflicting changes, resolve them manually by editing the affected files. Then
git add <file>\ngit commit\nRemote repositories allow teams to collaborate effectively.
"},{"location":"techdives/GeneralConcepts/git/#adding-a-remote","title":"Adding a Remote","text":"Link your local repository to a remote
git remote add origin <repository-url>\nSend your commits to the remote repository
git push origin <branch-name>\nFetch and integrate changes from the remote repository
git pull\nIf needed, you can remove a remote
git remote remove origin\nTemporarily save changes without committing
git stash\ngit stash apply\nApply a specific commit from another branch
git cherry-pick <commit-hash>\nRebase your branch onto another
git rebase <branch-name>\nFix the last commit message or contents
git commit --amend\nGit operates by storing snapshots of your project at each commit, not deltas (differences). The key components of Git's internal storage include:
All data is stored in the .git directory.
Teams often adopt workflows to streamline collaboration. Popular ones include:
main, develop, feature, and release for structured development.main branch, often behind feature flags.git switch <branch-name>\n.gitignore.Below are all the essential Git commands.
Git Cheat Sheet
# Git Commands Cheat Sheet\n\n# Configuration\ngit config --global user.name \"Your Name\" # Set user name\ngit config --global user.email \"your.email@uth.com\" # Set user email\ngit config --global core.editor \"code\" # Set default editor\ngit config --list # View current configuration\n\n# Repository Management\ngit init # Initialize a new repository\ngit clone <repository-url> # Clone an existing repository\ngit remote add origin <url> # Add remote repository\ngit remote -v # View remote repositories\ngit remote remove <name> # Remove a remote\n\n# Staging and Committing\ngit add <file> # Stage specific file\ngit add . # Stage all files\ngit status # Check status of repository\ngit commit -m \"Message\" # Commit with message\ngit commit --amend # Amend the last commit\n\n# Branching\ngit branch # List branches\ngit branch <branch-name> # Create a new branch\ngit checkout <branch-name> # Switch to a branch\ngit checkout -b <branch-name> # Create and Switch to a branch\ngit switch <branch-name> # Modern way to switch branches\ngit branch -d <branch-name> # Delete a branch\ngit branch -D <branch-name> # Force delete a branch\n\n# Merging\ngit merge <branch-name> # Merge a branch into the current branch\n\n# Pulling and Pushing\ngit pull # Fetch and merge from remote repository\ngit pull origin <branch-name> # Pull specific branch\ngit push origin <branch-name> # Push to remote repository\ngit push --all # Push all branches\ngit push --tags # Push tags to remote\n\n# Logs and History\ngit log # View commit history\ngit log --oneline # View concise commit history\ngit log --graph # View graphical commit history\n\n# Undo Changes\ngit reset HEAD <file> # Unstage a file\ngit checkout -- <file> # Discard changes in working directory\ngit revert <commit-hash> # Undo a specific commit (safe)\ngit reset <commit-hash> # Reset to a specific commit (dangerous)\n\n# Stashing\ngit stash # Stash changes\ngit stash list # List stashes\ngit stash apply # Apply the last stash\ngit stash drop # Remove the last stash\ngit stash clear # Clear all stashes\n\n# Rebasing\ngit rebase <branch-name> # Rebase current branch onto another\ngit rebase -i <commit-hash> # Interactive rebase\n\n# Tags\ngit tag <tag-name> # Create a tag\ngit tag -a <tag-name> -m \"Message\" # Create an annotated tag\ngit tag -d <tag-name> # Delete a tag locally\ngit push origin <tag-name> # Push a specific tag\ngit push --tags # Push all tags\n\n# Collaboration\ngit fetch # Fetch updates from remote\ngit pull # Fetch and merge updates\ngit pull origin <branch-name> # Pull specific branch\ngit push # Push changes to remote\ngit push origin <branch-name> # Push specific branch\n\n# Ignoring Files\necho \"filename\" >> .gitignore # Add file to .gitignore\ngit rm --cached <file> # Stop tracking a file\n\n# Viewing Changes\ngit diff # View unstaged changes\ngit diff --staged # View staged changes\ngit diff <commit-hash1> <commit-hash2> # Compare two commits\n\n# Cherry-Picking\ngit cherry-pick <commit-hash> # Apply a specific commit to the current branch\n\n# Aliases\ngit config --global alias.co checkout # Alias for checkout\ngit config --global alias.br branch # Alias for branch\ngit config --global alias.cm commit # Alias for commit\nWelcome to Deep Dives by MG
"},{"location":"blog/","title":"Blog","text":""},{"location":"blog/#coming-soon","title":"Coming soon","text":""},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#under-the-hood-by-mrudhul-guda","title":"Under the Hood by Mrudhul Guda","text":""},{"location":"changelog/#0.4.0","title":"0.4.0 November 16, 2024","text":"The Abstract Factory Pattern is a creational design pattern that provides an interface for creating families of related or dependent objects without specifying their concrete classes. It promotes loose coupling between client code and the actual implementations, allowing the code to be more flexible and scalable.
The Abstract Factory pattern works as a super-factory that creates other factories. Each factory produced by the abstract factory is responsible for creating a family of related objects.
Key Characteristics
Class Diagram
AbstractFactory\n\u251c\u2500\u2500 createProductA()\n\u2514\u2500\u2500 createProductB()\n\nConcreteFactory1 \u2500\u2500\u2500\u2500> ProductA1, ProductB1\nConcreteFactory2 \u2500\u2500\u2500\u2500> ProductA2, ProductB2\n\nClient \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500> AbstractFactory, AbstractProduct\nLet\u2019s go with an example Imagine you are creating a UI component factory. Your application can switch between two themes: Dark Theme and Light Theme. Both themes provide the same types of components (buttons, text fields) but with different appearances.
Step-1: Define the Abstract Products// Abstract Product: Button\npublic interface Button {\n void render();\n}\n\n// Abstract Product: TextField\npublic interface TextField {\n void render();\n}\n// Concrete Product: Light Button\npublic class LightButton implements Button {\n @Override\n public void render() {\n System.out.println(\"Rendering a Light Button\");\n }\n}\n\n// Concrete Product: Dark Button\npublic class DarkButton implements Button {\n @Override\n public void render() {\n System.out.println(\"Rendering a Dark Button\");\n }\n}\n\n// Concrete Product: Light TextField\npublic class LightTextField implements TextField {\n @Override\n public void render() {\n System.out.println(\"Rendering a Light Text Field\");\n }\n}\n\n// Concrete Product: Dark TextField\npublic class DarkTextField implements TextField {\n @Override\n public void render() {\n System.out.println(\"Rendering a Dark Text Field\");\n }\npublic interface UIFactory {\n Button createButton();\n TextField createTextField();\n}\n// Concrete Factory for Light Theme\npublic class LightUIFactory implements UIFactory {\n @Override\n public Button createButton() {\n return new LightButton();\n }\n\n @Override\n public TextField createTextField() {\n return new LightTextField();\n }\n}\n\n// Concrete Factory for Dark Theme\npublic class DarkUIFactory implements UIFactory {\n @Override\n public Button createButton() {\n return new DarkButton();\n }\n\n @Override\n public TextField createTextField() {\n return new DarkTextField();\n }\n}\npublic class Application {\n private Button button;\n private TextField textField;\n\n public Application(UIFactory factory) {\n this.button = factory.createButton();\n this.textField = factory.createTextField();\n }\n\n public void renderUI() {\n button.render();\n textField.render();\n }\n\n public static void main(String[] args) {\n // Client can choose between different factories.\n UIFactory factory = new DarkUIFactory(); // Could be switched to LightUIFactory\n Application app = new Application(factory);\n app.renderUI();\n }\n}\nOutput:
Rendering a Dark Button\nRendering a Dark Text Field\nIn Spring Boot, the Abstract Factory pattern can complement dependency injection (DI) by delegating object creation logic to the factory. Here\u2019s how to implement it with Spring Boot.
Step-1: Define Factory Beans@Configuration\npublic class UIFactoryConfig {\n\n @Bean\n public UIFactory uiFactory(@Value(\"${app.theme}\") String theme) {\n if (\"dark\".equalsIgnoreCase(theme)) {\n return new DarkUIFactory();\n } else {\n return new LightUIFactory();\n }\n }\n}\n@RestController\n@RequestMapping(\"/ui\")\npublic class UIController {\n\n private final UIFactory uiFactory;\n\n @Autowired\n public UIController(UIFactory uiFactory) {\n this.uiFactory = uiFactory;\n }\n\n @GetMapping(\"/render\")\n public void renderUI() {\n Button button = uiFactory.createButton();\n TextField textField = uiFactory.createTextField();\n\n button.render();\n textField.render();\n }\n}\n# application.properties\napp.theme=dark\nIn this example, the theme is configured through the application.properties file, and the factory selection is handled by the Spring context.
The Abstract Factory Pattern is a powerful tool when designing systems that need to create multiple families of related objects. While it adds complexity, the benefits include extensibility, maintainability, and loose coupling. In a Spring Boot application, it works well alongside dependency injection, especially when configurations like themes or environments vary.
"},{"location":"fundamentaldives/DesignPatterns/Adapter/","title":"Adapter Design Pattern","text":""},{"location":"fundamentaldives/DesignPatterns/Adapter/#what","title":"What ?","text":"The Adapter Pattern is a structural design pattern in software development that allows objects with incompatible interfaces to work together. It acts as a bridge between two incompatible interfaces, providing a wrapper or a mediator to enable their interaction without changing their existing code.
This Pattern converts the interface of a class into another interface that a client expects. This helps integrate two systems with different interfaces so they can work together without altering their code. It is often used when a legacy system needs to be integrated with new components or when third-party APIs are integrated into an existing codebase.
Analogy
Think of a power plug adapter You have an appliance with a US plug (two flat pins), but you need to connect it to a European socket (two round holes). The adapter ensures that both the incompatible interfaces (US and European plugs) work together without modifying either.
"},{"location":"fundamentaldives/DesignPatterns/Adapter/#when-to-use","title":"When to Use ?","text":"There are two common ways to implement the Adapter Pattern:
In this approach, the adapter class extends the adaptee (the class that has the incompatible interface) and implements the interface that the client expects.
Class Adapter Java Example// Target Interface - The desired interface that client expects\ninterface MediaPlayer {\n void play(String audioType, String fileName);\n}\n\n// Adaptee - Incompatible interface that needs adaptation\nclass AdvancedMediaPlayer {\n void playMp3(String fileName) {\n System.out.println(\"Playing mp3 file: \" + fileName);\n }\n\n void playMp4(String fileName) {\n System.out.println(\"Playing mp4 file: \" + fileName);\n }\n}\n\n// Class Adapter - Adapts AdvancedMediaPlayer to MediaPlayer\nclass MediaAdapter extends AdvancedMediaPlayer implements MediaPlayer {\n @Override\n public void play(String audioType, String fileName) {\n if (audioType.equalsIgnoreCase(\"mp3\")) {\n playMp3(fileName);\n } else if (audioType.equalsIgnoreCase(\"mp4\")) {\n playMp4(fileName);\n }\n }\n}\n\n// Client Code\npublic class AudioPlayer {\n public static void main(String[] args) {\n MediaPlayer player = new MediaAdapter();\n player.play(\"mp3\", \"song.mp3\");\n player.play(\"mp4\", \"video.mp4\");\n }\n}\nMediaAdapter extends AdvancedMediaPlayer (inheriting the original functionality) and implements the MediaPlayer interface (adapting it to what the client expects).
In this approach, the adapter contains an instance of the adaptee class and delegates calls to the appropriate methods.
Object Adapter Java Example// Target Interface\ninterface MediaPlayer {\n void play(String audioType, String fileName);\n}\n\n// Adaptee\nclass AdvancedMediaPlayer {\n void playMp3(String fileName) {\n System.out.println(\"Playing mp3 file: \" + fileName);\n }\n\n void playMp4(String fileName) {\n System.out.println(\"Playing mp4 file: \" + fileName);\n }\n}\n\n// Object Adapter\nclass MediaAdapter implements MediaPlayer {\n private AdvancedMediaPlayer advancedPlayer;\n\n public MediaAdapter(AdvancedMediaPlayer advancedPlayer) {\n this.advancedPlayer = advancedPlayer;\n }\n\n @Override\n public void play(String audioType, String fileName) {\n if (audioType.equalsIgnoreCase(\"mp3\")) {\n advancedPlayer.playMp3(fileName);\n } else if (audioType.equalsIgnoreCase(\"mp4\")) {\n advancedPlayer.playMp4(fileName);\n }\n }\n}\n\n// Client Code\npublic class AudioPlayer {\n public static void main(String[] args) {\n AdvancedMediaPlayer advancedPlayer = new AdvancedMediaPlayer();\n MediaPlayer adapter = new MediaAdapter(advancedPlayer);\n adapter.play(\"mp3\", \"song.mp3\");\n adapter.play(\"mp4\", \"video.mp4\");\n }\n}\nIn this version, MediaAdapter holds a reference to the AdvancedMediaPlayer instance and delegates method calls instead of extending the class.
In a Spring Boot context, the Adapter Pattern can be used to integrate an external or legacy service with your application's service layer.
Integrating a Legacy Payment Service// Legacy Payment Service - Adaptee\nclass LegacyPaymentService {\n public void payWithCreditCard(String cardNumber) {\n System.out.println(\"Payment made using Legacy Credit Card: \" + cardNumber);\n }\n}\n\n// Target Interface\ninterface PaymentService {\n void processPayment(String cardNumber);\n}\n\n// Adapter Implementation - Integrating LegacyPaymentService with PaymentService\n@Component\nclass PaymentServiceAdapter implements PaymentService {\n private final LegacyPaymentService legacyService;\n\n // Constructor injection\n public PaymentServiceAdapter(LegacyPaymentService legacyService) {\n this.legacyService = legacyService;\n }\n\n @Override\n public void processPayment(String cardNumber) {\n legacyService.payWithCreditCard(cardNumber);\n }\n}\n\n// Spring Boot Controller\n@RestController\n@RequestMapping(\"/payments\")\npublic class PaymentController {\n\n private final PaymentService paymentService;\n\n @Autowired\n public PaymentController(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n\n @PostMapping\n public String makePayment(@RequestParam String cardNumber) {\n paymentService.processPayment(cardNumber);\n return \"Payment Successful\";\n }\n}\nLegacyPaymentService is an old service with an incompatible method.PaymentServiceAdapter acts as an adapter by implementing the PaymentService interface and internally calling the legacy service.PaymentController depends on PaymentService, which can now work seamlessly with the legacy system.The Adapter Pattern enhances flexibility by decoupling client code from specific implementations, promotes reusability by enabling compatibility between systems, improves maintainability by isolating legacy or third-party code, and simplifies testing through easy mock or stub usage.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/","title":"Bridge","text":""},{"location":"fundamentaldives/DesignPatterns/Bridge/#what","title":"What ?","text":"The Bridge Pattern is a structural design pattern that helps to decouple an abstraction from its implementation so that both can vary independently. This pattern is especially useful when you need to manage complex class hierarchies or have multiple dimensions of variations.
When both the abstraction (interface) and its implementation (how it works internally) need to evolve, the code becomes complex and hard to manage. Bridge helps to separate these concerns. The pattern separates the abstraction (interface) from the actual implementation and lets them evolve independently by delegating the concrete work to another interface.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/#structure","title":"Structure ?","text":"The structure involves two key parts:
The Abstraction contains a reference to the Implementor (interface or class). This lets the abstraction delegate the implementation details to the concrete implementations.
Class Diagram
Abstraction --> Implementor\n | |\nRefinedAbstraction ConcreteImplementor\nCircleRed, CircleBlue, SquareRed, SquareBlue this can be avoided with the Bridge pattern.Let\u2019s look at a real-world example rendering shapes on different platforms. The rendering logic could vary depending on the platform (Windows, Linux, etc.), but the shape (e.g., Circle, Rectangle) stays the same.
Spring Boot ExampleIn a Spring Boot application, you can use the Bridge pattern to switch between different implementations of a service dynamically, such as switching between multiple ways of sending notifications (e.g., Email, SMS). This is helpful when services have multiple implementations, and you need to inject them dynamically without changing the client code.
"},{"location":"fundamentaldives/DesignPatterns/Bridge/#rendering-shapes-on-different-platforms","title":"Rendering Shapes on Different Platforms","text":"Step-1: Define the Implementor interfaceinterface Renderer {\n void render(String shape);\n}\nclass VectorRenderer implements Renderer {\n @Override\n public void render(String shape) {\n System.out.println(\"Rendering \" + shape + \" as vectors.\");\n }\n}\n\nclass RasterRenderer implements Renderer {\n @Override\n public void render(String shape) {\n System.out.println(\"Rendering \" + shape + \" as pixels.\");\n }\n}\nabstract class Shape {\n protected Renderer renderer;\n\n public Shape(Renderer renderer) {\n this.renderer = renderer;\n }\n\n public abstract void draw();\n}\nclass Circle extends Shape {\n public Circle(Renderer renderer) {\n super(renderer);\n }\n\n @Override\n public void draw() {\n renderer.render(\"Circle\");\n }\n}\n\nclass Rectangle extends Shape {\n public Rectangle(Renderer renderer) {\n super(renderer);\n }\n\n @Override\n public void draw() {\n renderer.render(\"Rectangle\");\n }\n}\npublic class BridgePatternDemo {\n public static void main(String[] args) {\n Shape circle = new Circle(new VectorRenderer());\n circle.draw(); // Output: Rendering Circle as vectors.\n\n Shape rectangle = new Rectangle(new RasterRenderer());\n rectangle.draw(); // Output: Rendering Rectangle as pixels.\n }\n}\npublic interface NotificationSender {\n void send(String message);\n}\n@Component\npublic class EmailSender implements NotificationSender {\n @Override\n public void send(String message) {\n System.out.println(\"Sending Email: \" + message);\n }\n}\n\n@Component\npublic class SmsSender implements NotificationSender {\n @Override\n public void send(String message) {\n System.out.println(\"Sending SMS: \" + message);\n }\n}\npublic abstract class Notification {\n protected NotificationSender sender;\n\n public Notification(NotificationSender sender) {\n this.sender = sender;\n }\n\n public abstract void notify(String message);\n}\n@Component\npublic class UrgentNotification extends Notification {\n\n @Autowired\n public UrgentNotification(NotificationSender sender) {\n super(sender);\n }\n\n @Override\n public void notify(String message) {\n System.out.println(\"Urgent Notification:\");\n sender.send(message);\n }\n}\n@RestController\n@RequestMapping(\"/notifications\")\npublic class NotificationController {\n\n private final UrgentNotification notification;\n\n @Autowired\n public NotificationController(UrgentNotification notification) {\n this.notification = notification;\n }\n\n @PostMapping(\"/send\")\n public String sendNotification(@RequestBody String message) {\n notification.notify(message);\n return \"Notification sent!\";\n }\n}\nIn this example, the Bridge Pattern allows you to switch between different ways of sending notifications (email or SMS) without changing the client code (the NotificationController).
The Bridge Pattern is an essential design pattern to consider when your class hierarchy is growing unmanageable due to multiple dimensions of variations. Use this pattern when you need to decouple abstraction from implementation and allow them to evolve independently, but avoid it when simpler solutions can suffice.
"},{"location":"fundamentaldives/DesignPatterns/Builder/","title":"Builder Design","text":""},{"location":"fundamentaldives/DesignPatterns/Builder/#what","title":"What ?","text":"The Builder Pattern is a creational design pattern that allows the construction of complex objects step by step. It separates the construction process from the actual object, giving more control over the construction process.
This Pattern simplifies the creation of complex objects with many optional fields by enabling incremental construction through method chaining, avoiding constructors with numerous parameters. It's ideal for objects requiring various configurations or optional parameters.
"},{"location":"fundamentaldives/DesignPatterns/Builder/#when-to-use","title":"When to Use ?","text":"Below is a basic Java example demonstrating the pattern. Assume we need to build a Car object with several optional fields.
public class Car {\n // Required fields\n private final String make;\n private final String model;\n\n // Optional fields\n private final String color;\n private final int year;\n private final boolean automatic;\n\n // Private constructor accessible only through Builder\n private Car(Builder builder) {\n this.make = builder.make;\n this.model = builder.model;\n this.color = builder.color;\n this.year = builder.year;\n this.automatic = builder.automatic;\n }\n\n // Getters (optional, based on your needs)\n public String getMake() { return make; }\n public String getModel() { return model; }\n public String getColor() { return color; }\n public int getYear() { return year; }\n public boolean isAutomatic() { return automatic; }\n\n // Static inner Builder class\n public static class Builder {\n // Required fields\n private final String make;\n private final String model;\n\n // Optional fields initialized to default values\n private String color = \"White\";\n private int year = 2020;\n private boolean automatic = true;\n\n // Builder constructor with required fields\n public Builder(String make, String model) {\n this.make = make;\n this.model = model;\n }\n\n // Setter-like methods for optional fields, returning the builder object\n public Builder color(String color) {\n this.color = color;\n return this;\n }\n\n public Builder year(int year) {\n this.year = year;\n return this;\n }\n\n public Builder automatic(boolean automatic) {\n this.automatic = automatic;\n return this;\n }\n\n // Build method to create the final Car object\n public Car build() {\n return new Car(this);\n }\n }\n\n @Override\n public String toString() {\n return \"Car [make=\" + make + \", model=\" + model + \n \", color=\" + color + \", year=\" + year + \n \", automatic=\" + automatic + \"]\";\n }\n}\npublic class Main {\n public static void main(String[] args) {\n // Using the builder to create a Car object\n Car car = new Car.Builder(\"Tesla\", \"Model S\")\n .color(\"Red\")\n .year(2023)\n .automatic(true)\n .build();\n\n System.out.println(car);\n }\n}\nOutput:
Car [make=Tesla, model=Model S, color=Red, year=2023, automatic=true]\nIn Spring Boot, you often need to build objects like DTOs, configurations, or entities with complex structures. Using the Builder Pattern can make object construction more manageable, especially when working with REST APIs.
Using Builder Pattern for DTOs in Spring Boot// Let's assume a UserDTO object for API responses.\npublic class UserDTO {\n private final String username;\n private final String email;\n private final String role;\n\n private UserDTO(Builder builder) {\n this.username = builder.username;\n this.email = builder.email;\n this.role = builder.role;\n }\n\n public static class Builder {\n private String username;\n private String email;\n private String role;\n\n public Builder username(String username) {\n this.username = username;\n return this;\n }\n\n public Builder email(String email) {\n this.email = email;\n return this;\n }\n\n public Builder role(String role) {\n this.role = role;\n return this;\n }\n\n public UserDTO build() {\n return new UserDTO(this);\n }\n }\n}\n@RestController\n@RequestMapping(\"/api/users\")\npublic class UserController {\n\n @GetMapping(\"/{id}\")\n public UserDTO getUserById(@PathVariable Long id) {\n // Simulate fetching user details from a database\n return new UserDTO.Builder()\n .username(\"johndoe\")\n .email(\"john.doe@example.com\")\n .role(\"ADMIN\")\n .build();\n }\n}\nThis approach ensures that the object returned from the API is constructed cleanly with only the necessary fields set.
Alternative WaysTelescoping Constructors Multiple overloaded constructors for different parameter combinations but Not ideal for readability and maintainability.
public Car(String make, String model) { ... }\npublic Car(String make, String model, String color) { ... }\npublic Car(String make, String model, String color, int year) { ... }\nSetter Methods Useful for mutable objects but doesn\u2019t guarantee immutability but less readable when constructing objects with many attributes.
Car car = new Car();\ncar.setMake(\"Tesla\");\ncar.setModel(\"Model S\");\ncar.setColor(\"Red\");\nThe Builder Pattern is an elegant way to handle object creation, especially when dealing with many fields or optional parameters. It ensures code readability, immutability, and flexibility while avoiding the need for numerous constructors. However, it should be used only when necessary, as simple objects may not benefit from it.
Note
In Spring Boot, the Builder Pattern can be effectively used for creating DTOs and other complex objects, improving both code readability and maintenance. This pattern fits well when dealing with REST API responses or configuration settings, ensuring your objects are built in a clear, consistent manner.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/","title":"Circuit Breakers","text":""},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#what","title":"What ?","text":"A circuit breaker is a design pattern used to prevent cascading failures and manage service availability. If a service call repeatedly fails, the circuit breaker \"trips\" and prevents further attempts, allowing the system to recover gracefully. This pattern mimics the behavior of electrical circuit breakers.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#why-is-it-needed","title":"Why is it Needed ?","text":"Real-world scenario
If Service A depends on Service B but Service B becomes unavailable, Service A will receive failures continuously. A circuit breaker prevents Service A from overloading itself and Service B by failing fast.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#types-of-circuit-breakers","title":"Types of Circuit Breakers","text":"There are multiple models of circuit breakers to choose from depending on use case:
Count-based Circuit Breaker - Trips if a predefined number of failures occur, eg: If there are 3 consecutive failed requests, the breaker opens.
Time-based Circuit Breaker - Monitors failures within a window of time and trips if the failure threshold is met, eg: If 5 requests out of 10 fail within 1 minute, it opens.
Sliding Window Circuit Breaker - A rolling window of requests over time determines if the circuit trips, Useful when failure patterns are sporadic.
"},{"location":"fundamentaldives/DesignPatterns/CircuitBreakers/#how-does-it-work","title":"How Does it Work?","text":"The basic mechanics of a circuit breaker involve three states:
Closed State
Open State
Half-Open State
State Transition Flow
Closed -> (failure threshold reached) -> Open -> (timeout) -> Half-Open -> (success) -> Closed\nSeveral popular libraries and frameworks make circuit breaker implementations simple. Below are code examples in Java and Python.
Java with Resilience4jPython with PyBreaker// Resilience4j is a library providing circuit breaker implementations.\nimport io.github.resilience4j.circuitbreaker.CircuitBreaker;\nimport io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;\nimport io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;\n\nimport java.time.Duration;\n\npublic class Example {\n public static void main(String[] args) {\n // Configuration\n CircuitBreakerConfig config = CircuitBreakerConfig.custom()\n .failureRateThreshold(50) // Open if 50% of requests fail\n .waitDurationInOpenState(Duration.ofSeconds(5)) // Wait 5 seconds before Half-Open\n .build();\n\n CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(config);\n CircuitBreaker circuitBreaker = registry.circuitBreaker(\"myService\");\n\n // Wrap a call in the circuit breaker\n String response = circuitBreaker.executeSupplier(() -> makeHttpRequest());\n\n System.out.println(response);\n }\n\n private static String makeHttpRequest() {\n // Simulate an HTTP request here\n return \"Success!\";\n }\n}\n# PyBreaker is a library implementing the Circuit Breaker pattern for Python applications.\n\nfrom pybreaker import CircuitBreaker, CircuitBreakerError\nimport requests\n\n# Define a circuit breaker\nbreaker = CircuitBreaker(fail_max=3, reset_timeout=5)\n\n@breaker\ndef fetch_data(url):\n response = requests.get(url)\n if response.status_code != 200:\n raise Exception(\"Service unavailable\")\n return response.json()\n\ntry:\n data = fetch_data('https://api.example.com/data')\n print(data)\nexcept CircuitBreakerError:\n print(\"Circuit is open. Service unavailable.\")\nCircuit breakers need to be monitored to ensure they perform correctly. You can integrate them with monitoring tools like Prometheus or Grafana. Many libraries offer hooks to capture metrics such as
Circuit breakers are crucial in modern, distributed systems, preventing unnecessary retries and protecting systems from cascading failures. As systems grow in complexity, using the circuit breaker pattern helps maintain high availability and resilience.
"},{"location":"fundamentaldives/DesignPatterns/Composite/","title":"Composite","text":""},{"location":"fundamentaldives/DesignPatterns/Composite/#what","title":"What ?","text":"The Composite Pattern is a structural design pattern used in software design to represent part whole hierarchies. It enables you to build complex object structures by treating both individual objects and compositions of objects uniformly.
This pattern allows you to treat a group of objects in the same way as a single object. This is especially useful when building tree structures (like directories or UI components).
Key Concepts
The idea is to define a common interface for all the objects, whether simple or complex, so they can be treated uniformly.
Component objects. It may contain both Leafs and other Composites.Basic Structure UML
Component (Interface or Abstract class)\n\u251c\u2500\u2500 Leaf (Concrete class)\n\u2514\u2500\u2500 Composite (Concrete class containing Components)\ndraw() should work for both individual shapes and a group of shapes.Component interface for both Leafs and Composites.// 1. Component Interface\ninterface Component {\n void showDetails(); // Common operation for both Leaf and Composite\n}\n\n// 2. Leaf Class (Single object)\nclass Employee implements Component {\n private String name;\n private String position;\n\n public Employee(String name, String position) {\n this.name = name;\n this.position = position;\n }\n\n @Override\n public void showDetails() {\n System.out.println(name + \" works as \" + position);\n }\n}\n\n// 3. Composite Class (Composite of Components)\nclass Department implements Component {\n private List<Component> employees = new ArrayList<>();\n\n public void addEmployee(Component employee) {\n employees.add(employee);\n }\n\n public void removeEmployee(Component employee) {\n employees.remove(employee);\n }\n\n @Override\n public void showDetails() {\n for (Component employee : employees) {\n employee.showDetails();\n }\n }\n}\n\n// 4. Client Code\npublic class CompositePatternDemo {\n public static void main(String[] args) {\n // Create individual employees\n Component emp1 = new Employee(\"John\", \"Developer\");\n Component emp2 = new Employee(\"Doe\", \"Tester\");\n\n // Create a department and add employees to it\n Department department = new Department();\n department.addEmployee(emp1);\n department.addEmployee(emp2);\n\n // Show details\n System.out.println(\"Department Details:\");\n department.showDetails();\n }\n}\nDepartment Details:\nJohn works as Developer\nDoe works as Tester\nIn Spring Boot, the Composite pattern can fit into cases where you model tree-based structures in your business logic, such as:
// 1. Component Interface\npublic interface ProductCategory {\n String getName();\n void showCategoryDetails();\n}\n\n// 2. Leaf Class\npublic class Product implements ProductCategory {\n private String name;\n\n public Product(String name) {\n this.name = name;\n }\n\n @Override\n public String getName() {\n return name;\n }\n\n @Override\n public void showCategoryDetails() {\n System.out.println(\"Product: \" + name);\n }\n}\n\n// 3. Composite Class\npublic class Category implements ProductCategory {\n private String name;\n private List<ProductCategory> children = new ArrayList<>();\n\n public Category(String name) {\n this.name = name;\n }\n\n public void add(ProductCategory category) {\n children.add(category);\n }\n\n public void remove(ProductCategory category) {\n children.remove(category);\n }\n\n @Override\n public String getName() {\n return name;\n }\n\n @Override\n public void showCategoryDetails() {\n System.out.println(\"Category: \" + name);\n for (ProductCategory child : children) {\n child.showCategoryDetails();\n }\n }\n}\n\n// 4. Controller in Spring Boot\n@RestController\n@RequestMapping(\"/categories\")\npublic class CategoryController {\n\n @GetMapping(\"/example\")\n public void example() {\n // Creating products\n ProductCategory p1 = new Product(\"Laptop\");\n ProductCategory p2 = new Product(\"Phone\");\n\n // Creating a category and adding products\n Category electronics = new Category(\"Electronics\");\n electronics.add(p1);\n electronics.add(p2);\n\n // Display details\n electronics.showCategoryDetails();\n }\n}\nCategory: Electronics\nProduct: Laptop\nProduct: Phone\n@OneToMany).The Composite Pattern is a powerful structural pattern for managing hierarchical, tree-like structures. It allows uniform handling of individual and composite objects, making it ideal for UI elements, filesystems, or business domains with nested elements. When integrating with Spring Boot, it works well in controllers, services, or JPA entities for modeling hierarchical data. However, avoid using it when there\u2019s no hierarchy or when performance is critical (deep recursion). Use it wisely, and it can help you reduce complexity and simplify your code.
"},{"location":"fundamentaldives/DesignPatterns/Decorator/","title":"Decorator","text":""},{"location":"fundamentaldives/DesignPatterns/Decorator/#what","title":"What ?","text":"The Decorator Pattern is a structural design pattern that allows behavior to be added to individual objects, either statically or dynamically, without affecting the behavior of other objects from the same class. This pattern is particularly useful when you need to add functionality to objects without subclassing and in scenarios where multiple combinations of behaviors are required.
This pattern is used to attach additional responsibilities or behaviors to an object dynamically. It wraps the original object, adding new behavior while keeping the object\u2019s interface intact. A decorator class has a reference to the original object and implements the same interface.
Key Concepts
Component interface.Component interface and holds a reference to a Component object.Decorator that add specific functionalities.Let's consider an example where we are building a coffee shop. Different types of coffees can be enhanced with add-ons like milk, sugar, etc. Using the decorator pattern, we can apply these add-ons dynamically without subclassing.
// Step 1: Component Interface\npublic interface Coffee {\n String getDescription();\n double getCost();\n}\n\n// Step 2: ConcreteComponent (Basic Coffee)\npublic class BasicCoffee implements Coffee {\n @Override\n public String getDescription() {\n return \"Basic Coffee\";\n }\n\n @Override\n public double getCost() {\n return 2.0;\n }\n}\n\n// Step 3: Decorator (Abstract)\npublic abstract class CoffeeDecorator implements Coffee {\n protected Coffee coffee; // The object being decorated\n\n public CoffeeDecorator(Coffee coffee) {\n this.coffee = coffee;\n }\n\n public String getDescription() {\n return coffee.getDescription();\n }\n\n public double getCost() {\n return coffee.getCost();\n }\n}\n\n// Step 4: Concrete Decorators (e.g., Milk, Sugar)\npublic class MilkDecorator extends CoffeeDecorator {\n public MilkDecorator(Coffee coffee) {\n super(coffee);\n }\n\n @Override\n public String getDescription() {\n return coffee.getDescription() + \", Milk\";\n }\n\n @Override\n public double getCost() {\n return coffee.getCost() + 0.5;\n }\n}\n\npublic class SugarDecorator extends CoffeeDecorator {\n public SugarDecorator(Coffee coffee) {\n super(coffee);\n }\n\n @Override\n public String getDescription() {\n return coffee.getDescription() + \", Sugar\";\n }\n\n @Override\n public double getCost() {\n return coffee.getCost() + 0.2;\n }\n}\n\n// Step 5: Usage\npublic class CoffeeShop {\n public static void main(String[] args) {\n Coffee coffee = new BasicCoffee();\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n\n coffee = new MilkDecorator(coffee);\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n\n coffee = new SugarDecorator(coffee);\n System.out.println(coffee.getDescription() + \" $\" + coffee.getCost());\n }\n}\nBasic Coffee $2.0\nBasic Coffee, Milk $2.5\nBasic Coffee, Milk, Sugar $2.7\nIn Spring Boot, the decorator pattern can be used in scenarios such as logging, monitoring, or security checks. You can implement a decorator pattern to enhance service classes without changing their core logic. Here's an example where we decorate a service class to add logging functionality.
Component Interface (Service Layer)public interface UserService {\n String getUserDetails(String userId);\n}\n@Service\npublic class UserServiceImpl implements UserService {\n @Override\n public String getUserDetails(String userId) {\n return \"User details for \" + userId;\n }\n}\n@Service\npublic class LoggingUserService implements UserService {\n\n private final UserService userService;\n\n public LoggingUserService(UserService userService) {\n this.userService = userService;\n }\n\n @Override\n public String getUserDetails(String userId) {\n System.out.println(\"Fetching details for user: \" + userId);\n return userService.getUserDetails(userId);\n }\n}\n@Configuration\npublic class ServiceConfig {\n\n @Bean\n public UserService userService(UserServiceImpl userServiceImpl) {\n return new LoggingUserService(userServiceImpl);\n }\n}\nUserServiceImpl provides the basic functionality.LoggingUserService acts as a decorator, adding logging before calling the original method.The Decorator Pattern is a powerful and flexible way to enhance objects with additional behaviors dynamically without altering their structure. It shines in scenarios requiring combinations of behaviors and helps maintain clean, modular code.
In Spring Boot, it can be used for decorating services with additional features like logging, security, or metrics, allowing these aspects to remain separate from the core business logic.
This pattern should be used thoughtfully since excessive use can introduce complexity and make debugging difficult. However, when applied correctly, it ensures that code remains extensible and adheres to the Single Responsibility Principle and Open Closed Principle.
"},{"location":"fundamentaldives/DesignPatterns/Facade/","title":"Facade","text":"The Facade Pattern is a structural design pattern commonly used to provide a simple, unified interface to a complex subsystem of classes, libraries, or frameworks. This pattern makes a complex library or system easier to use by hiding the underlying complexities and exposing only the functionality that is relevant for the client.
"},{"location":"fundamentaldives/DesignPatterns/Facade/#what","title":"What ?","text":"Let's go through a simple Java example demonstrating a Facade pattern for a Home Theater System:
// Subsystem classes\nclass Amplifier {\n public void on() { System.out.println(\"Amplifier is ON.\"); }\n public void off() { System.out.println(\"Amplifier is OFF.\"); }\n}\n\nclass DVDPlayer {\n public void play() { System.out.println(\"Playing movie.\"); }\n public void stop() { System.out.println(\"Stopping movie.\"); }\n}\n\nclass Projector {\n public void on() { System.out.println(\"Projector is ON.\"); }\n public void off() { System.out.println(\"Projector is OFF.\"); }\n}\n\n// Facade class\nclass HomeTheaterFacade {\n private Amplifier amplifier;\n private DVDPlayer dvdPlayer;\n private Projector projector;\n\n public HomeTheaterFacade(Amplifier amp, DVDPlayer dvd, Projector proj) {\n this.amplifier = amp;\n this.dvdPlayer = dvd;\n this.projector = proj;\n }\n\n public void watchMovie() {\n System.out.println(\"Setting up movie...\");\n amplifier.on();\n projector.on();\n dvdPlayer.play();\n }\n\n public void endMovie() {\n System.out.println(\"Shutting down movie...\");\n dvdPlayer.stop();\n projector.off();\n amplifier.off();\n }\n}\n\n// Client code\npublic class FacadePatternDemo {\n public static void main(String[] args) {\n Amplifier amp = new Amplifier();\n DVDPlayer dvd = new DVDPlayer();\n Projector proj = new Projector();\n\n HomeTheaterFacade homeTheater = new HomeTheaterFacade(amp, dvd, proj);\n\n homeTheater.watchMovie();\n homeTheater.endMovie();\n }\n}\nSetting up movie...\nAmplifier is ON.\nProjector is ON.\nPlaying movie.\nShutting down movie...\nStopping movie.\nProjector is OFF.\nAmplifier is OFF.\nIn Spring Boot, the Facade pattern can be applied to services or controllers to hide the complexity of business logic or external systems. For example, a facade class can wrap multiple service calls or integrate external APIs to provide a simplified interface to the client (like a REST controller).
Let's go through a example of how to apply the Facade pattern in a Spring Boot application.
Example Scenario: A Payment System interacts with several services (like PaymentGatewayService, NotificationService, and OrderService). We create a PaymentFacade to simplify the interaction.
Step-1: Subsystem Services@Service\npublic class PaymentGatewayService {\n public void processPayment(String orderId) {\n System.out.println(\"Processing payment for order: \" + orderId);\n }\n}\n\n@Service\npublic class NotificationService {\n public void sendNotification(String message) {\n System.out.println(\"Sending notification: \" + message);\n }\n}\n\n@Service\npublic class OrderService {\n public void completeOrder(String orderId) {\n System.out.println(\"Completing order: \" + orderId);\n }\n}\n@Service\npublic class PaymentFacade {\n\n private final PaymentGatewayService paymentGatewayService;\n private final NotificationService notificationService;\n private final OrderService orderService;\n\n @Autowired\n public PaymentFacade(PaymentGatewayService paymentGatewayService,\n NotificationService notificationService,\n OrderService orderService) {\n this.paymentGatewayService = paymentGatewayService;\n this.notificationService = notificationService;\n this.orderService = orderService;\n }\n\n public void makePayment(String orderId) {\n System.out.println(\"Initiating payment process...\");\n paymentGatewayService.processPayment(orderId);\n orderService.completeOrder(orderId);\n notificationService.sendNotification(\"Payment completed for order: \" + orderId);\n }\n}\n@RestController\n@RequestMapping(\"/api/payment\")\npublic class PaymentController {\n\n private final PaymentFacade paymentFacade;\n\n @Autowired\n public PaymentController(PaymentFacade paymentFacade) {\n this.paymentFacade = paymentFacade;\n }\n\n @PostMapping(\"/pay/{orderId}\")\n public ResponseEntity<String> pay(@PathVariable String orderId) {\n paymentFacade.makePayment(orderId);\n return ResponseEntity.ok(\"Payment successful for order: \" + orderId);\n }\n}\nPaymentGatewayService, OrderService, and NotificationService.PaymentFacade to expose a single endpoint for making payments.The Facade Pattern is a powerful tool for simplifying interactions with complex systems. It is especially useful when working with large subsystems or external APIs, as it encapsulates the internal workings and provides a simple interface to clients. In a Spring Boot application, you can use it to manage complex business logic or interactions with multiple services within a single, cohesive facade. However, it should be used judiciously to avoid over-abstraction or unnecessary complexity.
Note
This makes the Facade pattern a valuable asset in both object-oriented design and modern frameworks like Spring Boot.
"},{"location":"fundamentaldives/DesignPatterns/Facade/#this-pattern-is-best-used-when","title":"This pattern is best used when:","text":"- You need to simplify client interactions.\n- You want to decouple the client from a complex subsystem.\n- You aim to improve maintainability and reduce dependencies.\n- The system is already simple.\n- Performance is a key concern.\nFactory Method is a creational design pattern that provides an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created (decide which class to instantiate). This pattern delegates the responsibility of object creation to subclasses rather than using a direct constructor call, It is one of the most widely used creational design patterns, It helps in the creation of objects without specifying the exact class of the object that will be created.
You provide a \"factory\" method that the client code calls to get the object, but the actual object that gets created is determined at runtime (based on some logic).
"},{"location":"fundamentaldives/DesignPatterns/FactoryMethod/#when-to-use","title":"When to Use ?","text":"public interface Notification {\n void notifyUser();\n}\npublic class SMSNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an SMS notification.\");\n }\n}\n\npublic class EmailNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an Email notification.\");\n }\n}\npublic abstract class NotificationFactory {\n public abstract Notification createNotification();\n}\npublic class SMSNotificationFactory extends NotificationFactory {\n @Override\n public Notification createNotification() {\n return new SMSNotification();\n }\n}\n\npublic class EmailNotificationFactory extends NotificationFactory {\n @Override\n public Notification createNotification() {\n return new EmailNotification();\n }\n}\npublic class Client {\n public static void main(String[] args) {\n NotificationFactory factory = new SMSNotificationFactory();\n Notification notification = factory.createNotification();\n notification.notifyUser();\n\n factory = new EmailNotificationFactory();\n notification = factory.createNotification();\n notification.notifyUser();\n }\n}\nSpring Boot relies heavily on dependency injection (DI) and Inversion of Control (IoC), which means Spring beans can act as factory classes to produce the desired objects.
"},{"location":"fundamentaldives/DesignPatterns/FactoryMethod/#example-notification-factory-with-spring-boot","title":"Example: Notification Factory with Spring Boot","text":"Define the Product Interface and Implementations (Same as Before)public interface Notification {\n void notifyUser();\n}\n\npublic class SMSNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an SMS notification.\");\n }\n}\n\npublic class EmailNotification implements Notification {\n @Override\n public void notifyUser() {\n System.out.println(\"Sending an Email notification.\");\n }\n}\nimport org.springframework.stereotype.Service;\n// This class will act as the **Factory**. You can make it a Spring **`@Service` or `@Component`** bean, so Spring manages it.\n@Service\npublic class NotificationFactory {\n public Notification createNotification(String type) {\n if (type.equalsIgnoreCase(\"SMS\")) {\n return new SMSNotification();\n } else if (type.equalsIgnoreCase(\"Email\")) {\n return new EmailNotification();\n }\n throw new IllegalArgumentException(\"Unknown notification type: \" + type);\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.PathVariable;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class NotificationController {\n\n @Autowired\n private NotificationFactory notificationFactory;\n\n @GetMapping(\"/notify/{type}\")\n public String sendNotification(@PathVariable String type) {\n Notification notification = notificationFactory.createNotification(type);\n notification.notifyUser();\n return \"Notification sent: \" + type;\n }\n}\nWhen you access /notify/SMS or /notify/Email, it will dynamically create and send the corresponding notification after running the spring boot application.
The Factory Pattern enables dynamic object creation without specifying exact classes, reducing coupling and improving maintainability. It uses a factory method to determine which class to instantiate. In Spring Boot, this pattern integrates seamlessly through factory beans and dependency injection, providing flexible and condition-based object creation.
Note
Factory Method is especially helpful in modular systems where new functionalities might be added frequently, and we want to minimize the impact of changes to existing code.
"},{"location":"fundamentaldives/DesignPatterns/Iterator/","title":"Iterator","text":"The Iterator pattern is a behavioral design pattern that allows sequential access to elements of a collection without exposing its underlying structure. The goal is to provide a way to access the elements of an aggregate object (such as an array, list, or set) one by one, without needing to understand how the collection is implemented.
"},{"location":"fundamentaldives/DesignPatterns/Iterator/#what","title":"What ?","text":"Iterator is a behavioral design pattern that lets you traverse elements of a collection without exposing its underlying representation (list, stack, tree, etc.).
Key Components
hasNext(), next()).Iterator interface for specific collections.iterator()).ConcurrentModificationException.Let's go through with a simple Java example of a custom iterator for a list of strings.
Defining the Iterator Interfaceinterface Iterator<T> {\n boolean hasNext();\n T next();\n}\nclass NameIterator implements Iterator<String> {\n private String[] names;\n private int index;\n\n public NameIterator(String[] names) {\n this.names = names;\n }\n\n @Override\n public boolean hasNext() {\n return index < names.length;\n }\n\n @Override\n public String next() {\n if (this.hasNext()) {\n return names[index++];\n }\n return null;\n }\n}\ninterface NameCollection {\n Iterator<String> getIterator();\n}\nclass NameRepository implements NameCollection {\n private String[] names = {\"John\", \"Alice\", \"Robert\", \"Michael\"};\n\n @Override\n public Iterator<String> getIterator() {\n return new NameIterator(names);\n }\n}\npublic class IteratorPatternDemo {\n public static void main(String[] args) {\n NameRepository namesRepository = new NameRepository();\n Iterator<String> iterator = namesRepository.getIterator();\n\n while (iterator.hasNext()) {\n String name = iterator.next();\n System.out.println(\"Name: \" + name);\n }\n }\n}\nJava already provides built-in iterators for its collection framework (Iterator, ListIterator, and Spliterator).
import java.util.ArrayList;\nimport java.util.Iterator;\nimport java.util.List;\n\npublic class BuiltInIteratorExample {\n public static void main(String[] args) {\n List<String> names = new ArrayList<>();\n names.add(\"John\");\n names.add(\"Alice\");\n names.add(\"Robert\");\n\n Iterator<String> iterator = names.iterator();\n while (iterator.hasNext()) {\n System.out.println(\"Name: \" + iterator.next());\n }\n }\n}\nSpring Boot doesn\u2019t explicitly use the Iterator pattern as part of its core framework, but it does utilize Iterators internally (e.g., when dealing with ApplicationContext beans or data access layers). You can integrate the Iterator pattern within Spring Boot to handle collections of objects such as configurations, database entities, or APIs.
public class Book {\n private String title;\n\n public Book(String title) {\n this.title = title;\n }\n\n public String getTitle() {\n return title;\n }\n}\nimport java.util.ArrayList;\nimport java.util.List;\n\npublic class BookRepository {\n private List<Book> books = new ArrayList<>();\n\n public BookRepository() {\n books.add(new Book(\"Spring in Action\"));\n books.add(new Book(\"Java 8 in Action\"));\n books.add(new Book(\"Microservices Patterns\"));\n }\n\n public Iterator<Book> getIterator() {\n return books.iterator();\n }\n}\nimport org.springframework.stereotype.Service;\nimport java.util.Iterator;\n\n@Service\npublic class BookService {\n private final BookRepository bookRepository = new BookRepository();\n\n public void printBooks() {\n Iterator<Book> iterator = bookRepository.getIterator();\n while (iterator.hasNext()) {\n Book book = iterator.next();\n System.out.println(\"Book: \" + book.getTitle());\n }\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RequestMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\n@RequestMapping(\"/books\")\npublic class BookController {\n\n @Autowired\n private BookService bookService;\n\n @GetMapping\n public void listBooks() {\n bookService.printBooks();\n }\n}\nWhen you access http://localhost:8080/books, it will print the list of books using the iterator after running the application
The Iterator pattern is a powerful way to decouple iteration logic from collections, ensuring a clean separation of concerns. It\u2019s useful in Java projects where complex or multiple ways of traversal are required. When integrated into Spring Boot, the pattern can be applied to iterate over configurations, data models, or APIs efficiently.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/","title":"Prototype","text":""},{"location":"fundamentaldives/DesignPatterns/Prototype/#what","title":"What ?","text":"The Prototype Pattern is a creational design pattern used when the cost of creating a new object is expensive or complicated. Instead of creating new instances from scratch, this pattern suggests cloning existing objects to produce new ones.
The pattern allows cloning or copying existing instances to create new ones, ensuring that new objects are created without going through the expensive or complex instantiation process repeatedly.
Key Characteristics
In Java, the Prototype Pattern is implemented by making the class implement the Cloneable interface and overriding the clone() method from the Object class.
class Address {\n String street;\n String city;\n\n public Address(String street, String city) {\n this.street = street;\n this.city = city;\n }\n\n // Deep Copy\n public Address(Address address) {\n this.street = address.street;\n this.city = address.city;\n }\n\n @Override\n public String toString() {\n return street + \", \" + city;\n }\n}\n\nclass Employee implements Cloneable {\n String name;\n Address address;\n\n public Employee(String name, Address address) {\n this.name = name;\n this.address = address;\n }\n\n // Shallow Copy\n @Override\n protected Object clone() throws CloneNotSupportedException {\n return super.clone();\n }\n\n // Deep Copy\n public Employee deepClone() {\n return new Employee(this.name, new Address(this.address));\n }\n\n @Override\n public String toString() {\n return \"Employee: \" + name + \", Address: \" + address;\n }\n}\n\npublic class PrototypeExample {\n public static void main(String[] args) throws CloneNotSupportedException {\n Employee emp1 = new Employee(\"Alice\", new Address(\"123 Street\", \"New York\"));\n\n // Shallow Clone\n Employee emp2 = (Employee) emp1.clone();\n\n // Deep Clone\n Employee emp3 = emp1.deepClone();\n\n System.out.println(\"Original: \" + emp1);\n System.out.println(\"Shallow Copy: \" + emp2);\n System.out.println(\"Deep Copy: \" + emp3);\n\n // Modify the original object to see the effect on shallow vs deep copy\n emp1.address.street = \"456 Avenue\";\n\n System.out.println(\"After modifying the original object:\");\n System.out.println(\"Original: \" + emp1);\n System.out.println(\"Shallow Copy: \" + emp2);\n System.out.println(\"Deep Copy: \" + emp3);\n }\n}\nOriginal: Employee: Alice, Address: 123 Street, New York\nShallow Copy: Employee: Alice, Address: 123 Street, New York\nDeep Copy: Employee: Alice, Address: 123 Street, New York\n\nAfter modifying the original object:\nOriginal: Employee: Alice, Address: 456 Avenue, New York\nShallow Copy: Employee: Alice, Address: 456 Avenue, New York\nDeep Copy: Employee: Alice, Address: 123 Street, New York\nclone()): Modifying the original affects the copy (since both reference the same Address object).deepClone()): The copy remains unaffected because it contains a completely new Address object.Spring Framework allows defining prototype-scoped beans. Each time you request a bean with the prototype scope, Spring returns a new instance, effectively following the Prototype Pattern.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/#how-to-use-prototype-scope-in-spring-boot","title":"How to Use Prototype Scope in Spring Boot","text":"@Scope annotation to the bean definition.prototype scope to ensure each request gets a new object.import org.springframework.context.annotation.Scope;\nimport org.springframework.stereotype.Component;\n\n@Component\n@Scope(\"prototype\")\npublic class Employee {\n public Employee() {\n System.out.println(\"New Employee instance created.\");\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class EmployeeController {\n\n @Autowired\n private Employee employee1;\n\n @Autowired\n private Employee employee2;\n\n @GetMapping(\"/employees\")\n public String getEmployees() {\n return \"Employee 1: \" + employee1 + \" | Employee 2: \" + employee2;\n }\n}\nWhen you run this application and hit the /employees endpoint, you will see:
New Employee instance created.\nNew Employee instance created.\nThis shows that a new instance is created each time a prototype-scoped bean is injected.
"},{"location":"fundamentaldives/DesignPatterns/Prototype/#summary","title":"Summary","text":"The Prototype Pattern offers an elegant way to clone existing objects, saving the overhead of complex object creation. It fits well when objects are expensive to create or share the same initial configuration. With Java's cloning mechanisms and Spring Boot's prototype scope, it is easy to implement. However, care must be taken when handling deep versus shallow copies, and the pattern should be avoided when objects are inexpensive to create.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/","title":"Singleton","text":""},{"location":"fundamentaldives/DesignPatterns/Singleton/#what","title":"What ?","text":"The Singleton Pattern is a creational design pattern that ensures a class has only one instance and provides a global access point to that instance.
Singleton is useful when exactly one instance of a class is needed across the system, like for logging, configuration, database connection pools, etc.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#when-to-use","title":"When to Use ?","text":"The instance is created when the class is loaded. This is the simplest way, but it doesn\u2019t support lazy loading.
Eager Initialization Implementationpublic class EagerSingleton {\n private static final EagerSingleton INSTANCE = new EagerSingleton();\n\n // Private constructor to prevent instantiation\n private EagerSingleton() {}\n\n public static EagerSingleton getInstance() {\n return INSTANCE;\n }\n}\nWhen to Use Eager
When the instance is required throughout the application, and we are okay with it being created at startup.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#lazy-initialization","title":"Lazy Initialization","text":"The instance is created only when needed (on first access). But this version is not thread-safe.
Lazy Initialization Implementationpublic class LazySingleton {\n private static LazySingleton instance;\n\n private LazySingleton() {}\n\n public static LazySingleton getInstance() {\n if (instance == null) {\n instance = new LazySingleton();\n }\n return instance;\n }\n}\nIssue with Lazy
Lazy Initialization is not suitable for multithreaded environments.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#using-synchronized","title":"Using Synchronized","text":"This solves the issue of thread safety by synchronizing the access method.
Synchronized Implementationpublic class ThreadSafeSingleton {\n private static ThreadSafeSingleton instance;\n\n private ThreadSafeSingleton() {}\n\n public static synchronized ThreadSafeSingleton getInstance() {\n if (instance == null) {\n instance = new ThreadSafeSingleton();\n }\n return instance;\n }\n}\nIssue with Synchronized
Performance overhead due to synchronization.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#double-checked-locking","title":"Double-Checked Locking","text":"This improves the performance by reducing the overhead of synchronized block.
Double-Checked Locking Implementationpublic class DoubleCheckedLockingSingleton {\n private static volatile DoubleCheckedLockingSingleton instance;\n\n private DoubleCheckedLockingSingleton() {}\n\n public static DoubleCheckedLockingSingleton getInstance() {\n if (instance == null) {\n synchronized (DoubleCheckedLockingSingleton.class) {\n if (instance == null) {\n instance = new DoubleCheckedLockingSingleton();\n }\n }\n }\n return instance;\n }\n}\nThis approach leverages static inner classes, which ensures thread safety and lazy loading without synchronization overhead.
Bill Pugh Singleton Implementationpublic class BillPughSingleton {\n private BillPughSingleton() {}\n\n // Static inner class responsible for holding the instance\n private static class SingletonHelper {\n private static final BillPughSingleton INSTANCE = new BillPughSingleton();\n }\n\n public static BillPughSingleton getInstance() {\n return SingletonHelper.INSTANCE;\n }\n}\nBest Practice
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#enum-singleton","title":"Enum Singleton","text":"This approach is the most concise and prevents issues with serialization and reflection attacks.
Enum Singleton Implementationpublic enum EnumSingleton {\n INSTANCE;\n\n public void someMethod() {\n System.out.println(\"Enum Singleton Instance\");\n }\n}\nRecommended
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#in-spring-boot","title":"In Spring Boot","text":"In Spring Boot, Spring\u2019s IoC container (Inversion of Control) makes singleton beans by default. Each bean in Spring is, by default, a singleton. So, you don\u2019t need to explicitly implement the Singleton pattern. Instead, you annotate the class with @Component or @Service, and Spring ensures that only one instance is created and managed.
import org.springframework.stereotype.Component;\n\n@Component\npublic class MySingletonService {\n public void doSomething() {\n System.out.println(\"Singleton service is working\");\n }\n}\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.GetMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n@RestController\npublic class MyController {\n\n private final MySingletonService singletonService;\n\n @Autowired\n public MyController(MySingletonService singletonService) {\n this.singletonService = singletonService;\n }\n\n @GetMapping(\"/test\")\n public String test() {\n singletonService.doSomething();\n return \"Check logs for Singleton Service\";\n }\n}\nNote
Spring manages lifecycle and thread safety for you, ensuring it behaves like a Singleton without extra code.
"},{"location":"fundamentaldives/DesignPatterns/Singleton/#comparison","title":"Comparison","text":"Implementation Thread Safety Lazy Initialization Serialization Safe Ease of Implementation Eager Initialization Yes No No Easy Lazy Initialization No Yes No Easy Thread-safe Singleton (Synchronized) Yes Yes No Moderate Double-Checked Locking Singleton Yes Yes No Moderate Bill Pugh Singleton Yes Yes No Best Practice Enum Singleton Yes Yes Yes Recommended"},{"location":"fundamentaldives/DesignPatterns/Singleton/#potential-issues","title":"Potential Issues","text":"The Singleton Pattern is a powerful tool when used appropriately. However, misuse can lead to tightly coupled code, concurrency issues, and testing difficulties.
Note
If you are working with Spring Boot, rely on Spring\u2019s built-in singleton beans instead of implementing your own singleton logic. Where thread safety, serialization, or distributed behavior is required, choose the appropriate Singleton implementation like Enum Singleton or Bill Pugh Singleton.
By default, a single instance of the bean is created and shared across the entire application (singleton scope). If two or more components use the same bean, they will refer to the same instance. However, if you need a new instance every time a bean is requested, you can change the scope to prototype. But be mindful Spring\u2019s singleton scope simplifies things like caching and state consistency, while prototype beans may introduce complexity.
The Strategy Pattern is a behavioral design pattern that allows you to define a family of algorithms, encapsulate each one, and make them interchangeable. It lets the algorithm vary independently from clients that use it, promoting flexibility, scalability, and separation of concerns.
In simpler terms, It enables selecting a specific algorithm at runtime based on the context, without modifying the client code.
"},{"location":"fundamentaldives/DesignPatterns/Strategy/#when-to-use","title":"When to Use?","text":"if-else or switch statements).Let\u2019s implement a payment system that allows different payment methods using the Strategy Pattern. We will define multiple payment strategies (like Credit Card and PayPal) and switch between them dynamically.
Step-1: Create the Strategy Interface// PaymentStrategy.java\npublic interface PaymentStrategy {\n void pay(int amount);\n}\n// CreditCardStrategy.java\npublic class CreditCardStrategy implements PaymentStrategy {\n private String cardNumber;\n private String name;\n\n public CreditCardStrategy(String cardNumber, String name) {\n this.cardNumber = cardNumber;\n this.name = name;\n }\n\n @Override\n public void pay(int amount) {\n System.out.println(amount + \" paid with credit card.\");\n }\n}\n\n// PayPalStrategy.java\npublic class PayPalStrategy implements PaymentStrategy {\n private String email;\n\n public PayPalStrategy(String email) {\n this.email = email;\n }\n\n @Override\n public void pay(int amount) {\n System.out.println(amount + \" paid using PayPal.\");\n }\n}\n// PaymentContext.java\npublic class PaymentContext {\n private PaymentStrategy strategy;\n\n public PaymentContext(PaymentStrategy strategy) {\n this.strategy = strategy;\n }\n\n public void setPaymentStrategy(PaymentStrategy strategy) {\n this.strategy = strategy;\n }\n\n public void pay(int amount) {\n strategy.pay(amount);\n }\n}\npublic class StrategyPatternDemo {\n public static void main(String[] args) {\n PaymentContext context = new PaymentContext(new CreditCardStrategy(\"1234-5678-9012\", \"John Doe\"));\n context.pay(100);\n\n // Switch strategy at runtime\n context.setPaymentStrategy(new PayPalStrategy(\"john.doe@example.com\"));\n context.pay(200);\n }\n}\nIn a Spring Boot application, the Strategy Pattern can be applied by injecting different strategy implementations using Spring\u2019s dependency injection.
Let's build a simple notification service where the user can choose between sending notifications via Email or SMS.
Create the Strategy Interface// NotificationStrategy.java\npublic interface NotificationStrategy {\n void sendNotification(String message);\n}\n// EmailNotification.java\nimport org.springframework.stereotype.Service;\n\n@Service(\"email\")\npublic class EmailNotification implements NotificationStrategy {\n @Override\n public void sendNotification(String message) {\n System.out.println(\"Sending Email: \" + message);\n }\n}\n\n// SMSNotification.java\nimport org.springframework.stereotype.Service;\n\n@Service(\"sms\")\npublic class SMSNotification implements NotificationStrategy {\n @Override\n public void sendNotification(String message) {\n System.out.println(\"Sending SMS: \" + message);\n }\n}\n// NotificationContext.java\nimport org.springframework.stereotype.Component;\n\n@Component\npublic class NotificationContext {\n\n private final Map<String, NotificationStrategy> strategies;\n\n public NotificationContext(Map<String, NotificationStrategy> strategies) {\n this.strategies = strategies;\n }\n\n public void send(String type, String message) {\n NotificationStrategy strategy = strategies.get(type);\n if (strategy == null) {\n throw new IllegalArgumentException(\"No such notification type\");\n }\n strategy.sendNotification(message);\n }\n}\n// NotificationController.java\nimport org.springframework.web.bind.annotation.*;\n\n@RestController\n@RequestMapping(\"/notify\")\npublic class NotificationController {\n\n private final NotificationContext context;\n\n public NotificationController(NotificationContext context) {\n this.context = context;\n }\n\n @PostMapping(\"/{type}\")\n public void sendNotification(@PathVariable String type, @RequestBody String message) {\n context.send(type, message);\n }\n}\n// Application.java\nimport org.springframework.boot.SpringApplication;\nimport org.springframework.boot.autoconfigure.SpringBootApplication;\n\n@SpringBootApplication\npublic class Application {\n public static void main(String[] args) {\n SpringApplication.run(Application.class, args);\n }\n}\nemail or sms).@Service annotation./notify/email or /notify/sms).Using Enum-based Strategies: If the algorithms are simple and limited, you can use an enum with methods for strategy logic.
public enum Operation {\n ADD {\n @Override\n public int execute(int a, int b) {\n return a + b;\n }\n },\n SUBTRACT {\n @Override\n public int execute(int a, int b) {\n return a - b;\n }\n };\n\n public abstract int execute(int a, int b);\n}\nUsing Java 8 Lambdas: Since Java 8, you can use lambdas to avoid creating multiple strategy classes.
import java.util.function.Consumer;\n\npublic class LambdaStrategyDemo {\n public static void main(String[] args) {\n Consumer<String> emailStrategy = message -> System.out.println(\"Email: \" + message);\n Consumer<String> smsStrategy = message -> System.out.println(\"SMS: \" + message);\n\n emailStrategy.accept(\"Hello via Email!\");\n smsStrategy.accept(\"Hello via SMS!\");\n }\n}\nThe Strategy Pattern is a powerful way to manage dynamic behavior selection in a clean and decoupled way. In a Spring Boot application, you can easily integrate it by using dependency injection. However, it\u2019s essential to use the pattern wisely to avoid unnecessary complexity or overhead. Use it when multiple behaviors or algorithms need to vary independently without modifying client code. Avoid using it if the added complexity is not justified.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/","title":"Concurrency and Parallelism","text":"Both concurrency and parallelism refer to ways a computer performs multiple tasks, but they differ in how the tasks are executed. Let's go through them, along with related concepts like threads, processes, and programs in this article.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#what-is-process","title":"What is Process ?","text":"A process is an instance of a running program (an executable code). Each process runs in isolation and gets its own memory space. eg: Opening a browser or text editor creates a process.
Characteristics
A thread is the smallest unit of execution within a process. Threads run within a process and share the same memory space. eg: A web browser might use multiple threads one for rendering pages, one for handling user input, and another for downloading files.
Characteristics
Concurrency is when multiple tasks make progress within the same time frame, but not necessarily at the same exact moment. Tasks switch back and forth, sharing resources like CPU time.
Analogy
It\u2019s like a chef preparing multiple dishes, working on one dish for a few minutes, switching to another, and then returning to the previous dish.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#concurrency-with-threads","title":"Concurrency with Threads","text":"Note
Concurrency focuses on dealing with multiple tasks by time-sharing resources.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#issues-in-concurrency","title":"Issues in Concurrency","text":"Below are key issues associated with concurrency.
Race ConditionsWhen two or more threads/processes try to access and modify shared data simultaneously, the final result may depend on the sequence of execution, leading to unpredictable outcomes.
Example: Two bank transactions updating the same account balance at the same time might result in lost updates.
How to Mitigate
Use synchronization mechanisms like locks or mutexes to control access to shared resources.
DeadlocksOccurs when two or more threads/processes block each other by holding resources and waiting for resources held by the other. Example: Process A holds Resource 1 and waits for Resource 2, which is held by Process B, and vice versa.
How to Mitigate
Use techniques like resource ordering, deadlock detection, or timeouts to avoid deadlocks.
LivelockIn livelock, threads are constantly changing states to respond to each other but never make actual progress. It\u2019s similar to two people trying to step aside but always stepping in each other\u2019s way. Example: Two threads repeatedly yield control to avoid conflict, but neither progresses.
How to Mitigate
Add randomness or back-off mechanisms to break the cycle.
StarvationA thread or process may be blocked indefinitely because other higher-priority tasks consume all the resources. Example: A low-priority thread never gets CPU time because high-priority threads always take precedence.
How to Mitigate
Use fair scheduling algorithms to ensure all tasks eventually get a chance to execute.
Context Switching OverheadSwitching between multiple threads or processes incurs a cost, as the CPU saves and restores the state of each thread. Excessive context switching can reduce performance. Example: An overloaded web server with too many threads may spend more time switching contexts than doing actual work.
How to Mitigate
Minimize the number of threads and optimize the task scheduling.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#what-is-parallelism","title":"What is Parallelism ?","text":"Parallelism is when multiple tasks are executed simultaneously, usually on different processors or cores.
Analogy
It\u2019s like having multiple chefs, each cooking one dish at the same time.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#parallelism-with-threads","title":"Parallelism with Threads","text":"Note
Parallelism requires multiple CPUs or cores for real simultaneous execution.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#issues-in-parallelism","title":"Issues in Parallelism","text":"Below are key issues associated with parallelism.
Load ImbalanceIf the workload is not evenly distributed across threads or processes, some cores might remain underutilized while others are overloaded. Example: In a matrix multiplication task, if one thread processes a large chunk and another a small chunk, the first thread might take longer, slowing down the whole task.
How to Mitigate
Use dynamic load balancing or work stealing techniques to distribute the workload effectively.
Scalability BottlenecksAs more threads or processes are added, the overhead of synchronization and communication increases, limiting performance improvements. Example: A program may scale well with 4 threads but show diminishing returns with 16 threads due to synchronization overhead.
How to Mitigate
Optimize algorithms for scalability and minimize shared resources to reduce synchronization costs.
False SharingOccurs when multiple threads on different cores modify variables that are close in memory, leading to unnecessary cache invalidations and reduced performance. Example: Two threads updating variables in the same cache line can cause frequent cache synchronization, slowing execution.
How to Mitigate
Align data properly in memory to avoid false sharing.
Communication OverheadIn parallel systems, threads or processes may need to communicate with each other, which adds overhead. Example: In distributed computing, passing messages between nodes can slow down the computation.
How to Mitigate
Reduce communication frequency or use message batching techniques to minimize overhead.
Debugging and Testing ComplexityDebugging concurrent or parallel programs is harder because issues like race conditions or deadlocks may only appear under specific conditions, making them difficult to reproduce. Example: A race condition might only occur when threads execute in a specific order, which is hard to detect in testing.
How to Mitigate
Use debugging tools like thread analyzers and log events to trace execution paths.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#example-scenarios","title":"Example Scenarios","text":"Concurrent Programming (e.g., in Java, Python)
async/await in Python), tasks switch between each other, sharing the same thread.Parallel Programming (e.g., using Python's multiprocessing or CUDA for GPU computation)
Ensuring that shared data remains consistent when accessed by multiple threads or processes is challenging. Example: If multiple threads increment the same counter, the final result may be incorrect without proper synchronization.
How to Mitigate
Use locks, semaphores, or atomic operations to ensure data consistency.
Performance Trade-offsParallel or concurrent execution does not always lead to better performance. In some cases, overhead from synchronization, communication, and context switching can negate performance gains. Example: A parallel algorithm may run slower on a small dataset due to the overhead of managing multiple threads.
How to Mitigate
Assess whether the overhead is justified and use profiling tools to analyze performance.
Non-Deterministic BehaviorIn concurrent and parallel systems, the order of execution is not guaranteed, leading to non-deterministic results. Example: Running the same multi-threaded program twice may produce different outcomes, making testing and debugging difficult.
How to Mitigate
Use locks and barriers carefully, and design programs to tolerate or avoid non-determinism where possible.
Resource ContentionThreads and processes compete for shared resources, such as memory, I/O, and network bandwidth, leading to bottlenecks. Example: Multiple processes writing to the same disk simultaneously may degrade performance.
How to Mitigate
Optimize resource usage and avoid unnecessary contention by reducing shared resources.
"},{"location":"fundamentaldives/FundamentalConcepts/ConcurrencyParallelism/#summary","title":"Summary","text":"Concurrency deals with multiple tasks making progress within the same period (may or may not be simultaneous) whereas Parallelism deals with tasks running simultaneously on different cores or processors.
Processes and threads are core to both concurrency and parallelism, with threads sharing memory within a process and processes running independently with isolated memory.
While concurrency and parallelism offer significant benefits, they also come with substantial challenges. Managing issues such as race conditions, deadlocks, false sharing, and debugging complexity requires thoughtful design and appropriate use of synchronization techniques. Additionally, scalability bottlenecks and communication overhead can limit the effectiveness of parallel systems.
To mitigate these issues, some fixes are:
The DRY Principle stands for Don\u2019t Repeat Yourself. It is a fundamental software development principle aimed at reducing repetition of code and logic. The main idea is that duplication introduces potential risks, if you need to update logic in multiple places, you might forget some, leading to bugs and inconsistencies. When applied well, it improves maintainability, scalability, and clarity, yupp something that lot of codebases misses.
\"Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.\"
In other words, the DRY principle encourages developers to write modular, reusable code and avoid duplicating the same functionality in multiple places. It encourages us to minimize redundancy and write code that does one thing well, making our lives (and the lives of those who maintain our code) much easier.
"},{"location":"fundamentaldives/FundamentalPrinciples/DRY/#when-to-use","title":"When to Use ?","text":"Let's cover an example where we apply DRY principle to refactor duplicated logic into reusable.
Without DRY (Code Duplication) Examplepublic class OrderService {\n\n public void placeOrder(int productId, int quantity) {\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n // Logic to place order\n System.out.println(\"Order placed for product: \" + productId);\n }\n\n public void cancelOrder(int orderId) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n // Logic to cancel order\n System.out.println(\"Order cancelled: \" + orderId);\n }\n\n public void updateOrder(int orderId, int quantity) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n // Logic to update order\n System.out.println(\"Order updated with new quantity: \" + quantity);\n }\n}\nplaceOrder, cancelOrder, and updateOrder).We can extract the common logic into a reusable private method to apply the DRY principle.
With DRY (Refactored Code) Examplepublic class OrderService {\n\n public void placeOrder(int productId, int quantity) {\n validateQuantity(quantity);\n // Logic to place order\n System.out.println(\"Order placed for product: \" + productId);\n }\n\n public void cancelOrder(int orderId) {\n validateOrderId(orderId);\n // Logic to cancel order\n System.out.println(\"Order cancelled: \" + orderId);\n }\n\n public void updateOrder(int orderId, int quantity) {\n validateOrderId(orderId);\n validateQuantity(quantity);\n // Logic to update order\n System.out.println(\"Order updated with new quantity: \" + quantity);\n }\n\n // Reusable validation methods\n private void validateOrderId(int orderId) {\n if (orderId <= 0) {\n throw new IllegalArgumentException(\"Order ID must be greater than zero.\");\n }\n }\n\n private void validateQuantity(int quantity) {\n if (quantity <= 0) {\n throw new IllegalArgumentException(\"Quantity must be greater than zero.\");\n }\n }\n}\nvalidateOrderId and validateQuantity).The DRY principle ensures that code duplication is minimized for easier maintenance and improved consistency. In our example, we extracted common validation logic to private methods, adhering to DRY and making the code more maintainable. However, always be careful to avoid over-abstraction, as not all code repetition is bad. The goal is to achieve a balance between simplicity and reusability.
"},{"location":"fundamentaldives/FundamentalPrinciples/KISS/","title":"KISS Principle","text":""},{"location":"fundamentaldives/FundamentalPrinciples/KISS/#what","title":"What ?","text":"The KISS Principle stands for \"Keep It Simple, Stupid\". It\u2019s a design principle that emphasizes simplicity, stating that systems and code work best if they are kept simple rather than made unnecessarily complex. The main idea is to avoid over-engineering and unnecessary complications, which can introduce more bugs, make the code harder to maintain, and increase development time.
KISS encourages developers to create code or solutions that are easy to understand, maintain, and modify. The idea is not to use complicated approaches or unnecessary abstractions when a simpler, more straightforward approach will do.
\u201cSimple\u201d here doesn\u2019t mean incomplete or simplistic it means clear, focused, and straightforward.
"},{"location":"fundamentaldives/FundamentalPrinciples/KISS/#when-to-use","title":"When to Use ?","text":"While simplicity is valuable, there are situations where the KISS principle might not apply fully. Over-simplifying can sometimes lead to problems.
Let's Consider a example where we want to check if a number is even or odd.
Non-KISS Complex Codepublic class NumberUtils {\n public static boolean isEven(int number) {\n return isDivisibleByTwo(number);\n }\n\n private static boolean isDivisibleByTwo(int number) {\n if (number % 2 == 0) {\n return true;\n } else {\n return false;\n }\n }\n\n public static void main(String[] args) {\n System.out.println(\"Is 4 even? \" + isEven(4));\n }\n}\nisDivisibleByTwo.public class NumberUtils {\n public static boolean isEven(int number) {\n return number % 2 == 0;\n }\n\n public static void main(String[] args) {\n System.out.println(\"Is 4 even? \" + isEven(4));\n }\n}\nThe KISS principle encourages developers to create simple, maintainable, and readable code, it means avoiding unnecessary complexity. However, KISS must be balanced some projects or scenarios require complexity to meet performance, modularity, or security needs. In the end, applying KISS is about striking the right balance making the solution as simple as possible, but not simpler than necessary.
"},{"location":"fundamentaldives/FundamentalPrinciples/SOLID/","title":"SOLID Principles","text":""},{"location":"fundamentaldives/FundamentalPrinciples/SOLID/#single-responsibility-principle","title":"Single Responsibility Principle","text":"A class should have only one reason to change. This means that each class should focus on a single responsibility or feature.
Violation Example// Violates SRP: User class has multiple responsibilities.\npublic class User {\n private String name;\n private String email;\n\n public void sendEmail(String message) {\n // Sending email logic here...\n System.out.println(\"Email sent to \" + email);\n }\n\n public void saveUser() {\n // Save user to the database\n System.out.println(\"User saved to DB\");\n }\n}\n// Separate responsibilities into different classes.\npublic class User {\n private String name;\n private String email;\n\n // Getters and setters...\n}\n\npublic class UserRepository {\n public void save(User user) {\n System.out.println(\"User saved to DB\");\n }\n}\n\npublic class EmailService {\n public void sendEmail(User user, String message) {\n System.out.println(\"Email sent to \" + user.getEmail());\n }\n}\nSoftware components (classes, functions, etc.) should be open for extension but closed for modification. You shouldn\u2019t modify existing code to add new behavior instead, extend it.
Violation Example// Violates OCP: PaymentProcessor needs to be modified for new payment types.\npublic class PaymentProcessor {\n public void pay(String type) {\n if (type.equals(\"credit\")) {\n System.out.println(\"Processing credit card payment...\");\n } else if (type.equals(\"paypal\")) {\n System.out.println(\"Processing PayPal payment...\");\n }\n }\n}\n// Use an interface for extensibility.\ninterface PaymentMethod {\n void pay();\n}\n\npublic class CreditCardPayment implements PaymentMethod {\n public void pay() {\n System.out.println(\"Processing credit card payment...\");\n }\n}\n\npublic class PayPalPayment implements PaymentMethod {\n public void pay() {\n System.out.println(\"Processing PayPal payment...\");\n }\n}\n\npublic class PaymentProcessor {\n public void processPayment(PaymentMethod paymentMethod) {\n paymentMethod.pay();\n }\n}\nSubclasses should be substitutable for their base class without altering the correctness of the program.
Violation Example// Violates LSP: Square changes the behavior of Rectangle.\nclass Rectangle {\n protected int width, height;\n\n public void setWidth(int width) {\n this.width = width;\n }\n\n public void setHeight(int height) {\n this.height = height;\n }\n\n public int getArea() {\n return width * height;\n }\n}\n\nclass Square extends Rectangle {\n @Override\n public void setWidth(int width) {\n this.width = width;\n this.height = width; // Violates LSP: Unexpected behavior.\n }\n\n @Override\n public void setHeight(int height) {\n this.width = height;\n this.height = height;\n }\n}\n// Use separate classes to maintain correct behavior.\nclass Shape {\n public int getArea() {\n return 0;\n }\n}\n\nclass Rectangle extends Shape {\n protected int width, height;\n\n public Rectangle(int width, int height) {\n this.width = width;\n this.height = height;\n }\n\n @Override\n public int getArea() {\n return width * height;\n }\n}\n\nclass Square extends Shape {\n private int side;\n\n public Square(int side) {\n this.side = side;\n }\n\n @Override\n public int getArea() {\n return side * side;\n }\n}\nA client should not be forced to implement interfaces that it does not use. Instead, smaller, more specific interfaces should be preferred.
Violation Example// Violates ISP: Cat(r) needs to implement unnecessary methods.\ninterface Vehicle {\n void drive();\n void fly();\n}\n\nclass Car implements Vehicle {\n @Override\n public void drive() {\n System.out.println(\"Car is driving...\");\n }\n\n @Override\n public void fly() {\n // Car can't fly! This method is unnecessary.\n throw new UnsupportedOperationException(\"Car can't fly\");\n }\n}\n// Use separate interfaces for each capability.\ninterface Drivable {\n void drive();\n}\n\ninterface Flyable {\n void fly();\n}\n\nclass Car implements Drivable {\n @Override\n public void drive() {\n System.out.println(\"Car is driving...\");\n }\n}\n\nclass Plane implements Drivable, Flyable {\n @Override\n public void drive() {\n System.out.println(\"Plane is taxiing...\");\n }\n\n @Override\n public void fly() {\n System.out.println(\"Plane is flying...\");\n }\n}\nHigh-level modules should not depend on low-level modules. Both should depend on abstractions.
Violation Example// Violates DIP: High-level class depends on a specific implementation.\nclass SQLDatabase {\n public void connect() {\n System.out.println(\"Connected to SQL Database\");\n }\n}\n\nclass Application {\n private SQLDatabase database;\n\n public Application() {\n database = new SQLDatabase(); // Tight coupling to SQLDatabase.\n }\n\n public void start() {\n database.connect();\n }\n}\n// Depend on an abstraction instead of a specific implementation.\ninterface Database {\n void connect();\n}\n\nclass SQLDatabase implements Database {\n public void connect() {\n System.out.println(\"Connected to SQL Database\");\n }\n}\n\nclass NoSQLDatabase implements Database {\n public void connect() {\n System.out.println(\"Connected to NoSQL Database\");\n }\n}\n\nclass Application {\n private Database database;\n\n public Application(Database database) {\n this.database = database;\n }\n\n public void start() {\n database.connect();\n }\n}\nUser, EmailService, UserRepository. Open Closed Open for extension, closed for modification. Modify PaymentProcessor for new methods. Use PaymentMethod interface and extend classes. Liskov Substitution Subtypes should behave like their base type. Square modifies behavior of Rectangle. Separate Square and Rectangle classes. Interface Segregation Use small, specific interfaces. Car implements unnecessary fly() method. Split into Drivable and Flyable interfaces. Dependency Inversion Depend on abstractions, not implementations. App depends on SQLDatabase directly. Use Database interface for loose coupling."},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/","title":"YAGNI Principle","text":""},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/#what","title":"What ?","text":"YAGNI stands for \"You Aren\u2019t Gonna Need It.\" It is one of the core principles of Extreme Programming (XP) and Agile development. The principle advises developers not to add any functionality or code until it is truly needed. Essentially, YAGNI promotes simplicity and avoids speculative development.
Always implement things when you actually need them, never when you just foresee you might need them.
"},{"location":"fundamentaldives/FundamentalPrinciples/YAGNI/#why-use","title":"Why Use ?","text":"Lets go through a simple example to illustrate YAGNI in practice.
Without YAGNI Example Adding Unnecessary Functionalitypublic class UserService {\n\n // Current functionality: Retrieve a user by ID\n public User getUserById(int id) {\n // Logic to retrieve user\n return new User(id, \"John Doe\");\n }\n\n // Unnecessary feature: Speculating that we may need a \"deleteUser\" method in the future\n public void deleteUser(int id) {\n // Logic to delete user (unimplemented)\n System.out.println(\"User deleted: \" + id);\n }\n\n // Another unnecessary feature: Thinking we might need email notifications\n public void sendEmailNotification(User user) {\n // Logic to send email (unimplemented)\n System.out.println(\"Email sent to: \" + user.getEmail());\n }\n}\n\nclass User {\n private int id;\n private String name;\n\n public User(int id, String name) {\n this.id = id;\n this.name = name;\n }\n\n public String getEmail() {\n return name.toLowerCase() + \"@example.com\";\n }\n}\ndeleteUser() method.sendEmailNotification() method.public class UserService {\n\n // Only add the necessary method for now\n public User getUserById(int id) {\n // Logic to retrieve user\n return new User(id, \"John Doe\");\n }\n}\n\nclass User {\n private int id;\n private String name;\n\n public User(int id, String name) {\n this.id = id;\n this.name = name;\n }\n}\ndeleteUser() or sendEmailNotification().deleteUser() or notification feature, we can implement it when it becomes necessary.The YAGNI principle encourages developers to focus on delivering only the required features at a given point in time, avoiding speculative development that may never be used. This approach fosters simplicity, maintainability, and efficiency in the codebase. However, it should be applied carefully there are scenarios (like architecture or security) where anticipating needs is necessary, When used properly, YAGNI helps teams build better software, faster, and with fewer headaches down the line.
"},{"location":"langdives/Java/4Pillars/","title":"4 Pillars - What, How, and Why ?","text":""},{"location":"langdives/Java/4Pillars/#encapsulation","title":"Encapsulation","text":"What: Hiding the internal details of an object and only exposing necessary parts through public methods.
Why: It helps in data hiding and ensures controlled access to variables.
How: Use private variables to restrict direct access and provide getters and setters to access and modify the data.
Encapsulation Examplepublic class Person {\n private String name; // Encapsulated field\n\n // Getter\n public String getName() {\n return name;\n }\n\n // Setter\n public void setName(String name) {\n this.name = name;\n }\n}\nWhat: Allows a class (child/subclass) to acquire the properties and behaviors of another class (parent/superclass).
How: Use the extends keyword.
Why: Promotes code reusability and establishes a parent-child relationship.
Inheritance Exampleclass Animal {\n public void sound() {\n System.out.println(\"Animals make sound\");\n }\n}\n\nclass Dog extends Animal {\n @Override\n public void sound() {\n System.out.println(\"Dog barks\");\n }\n}\nWhat: Ability to process objects differently based on their data type or class.
Why: Increases flexibility and supports dynamic method invocation.
Polymorphism Example Method Overloadingclass Calculator {\n public int add(int a, int b) {\n return a + b;\n }\n\n public double add(double a, double b) {\n return a + b;\n }\n}\nclass Animal {\n public void sound() {\n System.out.println(\"Animal makes sound\");\n }\n}\n\nclass Cat extends Animal {\n @Override\n public void sound() {\n System.out.println(\"Cat meows\");\n }\n}\nWhat: Hiding the complex implementation details and only exposing the essential features.
Why: Helps in achieving modularity and loose coupling between components.
How: Use abstract classes and interfaces.
Abstraction Example Abstract Class Exampleabstract class Vehicle {\n abstract void start();\n}\n\nclass Car extends Vehicle {\n @Override\n void start() {\n System.out.println(\"Car starts with a key\");\n }\n}\ninterface Animal {\n void eat();\n}\n\nclass Dog implements Animal {\n @Override\n public void eat() {\n System.out.println(\"Dog eats bones\");\n }\n}\nextends keyword to derive subclasses. Overloading (compile-time) and overriding (runtime). Using abstract classes or interfaces. Key Benefit Data hiding and modular code. Reduces redundancy and promotes code reuse. Flexibility and extensibility of behavior. Promotes loose coupling and modularity. Access Modifiers Requires private, protected, or public. Involves all inheritance-accessible modifiers. Leverages method visibility across class hierarchies. Abstract methods can be protected or public (not private). Real-World Analogy A capsule with medicine inside it hides the internal components. A child inheriting traits from their parents. A shape object behaving differently as circle/square. A remote control exposing buttons without showing internal circuits. Code Dependency Independent within the class. Dependent on parent-child relationship. Involves multiple forms of a single method/class. Can work with unrelated classes sharing common behavior."},{"location":"langdives/Java/AccessModifPPPPP/","title":"Access modifiers","text":""},{"location":"langdives/Java/AccessModifPPPPP/#public","title":"Public","text":"Keyword: public Access: Accessible from anywhere (inside/outside the class, package, or project). Usage: Typically used for classes, methods, and variables that need global access.
public class MyClass {\n public int value = 10;\n public void display() {\n System.out.println(\"Public method\");\n }\n}\nKeyword: private Access: Accessible only within the same class. Usage: Used to hide class fields or methods, following the principle of encapsulation.
public class MyClass {\n private int value = 10; // Not accessible outside this class\n\n private void display() {\n System.out.println(\"Private method\");\n }\n}\nKeyword: protected Access: Accessible within the same package and by subclasses (even if outside the package). Usage: Useful when extending classes across packages.
public class MyClass {\n protected int value = 10;\n\n protected void display() {\n System.out.println(\"Protected method\");\n }\n}\nNote
If accessed by a subclass in a different package, it must be through inheritance (not directly via an instance).
"},{"location":"langdives/Java/AccessModifPPPPP/#package-private","title":"Package-Private","text":"Keyword: No keyword (Default Access) Access: Accessible only within the same package. Usage: Used for classes and members that don\u2019t need to be accessed outside their package.
Package-Private Exampleclass MyClass { // No access modifier, so it's package-private\n int value = 10;\n\n void display() {\n System.out.println(\"Package-private method\");\n }\n}\nNote
Package-private is the default if no modifier is specified.
"},{"location":"langdives/Java/AccessModifPPPPP/#access-summary","title":"Access Summary","text":"Modifier Same Class Same Package Subclass (Different Package) Other Packagespublic \u2705 \u2705 \u2705 \u2705 protected \u2705 \u2705 \u2705 (via inheritance) \u274c (default) \u2705 \u2705 \u274c \u274c private \u2705 \u274c \u274c \u274c"},{"location":"langdives/Java/Collections-JCF/","title":"Java Collections Framework","text":""},{"location":"langdives/Java/Collections-JCF/#categories-of-collections","title":"Categories of Collections","text":"Collections.synchronizedList().A resizable array, fast random access. It's backed by Array, When random access is needed and insertions are rare you can use this.
Operations & Complexities
O(1) O(1) (amortized) O(n) O(n) Thread Safety: Not synchronized, use Collections.synchronizedList() for thread safety.
List<String> list = new ArrayList<>();\nlist.add(\"Apple\");\nlist.get(0); // Fast access\nA Doubly linked list, better at frequent insertions and deletions. It's backed by Doubly Linked List, When insertion/deletion in the middle is frequent you can use this.
Operations & Complexities
O(n) O(1) (at head or tail) O(n)Thread Safety: Not synchronized.
ExampleList<String> list = new LinkedList<>();\nlist.add(\"Banana\");\nlist.addFirst(\"Apple\"); // O(1) insertion at head\nIt's Unordered, no duplicates, backed by Hash Table. You can use this When you need fast lookups and no duplicates.
Operations & Complexities
O(1) (on average) O(n)Thread Safety: Not synchronized, use Collections.synchronizedSet().
Set<String> set = new HashSet<>();\nset.add(\"Cat\");\nset.add(\"Dog\");\nThis maintains insertion order, backed by a Hash Table + Linked List. You can use this when you need order-preserving behavior.
Operations & Complexities: Same as HashSet (O(1) operations) but with slightly higher overhead due to linked list maintenance.
Set<String> set = new LinkedHashSet<>();\nset.add(\"Apple\");\nset.add(\"Banana\");\nA Sorted set, backed by Red-Black Tree, When you need sorted data.
Operations & Complexities
O(log n) O(n)Thread Safety: Not synchronized.
ExampleSet<Integer> set = new TreeSet<>();\nset.add(5);\nset.add(1); // Sorted automatically\nElements are ordered based on their natural order or a custom comparator. It's backed by Binary Heap, When priority-based retrieval is needed.
Operations & Complexities
O(log n) O(1) O(log n) Queue<Integer> queue = new PriorityQueue<>();\nqueue.add(10);\nqueue.add(5); // 5 will be at the top\nResizable-array-backed deque, allows adding/removing from both ends, When you need both stack and queue operations.
Operations & Complexities
O(1) O(n)Deque<String> deque = new ArrayDeque<>();\ndeque.addFirst(\"First\");\ndeque.addLast(\"Last\");\nStores key-value pairs, backed by Hash Table, Fast lookups for key-value pairs.
Operations & Complexities
O(1) (average) O(n)Thread Safety: Not synchronized, use ConcurrentHashMap for thread-safe operations.
Map<String, Integer> map = new HashMap<>();\nmap.put(\"Apple\", 1);\nmap.put(\"Banana\", 2);\nMaintains insertion order, backed by Hash Table + Linked List, When ordering of entries matters.
ExampleMap<String, Integer> map = new LinkedHashMap<>();\nmap.put(\"Apple\", 1);\nmap.put(\"Banana\", 2); // Maintains insertion order\nSorted map, backed by Red-Black Tree, When you need a sorted key-value store.
Operations & Complexities: Get/Put/Remove: O(log n)
Map<Integer, String> map = new TreeMap<>();\nmap.put(3, \"Three\");\nmap.put(1, \"One\"); // Sorted by key\nSynchronized Wrappers: Use Collections.synchronizedList() or Collections.synchronizedSet() to make collections thread-safe.
List<String> list = Collections.synchronizedList(new ArrayList<>());\nConcurrent Collections: Use ConcurrentHashMap, CopyOnWriteArrayList, or BlockingQueue for better thread-safe alternatives.
Garbage collection (GC) in Java is essential to automatic memory management, ensuring that objects no longer needed by an application are reclaimed, and the memory they occupied is freed. This allows Java developers to avoid memory leaks and other resource-management issues.
"},{"location":"langdives/Java/GarbageCollection/#basics","title":"Basics","text":"Heap Memory is divided into several areas, mainly
How GC Works ? Simply GC works by going through phases
Note
ZGC and Shenandoah use advanced algorithms that perform incremental marking, remapping, and concurrent sweeping. They avoid long pauses by offloading most GC work to concurrent threads.
"},{"location":"langdives/Java/GarbageCollection/#garbage-collection-concepts","title":"Garbage Collection Concepts","text":""},{"location":"langdives/Java/GarbageCollection/#generational-collection","title":"Generational Collection","text":"Java heap is divided into:
Young Generation: Divided into Eden Space and Survivor Spaces (S0, S1), New objects are created in Eden. After surviving a GC cycle, they move to a survivor space, and after multiple cycles, to the Old Generation.
Old Generation (Tenured): Contains long-lived objects.
Garbage collection types:
Dynamic Region Management:
ZGC uses regions independently to ensure ultra-low pause times, as each GC thread works on separate regions without blocking others.
Compressed Oops (Ordinary Object Pointers):
-XX:+UseCompressedOops\nJava threads stop only at specific safepoints for GC (or other JVM activities). A safepoint is a checkpoint where all threads must reach before GC can start for eg Executing bytecode that triggers allocation failure, method calls, or back branches in loops.
The JVM injects safepoint polls within running code to ensure threads hit these safepoints regularly, Too many safepoint pauses indicate GC tuning issues or excessive thread blocking.
JVM may delay triggering GC due to waiting for all threads to reach a safepoint, which introduces unpredictable latency. This is critical in low-latency systems like trading applications.
"},{"location":"langdives/Java/GarbageCollection/#stop-the-world-stw","title":"Stop the World (STW)","text":"A Stop-The-World (STW) event occurs when the garbage collector (GC) halts all application threads to perform critical tasks like marking live objects, reclaiming memory, and compacting the heap. These pauses, necessary to prevent heap inconsistencies, impact application performance, especially in latency-sensitive environments.
The duration of STW events depends on heap size, the number of live objects, and the GC algorithm. Traditional collectors like Serial GC and Parallel GC have longer STW pauses, while CMS reduces them with concurrent marking but still requires short pauses for initial and final marking. Modern GCs like G1 GC, ZGC, and Shenandoah GC minimize pauses by performing most work concurrently with application threads, achieving millisecond-range STW durations.
Optimizations include using low-latency collectors, tuning GC settings, reducing allocation pressure, and monitoring GC behavior with tools like JFR or VisualVM. For latency-critical applications, advanced collectors and careful memory management are essential to mitigate the impact of STW events.
"},{"location":"langdives/Java/GarbageCollection/#barriers-tables-fences","title":"Barriers, Tables & Fences","text":""},{"location":"langdives/Java/GarbageCollection/#write-barriers","title":"Write Barriers","text":"New Object Creation
Minor GC (Young Generation Collection)
Max Tenuring Threshold
Promotion Failures
When Old Generation Fills Up
Concurrent Collectors (CMS, G1, ZGC, Shenandoah)
What Causes Full GC?
What Happens During Full GC
Safepoints
Write Barriers
Soft, Weak, and Phantom References
GC Flow
Object Creation
Eden Full \u2192 Trigger Minor GC
Old Gen Full \u2192 Trigger Major GC
Concurrent Collections (G1, ZGC)
Full GC (Stop-the-World)
Fragmentation refers to the inefficient use of memory that occurs when free memory is split into small, non-contiguous blocks, making it difficult to allocate larger contiguous blocks even if the total free memory is sufficient. In Java, fragmentation can occur in both the young and old generations of the heap.
"},{"location":"langdives/Java/GarbageCollection/#types","title":"Types","text":"Internal Fragmentation: Occurs when a block of memory is larger than what is actually needed. For example, if an object requires 10 bytes but is allocated a 16-byte block, the remaining 6 bytes are wasted.
External Fragmentation: Happens when free memory is scattered in small chunks across the heap. This can lead to a situation where there isn\u2019t enough contiguous space available to fulfill a large allocation request, even if the total free memory is sufficient.
"},{"location":"langdives/Java/GarbageCollection/#causes","title":"Causes","text":"Object Lifetimes: Short-lived objects are frequently allocated and deallocated, especially in the young generation. This can create gaps in memory as these objects are collected, leading to external fragmentation.
Promotion of Objects: When objects in the young generation are promoted to the old generation, if the old generation is already fragmented, it may become difficult to allocate new objects.
Full GCs: In collectors like CMS (Concurrent Mark-Sweep), memory is reclaimed but not compacted, leaving fragmented free spaces.
"},{"location":"langdives/Java/GarbageCollection/#effects","title":"Effects","text":"OutOfMemoryError: Fragmentation can cause allocation failures, leading to OutOfMemoryError if there isn\u2019t enough contiguous memory available for new object allocations.
Increased GC Overhead: The JVM may spend more time during GC cycles trying to find suitable spaces for object allocation, which can degrade performance.
Heap Fragmentation: Some collectors (like CMS) suffer from heap fragmentation since they don\u2019t compact memory after reclaiming space.
Pinned Objects: Sometimes, objects cannot be moved during GC (e.g., JNI references or thread-local objects). This can lead to fragmentation.
"},{"location":"langdives/Java/GarbageCollection/#mitigating","title":"Mitigating","text":"Using G1 GC or ZGC: These collectors are designed to handle fragmentation better than older collectors. They manage memory in regions and perform compaction as part of their regular operations.
Heap Size Adjustments: Increasing the size of the old generation can help reduce the frequency of fragmentation issues.
Monitoring and Tuning: Regularly monitor memory usage and GC logs to identify fragmentation patterns. Tuning the JVM parameters can help alleviate fragmentation issues.
Object Pooling: Reusing objects instead of frequently allocating and deallocating them can help reduce fragmentation.
"},{"location":"langdives/Java/GarbageCollection/#configuring-garbage-collection","title":"Configuring garbage collection","text":"Configuring garbage collection and its parameters in Java is primarily done through JVM (Java Virtual Machine) options when starting your application.
"},{"location":"langdives/Java/GarbageCollection/#how-to-configure-params","title":"How to Configure Params","text":"Command-Line Options: You can specify GC options when you start your Java application using the java command.
java -Xms512m -Xmx4g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -jar my-application.jar\n-Xms512m: Sets the initial heap size to 512 MB.-Xmx4g: Sets the maximum heap size to 4 GB.-XX:+UseG1GC: Uses the G1 Garbage Collector.-XX:MaxGCPauseMillis=100: Aims for a maximum pause time of 100 milliseconds.Environment Variables: For containerized applications (like those running in Docker or Kubernetes), you can set JVM options through environment variables or directly in the configuration file.
Example in DockerENV JAVA_OPTS=\"-Xms512m -Xmx4g -XX:+UseG1GC\"\nCMD java $JAVA_OPTS -jar your-application.jar\nConfiguration Files: Some applications allow you to specify JVM options in a configuration file, which can be helpful for managing multiple parameters in one place.
"},{"location":"langdives/Java/GarbageCollection/#common-gc-options","title":"Common GC Options","text":"Basic Heap Size Configuration
-Xms<size>: Sets the initial heap size.-Xmx<size>: Sets the maximum heap size.Choosing a Garbage Collector
G1 GC-XX:+UseG1GC\n-XX:+UseParallelGC\n-XX:+UseConcMarkSweepGC\n-XX:+UseZGC\n-XX:+UseShenandoahGC\nTuning G1 GC
-XX:MaxGCPauseMillis=<time>: Target maximum pause time.-XX:G1HeapRegionSize=<size>: Size of the regions in G1.-XX:G1ReservePercent=<percentage>: Percentage of heap reserved for G1 to avoid Full GC.Tuning Parallel GC
-XX:ParallelGCThreads=<number>: Number of threads to use for parallel GC.-XX:ConcGCThreads=<number>: Number of concurrent threads during GC.Monitoring and Logging
-Xlog:gc*:file=gc.log: Logs detailed GC information to gc.log.-XX:+PrintGCDetails: Outputs detailed information about each GC event.Controlling Object Promotion
-XX:MaxTenuringThreshold=<N>: Number of times an object can be copied in the young generation before being promoted to the old generation.Metaspace Configuration (for class metadata)
-XX:MetaspaceSize=<size>: Initial size of Metaspace.-XX:MaxMetaspaceSize=<size>: Maximum size of Metaspace.Here\u2019s an example of a command to start a Java application with G1 GC and some tuning parameters
Examplejava -Xms1g -Xmx8g -XX:+UseG1GC \\\n -XX:MaxGCPauseMillis=200 \\\n -XX:G1HeapRegionSize=16m \\\n -XX:InitiatingHeapOccupancyPercent=30 \\\n -XX:ConcGCThreads=2 \\\n -Xlog:gc*:file=gc.log \\\n -jar my-application.jar\nHeap Size and GC Frequency
GC Latency and Response Times
Application Throughput vs Latency
Survivor Space Tuning
-XX:SurvivorRatio=<N>\n-XX:MaxTenuringThreshold=<N>\nTuning G1 GC
-XX:MaxGCPauseMillis=100\n-XX:G1ReservePercent=10 // Extra heap reserved to avoid Full GC\n-XX:G1HeapRegionSize=<N> // Larger region size helps handle humongous objects efficiently\nYoung GC Tuning
-Xlog:gc*:file=gc.log\nTuning Java GC for High Performance
Choose the Right GC:
Analyze and Tune:
Handling Multi-Terabyte Heaps
GC in Cloud and Microservices
-XX:MaxRAMPercentage=75.0\nLatency Monitoring
-Xlog:gc*:file=gc.log\nGC Logs: Capture GC details by adding -Xlog:gc* or -XX:+PrintGCDetails JVM options.
[GC (Allocation Failure) [PSYoungGen: 2048K->512K(2560K)] 4096K->2048K(8192K), 0.0103451 secs]\nVisualVM: A monitoring tool bundled with the JDK for real-time JVM performance monitoring.
Java Flight Recorder (JFR): An advanced profiling tool that collects detailed JVM metrics, including GC data.
JConsole: Visualize JVM statistics and monitor heap usage.
"},{"location":"langdives/Java/GarbageCollection/#diagnosing-troubleshooting","title":"Diagnosing & Troubleshooting","text":"OutOfMemoryError (OOM): Common causes
Heap Dump Analysis:
jmap -dump:format=b,file=heapdump.hprof <pid>\nDetecting Leaks: Look for large, unreachable objects with static references or growing collections (e.g., large HashMap or ArrayList).
Java Flight Recorder (JFR): JFR provides detailed memory profiling without heavy overhead. Collect a recording and analyze it for object lifetimes, GC events, and thread behavior.
"},{"location":"langdives/Java/GarbageCollection/#summary","title":"Summary","text":"Choose the Right GC Collector
Monitor and Analyze
Avoid Full GC
Pre-tuning Advice
Gradle is a modern, powerful build automation tool used for building, testing, and deploying applications. It is particularly popular in Java and Android projects due to its flexibility and performance. Gradle uses a Groovy/Kotlin-based DSL to configure and manage builds, allowing for easy customization.
Gradle organizes builds using a Directed Acyclic Graph (DAG) of tasks, ensuring that tasks are executed only when necessary.
The Build process has Three phases
Initialization Phase
Configuration Phase
Execution Phase
Gradle build configuration is done in the build.gradle file. This file uses Groovy or Kotlin DSL to describe:
build.gradle Example
plugins {\n id 'java' // Apply Java plugin\n id 'application' // Allow running the app from CLI\n}\n\nrepositories {\n mavenCentral() // Use Maven Central for dependencies\n}\n\ndependencies {\n implementation 'org.apache.commons:commons-lang3:3.12.0' // Runtime dependency\n testImplementation 'junit:junit:4.13.2' // Test dependency\n}\n\napplication {\n mainClass = 'com.example.UnderTheHood' // Entry point of the app\n}\nbuild.gradle","text":"Plugins Section
plugins {\n id 'java'\n id 'application'\n}\njava: Adds support for Java projects.application: Allows running applications directly from Gradle with gradle run.maven-publish: Publishes artifacts to Maven repositories.android: Used for Android app development.java plugin sets up the project to be built as a Java application.Repositories Section
repositories {\n mavenCentral()\n}\nmavenCentral() is a popular repository that hosts many open-source libraries.Dependencies Section
dependencies {\n implementation 'org.apache.commons:commons-lang3:3.12.0'\n testImplementation 'junit:junit:4.13.2'\n}\nimplementation: Adds a runtime dependency.testImplementation: Adds a dependency for testing purposes.Application Configuration
application {\n mainClass = 'com.example.UnderTheHood'\n}\ngradle run command.Gradle allows automatic dependency management. Dependencies (like libraries and frameworks) are fetched from repositories such as:
Gradle resolves dependencies in the following order:
~/.gradle/caches/.mavenLocal() is specified (~/.m2/repository/).# Forces Gradle to use only the local cache \n# and does not try to access remote repositories.\ngradle build --offline\nGradle allows developers to create custom tasks to automate specific workflows.
Custom Tasks Example Create Custom Tasktask uth {\n doLast {\n println 'Hello, UnderTheHood ;)'\n }\n}\ngradle uth\nHello, UnderTheHood ;)\nCustom tasks can be chained and made dependent on other tasks:
task compileCode {\n dependsOn clean\n doLast {\n println 'Compiling code...'\n }\n}\nYou can publish your project\u2019s artifacts (e.g., JARs) to Maven Local or Remote repositories using the maven-publish plugin.
Local Maven Publish Example
Apply Maven Publish Pluginplugins {\n id 'maven-publish'\n}\npublishing {\n publications {\n mavenJava(MavenPublication) {\n from components.java\n }\n }\n repositories {\n mavenLocal() // Publish to local Maven repository (~/.m2/repository)\n }\n}\n# This will install the JAR into your local Maven repository.\ngradle publishToMavenLocal\n/my-project\n\u2502\n\u251c\u2500\u2500 build.gradle # Build configuration file\n\u251c\u2500\u2500 settings.gradle # Project settings file\n\u251c\u2500\u2500 src\n\u2502 \u2514\u2500\u2500 main\n\u2502 \u2514\u2500\u2500 java # Source code\n\u2502 \u2514\u2500\u2500 test\n\u2502 \u2514\u2500\u2500 java # Unit tests\n\u2514\u2500\u2500 build # Output directory (JAR, WAR)\nsrc/main/java: Contains the application source code.src/test/java: Contains unit test code.build/: Contains compiled artifacts (like JARs).Gradle supports multi-module projects where different modules are part of the same build.
Example Multi-Project Structure/root-project\n\u2502\n\u251c\u2500\u2500 build.gradle # Root project configuration\n\u251c\u2500\u2500 settings.gradle # Lists sub-projects\n\u251c\u2500\u2500 module-1/\n\u2502 \u2514\u2500\u2500 build.gradle # Configuration for module 1\n\u2514\u2500\u2500 module-2/\n \u2514\u2500\u2500 build.gradle # Configuration for module 2\nrootProject.name = 'multi-project-example'\ninclude 'module-1', 'module-2'\n# This will build all modules in the correct order.\ngradle build\nThe Gradle Wrapper is a feature that allows a project to include a specific Gradle version along with scripts to execute builds. This ensures that anyone working on the project uses the same Gradle version without requiring a manual installation.
The Gradle Wrapper consists of:
gradlew, Windows: gradlew.bat).gradle/wrapper/gradle-wrapper.properties) that specifies the Gradle version to use.gradle-wrapper.jar) that downloads Gradle if needed.Here are some essential Gradle commands for working with projects:
Command Descriptiongradle init Initializes a new Gradle project. gradle build Compiles, tests, and packages the project. gradle run Runs the application (if using the Application plugin). gradle clean Removes the build/ directory for a fresh build. gradle tasks Lists all available tasks. gradle test Runs all tests in the project. gradle publish Publishes artifacts to Maven repositories."},{"location":"langdives/Java/Gradle/#performance-benefits","title":"Performance Benefits","text":"Gradle is designed for speed and efficiency
Gradle provides several advantages for modern projects
Java is a high-level, object-oriented programming language designed for portability, security, and ease of use. It is known for its \"write once, run anywhere\" capability, allowing developers to create software that can run on any device with a Java Virtual Machine (JVM).
"},{"location":"langdives/Java/JDK-JRE-JVM/#architecture","title":"Architecture","text":"The Java architecture is composed of three main components:
"},{"location":"langdives/Java/JDK-JRE-JVM/#jdk","title":"JDK","text":"The JDK Java Development Kit is a comprehensive development environment for building Java applications. It provides all the tools necessary for Java developers to create, compile, and package Java applications.
Components
javac): Translates Java source code (.java extension) into bytecode (.class extension).javadoc, jdb, and jar.The JRE-Java Runtime Environment provides the libraries, Java Virtual Machine (JVM), and other components necessary for running Java applications. It does not include development tools, making it ideal for end-users who only need to run Java applications.
Components
The JVM Java Virtual Machine is an abstract computing machine that enables a computer to run Java programs. It is responsible for interpreting and executing the bytecode generated by the Java compiler.
Functions
.class files containing the bytecode into memory.Hierarchical Structure
JDK (includes javac, JRE, Tools)\n\u2514\u2500\u2500 JRE (includes JVM and libraries)\n \u2514\u2500\u2500 JVM (executes bytecode)\nThe execution of Java code involves several steps, transitioning through the JDK, JRE, and JVM before reaching the machine code that the computer's CPU executes.
Code Writing: Java developers write source code in plain text files using the .java extension. This code defines classes and methods that make up the Java application.
Code Compilation: The developer uses the Java compiler (javac), which is part of the JDK, to compile the .java file. This process translates the human-readable Java code into an intermediate form known as bytecode. The output of this step is a .class file containing the bytecode.
Running the Application: To run the Java application, the developer executes a command using the Java runtime environment (e.g., java ClassName), which triggers the JRE, The JRE includes the JVM, which performs the following steps:
Loading: The JVM locates the specified .class file and loads it into memory.
Linking: The JVM links the bytecode:
Execution: The JVM executes the bytecode:
Machine Code Execution: The machine code generated by the JVM is executed by the host operating system's CPU. This process allows Java applications to be platform-independent, as the same bytecode can run on any system that has a compatible JVM.
"},{"location":"langdives/Java/Java8vs11vs17vs21/","title":"Java 8 vs 11 vs 17 vs 21","text":"A detailed comparison of Java 8, Java 11, Java 17, and Java 21, summarizing the key differences, improvements, and deprecations introduced across these versions:
"},{"location":"langdives/Java/Java8vs11vs17vs21/#java-8-released-march-2014","title":"Java 8 (Released March 2014)","text":"Java 8 is a long-term support (LTS) release, bringing significant new features:
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features","title":"Major Features","text":"NullPointerException.java.time): Replaces the old java.util.Date.Java 11 is also LTS and a significant milestone since it removed many outdated APIs and modularized the runtime.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_1","title":"Major Features","text":"var in lambda).lines(), strip(), repeat(), isBlank() methods.readString(), writeString(), isSameFile() methods.Java 17 is an LTS release, refining many features introduced in Java 9-16 and stabilizing the platform.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_2","title":"Major Features","text":"instanceof: Simplifies type casting.Java 21 is a non-LTS release (though with unofficial support from some vendors). It introduces many experimental and innovative features.
"},{"location":"langdives/Java/Java8vs11vs17vs21/#major-features_3","title":"Major Features","text":"List, Set, and Map with defined iteration order.instanceof) No No Yes Yes Virtual Threads (Loom) No No No Yes Garbage Collectors G1 GC ZGC ZGC, Shenandoah Improved ZGC, Shenandoah String Enhancements Basic strip(), repeat() Text Blocks String Templates TLS Version 1.2 1.3 1.3 1.3 Security Manager Available Deprecated Deprecated Removed Nashorn JavaScript Engine Yes Deprecated Removed Removed"},{"location":"langdives/Java/Java8vs11vs17vs21/#summary","title":"Summary","text":"For production systems, upgrading to Java 17 is generally recommended unless your project needs experimental features from Java 21.
"},{"location":"langdives/Java/JavaPassBy/","title":"Is Java Pass By Value or By Reference ?","text":"Java is stricly pass by value but lets go in depth.
"},{"location":"langdives/Java/JavaPassBy/#reference-vs-primitive","title":"Reference vs Primitive","text":"Primitive Types: These are the basic data types in Java (e.g., int, char, boolean). When you pass a primitive type to a method, a copy of the value is made.
Reference Types: These include objects, arrays, and instances of classes. When you pass a reference type to a method, you pass a reference (or pointer) to the actual object in memory, not the object itself.
"},{"location":"langdives/Java/JavaPassBy/#pass-by-value","title":"Pass by Value","text":"Java uses a mechanism called pass by value for method arguments, but it\u2019s important to clarify how this applies to primitive and reference types.
"},{"location":"langdives/Java/JavaPassBy/#primitive-types","title":"Primitive Types","text":"When you pass a primitive type to a method, the method receives a copy of the variable's value. Any changes made to this copy do not affect the original variable.
Examplepublic class PassByValueExample {\n public static void main(String[] args) {\n int num = 10;\n modifyValue(num); // Passing primitive\n System.out.println(num); // Output: 10\n }\n\n public static void modifyValue(int value) {\n value = 20; // Only modifies the copy\n }\n}\nWhen you pass a reference type to a method, the reference itself is passed by value. This means the method receives a copy of the reference to the object. While you can change the object's properties, you cannot change the reference to point to a different object.
Exampleclass MyClass {\n int value;\n\n MyClass(int value) {\n this.value = value;\n }\n}\n\npublic class PassByReferenceExample {\n public static void main(String[] args) {\n MyClass obj = new MyClass(10);\n modifyObject(obj); // Passing reference\n System.out.println(obj.value); // Output: 20\n }\n\n public static void modifyObject(MyClass myObject) {\n myObject.value = 20; // Modifies the object's property\n // myObject = new MyClass(30); // This would not affect the original object reference only changes local myObject.\n }\n}\nWhen a method is called, a new stack frame is created, and local variables (including method parameters) are stored in this stack frame. Objects are stored in the heap, and the reference to these objects is passed to methods. When you modify the object\u2019s state inside the method, it reflects outside the method because both the original reference and the parameter reference point to the same object in memory.
"},{"location":"langdives/Java/JavaPassBy/#scope-and-lifetime","title":"Scope and Lifetime","text":"Scope: Variables defined inside a method (local variables) can only be accessed within that method. Once the method completes, the local variables are destroyed.
Lifetime: Objects in the heap remain as long as they are referenced. When no references exist (e.g., after the method execution, and no references are held), they become eligible for garbage collection.
Java is pass-by-value:
Changes to the object through the reference affect the original object, but reassignment of the reference does not affect the original reference.
"},{"location":"langdives/Java/KeyWordsTerminolgies/","title":"Keywords and Terminolgies","text":""},{"location":"langdives/Java/KeyWordsTerminolgies/#class-object","title":"Class & Object","text":"class: Defines a class.interface: Declares an interface (a contract for classes).object: An instance of a class.new: Instantiates a new object.public / private / protected: Access control modifiers (already discussed). static: Indicates a field or method belongs to the class rather than instances, is shared across all instances of a class.final: Used to declare constants, prevent method overriding, or inheritance, A final method cannot be overridden by subclasses, A final class cannot be inherited.abstract: Used to declare incomplete methods or classes that must be extended. extends: A class inherits another class.implements: A class implements an interface.super: Refers to the parent class\u2019s members.this: Refers to the current instance of the class.if / else / switch: Conditional statements.for / while / do-while: Looping constructs.break / continue: Control loop execution.return: Exits from a method and optionally returns a value.try / catch / finally: Handle exceptions.throw / throws: Raise exceptions.assert: Check assumptions during development.new: Allocates memory for an object.null: A reference that points to no object.synchronized: Used to control access to methods or blocks in a multithreaded environment.volatile: Ensures visibility of changes to variables across threads.transient: Prevents serialization of a field.void: Specifies that a method does not return a value.primitive types: int, float, char, boolean, etc.instanceof: Tests if an object is of a particular type.enum: Declares an enumeration, a special type with predefined constant values.package: Defines a package (namespace) for classes.import: Brings other classes or packages into the current class.default: Provides default implementations in interfaces or switch statements.native: Declares methods implemented in native code (outside Java).strictfp: Ensures consistent floating-point calculations across platforms.const / goto: Reserved keywords (not used in Java).Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers synchronized blocks.
"},{"location":"langdives/Java/Locking-Intrinsic/#what-is-locking","title":"What is Locking?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock.
Java\u2019s synchronized key word is one of the primary ways to control access to shared resources in multithreaded programs. It ensures thread safety by providing mutual exclusion and visibility guarantees. Let's go further into every aspect of synchronized.
How synchronized Works ?
When a method or block is marked as synchronized, the JVM uses a monitor lock (intrinsic lock) for the associated object or class. The monitor is a synchronization construct provided by the JVM.
Two things happen when a thread enters a synchronized block or method:
Intrinsic Lock: Each Java object has an intrinsic lock (also called monitor lock) associated with it, The thread that enters the synchronized block acquires the intrinsic lock. When it leaves the block, it releases the lock, allowing other threads to acquire it.
"},{"location":"langdives/Java/Locking-Intrinsic/#synchronized-methods","title":"Synchronized Methods","text":""},{"location":"langdives/Java/Locking-Intrinsic/#instance-level-locking","title":"Instance-Level Locking","text":"When you synchronize a non-static method, the thread acquires the lock on the instance of the class (the this object).
public synchronized void increment() {\n // Lock acquired on the current instance (this)\n count++;\n}\nincrement() on the same object instance, only one thread will execute the method at a time. class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++;\n }\n\n public synchronized int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nWhy does this work ?
Since both threads are operating on the same Counter object, only one thread at a time can execute the increment() method due to instance-level locking.
A static synchronized method locks on the Class object (i.e., ClassName.class) rather than on an instance. This ensures that all threads calling static methods on the class are synchronized.
public synchronized static void staticIncrement() {\n // Lock acquired on the class object (Counter.class)\n}\nclass Counter {\n private static int count = 0;\n\n public synchronized static void increment() {\n count++;\n }\n\n public synchronized static int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) Counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) Counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + Counter.getCount()); // Output: 2000\n }\n}\nA synchronized block provides more control than a synchronized method. You can choose which object\u2019s intrinsic lock to use, instead of locking the entire method.
public void increment() {\n synchronized (this) { // Locking on the current instance\n count++;\n }\n}\nclass Counter {\n private int count = 0;\n private final Object lock = new Object();\n\n public void increment() {\n synchronized (lock) { // Locking on a custom object\n count++;\n }\n }\n}\npublic void updateBothCounters(Counter counter1, Counter counter2) {\n synchronized (counter1) { // Locking on the first Counter object\n counter1.increment();\n }\n synchronized (counter2) { // Locking on the second Counter object\n counter2.increment();\n }\n}\nEntry to the Monitor
Exit from the Monitor
Bias Locking and Lightweight Locking
Synchronizing unnecessary code slows down the application so use when necessary.
Minimize the scope of synchronization so use synchronized blocks rather than whole methods to reduce contention.
Ensure you synchronize on the same object across threads to avoid incorrect locking behavior.
Nested synchronized blocks can lead to deadlock. Use consistent lock ordering.
Deadlock, occurs if two or more threads are waiting for each other to release locks.
Performance Bottlenecks, overusing synchronization can lead to contention, where threads are constantly waiting to acquire locks.
Livelock, threads keep responding to each other without making progress (e.g., both threads keep yielding the lock to each other).
Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how deadlocks occur, how to prevent them, and practical examples of various techniques to detect and resolve deadlocks. A deadlock is a common concurrency issue in multithreaded programs and can severely impact performance.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#what-is-deadlock","title":"What is Deadlock ?","text":"A deadlock occurs when two or more threads are blocked indefinitely, Each thread is waiting for a lock held by the other, and neither can proceed.
This results in a circular wait, where no thread can release the locks it holds, leading to a deadlock condition.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#how-deadlock-occurs","title":"How Deadlock Occurs ?","text":"Let\u2019s revisit the classic deadlock example.
Deadlock Exampleclass A {\n public synchronized void methodA(B b) {\n System.out.println(Thread.currentThread().getName() + \": Locked A, waiting for B...\");\n try {\n Thread.sleep(50); // Simulate some work\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n b.last(); // Waiting for lock on object B\n }\n\n public synchronized void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside A.last()\");\n }\n}\n\nclass B {\n public synchronized void methodB(A a) {\n System.out.println(Thread.currentThread().getName() + \": Locked B, waiting for A...\");\n try {\n Thread.sleep(50); // Simulate some work\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n a.last(); // Waiting for lock on object A\n }\n\n public synchronized void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside B.last()\");\n }\n}\n\npublic class DeadlockDemo {\n public static void main(String[] args) {\n A a = new A();\n B b = new B();\n\n Thread t1 = new Thread(() -> a.methodA(b), \"Thread 1\");\n Thread t2 = new Thread(() -> b.methodB(a), \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nFlow Analysis
Thread 1 starts and calls a.methodA(b). It acquires the lock on object A and prints:
Thread 1: Locked A, waiting for B...\nThread 2 starts and calls b.methodB(a). It acquires the lock on object B and prints:
Thread 2: Locked B, waiting for A...\nNow:
A and waits for Thread 2 to release the lock on B.B and waits for Thread 1 to release the lock on A.Both threads are waiting indefinitely, resulting in a deadlock.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#how-to-avoid","title":"How to Avoid ?","text":"tryLock() with TimeoutIf all threads acquire locks in the same order, deadlock can be prevented.
Modified Example: Acquiring Locks in the Same Orderclass A {\n public void methodA(B b) {\n synchronized (this) {\n System.out.println(Thread.currentThread().getName() + \": Locked A, waiting for B...\");\n synchronized (b) {\n System.out.println(Thread.currentThread().getName() + \": Acquired lock on B\");\n b.last();\n }\n }\n }\n\n public void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside A.last()\");\n }\n}\n\nclass B {\n public void methodB(A a) {\n synchronized (this) {\n System.out.println(Thread.currentThread().getName() + \": Locked B, waiting for A...\");\n synchronized (a) {\n System.out.println(Thread.currentThread().getName() + \": Acquired lock on A\");\n a.last();\n }\n }\n }\n\n public void last() {\n System.out.println(Thread.currentThread().getName() + \": Inside B.last()\");\n }\n}\n\npublic class DeadlockResolved {\n public static void main(String[] args) {\n A a = new A();\n B b = new B();\n\n Thread t1 = new Thread(() -> a.methodA(b), \"Thread 1\");\n Thread t2 = new Thread(() -> b.methodB(a), \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nExplanation
Both threads now acquire locks in the same order (A \u2192 B). This ensures that deadlock cannot occur.
tryLock() with Timeout","text":"The tryLock() method attempts to acquire a lock and fails gracefully if the lock is not available within a specified time.
tryLock() example import java.util.concurrent.TimeUnit;\nimport java.util.concurrent.locks.ReentrantLock;\n\npublic class TryLockDemo {\n private final ReentrantLock lockA = new ReentrantLock();\n private final ReentrantLock lockB = new ReentrantLock();\n\n public void methodA() {\n try {\n if (lockA.tryLock(1, TimeUnit.SECONDS)) {\n System.out.println(Thread.currentThread().getName() + \": Locked A\");\n Thread.sleep(50); // Simulate some work\n\n if (lockB.tryLock(1, TimeUnit.SECONDS)) {\n try {\n System.out.println(Thread.currentThread().getName() + \": Locked B\");\n } finally {\n lockB.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \": Could not acquire lock B, releasing A\");\n }\n\n lockA.unlock();\n }\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n public void methodB() {\n try {\n if (lockB.tryLock(1, TimeUnit.SECONDS)) {\n System.out.println(Thread.currentThread().getName() + \": Locked B\");\n Thread.sleep(50); // Simulate some work\n\n if (lockA.tryLock(1, TimeUnit.SECONDS)) {\n try {\n System.out.println(Thread.currentThread().getName() + \": Locked A\");\n } finally {\n lockA.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \": Could not acquire lock A, releasing B\");\n }\n\n lockB.unlock();\n }\n } catch (InterruptedException e) {\n e.printStackTrace();\n }\n }\n\n public static void main(String[] args) {\n TryLockDemo demo = new TryLockDemo();\n\n Thread t1 = new Thread(demo::methodA, \"Thread 1\");\n Thread t2 = new Thread(demo::methodB, \"Thread 2\");\n\n t1.start();\n t2.start();\n }\n}\nExplanation
If a thread fails to acquire a lock within the timeout, it releases any locks it holds, avoiding a deadlock.
"},{"location":"langdives/Java/Locking-Issues-DeadLock/#detecting-using-monitoring-tools","title":"Detecting Using Monitoring Tools","text":"You can detect deadlocks using tools like:
tryLock() with timeout to avoid indefinite blocking.AtomicInteger or ConcurrentHashMap when possible.Deadlocks are one of the most common and dangerous issues in multithreaded programming.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/","title":"Issues with Locking - LiveLock","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how livelock occur, how to prevent them, and practical examples of various techniques to detect and resolve livelock.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#what-is-livelock","title":"What is Livelock ?","text":"In a livelock, multiple threads remain active but unable to make progress because they keep responding to each other\u2019s actions. Unlike deadlock, where threads are stuck waiting for locks indefinitely, threads in a livelock keep changing their states in response to each other, but they fail to reach a final state or make useful progress.
Key difference from deadlock
In deadlock, threads are blocked waiting for each other, while in livelock, threads are not blocked, but they keep releasing and reacquiring locks or changing states in a way that prevents progress.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#example-of-livelock","title":"Example of Livelock","text":"Consider two people trying to pick up a spoon to eat, but they keep yielding to each other in an attempt to be polite. Neither person can make progress because they\u2019re constantly checking and responding to each other\u2019s actions.
Livelock Exampleclass Spoon {\n private boolean isAvailable = true;\n\n public synchronized boolean pickUp() {\n if (isAvailable) {\n isAvailable = false;\n return true;\n }\n return false;\n }\n\n public synchronized void putDown() {\n isAvailable = true;\n }\n}\n\npublic class LivelockDemo {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n while (!spoon.pickUp()) {\n System.out.println(\"Person 1: Waiting for spoon...\");\n Thread.yield(); // Yield control to other threads\n }\n System.out.println(\"Person 1: Picked up spoon!\");\n });\n\n Thread person2 = new Thread(() -> {\n while (!spoon.pickUp()) {\n System.out.println(\"Person 2: Waiting for spoon...\");\n Thread.yield(); // Yield control to other threads\n }\n System.out.println(\"Person 2: Picked up spoon!\");\n });\n\n person1.start();\n person2.start();\n }\n}\nExplanation
Excessive Yielding or Giving Way: Threads are too polite and keep yielding to each other and In Java, using Thread.yield() repeatedly can lead to livelock.
Conflicting Retrying Logic: Both threads retry in response to each other\u2019s behavior, creating a circular dependency.
Improper Design of Locking Mechanisms, Threads repeatedly release and reacquire locks without making useful progress.
Poor Handling of Shared Resources, When shared resources are locked and unlocked too frequently, threads can repeatedly fail to acquire them.
Using timeouts helps threads avoid indefinite waiting. If a thread cannot acquire the lock within a certain time, it can stop trying or take an alternative path.
UsingtryLock() with Timeout import java.util.concurrent.TimeUnit;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private final ReentrantLock lock = new ReentrantLock();\n\n public boolean pickUp() throws InterruptedException {\n // Try to acquire the lock with a timeout\n return lock.tryLock(1, TimeUnit.SECONDS);\n }\n\n public void putDown() {\n lock.unlock();\n }\n}\n\npublic class LivelockFixed {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n try {\n if (spoon.pickUp()) {\n System.out.println(\"Person 1: Picked up spoon!\");\n spoon.putDown();\n } else {\n System.out.println(\"Person 1: Couldn't get the spoon in time.\");\n }\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread person2 = new Thread(() -> {\n try {\n if (spoon.pickUp()) {\n System.out.println(\"Person 2: Picked up spoon!\");\n spoon.putDown();\n } else {\n System.out.println(\"Person 2: Couldn't get the spoon in time.\");\n }\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n person1.start();\n person2.start();\n }\n}\nWhy it works ?
If a thread fails to acquire the lock within 1 second, it backs off instead of trying indefinitely.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#use-back-off-strategies","title":"Use Back-off Strategies","text":"A back-off strategy makes threads wait for a random amount of time before retrying. This avoids a situation where two threads keep checking the same lock in sync.
Back-off Strategy Exampleimport java.util.Random;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private final ReentrantLock lock = new ReentrantLock();\n private final Random random = new Random();\n\n public boolean tryPickUp() {\n return lock.tryLock();\n }\n\n public void putDown() {\n lock.unlock();\n }\n\n public void backOff() throws InterruptedException {\n Thread.sleep(random.nextInt(100)); // Wait for a random time\n }\n}\n\npublic class LivelockWithBackoff {\n public static void main(String[] args) {\n Spoon spoon = new Spoon();\n\n Thread person1 = new Thread(() -> {\n try {\n while (!spoon.tryPickUp()) {\n System.out.println(\"Person 1: Waiting...\");\n spoon.backOff(); // Wait before retrying\n }\n System.out.println(\"Person 1: Picked up spoon!\");\n spoon.putDown();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread person2 = new Thread(() -> {\n try {\n while (!spoon.tryPickUp()) {\n System.out.println(\"Person 2: Waiting...\");\n spoon.backOff(); // Wait before retrying\n }\n System.out.println(\"Person 2: Picked up spoon!\");\n spoon.putDown();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n person1.start();\n person2.start();\n }\n}\nWhy it works ?
The random back-off time prevents threads from retrying in lockstep, avoiding livelock.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#avoid-excessive-yielding","title":"Avoid Excessive Yielding","text":"Frequent use of Thread.yield() can lead to livelock. Instead, use timeouts or back-off strategies to prevent threads from constantly giving way to each other.
Use Condition variables (available with ReentrantLock) to properly coordinate threads waiting on specific conditions.
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass Spoon {\n private boolean isAvailable = true;\n private final ReentrantLock lock = new ReentrantLock();\n private final Condition spoonAvailable = lock.newCondition();\n\n public void pickUp() throws InterruptedException {\n lock.lock();\n try {\n while (!isAvailable) {\n spoonAvailable.await(); // Wait until spoon is available\n }\n isAvailable = false;\n } finally {\n lock.unlock();\n }\n }\n\n public void putDown() {\n lock.lock();\n try {\n isAvailable = true;\n spoonAvailable.signal(); // Notify waiting thread\n } finally {\n lock.unlock();\n }\n }\n}\nWhy it works ?
Using condition variables ensures that only one thread proceeds when the spoon becomes available, avoiding busy-waiting and yielding.
"},{"location":"langdives/Java/Locking-Issues-LiveLock/#best-practices","title":"Best Practices","text":"yield() with smarter coordination mechanisms like conditions.Livelocks can be tricky to detect because threads remain active, but they fail to make meaningful progress. By using timeouts, back-off strategies, condition variables, and proper locking mechanisms.
"},{"location":"langdives/Java/Locking-Issues-Others/","title":"Issues with Locking - Other Issues","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
We will cover key locking issues in Java in this article like race conditions, thread contention, missed signals, nested locks, overuse of locks, and performance impact. Each section contains causes, examples, solutions, and best practices to avoid or mitigate these issues.
"},{"location":"langdives/Java/Locking-Issues-Others/#race-conditions-despite-locking","title":"Race Conditions Despite Locking","text":"CauseA race condition occurs when multiple threads access a shared resource without proper synchronization, leading to inconsistent results based on the timing of thread execution. Even with partial locks, a shared variable may still be accessed inconsistently if not protected properly.
Race Condition Example
class Counter {\n private int count = 0;\n\n public void increment() {\n synchronized (this) {\n count++;\n }\n }\n\n public int getCount() {\n // Not synchronized, potential race condition.\n return count;\n }\n}\nProblem
count to 1.count can be read by Thread 2, Thread 3 increments it again.getCount() is not synchronized, the returned value may skip increments due to improper timing.Always synchronize access to shared variables, even on read operations if other threads can modify the data.
class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++;\n }\n\n public synchronized int getCount() {\n return count;\n }\n}\nUse AtomicInteger if possible for thread-safe increments:
import java.util.concurrent.atomic.AtomicInteger;\n\nclass Counter {\n private final AtomicInteger count = new AtomicInteger(0);\n\n public void increment() {\n count.incrementAndGet();\n }\n\n public int getCount() {\n return count.get();\n }\n}\nWhen multiple threads compete for the same lock, they spend time waiting for the lock to become available, reducing throughput and performance.
Contention Example
class BankAccount {\n private int balance = 100;\n\n public synchronized void withdraw(int amount) {\n balance -= amount;\n }\n\n public synchronized int getBalance() {\n return balance;\n }\n}\nProblem
If multiple threads frequently access the withdraw() method, contention for the lock will occur, degrading performance.
Minimize the Scope of Synchronized Blocks:
public void withdraw(int amount) {\n if (amount <= 0) return;\n\n synchronized (this) {\n balance -= amount;\n }\n}\nUse ReadWriteLock if reads dominate writes:
import java.util.concurrent.locks.ReentrantReadWriteLock;\n\nclass BankAccount {\n private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();\n private int balance = 100;\n\n public void withdraw(int amount) {\n lock.writeLock().lock();\n try {\n balance -= amount;\n } finally {\n lock.writeLock().unlock();\n }\n }\n\n public int getBalance() {\n lock.readLock().lock();\n try {\n return balance;\n } finally {\n lock.readLock().unlock();\n }\n }\n}\nUse lock-free data structures like AtomicInteger or ConcurrentHashMap to reduce contention.
When a thread misses a notify() signal because it was not yet waiting on the lock, a lost wake-up occurs. This results in threads waiting indefinitely for a signal that has already been sent.
Lost Wake-Up Example
public synchronized void produce() throws InterruptedException {\n while (available) {\n wait(); // Missed if notify() was called before reaching here.\n }\n available = true;\n notify();\n}\nAlways Use a while Loop Instead of if, The while loop ensures the thread rechecks the condition after being notified (to avoid spurious wake-ups).
public synchronized void produce() throws InterruptedException {\n while (available) {\n wait();\n }\n available = true;\n notify();\n}\nUse Condition Variables with ReentrantLock for finer control:
private final ReentrantLock lock = new ReentrantLock();\nprivate final Condition condition = lock.newCondition();\nprivate boolean available = false;\n\npublic void produce() throws InterruptedException {\n lock.lock();\n try {\n while (available) {\n condition.await(); // Wait on condition.\n }\n available = true;\n condition.signal();\n } finally {\n lock.unlock();\n }\n}\nUsing multiple locks can cause deadlocks if threads acquire locks in different orders.
Deadlock Example
synchronized (lock1) {\n synchronized (lock2) {\n // Critical section\n }\n}\nProblem
If Thread 1 acquires lock1 and Thread 2 acquires lock2, both threads will wait indefinitely for each other\u2019s lock, resulting in a deadlock.
tryLock() with timeout to avoid blocking indefinitely: if (lock1.tryLock(1, TimeUnit.SECONDS)) {\n if (lock2.tryLock(1, TimeUnit.SECONDS)) {\n try {\n // Critical section\n } finally {\n lock2.unlock();\n }\n }\n lock1.unlock();\n}\nUsing too many locks or locking too frequently can reduce parallelism, resulting in poor scalability, If every method in a class is synchronized, threads will frequently block each other, reducing concurrency and efficiency.
Solution and Best PracticesConcurrentHashMap) instead of traditional synchronized collections: ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();\nLocking adds overhead in the form of: - Context switches between threads. - CPU cache invalidation. - JVM's monitor management for intrinsic locks.
Performance Issues with Synchronized Code
Excessive locking causes contention and frequent context switches, impacting throughput and latency.
Solution and Best PracticesAtomicInteger for simple counters.Use tryLock() for non-blocking operations:
if (lock.tryLock()) {\n try {\n // Critical section\n } finally {\n lock.unlock();\n }\n}\nMinimize the scope of synchronized blocks to reduce contention.
Use fair locks to avoid starvation:
ReentrantLock lock = new ReentrantLock(true); // Fair lock\nUse lock-free data structures when possible (e.g., ConcurrentHashMap).
Monitor and detect deadlocks using tools like VisualVM or JConsole.
Follow consistent lock ordering to prevent deadlocks.
Locking is essential to ensure thread safety, but improper use can lead to issues such as race conditions, deadlocks, livelocks, contention, and performance degradation. Understanding these issues and following best practices will help you write efficient, scalable, and thread-safe code. Using fine-grained locks and concurrent utilities wisely to maximize concurrency while minimizing risks.
"},{"location":"langdives/Java/Locking-Issues-Starvation/","title":"Issues with Locking - Starvation","text":"Locking mechanisms in Java, while essential for ensuring thread safety in multithreaded applications, can introduce various issues if not used properly.
In this article, we\u2019ll explore how starvation occur, how to prevent them, and practical examples of various techniques to detect and resolve starvation.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#what-is-starvation","title":"What is Starvation ?","text":"Starvation is a condition where low-priority threads are unable to gain access to resources because higher-priority threads or unfair scheduling policies monopolize CPU time or locks. As a result, the low-priority thread starves and never gets the chance to run, even though it is ready to execute.
This issue can manifest in multithreaded programs when locks or resources are continuously granted to specific threads, leaving others waiting indefinitely. It can occur not only due to CPU scheduling but also due to improper locking strategies, unfair algorithms, or resource starvation.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#causes-of-starvation","title":"Causes of Starvation","text":"Unfair Locking Mechanism, When using unfair locks, the lock may always favor threads that request it recently over those that have been waiting longer.
Priority Inversion, High-priority threads monopolize the CPU and low-priority threads rarely get scheduled. This is especially problematic in priority-based scheduling systems.
Thread Contention and Resource Starvation, Too many threads competing for a limited number of resources (like I/O or locks), some threads may always be able to acquire the resources, leaving others waiting indefinitely.
Busy-Waiting Loops, Threads that continuously request resources or repeatedly try to acquire locks without releasing the CPU can cause other threads to starve.
Thread Prioritization in Java, If threads have different priorities, the JVM might schedule higher-priority threads more frequently, leading to starvation of lower-priority threads.
import java.util.concurrent.locks.ReentrantLock;\n\npublic class StarvationDemo {\n private static final ReentrantLock lock = new ReentrantLock(); // Unfair lock\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n }\n }\n };\n\n Thread highPriority = new Thread(task, \"High-Priority\");\n highPriority.setPriority(Thread.MAX_PRIORITY);\n\n Thread lowPriority = new Thread(task, \"Low-Priority\");\n lowPriority.setPriority(Thread.MIN_PRIORITY);\n\n highPriority.start();\n lowPriority.start();\n }\n}\nExplanation
Using fair locks ensures that the longest-waiting thread gets the lock first. This prevents threads from skipping the queue and ensures all threads get a chance to execute.
Fair Lock Exampleimport java.util.concurrent.locks.ReentrantLock;\n\npublic class FairLockDemo {\n private static final ReentrantLock lock = new ReentrantLock(true); // Fair lock\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n }\n }\n };\n\n Thread highPriority = new Thread(task, \"High-Priority\");\n Thread lowPriority = new Thread(task, \"Low-Priority\");\n\n highPriority.setPriority(Thread.MAX_PRIORITY);\n lowPriority.setPriority(Thread.MIN_PRIORITY);\n\n highPriority.start();\n lowPriority.start();\n }\n}\nNote
Although Java allows you to assign priorities to threads, the JVM\u2019s thread scheduler may not always honor them consistently. It\u2019s generally recommended to avoid relying on thread priorities for critical tasks, If you need to control thread scheduling, use fair locks or condition variables instead of thread priorities.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#backoff-strategies","title":"Backoff Strategies","text":"Using backoff strategies (delays) between retries can help reduce contention for resources. This ensures that no thread monopolizes the CPU by continuously attempting to acquire a resource.
Backoff Strategy Exampleimport java.util.concurrent.locks.ReentrantLock;\n\npublic class BackoffDemo {\n private static final ReentrantLock lock = new ReentrantLock();\n\n public static void main(String[] args) {\n Runnable task = () -> {\n while (true) {\n if (lock.tryLock()) {\n try {\n System.out.println(Thread.currentThread().getName() + \" got the lock!\");\n break;\n } finally {\n lock.unlock();\n }\n } else {\n System.out.println(Thread.currentThread().getName() + \" waiting...\");\n try {\n Thread.sleep((long) (Math.random() * 100)); // Random delay\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }\n }\n };\n\n Thread t1 = new Thread(task, \"Thread-1\");\n Thread t2 = new Thread(task, \"Thread-2\");\n\n t1.start();\n t2.start();\n }\n}\nNote
Random delays ensure that threads do not engage in busy-waiting loops, reducing contention and improving fairness.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#use-thread-pools","title":"Use Thread Pools","text":"When dealing with many concurrent tasks, using a thread pool ensures that threads are fairly scheduled and resources are shared efficiently.
Using ThreadPoolExecutor Exampleimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class ThreadPoolDemo {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(2);\n\n Runnable task = () -> {\n System.out.println(Thread.currentThread().getName() + \" is running\");\n };\n\n for (int i = 0; i < 5; i++) {\n executor.submit(task);\n }\n\n executor.shutdown();\n }\n}\nNote
Using thread pools avoids creating too many threads and ensures fair resource sharing.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#avoid-long-critical-sections","title":"Avoid Long Critical Sections","text":"Instead of relying on priorities or busy-waiting, use Condition objects with ReentrantLock to manage thread coordination efficiently.
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.Lock;\nimport java.util.concurrent.locks.ReentrantLock;\n\npublic class ConditionDemo {\n private static final Lock lock = new ReentrantLock();\n private static final Condition condition = lock.newCondition();\n\n public static void main(String[] args) {\n new Thread(() -> {\n lock.lock();\n try {\n System.out.println(\"Waiting...\");\n condition.await(); // Wait for a signal\n System.out.println(\"Resumed\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n } finally {\n lock.unlock();\n }\n }).start();\n\n new Thread(() -> {\n lock.lock();\n try {\n Thread.sleep(1000);\n condition.signal(); // Signal the waiting thread\n System.out.println(\"Signaled\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n } finally {\n lock.unlock();\n }\n }).start();\n }\n}\nNote
Using conditions helps avoid busy-waiting and ensures efficient thread signaling.
"},{"location":"langdives/Java/Locking-Issues-Starvation/#best-practices","title":"Best Practices","text":"Starvation is a subtle but serious issue in multithreaded programs, particularly when some threads are prioritized over others or when resources are monopolized by specific threads. By using fair locks, thread pools, backoff strategies, and avoiding long critical sections.
"},{"location":"langdives/Java/Locking-Reentrant/","title":"Locking.","text":"Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers reentrant locks.
"},{"location":"langdives/Java/Locking-Reentrant/#what-is-locking","title":"What is Locking ?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock.
ReentrantLock ?","text":"The ReentrantLock class, introduced in Java 5, offers more control over thread synchronization than the synchronized keyword. It allows for advanced locking techniques such as fairness policies, tryLock, and interruptible locks. Let\u2019s explore everything about ReentrantLock, including its use cases, internal mechanisms, and best practices.
ReentrantLock is a concrete class in the java.util.concurrent.locks package that implements the Lock interface.
Note
synchronized keyword, which automatically releases the lock when the block exits.import java.util.concurrent.locks.ReentrantLock;\n\nclass Counter {\n private int count = 0;\n private final ReentrantLock lock = new ReentrantLock();\n\n public void increment() {\n lock.lock(); // Acquire the lock\n try {\n count++;\n } finally {\n lock.unlock(); // Release the lock\n }\n }\n\n public int getCount() {\n lock.lock();\n try {\n return count;\n } finally {\n lock.unlock();\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nLock Acquisition: When a thread calls lock(), it tries to acquire the lock. If the lock is available, the thread proceeds otherwise, it blocks until the lock becomes available.
Reentrancy: A thread that holds the lock can acquire the lock again without blocking. This is useful when a thread *nters a method that also calls another synchronized method or block that requires the same lock.
Fair vs Unfair Locking:
Fair Lock: Threads are granted access in the order they requested the lock. this lock ensures that the longest-waiting thread gets the lock first, the main advantage is this avoids thread starvation and the disadvantage is it may have lower performence due to increased overhead
ReentrantLock lock = new ReentrantLock(true); // Fair lock\nUnfair Lock: Threads may skip the queue if the lock is released, improving performance but reducing fairness the main advantage is better throughout because threads are allowed to \"Jump the queue\" but disadvantage is it can lead to thread starvation where some threads may not get chance to execute.
ReentrantLock lock = new ReentrantLock(); // Unfair lock (default)\nThe tryLock() method attempts to acquire the lock without blocking. It returns true if the lock is acquired, otherwise false.
if (lock.tryLock()) {\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} else {\n System.out.println(\"Could not acquire lock, doing something else...\");\n}\nWhen you want to avoid blocking indefinitely if the lock is not available.
"},{"location":"langdives/Java/Locking-Reentrant/#trylock-with-timeout","title":"tryLock with Timeout","text":"The tryLock(long timeout, TimeUnit unit) method waits for a specific amount of time to acquire the lock.
import java.util.concurrent.TimeUnit;\n\nif (lock.tryLock(1, TimeUnit.SECONDS)) {\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} else {\n System.out.println(\"Could not acquire lock within timeout.\");\n}\nWhen waiting indefinitely is not practical, such as network operations or I/O tasks.
"},{"location":"langdives/Java/Locking-Reentrant/#interruptible-lock-acquisition","title":"Interruptible Lock Acquisition","text":"The lockInterruptibly() method allows a thread to acquire the lock but respond to interrupts while waiting.
try {\n lock.lockInterruptibly(); // Wait for lock, but respond to interrupts\n try {\n // Perform task\n } finally {\n lock.unlock();\n }\n} catch (InterruptedException e) {\n System.out.println(\"Thread was interrupted.\");\n}\nUse when a thread needs to be interrupted while waiting for a lock.
"},{"location":"langdives/Java/Locking-Reentrant/#behavior","title":"Behavior","text":"A reentrant lock means that the same thread can acquire the lock multiple times without blocking itself. However, the thread must release the lock the same number of times to fully unlock it.
Behavior Exampleclass ReentrantExample {\n private final ReentrantLock lock = new ReentrantLock();\n\n public void outerMethod() {\n lock.lock();\n try {\n System.out.println(\"In outer method\");\n innerMethod();\n } finally {\n lock.unlock();\n }\n }\n\n public void innerMethod() {\n lock.lock();\n try {\n System.out.println(\"In inner method\");\n } finally {\n lock.unlock();\n }\n }\n}\nIn this example, outerMethod calls innerMethod, and both methods acquire the same lock. This works without issues because ReentrantLock allows reentrant locking.
The Condition interface (associated with a ReentrantLock) allows a thread to wait for a condition to be met before proceeding. It provides better control than the traditional wait()/notify().
import java.util.concurrent.locks.Condition;\nimport java.util.concurrent.locks.ReentrantLock;\n\nclass ConditionExample {\n private final ReentrantLock lock = new ReentrantLock();\n private final Condition condition = lock.newCondition();\n private boolean ready = false;\n\n public void awaitCondition() throws InterruptedException {\n lock.lock();\n try {\n while (!ready) {\n condition.await(); // Wait for signal\n }\n System.out.println(\"Condition met, proceeding...\");\n } finally {\n lock.unlock();\n }\n }\n\n public void signalCondition() {\n lock.lock();\n try {\n ready = true;\n condition.signal(); // Signal waiting thread\n } finally {\n lock.unlock();\n }\n }\n}\nReentrantLock has more overhead than synchronized due to fairness policies and explicit lock management, Use synchronized for simple scenarios, use reentrantLock for more complex locking requirements(eg: tryLock, fairness).
Locking is an essential concept in multithreaded programming to prevent race conditions and ensure thread safety. When multiple threads access shared resources, locks ensure that only one thread accesses the critical section at a time.
This article covers read-write locks.
"},{"location":"langdives/Java/Locking-ReentrantReadWrite/#what-is-locking","title":"What is Locking ?","text":"Locking is a way to ensure that only one thread at a time executes a critical section or modifies a shared resource, Without proper locks, multiple threads may interfere with each other, leading to data inconsistency or unexpected behavior (race conditions).
Java offers various locking mechanisms, from synchronized blocks to explicit locks like ReentrantLock and ReentrantReadWriteLock.
ReentrantReadWriteLock?","text":"The ReentrantReadWriteLock is a specialized lock from Java\u2019s java.util.concurrent.locks package, designed to improve performance in concurrent systems by separating read and write operations. It provides more efficient locking behavior when the majority of operations are read-only, allowing multiple readers to access the shared resource simultaneously but blocking writers until all reading operations are complete.
A ReentrantReadWriteLock maintains two types of locks:
This separation helps optimize performance for read-heavy workloads.
Exampleimport java.util.concurrent.locks.ReentrantReadWriteLock;\n\nclass SharedResource {\n private int data = 0;\n private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();\n\n public void write(int value) {\n lock.writeLock().lock(); // Acquire write lock\n try {\n data = value;\n System.out.println(\"Data written: \" + value);\n } finally {\n lock.writeLock().unlock(); // Release write lock\n }\n }\n\n public int read() {\n lock.readLock().lock(); // Acquire read lock\n try {\n System.out.println(\"Data read: \" + data);\n return data;\n } finally {\n lock.readLock().unlock(); // Release read lock\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n SharedResource resource = new SharedResource();\n\n // Writer thread\n Thread writer = new Thread(() -> resource.write(42));\n\n // Reader threads\n Thread reader1 = new Thread(() -> resource.read());\n Thread reader2 = new Thread(() -> resource.read());\n\n writer.start();\n reader1.start();\n reader2.start();\n\n writer.join();\n reader1.join();\n reader2.join();\n }\n}\nFair vs Unfair Locking: By default, ReentrantReadWriteLock uses unfair locking for better performance, but they may introduce performance overhead
ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true); // Fair lock\nReentrancy: A thread holding a write lock can acquire the read lock without blocking, This is useful for scenarios where a thread needs to read and modify shared data atomically.
Downgrading: A thread can downgrade a write lock to a read lock without releasing the lock entirely.
lock.writeLock().lock();\ntry {\n // Critical write section\n lock.readLock().lock(); // Downgrade to read lock\n} finally {\n lock.writeLock().unlock(); // Release write lock\n}\nIn scenarios with frequent readers, a writer may starve because readers keep acquiring the read lock, delaying the writer indefinitely.
Solution
Use a fair lock
ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true); // Enable fairness\nFair locks ensure that waiting writers get a chance to execute after the current readers finish.
DeadLockIf a read lock and write lock are acquired in an inconsistent order across multiple threads, it can lead to deadlock.
Deadlock example
Thread 1: Acquire write lock -> Attempt to acquire read lock (blocks)\nThread 2: Acquire read lock -> Attempt to acquire write lock (blocks)\nSolution
If there are frequent write operations, the system behaves similarly to using a normal ReentrantLock, as readers must wait for writers to release the lock.
Solution
Use lock-free data structures (like AtomicReference) or ReadWriteLock only when reads significantly outnumber writes.
If a thread holding the write lock tries to release it before acquiring the read lock, data inconsistencies can occur.
Correct Lock Downgrading Example
lock.writeLock().lock();\ntry {\n // Write critical section\n lock.readLock().lock(); // Downgrade to read lock\n} finally {\n lock.writeLock().unlock(); // Release write lock\n}\n// Perform read operations under the read lock.\nRead-heavy Workloads, when the majority of operations are reads and only a few writes occur.
Improved Performance over ReentrantLock, In scenarios where multiple threads need to read data simultaneously, ReentrantReadWriteLock improves performance by allowing concurrent reads.
When You Need to Separate Read and Write Operations, use it to prevent blocking readers unnecessarily during write operations.
Write-heavy Applications, If writes occur frequently, the write lock will block readers, negating the benefits of the read-write separation.
Simple Synchronization Requirements, If your application doesn\u2019t require complex locking behavior, use synchronized or ReentrantLock instead.
Lock-free Alternatives Available, consider atomic classes or concurrent data structures (e.g., ConcurrentHashMap) if they meet your needs.
Use Fair Locks Only When Necessary, Use unfair locks by default for better performance. Enable fairness only if starvation becomes an issue.
Minimize the Duration of Locks, Avoid holding read or write locks longer than necessary to reduce contention.
Use Lock Downgrading Cautiously, Always acquire the read lock before releasing the write lock to avoid inconsistencies.
Monitor Lock Usage, Use tools like VisualVM or JConsole to monitor thread activity and detect potential contention or deadlocks.
Use Lock-Free Data Structures When Possible, For simple counters or flags, use atomic classes (e.g., AtomicInteger) to avoid locking overhead.
ReentrantReadWriteLock is a powerful tool that allows multiple threads to read concurrently while ensuring exclusive access for writes. However, it is most effective in read-heavy scenarios. Understanding potential issues like write starvation, deadlocks, and performance degradation is essential for using this lock effectively, by following best practices like consistent lock ordering, minimizing lock duration, and monitoring lock usage, you can avoid common pitfalls and maximize the performance benefits of ReentrantReadWriteLock.
synchronized vs ReentrantLock","text":""},{"location":"langdives/Java/LockingIntrinsicReentrant/#differences","title":"Differences","text":"Feature synchronized ReentrantLock Basic Concept Uses intrinsic lock (monitor) on objects. Uses an explicit lock from java.util.concurrent.locks. Lock Acquisition Acquired implicitly when entering a synchronized block or method. Acquired explicitly via lock() method. Release of Lock Automatically released when the thread exits the synchronized block or method. Must be explicitly released via unlock(). Reentrancy Supports reentrancy (same thread can acquire the same lock multiple times). Supports reentrancy just like synchronized. Fairness Unfair by default (no control over thread access order). Can be fair or unfair (configurable with ReentrantLock(true)). Interruptibility Cannot respond to interrupts while waiting for the lock. Supports interruptible locking via lockInterruptibly(). Try Locking Not supported. A thread will block indefinitely if the lock is not available. Supports tryLock() to attempt locking without blocking or with timeout. Condition Variables Uses wait() / notify() / notifyAll() methods on the intrinsic lock. Supports multiple Condition objects for finer-grained wait/notify control. Timeout Support Not supported. If the lock is held by another thread, it will wait indefinitely. Supports timeout locking with tryLock(long timeout, TimeUnit unit). Performance Overhead Low for simple scenarios with little contention. Higher overhead but provides greater control over locking behavior. Fair Locking Option Not supported (always unfair). Fair locking can be enabled with ReentrantLock(true). Use in Non-blocking Operations Not possible. Possible with tryLock() (non-blocking). Flexibility and Control Limited to synchronized methods or blocks. Greater flexibility: lock multiple sections, lock only part of a method, or use multiple conditions. Suitability for Deadlock Avoidance Requires external logic to prevent deadlocks (acquire locks in the same order). Easier to prevent deadlocks using tryLock() and timeouts. Memory Usage No additional memory overhead. Uses the object\u2019s monitor. Requires additional memory for lock objects and lock metadata. Readability and Simplicity Easier to read and maintain (especially for small, simple use cases). More complex code with explicit lock management. Error Handling No need to manage lock release in a finally block. The lock is automatically released. Requires explicit unlock() in finally blocks to avoid deadlocks or memory leaks. Thread Starvation Prone to thread starvation in high contention scenarios. Can prevent starvation using fair lock mode. Recommended Use Case Best for simple synchronization needs where you don\u2019t need advanced control. Recommended for complex concurrency scenarios needing fine-grained locking, fairness, tryLock, or interruptibility."},{"location":"langdives/Java/LockingIntrinsicReentrant/#when-to-use","title":"When to Use ?","text":"Use synchronized Use ReentrantLock When you need simple, block-level or method-level synchronization. When you need advanced control over locking behavior (e.g., tryLock, fairness, or interruptibility). When you want automatic lock release (less error-prone). When you need multiple locks or condition variables. When performance matters in low-contention scenarios (lower overhead). When dealing with high contention and you need fair scheduling to prevent starvation. When you don't need non-blocking operations or timeouts. When you want non-blocking operations using tryLock() or timeout-based locking. When the code needs to be simple and easy to read. When code complexity is acceptable for greater flexibility."},{"location":"langdives/Java/LockingIntrinsicReentrant/#summary","title":"Summary","text":"Both synchronized and ReentrantLock have their own strengths and use cases. Use synchronized for simpler, lower-level concurrency needs, and ReentrantLock when you need more control, fairness, or advanced features like non-blocking locking and condition variables.
In general: - synchronized is easier to use and less error-prone. - ReentrantLock is more powerful and flexible, but with more overhead and complexity.
Apache Maven is a build automation and project management tool primarily for Java projects. It uses XML (pom.xml) to describe the project's structure, dependencies, and build lifecycle. Maven focuses on the \u201cconvention over configuration\u201d principle, meaning it provides a standard way to structure and build projects with minimal configuration.
"},{"location":"langdives/Java/Maven/#how-maven-works","title":"How Maven Works ?","text":"Maven operates using a build lifecycle consisting of pre-defined phases. When you execute a specific phase, all preceding phases are executed as well.
Maven Lifecycle Phases Phase Descriptionvalidate Validates the project structure. compile Compiles the source code. test Runs the unit tests. package Packages the compiled code into a JAR or WAR. verify Verifies the package meets specifications. install Installs the JAR into the local Maven repository. deploy Deploys the artifact to a remote repository. Maven revolves around the POM (Project Object Model), which defines:
We will understand pom.xml in next section more.
pom.xml","text":"The POM (Project Object Model) file is the heart of a Maven project. It defines dependencies, build plugins, and project metadata.
Basic Example of pom.xml
<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 \n http://maven.apache.org/xsd/maven-4.0.0.xsd\">\n <modelVersion>4.0.0</modelVersion>\n\n <groupId>com.example</groupId>\n <artifactId>my-app</artifactId>\n <version>1.0-SNAPSHOT</version>\n <packaging>jar</packaging>\n\n <dependencies>\n <dependency>\n <groupId>org.apache.commons</groupId>\n <artifactId>commons-lang3</artifactId>\n <version>3.12.0</version>\n </dependency>\n </dependencies>\n\n <build>\n <plugins>\n <plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.8.1</version>\n <configuration>\n <source>1.8</source>\n <target>1.8</target>\n </configuration>\n </plugin>\n </plugins>\n </build>\n</project>\nKey Components of pom.xml
com.example).my-app).1.0-SNAPSHOT).maven-compiler-plugin).Maven simplifies dependency management by automatically downloading required libraries from repositories.
Scopes of Dependencies
compile: Available at compile time and runtime (default scope).test: Only used for testing purposes.provided: Available at compile time but not packaged (e.g., Servlet API).runtime: Available at runtime only.Example Dependency Declaration
<dependency>\n <groupId>org.apache.commons</groupId>\n <artifactId>commons-lang3</artifactId>\n <version>3.12.0</version>\n</dependency>\nMaven resolves dependencies from repositories:
~/.m2/repository, stores downloaded artifacts.Adding a Custom Repository
<repositories>\n <repository>\n <id>my-repo</id>\n <url>https://my-repo-url</url>\n </repository>\n</repositories>\nMaven plugins extend its functionality. Plugins can handle tasks such as compiling, testing, or packaging.
Maven Compiler Plugin
<plugin>\n <groupId>org.apache.maven.plugins</groupId>\n <artifactId>maven-compiler-plugin</artifactId>\n <version>3.8.1</version>\n <configuration>\n <source>1.8</source>\n <target>1.8</target>\n </configuration>\n</plugin>\n/my-project\n\u2502\n\u251c\u2500\u2500 pom.xml # Project Object Model configuration\n\u251c\u2500\u2500 src\n\u2502 \u2514\u2500\u2500 main\n\u2502 \u2514\u2500\u2500 java # Source code\n\u2502 \u2514\u2500\u2500 test\n\u2502 \u2514\u2500\u2500 java # Unit tests\n\u2514\u2500\u2500 target # Output directory (JAR, WAR)\nsrc/main/java: Contains the application source code.src/test/java: Contains unit test code.target/: Contains compiled artifacts (like JARs).Maven supports multi-module projects, allowing multiple related projects to be managed together.
Directory Structure/parent-project\n\u2502\n\u251c\u2500\u2500 pom.xml (parent)\n\u251c\u2500\u2500 module-1/\n\u2502 \u2514\u2500\u2500 pom.xml\n\u2514\u2500\u2500 module-2/\n \u2514\u2500\u2500 pom.xml\n<modules>\n <module>module-1</module>\n <module>module-2</module>\n</modules>\nmvn install\nSimilar to Gradle, Maven has a wrapper (mvnw) that ensures the project uses a specific Maven version.
mvn -N io.takari:maven:wrapper\n./mvnw clean installmvnw.cmd clean installHere are some common Maven commands
Command Descriptionmvn compile Compiles the source code. mvn test Runs unit tests. mvn package Packages the code into a JAR/WAR. mvn install Installs the artifact to the local repository. mvn deploy Deploys the artifact to a remote repository. mvn clean Cleans the target/ directory. mvn dependency:tree Displays the project's dependency tree."},{"location":"langdives/Java/Maven/#best-practices","title":"Best Practices","text":"Maven is a mature, stable tool that simplifies building and managing Java applications. Its focus on conventions reduces the need for complex configurations, making it ideal for enterprise projects. While Maven may lack some of the flexibility and speed of Gradle, it is widely used in large organizations for its reliability and standardization. For projects requiring strict conventions and extensive dependency management, Maven remains a popular choice.
"},{"location":"langdives/Java/MavenVsGradle/","title":"Maven vs Gradle","text":""},{"location":"langdives/Java/MavenVsGradle/#comparision","title":"Comparision","text":"Aspect Maven Gradle Configuration Style Uses XML (pom.xml). Uses Groovy/Kotlin DSL (build.gradle). Performance Slower, especially for large projects (no build caching). Faster with incremental builds and caching. Flexibility Follows convention over configuration, less customizable. Highly customizable, supports custom build logic. Dependency Management Maven Central and custom repositories. Supports Maven Central, JCenter, Ivy, and custom repositories. Plugin System Pre-built Maven plugins (strict lifecycle integration). More flexible plugins with multiple custom task types. Build Output Produces JARs, WARs, and other artifacts. Produces JARs, WARs, and custom artifacts more easily. Multi-Project Support Good for enterprise projects with structured multi-module builds. Excellent for multi-module projects, especially in complex setups. Integration with CI/CD Easily integrates with Jenkins, GitHub Actions, Bamboo. Same level of integration with Jenkins, CircleCI, GitHub Actions. Use in Android Development Not suitable. Preferred build tool for Android development. Incremental Builds Not supported. Supported, resulting in faster builds. Offline Mode Uses the local Maven repository (~/.m2/repository). Uses a local cache (~/.gradle/caches/) and has offline mode. Version Control of Build Tool Maven Wrapper (mvnw) ensures consistent versions. Gradle Wrapper (gradlew) ensures consistent versions. Preferred Projects Enterprise Java applications with well-defined standards. Android apps, complex and large projects with custom build requirements."},{"location":"langdives/Java/MavenVsGradle/#when-to-use-gradle","title":"When to Use Gradle ?","text":"Use Maven if:
Use Gradle if:
In conclusion, both Maven and Gradle are excellent tools, and the choice depends on the project requirements. For enterprise applications, Maven remains a solid choice. For Android apps, large multi-module projects, or performance-critical builds, Gradle stands out as the preferred option.
"},{"location":"langdives/Java/MemoryModel/","title":"Java Memory Model","text":"Java uses a memory model that divides memory into different areas, primarily the heap and stack.
"},{"location":"langdives/Java/MemoryModel/#heap-memory","title":"Heap Memory","text":"The heap is mainly used for dynamic memory allocation. Objects created using the new keyword are stored in the heap. Coming to it's life time objects in the heap remain in memory until they are no longer referenced and are garbage collected. This means the lifetime of an object is not tied to the scope of a method and Accessing memory in the heap is slower than in the stack due to its dynamic nature and the potential for fragmentation.
Note
-Xms for initial heap size and -Xmx for maximum heap size). new keyword.MyClass obj = new MyClass(); // Heap allocation\nThe stack is mainly used for static memory allocation. It stores method call frames, which contain local variables, method parameters, and return addresses. coming to the lifetime of a variable in the stack is limited to the duration of the method call. Once the method returns, the stack frame is popped off, and the memory is reclaimed and accessing stack memory is faster than heap memory because it follows a Last In, First Out (LIFO) order, allowing for quick allocation and deallocation.
Note
-Xss).Example
public class MemoryExample {\n public static void main(String[] args) {\n int localVar = 10; // Stack memory\n\n MemoryExample obj = new MemoryExample(); // Heap memory\n obj.display(localVar); // Passing parameter, stack memory\n }\n\n public void display(int param) { // Stack memory\n System.out.println(param);\n String message = \"Hello\"; // Heap memory (String object)\n }\n}\nJava has 8 primitive data types that store simple values directly in memory.
Type Size Default Value Range Examplebyte 1 byte (8 bits) 0 -128 to 127 byte b = 100; short 2 bytes (16 bits) 0 -32,768 to 32,767 short s = 30000; int 4 bytes (32 bits) 0 -2^31 to (2^31)-1 int i = 100000; long 8 bytes (64 bits) 0L -2^63 to (2^63)-1 long l = 100000L; float 4 bytes (32 bits) 0.0f ~\u00b13.4E38 (7 decimal digits precision) float f = 3.14f; double 8 bytes (64 bits) 0.0 ~\u00b11.8E308 (15 decimal digits precision) double d = 3.14159; char 2 bytes (16 bits) '\\u0000' Unicode characters (0 to 65,535) char c = 'A'; boolean 1 bit (virtual) false true or false boolean b = true;"},{"location":"langdives/Java/PrimitiveReferenceTypes/#reference-types","title":"Reference Types","text":"Reference types store references (addresses) to objects in memory, unlike primitive types that store values directly.
String: Represents a sequence of characters.
String str = \"Hello, World!\";\nArrays: Collections of elements of the same type.
int[] numbers = {1, 2, 3};\nClasses and Objects: Custom data types representing real-world entities.
class Person {\n String name;\n}\nPerson p = new Person();\nInterfaces: Contracts that classes can implement.
interface Animal {\n void sound();\n}\nEnums: Special classes that define a set of constants.
enum Day {\n MONDAY, TUESDAY, WEDNESDAY;\n}\nWrapper Classes: Used to convert primitive types into objects (auto-boxing/unboxing).
byte Byte short Short int Integer long Long float Float double Double char Character boolean Boolean"},{"location":"langdives/Java/PrimitiveReferenceTypes/#differences","title":"Differences","text":"Aspect Primitive Types Reference Types Storage Store actual values. Store references to objects in memory. Memory Allocation Stored in stack memory. Stored in heap memory. Default Values Zero/false equivalents. null for uninitialized references. Examples int, char, boolean. String, Arrays, Classes, Interfaces, etc."},{"location":"langdives/Java/ReferenceTypesInDepth/","title":"Reference Types In Depth.","text":"Let's deep dive to understand How memory management, object references, and behaviors work in Java in this article, with a focus on String handling and other reference types like arrays, classes, and wrapper objects.
"},{"location":"langdives/Java/ReferenceTypesInDepth/#intro","title":"Intro","text":"Primitive types store values directly in stack memory Where as Reference types store references (addresses) to objects located in heap memory, When you assign a reference type (e.g., String or an array), only the reference (address) is copied, not the actual data. This means multiple references can point to the same object.
The String class in Java is a special reference type with some unique behaviors. Strings are immutable once a String object is created, it cannot be changed. Any modification on a String results in the creation of a new object in memory.
A special memory area inside the heap used to store string literals. If a string literal like \"Hello\" is created, Java first checks the string pool to see if it already exists. If it does, it returns the reference from the pool. If not, the string is added to the pool.
String s1 = \"Hello\"; // Stored in the String Pool\nString s2 = \"Hello\"; // s2 points to the same object as s1\n\nSystem.out.println(s1 == s2); // true (same reference)\nWhen you use the new keyword, a new String object is always created in the heap memory. Even if the same string already exists in the string pool, the new keyword forces the creation of a separate instance in the heap.
String s1 = new String(\"Hello\"); // creates a new object outside the pool in the heap.\n\nString s2 = \"Hello\"; // Stored in the String Pool\n\nSystem.out.println(s1 == s2); // false (different references)\nnew String(), Java forces the creation of a new object in heap even if the same string exists in the pool."},{"location":"langdives/Java/ReferenceTypesInDepth/#arrays","title":"Arrays","text":"Arrays are reference types, meaning the array variable stores a reference to the memory location where the array data is stored.
Exampleint[] arr1 = {1, 2, 3};\nint[] arr2 = arr1; // arr2 now references the same array as arr1\n\narr2[0] = 10; // Modifies the original array\n\nSystem.out.println(arr1[0]); // Output: 10 (both arr1 and arr2 reference the same array)\nHow Array References Work:
arr1 and arr2 point to the same memory location in the heap.arr2, the change will reflect in arr1 because both refer to the same object.When you create an object using new, the reference variable points to the object in heap memory.
class Person {\n String name;\n}\n\nPerson p1 = new Person();\np1.name = \"Alice\";\n\nPerson p2 = p1; // p2 points to the same object as p1\np2.name = \"Bob\";\n\nSystem.out.println(p1.name); // Output: Bob (both references point to the same object)\nHow References Work with Objects:
p1 and p2 point to the same Person object. Changing the object via p2 affects p1.Wrapper classes (Integer, Double, Boolean, etc.) wrap primitive types into objects. These are reference types, and Java performs autoboxing/unboxing to convert between primitive types and wrapper objects.
Integer num1 = 100;\nInteger num2 = 100;\n\nSystem.out.println(num1 == num2); // true (for values within -128 to 127)\n\nInteger num3 = 200;\nInteger num4 = 200;\n\nSystem.out.println(num3 == num4); // false (new objects for values beyond 127)\nWrapper Caching
Integer objects in the range -128 to 127 for performance.Shallow Copy: Copies only the reference, so both variables refer to the same object.
Exampleint[] original = {1, 2, 3};\nint[] shallowCopy = original; // Points to the same array\n\nshallowCopy[0] = 100;\nSystem.out.println(original[0]); // Output: 100\nDeep Copy: Creates a new object with the same data.
Exampleint[] original = {1, 2, 3};\nint[] deepCopy = original.clone(); // Creates a new array\n\ndeepCopy[0] = 100;\nSystem.out.println(original[0]); // Output: 1\nNullPointerException","text":"When a reference is not initialized, it holds the value null. Accessing a field or method on a null reference throws a NullPointerException.
Person p = null;\nSystem.out.println(p.name); // Throws NullPointerException\nJava uses Garbage Collection to manage memory. When no references point to an object, it becomes eligible for garbage collection.
ExamplePerson p1 = new Person(); // Object created\np1 = null; // Now eligible for garbage collection\nWe will learn about garbage collection more in depth in another article.
"},{"location":"langdives/Java/ReferenceTypesInDepth/#summary","title":"Summary","text":"new String() creates a separate object.Integer values from -128 to 127).The String Pool (also called the intern pool) in Java is implemented using a Hash Table-like data structure internally. Let\u2019s explore the design and behavior behind this structure:
"},{"location":"langdives/Java/ReferenceTypesInDepth/#internals","title":"Internals","text":"Hash Table Concept: The String Pool uses a Hash Map-like structure internally, where the key is the string literal, and the value is a reference to the interned string in the pool. This ensures fast lookups and deduplication of identical strings because hash-based structures allow O(1) average-time complexity for lookup operations.
Behavior of String Pool: If a new string literal is created, the pool Checks if the string already exists (by computing its hash), If the string exists, it returns the existing reference. If not, the string is added to the pool.
Rehashing and Thread Safety: The interned pool is part of the JVM's heap and managed by the Java String class. Since Java 7, the pool is stored in the heap instead of the PermGen space to allow better memory management. The pool structure is thread-safe, meaning multiple threads can safely use the pool.
How the pool works internally
class StringPool {\n private static Map<String, String> pool = new HashMap<>();\n\n public static String intern(String str) {\n if (pool.containsKey(str)) {\n return pool.get(str); // Return existing reference\n } else {\n pool.put(str, str); // Add to the pool\n return str;\n }\n }\n}\nString.intern(), Java interns the string, meaning it adds the string to the pool if it's not already present. String Pool Usage Example public class Main {\n public static void main(String[] args) {\n String s1 = new String(\"Hello\");\n String s2 = s1.intern(); // Adds \"Hello\" to the pool, if not already present\n\n String s3 = \"Hello\"; // Uses the interned string from the pool\n\n System.out.println(s2 == s3); // true (same reference from the pool)\n }\n}\nKey Takeaways
OutOfMemoryError if too many strings were interned.intern() on every string can increase the overhead slightly (because of the hash lookup). It\u2019s only beneficial for reducing memory usage when you expect many repeated strings.The String Pool is implemented using a Hash Table-like data structure, This allows for efficient string reuse through fast lookups and ensures no duplicate literals are created. Strings added via literals or intern() are stored in the pool, with existing references returned on subsequent requests.
Enables functional programming by treating functions as first-class citizens.
(parameters) -> expression or (parameters) -> { statements }List<String> names = Arrays.asList(\"Alice\", \"Bob\", \"Charlie\");\nnames.forEach(name -> System.out.println(name));\nA functional interface is an interface with only one abstract method. This is important because lambda expressions can be used to provide the implementation for these interfaces.
Example Example functional interface@FunctionalInterface // Optional but ensures the interface has only one abstract method.\ninterface MyFunction {\n int apply(int a, int b); // Single abstract method\n}\nNow, when you want to use this interface, you don\u2019t need to create a class and provide an implementation like before. Instead, you can use a lambda expression to quickly provide the logic.
Using Lambda with MyFunctionMyFunction addition = (a, b) -> a + b; // Lambda expression for addition\nSystem.out.println(addition.apply(5, 3)); // Output: 8\n(a, b) -> a + b is the lambda expression.apply(int a, int b) method of the MyFunction interface.A method reference is a shorthand way of writing a lambda when a method already exists that matches the lambda\u2019s purpose. This makes the code more concise and readable.
Example withforEach and Method Reference Consider the following list of names:
List<String> names = Arrays.asList(\"Alice\", \"Bob\", \"Charlie\");\nYou want to print all names using forEach(). You could do it with a lambda like this:
names.forEach(name -> System.out.println(name)); // Lambda expression\nNow, Java provides a shorthand: Method Reference. Since System.out.println() already matches the structure (String) -> void, you can write:
names.forEach(System.out::println); // Method reference\nSystem.out::println is a method reference to the println() method of System.out.name -> System.out.println(name) but is cleaner.Use method references when:
// 1. Static method reference\nFunction<String, Integer> parse = Integer::parseInt;\nSystem.out.println(parse.apply(\"123\")); // Output: 123\n\n// 2. Instance method reference on an arbitrary object\nList<String> words = Arrays.asList(\"one\", \"two\", \"three\");\nwords.sort(String::compareToIgnoreCase); // Sorts case-insensitively\nIntroduced in Java 8 to process collections in a declarative way.
Core Stream Operations
"},{"location":"langdives/Java/StreamsLambdas/#creation","title":"Creation","text":"Stream<Integer> stream = Stream.of(1, 2, 3, 4);\nList<String> list = Arrays.asList(\"A\", \"B\", \"C\");\nStream<String> streamFromList = list.stream();\n// Filters elements based on a predicate.\nList<Integer> evenNumbers = stream.filter(n -> n % 2 == 0).toList();\n// Transforms elements.\nList<Integer> lengths = list.stream().map(String::length).toList();\n// Sorts elements.\nList<Integer> sortedList = stream.sorted().toList();\n// Iterates through elements.\nlist.stream().forEach(System.out::println);\n// Collects elements into a collection.\nList<String> newList = list.stream().filter(s -> s.startsWith(\"A\")).collect(Collectors.toList());\n// Reduces the elements to a single result.\nint sum = Stream.of(1, 2, 3, 4).reduce(0, Integer::sum);\n// Used to process elements in parallel for better performance.\nlist.parallelStream().forEach(System.out::println);\nint sumOfEvens = Stream.of(1, 2, 3, 4, 5, 6)\n .filter(n -> n % 2 == 0)\n .reduce(0, Integer::sum);\nSystem.out.println(sumOfEvens); // Output: 12\nList<String> upperCaseNames = list.stream()\n .map(String::toUpperCase)\n .collect(Collectors.toList());\nMap<Integer, List<String>> groupedByLength = list.stream()\n .collect(Collectors.groupingBy(String::length));\nmap()).Thread pool configuration is critical for optimizing the performance of your applications. Poorly configured thread pools can lead to problems such as CPU starvation, thread contention, memory exhaustion, or poor resource utilization. In this Article, we\u2019ll dive deep into CPU-bound vs I/O-bound tasks, explore how to determine optimal thread pool sizes, and discuss key considerations such as queue types and rejection policies.
"},{"location":"langdives/Java/ThreadPoolTuning/#cpu-vs-io-bound-tasks","title":"CPU vs I/O Bound Tasks","text":"When configuring thread pools, it is essential to classify your tasks as CPU-bound or I/O-bound, as this distinction guides the number of threads your pool should maintain.
"},{"location":"langdives/Java/ThreadPoolTuning/#cpu-bound-tasks","title":"CPU-Bound Tasks","text":"Tasks that perform intensive computations (e.g., mathematical calculations, data processing, encoding), and here limiting factor is the CPU core availability. So its better to avoid context switching overhead by keeping the number of threads close to the available CPU cores.
Optimal Thread Pool Size for CPU-Bound Tasksint coreCount = Runtime.getRuntime().availableProcessors();\nExecutorService cpuBoundPool = Executors.newFixedThreadPool(coreCount);\nNote
If more threads than CPU cores are running, threads will compete for CPU cycles, causing context switching, which adds overhead.
Optimal Threads = Number of Cores\nTasks that spend most of the time waiting for I/O operations (e.g., network, database, file I/O). and here the limiting factor is the time spent waiting on I/O. So it's better to use more threads than the number of cores to ensure that idle CPU cycles are used efficiently while waiting for I/O.
Optimal Thread Pool Size for I/O-Bound Tasksint coreCount = Runtime.getRuntime().availableProcessors();\nint optimalThreads = coreCount * 2 + 1;\nExecutorService ioBoundPool = Executors.newFixedThreadPool(optimalThreads);\nNote
Since the tasks spend significant time waiting for I/O, more threads can be created to make sure the CPU is not idle while other threads wait for input/output operations.
Optimal Threads = Number of Cores * (1 + Wait Time / Compute Time)\nChoosing the right work queue is crucial for memory management and task scheduling. The queue holds tasks waiting to be executed when all threads are busy.
"},{"location":"langdives/Java/ThreadPoolTuning/#unbounded-queue","title":"Unbounded Queue","text":"A queue with no size limit, but if too many tasks are submitted, it can lead to memory exhaustion (out-of-memory errors).
LinkedBlockingQueueBlockingQueue<Runnable> queue = new LinkedBlockingQueue<>();\nSuitable only if you expect tasks to complete quickly and the queue will not grow indefinitely.
"},{"location":"langdives/Java/ThreadPoolTuning/#bounded-queue","title":"Bounded Queue","text":"A queue with a fixed size limit, it prevents unbounded memory usage, and If the queue is full, tasks will be rejected or handled based on a rejection policy.
ArrayBlockingQueueBlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(10);\nIdeal for controlled environments where you want to cap the number of waiting tasks.
"},{"location":"langdives/Java/ThreadPoolTuning/#thread-pool-size-tuning","title":"Thread Pool Size Tuning","text":"For CPU-Bound TasksOptimal Threads = Number of Cores\nOptimal Threads = Number of Cores * (1 + Wait Time / Compute Time)\nIf a thread spends 70% of the time waiting on I/O, and only 30% performing work:
Optimal Threads = 4 * (1 + 0.7 / 0.3) = 12\nWhen the task queue is full and the pool is at its maximum size, the ThreadPoolExecutor must decide what to do with new tasks. You can configure rejection policies to handle these situations.
RejectedExecutionException.new ThreadPoolExecutor.AbortPolicy();\nnew ThreadPoolExecutor.CallerRunsPolicy();\nnew ThreadPoolExecutor.DiscardPolicy();\nnew ThreadPoolExecutor.DiscardOldestPolicy();\nMonitoring thread pools ensures that your configuration is correct and performing well. You can monitor the following metrics:
Key Metrics to Monitor
ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS,\n new ArrayBlockingQueue<>(2));\n\nSystem.out.println(\"Active Threads: \" + executor.getActiveCount());\nSystem.out.println(\"Task Count: \" + executor.getTaskCount());\nSystem.out.println(\"Completed Tasks: \" + executor.getCompletedTaskCount());\nSometimes, you may need to adjust the pool size at runtime to respond to changing workloads.
Example: Adjusting Thread Pool Size DynamicallyThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS,\n new ArrayBlockingQueue<>(10));\n\n// Adjust core and max pool size dynamically\nexecutor.setCorePoolSize(3);\nexecutor.setMaximumPoolSize(6);\nint poolSize = Runtime.getRuntime().availableProcessors();\nExecutorService executor = Executors.newFixedThreadPool(poolSize);\nFor I/O-bound tasks, use more threads than the number of cores.
Use bounded queues to prevent memory issues.
Monitor and tune Continuously monitor thread pools and adjust configuration as needed.
Gracefully shutdown thread pools to prevent resource leaks
executor.shutdown();\ntry {\n if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {\n executor.shutdownNow();\n }\n} catch (InterruptedException e) {\n executor.shutdownNow();\n}\nUse CallerRunsPolicy for tasks that must be executed but are non-critical (to prevent task loss).
A thread pool is a collection of worker threads that are created at the start and reused to perform multiple tasks. When tasks are submitted to the pool, a free thread picks up the task and executes it. If no threads are free, the tasks wait in a queue until one becomes available.
"},{"location":"langdives/Java/ThreadPools/#advantages-of-thread-pooling","title":"Advantages of Thread Pooling","text":"The Executors class provides convenient factory methods to create thread pools:
newFixedThreadPool()newCachedThreadPool()newSingleThreadExecutor()newScheduledThreadPool()For greater control, you can instantiate ThreadPoolExecutor directly.
Creates a pool with a fixed number of threads. When all threads are busy, tasks are placed in a queue and executed as soon as a thread becomes available.
newFixedThreadPool import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class FixedThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(3);\n\n for (int i = 1; i <= 6; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA dynamic thread pool where threads are created as needed. If threads are idle for 60 seconds, they are terminated. If a thread is available, it will be reused for a new task.
newCachedThreadPool import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class CachedThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newCachedThreadPool();\n\n for (int i = 1; i <= 5; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA single-threaded executor that ensures tasks are executed sequentially in the order they are submitted. If the thread dies due to an exception, a new thread is created to replace it.
newSingleThreadExecutor import java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class SingleThreadExecutorExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newSingleThreadExecutor();\n\n for (int i = 1; i <= 3; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nA scheduled thread pool allows you to schedule tasks to run after a delay or periodically at a fixed rate.
newScheduledThreadPool import java.util.concurrent.Executors;\nimport java.util.concurrent.ScheduledExecutorService;\nimport java.util.concurrent.TimeUnit;\n\npublic class ScheduledThreadPoolExample {\n public static void main(String[] args) {\n ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(2);\n\n Runnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\n\n // Schedule task to run after 3 seconds\n scheduler.schedule(task, 3, TimeUnit.SECONDS);\n\n // Schedule task to run repeatedly every 2 seconds\n scheduler.scheduleAtFixedRate(task, 1, 2, TimeUnit.SECONDS);\n\n // Allow the tasks to complete after 10 seconds\n scheduler.schedule(() -> scheduler.shutdown(), 10, TimeUnit.SECONDS);\n }\n}\nThreadPoolExecutor is the core implementation of thread pools in Java. Using it allows you to fine-tune the thread pool\u2019s behavior with more control over the number of threads, queue type, and rejection policy.
ThreadPoolExecutor executor = new ThreadPoolExecutor(\n corePoolSize, // Minimum number of threads\n maximumPoolSize, // Maximum number of threads\n keepAliveTime, // Idle time before a thread is terminated\n timeUnit, // Time unit for keepAliveTime\n workQueue, // Queue to hold waiting tasks\n threadFactory, // Factory to create new threads\n handler // Rejection policy when the queue is full\n);\nimport java.util.concurrent.*;\n\npublic class CustomThreadPoolExecutorExample {\n public static void main(String[] args) {\n ThreadPoolExecutor executor = new ThreadPoolExecutor(\n 2, 4, 30, TimeUnit.SECONDS,\n new LinkedBlockingQueue<>(2), // Task queue with capacity 2\n Executors.defaultThreadFactory(),\n new ThreadPoolExecutor.CallerRunsPolicy() // Rejection policy\n );\n\n // Submit 6 tasks to the pool\n for (int i = 1; i <= 6; i++) {\n int taskId = i;\n executor.execute(() -> {\n System.out.println(\"Task \" + taskId + \" executed by \" + Thread.currentThread().getName());\n });\n }\n\n executor.shutdown();\n }\n}\nCommon Rejection Policies in ThreadPoolExecutor
RejectedExecutionException when a task is rejected.The Runnable interface represents a task that can run asynchronously in a thread but does not return any result or throw a checked exception.
@FunctionalInterface\npublic interface Runnable {\n void run();\n}\npublic class RunnableExample {\n public static void main(String[] args) {\n Runnable task = () -> {\n System.out.println(\"Executing task in: \" + Thread.currentThread().getName());\n };\n\n Thread thread = new Thread(task);\n thread.start();\n }\n}\nRunnable when no result is expected from the task.The Callable interface is similar to Runnable, but it can return a result and throw a checked exception.
@FunctionalInterface\npublic interface Callable<V> {\n V call() throws Exception;\n}\nimport java.util.concurrent.Callable;\n\npublic class CallableExample {\n public static void main(String[] args) throws Exception {\n Callable<Integer> task = () -> {\n System.out.println(\"Executing task in: \" + Thread.currentThread().getName());\n return 42;\n };\n\n // Direct call (for demonstration)\n Integer result = task.call();\n System.out.println(\"Task result: \" + result);\n }\n}\nCallable when a result or an exception is expected.A Future represents the result of an asynchronous computation. It provides methods to check if the computation is complete, wait for the result, and cancel the task if necessary.
public interface Future<V> {\n boolean cancel(boolean mayInterruptIfRunning);\n boolean isCancelled();\n boolean isDone();\n V get() throws InterruptedException, ExecutionException;\n V get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException;\n}\nimport java.util.concurrent.*;\n\npublic class FutureExample {\n public static void main(String[] args) throws ExecutionException, InterruptedException {\n ExecutorService executor = Executors.newSingleThreadExecutor();\n\n Callable<Integer> task = () -> {\n Thread.sleep(2000); // Simulate some work\n return 42;\n };\n\n Future<Integer> future = executor.submit(task);\n\n // Do something else while the task executes asynchronously\n System.out.println(\"Task is running...\");\n\n // Wait for the result\n Integer result = future.get();\n System.out.println(\"Task result: \" + result);\n\n executor.shutdown();\n }\n}\nFuture allows you to submit tasks to thread pools and retrieve their results once completed.get(): Blocks until the task completes and returns the result.isDone(): Checks if the task is completed.cancel(): Cancels the task if it's still running.BlockingQueue is a thread-safe queue that blocks the calling thread when:
public interface BlockingQueue<E> extends Queue<E> {\n void put(E e) throws InterruptedException;\n E take() throws InterruptedException;\n // Other methods for timed operations, size, etc.\n}\nimport java.util.concurrent.*;\n\npublic class BlockingQueueExample {\n public static void main(String[] args) {\n BlockingQueue<Integer> queue = new ArrayBlockingQueue<>(2);\n\n // Producer thread\n new Thread(() -> {\n try {\n queue.put(1);\n System.out.println(\"Added 1 to the queue\");\n queue.put(2);\n System.out.println(\"Added 2 to the queue\");\n queue.put(3); // This will block until space is available\n System.out.println(\"Added 3 to the queue\");\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }).start();\n\n // Consumer thread\n new Thread(() -> {\n try {\n Thread.sleep(1000); // Simulate some delay\n System.out.println(\"Removed from queue: \" + queue.take());\n System.out.println(\"Removed from queue: \" + queue.take());\n System.out.println(\"Removed from queue: \" + queue.take());\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n }).start();\n }\n}\nBlockingQueue is commonly used for task queues in thread pools (ThreadPoolExecutor).Types of BlockingQueues
ArrayBlockingQueue: A fixed-size queue.LinkedBlockingQueue: A potentially unbounded queue.PriorityBlockingQueue: A priority-based queue.run() method) Yes (call() method) Use Case Simple background tasks Tasks that need to return a value or throw an exception"},{"location":"langdives/Java/ThreadPools/#how-these-work-together","title":"How These Work Together","text":"Using Runnable in a Thread PoolExecutorService executor = Executors.newFixedThreadPool(2);\nRunnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\nexecutor.execute(task);\nexecutor.shutdown();\nExecutorService executor = Executors.newFixedThreadPool(2);\nCallable<Integer> task = () -> 42;\nFuture<Integer> future = executor.submit(task);\nSystem.out.println(\"Result: \" + future.get());\nexecutor.shutdown();\nBlockingQueue<Runnable> queue = new LinkedBlockingQueue<>(2);\nThreadPoolExecutor executor = new ThreadPoolExecutor(2, 4, 30, TimeUnit.SECONDS, queue);\nRunnable task = () -> System.out.println(\"Task executed by \" + Thread.currentThread().getName());\nexecutor.execute(task);\nexecutor.shutdown();\nAtomicity is a fundamental concept in multithreading and concurrency that ensures operations are executed entirely or not at all, with no intermediate states visible to other threads. In Java, atomicity plays a crucial role in maintaining data consistency in concurrent environments.
This Article covers everything about atomic operations, issues with atomicity, atomic classes in Java, and best practices to ensure atomic behavior in your code.
"},{"location":"langdives/Java/Threads-Atomicity/#what-is-atomicity","title":"What is Atomicity ?","text":"In a multithreaded program, atomicity guarantees that operations are executed as a single, indivisible unit. When an operation is atomic, it ensures that:
Without atomic operations, multiple threads could interfere with each other, leading to race conditions and data inconsistencies. For example, if two threads try to increment a shared counter simultaneously, the result may not reflect both increments due to interleaving of operations.
"},{"location":"langdives/Java/Threads-Atomicity/#problems","title":"Problems ?","text":"Non-Atomic Operations on Primitive Data Types Counter Increment Exampleclass Counter {\n private int count = 0;\n\n public void increment() {\n count++; // Not atomic\n }\n\n public int getCount() {\n return count;\n }\n}\nProblem
The statement count++ is not atomic. It consists of three operations
count.count.If two threads execute count++ simultaneously, one increment might be lost due to race conditions.
Java provides several ways to ensure atomicity, including:
synchronized blocks and methods.ReentrantLock.java.util.concurrent.atomic package (recommended for simple atomic operations).Atomic Classes","text":"The java.util.concurrent.atomic package offers classes that support lock-free, thread-safe operations on single variables. These classes rely on low-level atomic operations (like CAS \u2014 Compare-And-Swap) provided by the underlying hardware.
AtomicInteger \u2013 Atomic operations on integers.AtomicLong \u2013 Atomic operations on long values.AtomicBoolean \u2013 Atomic operations on boolean values.AtomicReference<V> \u2013 Atomic operations on reference variables.AtomicStampedReference<V> \u2013 Supports versioned references to prevent ABA problems.AtomicInteger","text":"Example: Solving the Increment Problem import java.util.concurrent.atomic.AtomicInteger;\n\nclass AtomicCounter {\n private final AtomicInteger count = new AtomicInteger(0);\n\n public void increment() {\n count.incrementAndGet(); // Atomic increment\n }\n\n public int getCount() {\n return count.get();\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n AtomicCounter counter = new AtomicCounter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) {\n counter.increment();\n }\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) {\n counter.increment();\n }\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final Count: \" + counter.getCount()); // Output: 2000\n }\n}\nincrementAndGet() method ensures atomicity without using locks.synchronized or ReentrantLock.AtomicBoolean","text":"Example: Managing Flags Safely import java.util.concurrent.atomic.AtomicBoolean;\n\nclass FlagManager {\n private final AtomicBoolean isActive = new AtomicBoolean(false);\n\n public void activate() {\n if (isActive.compareAndSet(false, true)) {\n System.out.println(\"Flag activated.\");\n } else {\n System.out.println(\"Flag already active.\");\n }\n }\n\n public void deactivate() {\n if (isActive.compareAndSet(true, false)) {\n System.out.println(\"Flag deactivated.\");\n } else {\n System.out.println(\"Flag already inactive.\");\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n FlagManager manager = new FlagManager();\n\n Thread t1 = new Thread(manager::activate);\n Thread t2 = new Thread(manager::activate);\n\n t1.start();\n t2.start();\n }\n}\ncompareAndSet() changes the flag only if it matches the expected value, ensuring thread safety.
AtomicReference","text":"Example: Atomic Operations on Objects import java.util.concurrent.atomic.AtomicReference;\n\nclass Person {\n String name;\n\n Person(String name) {\n this.name = name;\n }\n}\n\npublic class AtomicReferenceExample {\n public static void main(String[] args) {\n AtomicReference<Person> personRef = new AtomicReference<>(new Person(\"Alice\"));\n\n // Atomic update of the reference\n personRef.set(new Person(\"Bob\"));\n System.out.println(\"Updated Person: \" + personRef.get().name);\n }\n}\nWhen to Use ?
Use AtomicReference when you need atomic operations on object references.
AtomicStampedReference","text":"The ABA problem occurs when a value changes from A to B and then back to A. AtomicStampedReference solves this by associating a version (stamp) with the value.
import java.util.concurrent.atomic.AtomicStampedReference;\n\npublic class AtomicStampedReferenceExample {\n public static void main(String[] args) {\n AtomicStampedReference<Integer> ref = new AtomicStampedReference<>(1, 0);\n\n int[] stamp = new int[1];\n Integer value = ref.get(stamp);\n System.out.println(\"Initial Value: \" + value + \", Stamp: \" + stamp[0]);\n\n boolean success = ref.compareAndSet(1, 2, stamp[0], stamp[0] + 1);\n System.out.println(\"CAS Success: \" + success + \", New Value: \" + ref.get(stamp) + \", New Stamp: \" + stamp[0]);\n }\n}\nAtomicStampedReference ensures that the same value change does not go undetected by tracking the version.
Use AtomicInteger or AtomicBoolean instead of synchronized methods like simple counters or flags.
Use AtomicReference for lock-free algorithms (Non blocking Algos).
Use AtomicStampedReference for versioned updates to prevent ABA problems.
Atomic classes only work with single variables. For multiple variables, use synchronized or ReentrantLock.
For complex operations involving multiple state changes, atomic classes are insufficient.
Atomic operations avoid deadlocks but can still suffer from livelocks (threads continuously retrying without progress).
synchronized or ReentrantLock.The atomic classes in Java\u2019s java.util.concurrent.atomic package offer lock-free, thread-safe operations that are ideal for simple state management. By ensuring atomicity, these classes help avoid race conditions and improve the performance and scalability of multithreaded applications. However, they are best suited for single-variable updates for more complex operations, locks or transactional mechanisms may still be necessary.
Java offers multithreading to perform multiple tasks concurrently, improving performance and responsiveness. This deep dive covers every key concept of Java threading with detailed explanations and code examples.
Before that let's have a quick rewind of fundamental concept of concurrency and parallelism.
"},{"location":"langdives/Java/Threads/#concurrency-and-parallelism","title":"Concurrency and Parallelism","text":"Concurrency: Multiple tasks start, run, and complete in overlapping time periods (not necessarily simultaneously).
Parallelism: Multiple tasks run exactly at the same time (requires multi-core processors).
We have another article where we gone through fundamentals of concurrency and parallelism in depth though we cover some of the stuff here to but its recommeneded to go through this artice Concurrency and Parallelism
Java achieves both using threads, thread pools, and various libraries such as Executors, Fork/Join Framework, and Streams API, We will go through them one by one and in this article we mostly cover Threads.
"},{"location":"langdives/Java/Threads/#what-is-a-thread","title":"What is a Thread?","text":"A thread is a lightweight sub-process. A Java program has at least one thread \u2014 the main thread, which starts with the main() method. You can create additional threads to execute code concurrently. Each thread shares the same process memory, but has its own stack, registers, and program counter.
You can create a thread in two ways:
Extending the Thread classclass MyThread extends Thread {\n public void run() {\n System.out.println(\"Thread running: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n MyThread t1 = new MyThread();\n t1.start(); // Start the thread\n }\n}\nclass MyRunnable implements Runnable {\n public void run() {\n System.out.println(\"Runnable running: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n Thread t1 = new Thread(new MyRunnable());\n t1.start(); // Start the thread\n }\n}\nWhen to Use ?
Runnable: When you need to inherit from another class, use Runnable since Java does not support multiple inheritance.Thread: If your class does not need to extend any other class, extending Thread might be more intuitive.A thread in Java goes through the following states:
Thread Lifecycle
class MyThread extends Thread {\n public void run() {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName());\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n MyThread t1 = new MyThread(); // NEW\n t1.start(); // RUNNABLE\n\n // Join to wait for the thread to complete\n t1.join(); // Terminated once finished\n System.out.println(\"Thread has terminated.\");\n }\n}\nNote
wait() or waits indefinitely.sleep() or join(time) is invoked, meaning the thread will resume after a specific time.A daemon thread is a background thread that provides support services, like the garbage collector. It does not prevent the JVM from shutting down once all user threads are completed.
Example of Daemon Threadclass DaemonThread extends Thread {\n public void run() {\n if (Thread.currentThread().isDaemon()) {\n System.out.println(\"This is a daemon thread.\");\n } else {\n System.out.println(\"This is a user thread.\");\n }\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n DaemonThread t1 = new DaemonThread();\n t1.setDaemon(true); // Set as daemon thread\n t1.start();\n\n DaemonThread t2 = new DaemonThread();\n t2.start();\n }\n}\nWhen to use Daemon Threads ?
For background tasks like logging, garbage collection, or monitoring services.
"},{"location":"langdives/Java/Threads/#thread-priority","title":"Thread Priority","text":"Java assigns a priority to each thread, ranging from 1 (MIN_PRIORITY) to 10 (MAX_PRIORITY). The default priority is 5 (NORM_PRIORITY). Thread priority affects scheduling, but it\u2019s platform-dependent \u2014 meaning it doesn\u2019t guarantee execution order.
Setting Thread Priority Exampleclass PriorityThread extends Thread {\n public void run() {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName() +\n \" with priority: \" + Thread.currentThread().getPriority());\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n PriorityThread t1 = new PriorityThread();\n PriorityThread t2 = new PriorityThread();\n\n t1.setPriority(Thread.MIN_PRIORITY); // Priority 1\n t2.setPriority(Thread.MAX_PRIORITY); // Priority 10\n\n t1.start();\n t2.start();\n }\n}\nWhen to use Priority Threads
Only when certain tasks should have preferential scheduling over others. However, Java thread scheduling is not guaranteed, so don't rely solely on priority.
"},{"location":"langdives/Java/Threads/#thread-synchronization","title":"Thread Synchronization","text":"When multiple threads access shared resources (like variables), synchronization ensures that only one thread modifies the resource at a time. Use the synchronized keyword to prevent race conditions.
class Counter {\n private int count = 0;\n\n public synchronized void increment() {\n count++; // Only one thread can increment at a time\n }\n\n public int getCount() {\n return count;\n }\n}\n\npublic class Main {\n public static void main(String[] args) throws InterruptedException {\n Counter counter = new Counter();\n\n Thread t1 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n Thread t2 = new Thread(() -> {\n for (int i = 0; i < 1000; i++) counter.increment();\n });\n\n t1.start();\n t2.start();\n t1.join();\n t2.join();\n\n System.out.println(\"Final count: \" + counter.getCount());\n }\n}\nWhen to use Synchronization ?
When multiple threads access critical sections of code to avoid inconsistent data.
"},{"location":"langdives/Java/Threads/#inter-thread-communication","title":"Inter-thread Communication","text":"Java allows threads to communicate using wait-notify methods, avoiding busy waiting.
wait(): Makes the thread wait until another thread notifies it.notify(): Wakes up a single waiting thread.notifyAll(): Wakes up all waiting threads.class SharedResource {\n private int value;\n private boolean available = false;\n\n public synchronized void produce(int val) throws InterruptedException {\n while (available) {\n wait(); // Wait if value is already available\n }\n value = val;\n available = true;\n System.out.println(\"Produced: \" + value);\n notify(); // Notify the consumer thread\n }\n\n public synchronized void consume() throws InterruptedException {\n while (!available) {\n wait(); // Wait if value is not available\n }\n System.out.println(\"Consumed: \" + value);\n available = false;\n notify(); // Notify the producer thread\n }\n}\n\npublic class Main {\n public static void main(String[] args) {\n SharedResource resource = new SharedResource();\n\n Thread producer = new Thread(() -> {\n try {\n for (int i = 1; i <= 5; i++) resource.produce(i);\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n Thread consumer = new Thread(() -> {\n try {\n for (int i = 1; i <= 5; i++) resource.consume();\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n }\n });\n\n producer.start();\n consumer.start();\n }\n}\nThreadLocal provides a way to create thread-isolated variables. Each thread gets its own copy of the variable, and changes made by one thread do not affect others. This is useful when you don\u2019t want threads to share a common state.
public class ThreadLocalExample {\n private static ThreadLocal<Integer> threadLocal = ThreadLocal.withInitial(() -> 1);\n\n public static void main(String[] args) {\n Thread t1 = new Thread(() -> {\n threadLocal.set(100);\n System.out.println(\"Thread 1: \" + threadLocal.get());\n });\n\n Thread t2 = new Thread(() -> {\n threadLocal.set(200);\n System.out.println(\"Thread 2: \" + threadLocal.get());\n });\n\n t1.start();\n t2.start();\n }\n}\nWhen to use ?
Useful in multi-threaded environments (like database transactions) where each thread needs its own context without interference from other threads.
"},{"location":"langdives/Java/Threads/#volatile-variables","title":"Volatile Variables","text":"The volatile keyword ensures visibility of changes to variables across threads. Without volatile, thread-local caches may not reflect the latest changes made by other threads.
public class VolatileExample {\n private static volatile boolean running = true;\n\n public static void main(String[] args) {\n Thread t = new Thread(() -> {\n while (running) {\n // Busy-wait\n }\n System.out.println(\"Thread stopped.\");\n });\n\n t.start();\n\n try { Thread.sleep(1000); } catch (InterruptedException e) { }\n running = false; // Change will be visible to other threads\n }\n}\nWhen to Use Volatile
Use volatile for variables accessed by multiple threads without needing synchronization (e.g., flags).
"},{"location":"langdives/Java/Threads/#when-to-use-volatile","title":"When to Usevolatile ?","text":"volatile is Necessary class VolatileExample {\n private volatile boolean running = true;\n\n public void stop() {\n running = false; // Change becomes immediately visible to other threads\n }\n\n public void run() {\n while (running) {\n // Do something\n }\n System.out.println(\"Thread stopped.\");\n }\n\n public static void main(String[] args) throws InterruptedException {\n VolatileExample example = new VolatileExample();\n\n Thread t = new Thread(example::run);\n t.start();\n\n Thread.sleep(1000);\n example.stop(); // Stop the thread\n }\n}\nHere, volatile ensures that the change to running made by the stop() method is immediately visible to the thread executing run(). Without volatile, the run() thread might never see the change and keep running indefinitely.
volatile ?","text":"count++).Atomic classes).class Counter {\n private volatile int count = 0;\n\n public void increment() {\n count++; // Not atomic! Two threads can still read the same value.\n }\n\n public int getCount() {\n return count;\n }\n}\nIssue
Even though count is marked volatile, count++ is not atomic. Two threads could read the same value and increment it, leading to lost updates. To fix this, use synchronized or AtomicInteger.
synchronized or volatile ?","text":"If you don\u2019t use volatile or synchronized, some dangerous scenarios can occur. Like this:
class SharedResource {\n private boolean available = false;\n\n public void produce() {\n available = true; // Change not guaranteed to be visible immediately\n }\n\n public void consume() {\n while (!available) {\n // Busy-waiting, might never see the change to `available`\n }\n System.out.println(\"Consumed!\");\n }\n}\nProblem
If available is not marked volatile, the change made by produce() might not be visible to the consume() thread immediately. The consumer thread might be stuck in an infinite loop because it doesn't see the latest value of available.
Note
volatile or synchronized is used.Synchronized over volatile ?","text":"Let's go through an example where its okay to use just synchroinized instead of volatile
Examplepublic synchronized void produce(int val) throws InterruptedException {\n while (available) {\n wait(); // Wait if value is already available\n }\n value = val;\n available = true;\n System.out.println(\"Produced: \" + value);\n notify(); // Notify the consumer thread\n}\nSynchronized Keyword:
synchronized block or method (like produce() or consume()), it acquires a lock on the SharedResource object. This ensures mutual exclusion \u2014 only one thread can access the critical section at a time.synchronized block. This ensures that the latest value of available is visible to all other threads.Wait-Notify Mechanism:
wait() and notify() methods release and acquire the lock, enforcing happens-before relationships (meaning operations before notify() or wait() are visible to other threads).available = true in produce(), another thread waiting in consume() will see the updated value when it wakes up.Because this code uses synchronized methods and wait-notify, the necessary memory visibility is achieved without needing volatile.
volatile synchronized Visibility Ensures visibility of changes. Ensures visibility and atomicity. Atomicity Not guaranteed. Guaranteed (only one thread at a time). Performance Faster (no locking). Slower (locking involved). Use Case For flags, simple state updates. For complex operations, critical sections. Overhead Low (no blocking). High (involves blocking and context switches)."},{"location":"langdives/Java/Threads/#thread-memory","title":"Thread Memory","text":"The memory consumption per thread and the maximum number of threads in Java depend on several factors, such as:
Each Java thread consumes two key areas of memory:
Thread Stack Memory: Each thread gets its own stack, which holds Local variables (primitives, references), Method call frames, Intermediate results during method execution.
Note
The default stack size depends on the JVM and platform:
You can change the stack size with the -Xss JVM option:
java -Xss512k YourProgram\nNative Thread Metadata: In addition to stack memory, the OS kernel allocates metadata per thread (for thread control structures). This varies by platform but is typically in the range of 8 KB to 16 KB per thread.
"},{"location":"langdives/Java/Threads/#memory-per-thread","title":"Memory per Thread ?","text":"The typical memory consumption per thread:
Thus, a single thread could use ~1 MB to 1.1 MB of memory.
"},{"location":"langdives/Java/Threads/#max-threads-you-can-create","title":"Max Threads you Can Create ?","text":"The number of threads you can create depends on:
-Xmx).-Xss).Let's say:
-Xmx2g).-Xss1m).Maximum threads = 6 GB / 1 MB per thread = ~6000 threads.
OS Limits on Threads
Even if memory allows for thousands of threads, the OS imposes limits:
/etc/security/limits.conf).OutOfMemoryError: unable to create new native thread Occurs when the OS can't allocate more native threads.
Performance Degradation as context switching between many threads becomes expensive, leading to CPU thrashing.
Rather than creating many threads manually, use thread pools to manage a fixed number of threads efficiently:
Thread Pools Exampleimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\n\npublic class ThreadPoolExample {\n public static void main(String[] args) {\n ExecutorService executor = Executors.newFixedThreadPool(10);\n for (int i = 0; i < 100; i++) {\n executor.submit(() -> {\n System.out.println(\"Running thread: \" + Thread.currentThread().getName());\n });\n }\n executor.shutdown();\n }\n}\nThread pools reuse threads, reducing memory usage and improving performance.
"},{"location":"langdives/Java/Threads/#how-to-increase-max-thread","title":"How to Increase Max Thread ?","text":"On Linux, you can increase the maximum threads per process
Check current limitsulimit -u # Max user processes\nulimit -u 65535\nyour_user_name hard nproc 65535\nyour_user_name soft nproc 65535\nKey points
Spring is a popular, open-source Java-based framework used to create enterprise-level applications. It provides a comprehensive programming and configuration model that simplifies Java development. At its core, Spring focuses on dependency injection and inversion of control (IoC), providing an abstraction over Java's complexity.
"},{"location":"langdives/Java/Spring/#core-goals-of-spring","title":"Core goals of Spring","text":"The Spring ecosystem consists of various projects for different use cases:
A comprehensive list of Spring Boot annotations, covering core Spring Boot, configuration, web, data, testing, and more. I'll organize them by categories with keys (annotation names) and values (purpose/use cases) for easy reference.
"},{"location":"langdives/Java/Spring/SpringAnnotations/#core-annotations","title":"Core Annotations","text":"Annotation Purpose/Use Case@SpringBootApplication Main entry point for a Spring Boot application. Combines @Configuration, @ComponentScan, and @EnableAutoConfiguration. @EnableAutoConfiguration Enables automatic configuration of Spring beans based on the classpath and defined properties. @ComponentScan Scans the package and its sub-packages for Spring components (e.g., @Component, @Service). @Configuration Marks a class as a source of bean definitions. Used to define Spring beans programmatically. @Bean Declares a method as a Spring bean, registered in the application context. @Import Imports additional configuration classes. @ImportResource Loads bean definitions from external XML configuration files."},{"location":"langdives/Java/Spring/SpringAnnotations/#web-and-rest-annotations","title":"Web and REST Annotations","text":"Annotation Purpose/Use Case @RestController Marks a class as a REST API controller. Combines @Controller and @ResponseBody. @Controller Marks a class as a web controller. Works with view templates (like Thymeleaf). @RequestMapping Maps HTTP requests to specific handler methods or classes. Can be used on classes or methods. @GetMapping Maps HTTP GET requests to specific handler methods. @PostMapping Maps HTTP POST requests to specific handler methods. @PutMapping Maps HTTP PUT requests to specific handler methods. @DeleteMapping Maps HTTP DELETE requests to specific handler methods. @PatchMapping Maps HTTP PATCH requests to specific handler methods. @RequestBody Binds the HTTP request body to a Java object. Used in REST controllers. @ResponseBody Binds the return value of a method directly to the HTTP response body. @RequestParam Binds HTTP query parameters to method arguments. @PathVariable Binds URI template variables to method parameters. @RequestHeader Binds HTTP request headers to method parameters. @CookieValue Binds cookie values to method parameters. @ModelAttribute Binds form data to a model object. @SessionAttributes Declares session-scoped model attributes. @CrossOrigin Enables cross-origin requests (CORS) for specific endpoints."},{"location":"langdives/Java/Spring/SpringAnnotations/#jpa-jdbc-annotations","title":"JPA, JDBC Annotations","text":"Annotation Purpose/Use Case @Entity Marks a class as a JPA entity. @Table Specifies the database table for a JPA entity. @Id Marks a field as the primary key of a JPA entity. @GeneratedValue Specifies how the primary key value should be generated. @Column Specifies the mapping of a field to a database column. @OneToOne Establishes a one-to-one relationship between entities. @OneToMany Establishes a one-to-many relationship between entities. @ManyToOne Establishes a many-to-one relationship between entities. @ManyToMany Establishes a many-to-many relationship between entities. @JoinColumn Specifies the foreign key column for a relationship. @Query Defines a custom JPQL or SQL query on a repository method. @Transactional Marks a method or class as transactional. Ensures ACID properties in data operations. @EnableJpaRepositories Enables JPA repositories for data access. @Repository Marks a class as a data repository. @EnableTransactionManagement Enables declarative transaction management."},{"location":"langdives/Java/Spring/SpringAnnotations/#security-annotations","title":"Security Annotations","text":"Annotation Purpose/Use Case @EnableWebSecurity Enables Spring Security for web applications. @EnableGlobalMethodSecurity Enables method-level security annotations like @PreAuthorize and @PostAuthorize. @PreAuthorize Applies authorization logic before a method is invoked. @PostAuthorize Applies authorization logic after a method has executed. @Secured Secures a method by roles (deprecated in favor of @PreAuthorize). @RolesAllowed Specifies which roles are allowed to access a method. @WithMockUser Simulates a user for testing security."},{"location":"langdives/Java/Spring/SpringAnnotations/#testing-annotations","title":"Testing Annotations","text":"Annotation Purpose/Use Case @SpringBootTest Runs integration tests for a Spring Boot application. Loads the full application context. @WebMvcTest Tests only web layer components (e.g., controllers). @DataJpaTest Tests only JPA repositories. Configures an in-memory database. @MockBean Replaces a bean with a mock during tests. @SpyBean Replaces a bean with a spy during tests. @TestConfiguration Provides additional bean configurations for tests. @BeforeEach Runs before each test method in a test class. @AfterEach Runs after each test method in a test class."},{"location":"langdives/Java/Spring/SpringAnnotations/#profiles-annotations","title":"Profiles Annotations","text":"Annotation Purpose/Use Case @ConfigurationProperties Binds external configuration properties to a Java bean. @EnableConfigurationProperties Enables support for @ConfigurationProperties beans. @Profile Specifies the profile under which a bean is active (e.g., dev, prod). @Value Injects a value from the properties or environment. @PropertySource Loads properties from an external file. @Environment Provides access to the current environment settings."},{"location":"langdives/Java/Spring/SpringAnnotations/#actuator-metrics-annotations","title":"Actuator & Metrics Annotations","text":"Annotation Purpose/Use Case @Endpoint Defines a custom Actuator endpoint. @ReadOperation Marks a method as a read operation for an Actuator endpoint. @WriteOperation Marks a method as a write operation for an Actuator endpoint. @DeleteOperation Marks a method as a delete operation for an Actuator endpoint. @Timed Measures the execution time of a method. @Gauge Exposes a gauge metric to Actuator. @Metered Marks a method to be counted as a metric (deprecated in favor of @Timed)."},{"location":"langdives/Java/Spring/SpringAnnotations/#microservices-annotations","title":"Microservices Annotations","text":"Annotation Purpose/Use Case @EnableDiscoveryClient Enables service registration with Eureka, Consul, or Zookeeper. @EnableFeignClients Enables Feign clients for inter-service communication. @CircuitBreaker Implements circuit-breaking logic using Resilience4j. @Retryable Enables retry logic for a method. @LoadBalanced Enables load balancing for REST clients."},{"location":"langdives/Java/Spring/SpringAnnotations/#miscellaneous-annotations","title":"Miscellaneous Annotations","text":"Annotation Purpose/Use Case @Conditional Conditionally registers a bean based on custom logic. @Async Marks a method to run asynchronously. @Scheduled Schedules a method to run at fixed intervals or cron expressions. @EventListener Marks a method to listen for application events. @Cacheable Caches the result of a method. @CacheEvict Evicts entries from a cache."},{"location":"langdives/Java/Spring/SpringAnnotations/#summary","title":"Summary","text":"This is a comprehensive list of all major Spring Boot annotations, categorized by their functionality. With these annotations, Spring Boot makes it easier to develop applications by reducing boilerplate code, automating configuration, and offering powerful tools for testing, security, and microservices development.
"},{"location":"langdives/Java/Spring/SpringBoot/","title":"Spring Boot","text":"This article covers how Spring Boot automates configurations, deals with microservices, and manages monitoring, security, and performance.
"},{"location":"langdives/Java/Spring/SpringBoot/#what-is-spring-boot","title":"What is Spring Boot ?","text":"Spring Boot is an extension of the Spring Framework that simplifies the development of Java applications by offering:
The goal of Spring Boot is to help developers build stand-alone, production-grade applications quickly and with less fuss.
"},{"location":"langdives/Java/Spring/SpringBoot/#application-architecture","title":"Application Architecture","text":"A Spring Boot application consists of
spring-boot-starter-web for web apps).application.properties or application.yml.@SpringBootApplication: Combines three core annotations
@Configuration: Marks the class as a source of bean definitions.@EnableAutoConfiguration: Enables auto-configuration based on classpath contents.@ComponentScan: Scans for Spring components (beans, services, controllers).@RestController: Marks a class as a controller with REST endpoints.
@Autowired: Injects dependencies automatically by type.
@Value: Injects values from properties files.
@Profile: Activates beans only under specific profiles (e.g., dev, prod).
Starters are pre-configured dependency bundles for common functionalities
spring-boot-starter-web: For web applications.spring-boot-starter-data-jpa: For working with databases using JPA.spring-boot-starter-security: For authentication and authorization.spring-boot-starter-actuator: For monitoring and management.Spring Boot uses @EnableAutoConfiguration to detect dependencies and automatically configure beans for you.
For example: If spring-boot-starter-data-jpa is present, it will
DataSource.EntityManagerFactory to manage JPA entities.@EnableTransactionManagement.How to Debug Auto-Configuration:
Add --debug to the application start command to list all auto-configurations.
$ java -jar myapp.jar --debug\nIf you need to exclude an auto-configuration, use:
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})\nStartup: Spring Boot applications initialize with SpringApplication.run(), The lifecycle involves loading beans, initializing contexts, and wiring dependencies.
Embedded Server: By default, Spring Boot uses Tomcat as the embedded server. Others include Jetty and Undertow, The server listens on a configurable port (default: 8080).
Shutdown: Spring Boot provides graceful shutdown using @PreDestroy or hooks via SpringApplication.addShutdownHook().
application.properties","text":"Spring Boot applications are configured using either application.properties
application.properties server.port=8081\nspring.datasource.url=jdbc:mysql://localhost:3306/mydb\nspring.datasource.username=root\nspring.datasource.password=password\napplication.yml","text":"Using Profiles (e.g., Dev vs. Prod) in application.yml
application.yml server:\n port: 8080\nspring:\n profiles:\n active: dev\n\n---\nspring:\n profiles: dev\n datasource:\n url: jdbc:h2:mem:testdb\n\n---\nspring:\n profiles: prod\n datasource:\n url: jdbc:mysql://localhost:3306/proddb\nYou can activate profiles programmatically or through command-line options
$ java -Dspring.profiles.active=prod -jar myapp.jar\nYou can define your own custom properties and inject them into beans using @ConfigurationProperties.
@ConfigurationProperties(prefix = \"custom\")\npublic class CustomConfig {\n private String name;\n private int timeout;\n\n // Getters and Setters\n}\ncustom.name=SpringApp\ncustom.timeout=5000\n@Autowired\nprivate CustomConfig customConfig;\nYou can customize the embedded server by configuring the EmbeddedServletContainerFactory.
@Bean\npublic ConfigurableServletWebServerFactory webServerFactory() {\n TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();\n factory.setPort(9090);\n factory.addConnectorCustomizers(connector -> {\n connector.setAttribute(\"maxThreads\", 200);\n });\n return factory;\n}\nExposes application management and monitoring endpoints.
Some common endpoints:
/actuator/health: Health status./actuator/info: Application information./actuator/metrics: Metrics data (e.g., memory usage, request counts).management:\n endpoints:\n web:\n exposure:\n include: health, info\n security:\n enabled: true\nYou can customize or add your own metrics by using @Timed or MeterRegistry.
Basic Authentication Setup: Add spring-boot-starter-security dependency, by default, Spring Boot enables basic authentication with a generated password.
Custom Authentication
@Configuration\npublic class SecurityConfig extends WebSecurityConfigurerAdapter {\n @Override\n protected void configure(HttpSecurity http) throws Exception {\n http\n .authorizeRequests()\n .antMatchers(\"/public/\").permitAll()\n .anyRequest().authenticated()\n .and()\n .httpBasic();\n }\n}\nJWT Authentication: Use JSON Web Tokens (JWT) for token-based authentication in microservices architectures.
Unit Testing: Use @SpringBootTest for integration testing, Mock beans with @MockBean.
@SpringBootTest\npublic class MyServiceTest {\n @MockBean\n private MyRepository repository;\n\n @Autowired\n private MyService service;\n\n @Test\n public void testService() {\n when(repository.findById(1L)).thenReturn(Optional.of(new MyEntity()));\n assertTrue(service.findById(1L).isPresent());\n }\n}\nTest Containers: Use Testcontainers for running databases or services in Docker containers during tests.
Spring Boot streamlines the development process by providing auto-configuration, embedded servers, and a production-ready environment. It empowers developers to build and deploy microservices quickly, backed by powerful features like Spring Security, Spring Data, Actuator, and more. With its opinionated defaults and deep customizability, Spring Boot strikes a balance between simplicity and flexibility making it ideal for both beginners and advanced developers.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/","title":"Spring Core Framework","text":"the foundation of the entire Spring ecosystem. We'll explore each component and mechanism in detail, so by the end, you\u2019ll have a thorough understanding of how Spring Core works, including the IoC container, Dependency Injection (DI), Beans, ApplicationContext, Bean Lifecycle, AOP, and more.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#what-is-spring-core-framework","title":"What is Spring Core Framework","text":"The Spring Core Framework is the heart of the Spring ecosystem. It provides the essential features required to build Java applications, with a focus on dependency injection (DI) and inversion of control (IoC). At its core, Spring aims to eliminate the complexities of creating objects, managing dependencies, and wiring different components together.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#modules","title":"Modules","text":""},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-core","title":"Spring Core","text":"The foundational module that provides the IoC container and the basic tools for dependency injection (DI), It includes the core interfaces and classes like BeanFactory, ApplicationContext, BeanPostProcessor, BeanDefinition, and others.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-beans","title":"Spring Beans","text":"Manages the configuration, creation, and lifecycle of Spring beans.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-context","title":"Spring Context","text":"Provides a runtime environment for applications using the IoC container. It builds on the Spring Core and adds additional functionality like events and internationalization (i18n).
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#spring-spel","title":"Spring SpEL","text":"SpEL(Spring Expression Language) A powerful expression language that can be used to dynamically query or manipulate bean properties.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#core-concepts","title":"Core Concepts","text":""},{"location":"langdives/Java/Spring/SpringCoreFramework/#inversion-of-control-ioc","title":"Inversion of Control (IoC)","text":"The Inversion of Control (IoC) principle is at the core of the Spring Framework. It shifts the control of object creation and management from the developer to the IoC container, promoting loose coupling and enhancing testability. Let\u2019s break it down conceptually and then dive into the Spring implementation.
In traditional programming, the application code creates and manages its dependencies directly.
Examplepublic class OrderService {\n private PaymentService paymentService;\n\n public OrderService() {\n this.paymentService = new PaymentService(); // Tight coupling\n }\n}\nExplanation
OrderService class directly instantiates PaymentService, meaning these two classes are tightly coupled. If the PaymentService class changes, you must modify the OrderService code as well.OrderService always uses a real PaymentService instead of a mock service.IoC Solution With Inversion of Control (IoC), the responsibility of creating the PaymentService is \"inverted\" and delegated to the Spring IoC container. Now, the IoC container injects the dependency into OrderService.
@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService; // Dependency injection\n }\n}\nExplanation
OrderService no longer creates the PaymentService itself.PaymentService and provides it to OrderService.BeanFactory is the basic IoC container in Spring. It provides basic dependency injection and bean management functionality.
Features of BeanFactory:
BeanFactory factory = new XmlBeanFactory(new FileSystemResource(\"beans.xml\"));\nOrderService service = (OrderService) factory.getBean(\"orderService\");\nHowever, BeanFactory is rarely used now because it lacks advanced features like event propagation, internationalization, and eager initialization, which are provided by ApplicationContext.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#applicationcontext","title":"ApplicationContext","text":"ApplicationContext is a more powerful IoC container that extends BeanFactory. It is widely used in modern Spring applications because of its rich features.
Features of ApplicationContext:
ContextRefreshedEvent).ApplicationContext context = new ClassPathXmlApplicationContext(\"beans.xml\");\nOrderService service = context.getBean(OrderService.class);\nIn most cases, developers use AnnotationConfigApplicationContext or Spring Boot to load configurations and manage beans.
Step-by-Step
Define Beans and Dependencies: Beans can be defined in XML, Java Configuration, or through Annotations like @Component.
<bean id=\"paymentService\" class=\"com.example.PaymentService\"/>\n<bean id=\"orderService\" class=\"com.example.OrderService\">\n <constructor-arg ref=\"paymentService\"/>\n</bean>\nSpring IoC Container Loads Configuration: The IoC container reads the configuration (XML, annotations, or Java-based) during startup.
Dependency Injection (DI): The IoC container identifies the dependencies and injects them using constructor, setter, or field injection.
Bean Initialization: The IoC container initializes all necessary beans (eagerly or lazily).
Bean Usage: The beans are now available for use by the application.
Dependency Injection (DI) is a pattern where objects are provided with their dependencies at runtime by the IoC container instead of creating them directly. Spring supports multiple types of dependency injection:
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#types-of-di","title":"Types of DI","text":"Constructor Injection: Dependencies are provided through the class constructor, Recommended for mandatory dependencies.
Constructor Injection Example@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\nSetter Injection: Dependencies are injected using setter methods, Useful for optional dependencies.
Setter Injection Example@Component\npublic class OrderService {\n private PaymentService paymentService;\n\n @Autowired\n public void setPaymentService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\nField Injection: Dependencies are injected directly into class fields, Not recommended since it makes unit testing harder.
Field Injection@Component\npublic class OrderService {\n @Autowired\n private PaymentService paymentService;\n}\nLoose Coupling: Objects don\u2019t manage their dependencies, making them easier to modify, extend, or replace.
Testability: Dependencies can be easily mocked during unit tests, enhancing test coverage.
Reusability: Components are easier to reuse across different parts of the application.
Configuration Management: Centralized configuration of beans ensures that all components behave consistently.
Configuration Overhead: Without frameworks like Spring, managing dependencies can lead to complex code. Spring solves this by centralizing configurations.
Circular Dependencies: Two beans dependent on each other can cause issues. Spring detects such circular dependencies and throws exceptions.
Managing Large Applications: With many components, configuration can get complicated. Spring Boot simplifies this with auto-configuration and starters.
Let\u2019s look at a complete example using Spring Core with constructor injection:
Full IoC Implementation Example Java Configuration Example@Configuration\npublic class AppConfig {\n\n @Bean\n public PaymentService paymentService() {\n return new PaymentService();\n }\n\n @Bean\n public OrderService orderService(PaymentService paymentService) {\n return new OrderService(paymentService);\n }\n}\n@Component\npublic class PaymentService {\n public void processPayment() {\n System.out.println(\"Payment processed.\");\n }\n}\n\n@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n\n public void placeOrder() {\n System.out.println(\"Order placed.\");\n paymentService.processPayment();\n }\n}\npublic class Main {\n public static void main(String[] args) {\n ApplicationContext context = new AnnotationConfigApplicationContext(AppConfig.class);\n OrderService orderService = context.getBean(OrderService.class);\n orderService.placeOrder();\n }\n}\nSpring beans are the building blocks of any Spring application. They represent the objects that the Spring IoC container manages throughout their lifecycle. Understanding beans and their lifecycle is critical for mastering the Spring Core framework. Let\u2019s explore everything about beans\u2014from creation, scopes, lifecycle, initialization, destruction, and more\u2014in detail.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#what-is-a-bean","title":"What is a Bean ?","text":"A bean in Spring is an object that is instantiated, assembled, and managed by the IoC container. The container creates, initializes, and wires these beans, ensuring that all dependencies are injected as needed. Beans are usually defined using:
@Component, @Service, @Repository, @Controller)Spring provides multiple ways to declare beans and register them with the IoC container:
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#xml-based-configuration","title":"XML-based Configuration","text":"Traditional way of defining beans using XML files.
XML Config Example<beans xmlns=\"http://www.springframework.org/schema/beans\" \n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n xsi:schemaLocation=\"http://www.springframework.org/schema/beans \n http://www.springframework.org/schema/beans/spring-beans.xsd\">\n\n <bean id=\"paymentService\" class=\"com.example.PaymentService\"/>\n <bean id=\"orderService\" class=\"com.example.OrderService\">\n <constructor-arg ref=\"paymentService\"/>\n </bean>\n</beans>\nSpring allows you to use Java classes to define beans. This is cleaner and avoids XML boilerplate.
Java Config Example@Configuration\npublic class AppConfig {\n\n @Bean\n public PaymentService paymentService() {\n return new PaymentService();\n }\n\n @Bean\n public OrderService orderService() {\n return new OrderService(paymentService());\n }\n}\nYou can annotate classes with @Component, @Service, @Repository, or @Controller. Spring automatically detects these beans if @ComponentScan is enabled.
Annotations Example@Component\npublic class PaymentService { }\n\n@Component\npublic class OrderService {\n private final PaymentService paymentService;\n\n @Autowired\n public OrderService(PaymentService paymentService) {\n this.paymentService = paymentService;\n }\n}\n@Configuration\n@ComponentScan(basePackages = \"com.example\")\npublic class AppConfig { }\nThe scope of a bean defines the lifecycle and visibility of that bean within the Spring IoC container.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#types-of-bean-scopes","title":"Types of Bean Scopes","text":"singleton (default): A single instance of the bean is created and shared across the entire application, Used for stateless beans.
Example@Scope(\"singleton\")\n@Component\npublic class SingletonBean { }\nprototype: A new instance is created every time the bean is requested, Useful for stateful objects or temporary tasks.
Example@Scope(\"prototype\")\n@Component\npublic class PrototypeBean { }\nrequest: A new bean instance is created for each HTTP request, Used in web applications.
session: A single instance is created per HTTP session.
globalSession: A global session scope for applications using portlets.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#bean-lifecycle","title":"Bean Lifecycle","text":"Each Spring bean goes through several lifecycle phases, starting from instantiation to destruction. The Spring IoC container manages this lifecycle internally.
Bean Lifecycle Phases
Bean Instantiation: The IoC container instantiates the bean using the constructor or factory method.
Dependency Injection (DI): The container injects dependencies (via constructor, setter, or field injection).
Custom Initialization: If the bean implements InitializingBean or uses the @PostConstruct annotation, the initialization logic is executed here.
Bean Ready to Use: After initialization, the bean is ready for use by the application.
Custom Destruction: When the IoC container is shutting down, beans with @PreDestroy or DisposableBean are given a chance to release resources.
InitializingBean Interface: If a bean implements the InitializingBean interface, it must override the afterPropertiesSet() method, which is called after all properties are set.
public class MyService implements InitializingBean {\n @Override\n public void afterPropertiesSet() {\n System.out.println(\"MyService is initialized.\");\n }\n}\nDisposableBean Interface: If a bean implements the DisposableBean interface, it must override the destroy() method, which is called during the destruction phase.
public class MyService implements DisposableBean {\n @Override\n public void destroy() {\n System.out.println(\"MyService is being destroyed.\");\n }\n}\nUsing @PostConstruct and @PreDestroy: These annotations are the recommended way to manage initialization and destruction callbacks.
@Component\npublic class MyService {\n\n @PostConstruct\n public void init() {\n System.out.println(\"Initialization logic in @PostConstruct.\");\n }\n\n @PreDestroy\n public void cleanup() {\n System.out.println(\"Cleanup logic in @PreDestroy.\");\n }\n}\nCustom Initialization and Destruction Methods: You can also specify custom methods in the bean configuration.
Example (Java Config)@Bean(initMethod = \"init\", destroyMethod = \"cleanup\")\npublic MyService myService() {\n return new MyService();\n}\nAll singleton beans are created at the time of application startup (default behavior), Useful for performance since all dependencies are resolved upfront.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#lazy-initialization","title":"Lazy Initialization","text":"Beans are created only when they are first requested, You can enable lazy initialization at the bean level using @Lazy.
@Lazy\n@Component\npublic class LazyBean { }\nSpring IoC container resolves bean dependencies through constructor injection, setter injection, or field injection, You can specify bean dependencies explicitly using the depends-on attribute in XML.
<bean id=\"databaseConnection\" class=\"com.example.DatabaseConnection\"/>\n<bean id=\"orderService\" class=\"com.example.OrderService\" depends-on=\"databaseConnection\"/>\nThis ensures that the databaseConnection bean is initialized before the orderService bean.
A circular dependency occurs when two or more beans are mutually dependent on each other.
Example@Component\npublic class A {\n @Autowired\n private B b;\n}\n\n@Component\npublic class B {\n @Autowired\n private A a;\n}\nSpring tries to resolve circular dependencies using singleton beans by injecting proxies, but it fails for constructor injection. To avoid circular dependencies: - Refactor the code to reduce dependencies. - Use setter injection instead of constructor injection.
"},{"location":"langdives/Java/Spring/SpringCoreFramework/#bean-definition-inheritance","title":"Bean Definition Inheritance","text":"Spring allows bean definitions to inherit properties from a parent bean. This helps reduce configuration duplication.
Example (XML)<bean id=\"parentBean\" class=\"com.example.BaseService\">\n <property name=\"name\" value=\"Base Service\"/>\n</bean>\n\n<bean id=\"childBean\" class=\"com.example.ChildService\" parent=\"parentBean\">\n <property name=\"name\" value=\"Child Service\"/>\n</bean>\nAOP allows you to separate cross-cutting concerns (like logging, security, or transaction management) from the business logic. In Spring, AOP is implemented using aspects, advice, and pointcuts.
@Aspect\n@Component\npublic class LoggingAspect {\n @Before(\"execution(* com.example.*.*(..))\")\n public void logBefore(JoinPoint joinPoint) {\n System.out.println(\"Before method: \" + joinPoint.getSignature().getName());\n }\n}\n@Configuration\n@EnableAspectJAutoProxy\npublic class AppConfig { }\nSpring supports an event-driven model that allows you to build decoupled components. The ApplicationContext can publish events and allow listeners to respond to them.
Example of a Custom Eventpublic class CustomEvent extends ApplicationEvent {\n public CustomEvent(Object source) {\n super(source);\n }\n}\n@Component\npublic class CustomEventListener implements ApplicationListener<CustomEvent> {\n @Override\n public void onApplicationEvent(CustomEvent event) {\n System.out.println(\"Received custom event: \" + event);\n }\n}\nSpEL allows you to manipulate and query beans dynamically. It can be used inside XML or annotations.
Example of SpEL<bean id=\"myBean\" class=\"com.example.MyClass\">\n <property name=\"value\" value=\"#{2 + 3}\"/>\n</bean>\nA detailed comparison table that covers all possible differences between Spring Boot and Spring Framework. This comparison covers everything from setup, configuration, embedded servers, web applications, testing, microservices support, and much more.
"},{"location":"langdives/Java/Spring/SpringFrameworkVsSpringBoot/#differences","title":"Differences","text":"Category Spring Framework Spring Boot Purpose A comprehensive framework for building Java applications. An extension of Spring Framework to simplify configuration and create stand-alone applications. Setup Requires manual setup, including XML or Java-based configuration. Minimal setup with auto-configuration based on the classpath. Main Focus Provides flexibility and control over every aspect of the application. Focuses on rapid development with sensible defaults and opinions. Configuration Can use XML, Java-based, or annotation-based configuration. Uses annotations and properties/YAML files for configuration. Learning Curve Requires more learning time due to complexity. Easier to get started with for beginners due to auto-configuration and pre-built setups. Project Dependencies Requires managing multiple dependencies for each feature manually. Provides starters (e.g.,spring-boot-starter-web) to include required dependencies. Embedded Server No embedded server support; WAR files must be deployed to external servers (Tomcat, Jetty, etc.). Comes with embedded servers (Tomcat, Jetty, or Undertow) for running stand-alone applications. Deployment Deploy WAR/EAR files to external servers. Runs applications directly as JAR files with embedded servers. Auto-Configuration No auto-configuration; requires manual configuration of components. Auto-configures components based on available classpath dependencies. Application Entry Point Relies on external servlet containers to manage the lifecycle. Uses @SpringBootApplication as the entry point with SpringApplication.run(). Microservices Support Not specialized for microservices; requires additional tools. Built with microservices architecture in mind. Supports Spring Cloud, Eureka, Feign, etc. Performance Requires more configuration to optimize performance. Better suited for lightweight and high-performance microservices. Profiles Supports profiles for environment-specific configurations but requires more setup. Supports profiles easily through application.yml or application.properties. Testing Provides JUnit support but requires manual configuration for context loading. Provides easy testing with @SpringBootTest, @MockBean, @DataJpaTest, and others. Security Uses Spring Security but requires manual integration. Integrates Spring Security easily with spring-boot-starter-security. Database Access Provides JDBC, JPA, ORM support, but requires more configuration. Simplifies database access with Spring Data JPA and auto-configuration of DataSource. Starters and Dependency Management Requires manual management of dependencies and configurations. Provides Spring Boot Starters that bundle all required dependencies for specific use cases. Template Engines Supports Thymeleaf, JSP, and others with manual setup. Supports template engines with starters (e.g., spring-boot-starter-thymeleaf). Command-Line Interface (CLI) No built-in CLI support. Provides Spring Boot CLI to run Groovy scripts for quick development. Actuator and Monitoring Requires external monitoring tools or custom configurations. Comes with Spring Boot Actuator to monitor application health, metrics, and endpoints. DevTools for Hot Reload Requires manual setup for hot reloading of code changes. Provides Spring Boot DevTools for hot reloading during development. Support for Reactive Programming Supports Spring WebFlux and Project Reactor (from version 5.x). Fully supports Spring WebFlux for reactive, non-blocking programming. Circuit Breakers & Resilience Requires integration with third-party libraries like Hystrix. Seamlessly integrates with Resilience4j and Spring Cloud for resilience. Integration with Cloud Platforms Requires Spring Cloud or manual setup for cloud integration. Seamlessly integrates with Spring Cloud for cloud-native development. Logging Configuration Requires manual configuration of logging frameworks (e.g., Log4j, SLF4J). Provides auto-configured logging using Logback by default. Health Checks and Metrics Requires manual configuration to expose health metrics. Provides Actuator endpoints (/actuator/health, /actuator/metrics) out-of-the-box. Web Framework Uses Spring MVC for building web applications. Uses Spring MVC or Spring WebFlux with easy setup through starters. Restful API Development Requires manual setup of controllers and components. Provides easy development with @RestController and auto-configuration of REST endpoints. Command-Line Arguments Support Requires manual handling of command-line arguments. Easily reads command-line arguments with SpringApplication or @Value. Caching Support Requires setting up EhCache, Guava, or other caching solutions manually. Provides easy caching configuration with @EnableCaching and auto-configuration. Internationalization (i18n) Supports i18n but requires more setup. Supports i18n with minimal configuration through application.properties. Job Scheduling Requires integration with Quartz or other scheduling libraries. Supports scheduling with @Scheduled and Task Executors. Dependency Injection (DI) Provides dependency injection with IoC container. Same as Spring Framework but simplifies it with auto-wiring using @Autowired. Backward Compatibility Must manually update configurations when upgrading versions. Provides backward compatibility with most Spring projects. Community and Ecosystem Large community and extensive ecosystem. Built on top of Spring Framework with additional tools for modern development."},{"location":"langdives/Java/Spring/SpringFrameworkVsSpringBoot/#summary","title":"Summary","text":"Spring Boot simplifies Spring Framework by:
Spring Framework gives more control and flexibility but at the cost of manual setup and configuration.
Spring Boot is optimized for rapid development, especially for microservices and cloud-native applications.
Spring Framework is still relevant for legacy applications or when fine-grained control is necessary.
"},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/","title":"High Availability and Fault Tolerance","text":""},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/#high-availability-ha","title":"High Availability (HA)","text":"High Availability (HA) refers to a system or infrastructure's ability to remain operational and accessible for a very high percentage of time, minimizing downtime. In essence, it's a design principle used in IT to ensure that services, applications, or systems are continuously available, even in the event of hardware failures, software issues, or unexpected disruptions.
Key Aspects of High Availability
Redundancy: Multiple components (like servers, databases, or network paths) are set up so that if one fails, another can take over immediately without interrupting the service.
Failover: If a primary system component fails, the workload automatically switches to a backup or standby system to avoid service disruptions.
Load Balancing: Distribution of workloads across multiple servers or resources helps prevent any single system from becoming overwhelmed and failing. This also increases performance and resiliency.
Clustering: Multiple servers or resources act as a \"cluster,\" functioning together as a single system. If one server fails, the cluster can redistribute the tasks to the remaining ones.
Monitoring and Alerts: Constant health checks and monitoring tools are used to detect potential problems early. Alerts can notify administrators to address issues before they impact availability.
Data Replication: Critical data is often replicated across multiple locations or systems. This ensures that in case of a failure, there\u2019s no data loss and services can continue with minimal disruption.
Availability is often expressed as a percentage. For instance, an uptime of 99.99% means the service is expected to be down for only 52 minutes in a year.
Common availability standards
Data Centers: Cloud providers like AWS, Microsoft Azure, and Google Cloud offer high availability zones to ensure service uptime even during hardware failures or network outages.
Web Applications: E-commerce sites (like Amazon) employ HA architecture to ensure they remain operational at all times, as any downtime directly impacts revenue.
Banking Systems: Financial services need HA to ensure uninterrupted access to customer transactions, ATMs, and payment processing services.
Telecommunication Networks: Cellular networks and ISPs use HA to maintain voice, data, and internet services without interruptions.
Fault Tolerance refers to the ability of a system, network, or application to continue functioning correctly even when one or more of its components fail. It ensures continuous operation without loss of service, despite hardware, software, or other types of faults occurring in the system. Fault tolerance plays a crucial role in ensuring high reliability and availability of critical systems.
Key Principles of Fault Tolerance
Redundancy: Multiple components are deployed so that if one component fails, another takes over seamlessly. Redundant systems include extra servers, power supplies, storage drives, and networks.
Failover: When a primary component or service fails, the workload automatically \"fails over\" to a backup system with minimal disruption.
Error Detection and Correction: The system includes mechanisms to detect, isolate, and correct errors to maintain proper functioning. This is often implemented using parity bits, checksums, or watchdog timers.
Replication: Critical data is copied across multiple devices or systems (e.g., in distributed systems). This way, even if a fault occurs in one part, other replicas maintain the system\u2019s integrity.
Graceful Degradation: If a fault occurs, the system may reduce performance or disable non-essential services instead of shutting down entirely, thus ensuring the most critical functions remain operational.
Hardware Fault Tolerance:
Software Fault Tolerance:
Network Fault Tolerance:
Error Detection and Recovery:
Airplane Control Systems: Modern aircraft use redundant flight control systems, so if one control unit fails, another takes over to ensure passenger safety.
Nuclear Power Plants: Fault tolerance is critical to ensure safe operations. Redundant sensors, control systems, and cooling systems prevent catastrophic failures.
Financial Systems: Banks and payment processing systems must tolerate faults to ensure that transactions are processed without downtime or data corruption.
Cloud Computing Services: Cloud providers like AWS or Google Cloud implement fault tolerance using distributed systems, replication, and redundant storage across multiple locations.
Cost: Building redundant systems is expensive, as it requires additional hardware, software, and management resources.
Complexity: Managing fault-tolerant systems is complex due to the need for real-time monitoring, synchronization, and failover testing.
Performance Overhead: Some fault tolerance techniques, like replication or checkpointing, can introduce performance bottlenecks.
High Availability (HA) and Fault Tolerance (FT) are two strategies aimed at keeping systems operational, but they approach this goal differently. Here's a detailed comparison:
In other words, high availability aims to reduce downtime, whereas fault-tolerant systems aim for zero downtime, even during failures.
Aspect High Availability (HA) Fault Tolerance (FT) Definition Ensures minimal downtime by quickly switching to backup systems when a failure occurs. Ensures continuous operation even during failures, with no noticeable interruption. Goal Minimize downtime and ensure service is restored quickly. Eliminate downtime and maintain seamless operation during faults. Approach Uses redundant components and failover systems to switch operations when needed. Uses duplication of systems to ensure tasks are always mirrored on another system. Downtime Small amount of downtime during failover (milliseconds to minutes). No downtime; systems operate continuously, even during faults. Example Use Cases E-commerce websites (e.g., Amazon) that switch servers when one fails. Airplane control systems, which cannot afford any interruptions. Redundancy Type Active-Passive: Backup components are activated only when primary systems fail. Active-Active: All components are working simultaneously, and one continues if the other fails. Cost Less expensive since backup systems are not always active. More expensive due to constant replication and active systems running in parallel. Complexity Easier to implement and manage due to reliance on failover mechanisms. More complex, requiring real-time synchronization and parallel operation. Performance Impact Some performance hit during failover but minimal. Higher overhead, as multiple systems operate simultaneously. Use Case Example Cloud platforms (like AWS) use high availability to ensure that servers recover quickly after a failure. Nuclear power plants employ fault-tolerant systems to keep critical processes running with no interruptions. Failure Handling Handles component failures through redundancy and quick recovery mechanisms. Prevents failure from affecting the system by running identical processes or systems in parallel."},{"location":"techdives/DistrubutedConcepts/HighAvailabilityFaultTolerance/#summary","title":"Summary","text":"In summary, high availability ensures that critical systems are always accessible with minimal interruptions. Organizations rely on HA strategies to meet customer expectations, protect revenue, and ensure business continuity, especially in industries where even a small amount of downtime can have serious consequences and fault tolerance is the ability of a system to keep operating without interruption despite experiencing faults or failures. It is crucial for mission-critical systems in industries like aviation, finance, and healthcare, where downtime or errors could lead to catastrophic outcomes.
In essence, High Availability focuses on minimizing downtime by recovering quickly from failures, while Fault Tolerance eliminates downtime altogether by ensuring the system continues running seamlessly. HA is less costly and easier to implement, while FT is expensive and complex but essential for critical environments where even a few seconds of downtime are unacceptable.
"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/","title":"Docker and Kubernetes","text":""},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#overview","title":"Overview","text":""},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#docker","title":"Docker","text":"Docker is a platform that enables developers to build, package, and run applications in lightweight containers. It ensures applications are portable and can run consistently across different environments, from development to production.
Key Components
Dockerfile. Images are layered and reusable.Kubernetes (often abbreviated as K8s) is an open-source platform for automating the deployment, scaling, and management of containerized applications. It abstracts away the complexity of running containers at scale across multiple machines.
Key Components
Docker focuses on building, packaging, and running containers. It handles application-level concerns whereas Kubernetes focuses on orchestrating, scaling, and managing containers across distributed environments.
"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#how-they-work-together","title":"How they Work Together","text":"Create a Dockerfile for your application and build the image Step-1: Build Docker Images
docker build -t my-app:latest .\nStore the image in a registry (e.g., Docker Hub) Step-2: Push to Registry
docker push my-app:latest\nUse the built Docker image in a Kubernetes Deployment YAML file and apply it with: Deploy on Kubernetes
kubectl apply -f deployment.yaml\nInstall Docker: Follow Docker\u2019s official installation guide for your operating system.
Create a Dockerfile: Example:
FROM python:3.9\nWORKDIR /app\nCOPY . .\nRUN pip install -r requirements.txt\nCMD [\"python\", \"app.py\"]\nBuild and Run Docker Containers:
docker build -t my-app:latest .\nRun Container:
docker run -d -p 5000:5000 my-app:latest\nUse Docker Compose:
docker-compose.yml: version: '3.8'\nservices:\n web:\n image: my-app:latest\n ports:\n - \"5000:5000\"\ndocker-compose up\nInstall Kubernetes: Use Minikube for local development or create a production cluster using cloud providers like GKE, AKS, or EKS.
Define Kubernetes Resources: Create YAML manifests for Deployments and Services.
Deployment YAML:
apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: my-app\nspec:\n replicas: 2\n selector:\n matchLabels:\n app: my-app\n template:\n metadata:\n labels:\n app: my-app\n spec:\n containers:\n - name: my-app\n image: my-app:latest\n ports:\n - containerPort: 5000\nService YAML:
apiVersion: v1\nkind: Service\nmetadata:\n name: my-app-service\nspec:\n type: NodePort\n ports:\n - port: 5000\n targetPort: 5000\n selector:\n app: my-app\nDeploy to Kubernetes:
kubectl apply -f deployment.yaml\nkubectl apply -f service.yaml\nWith Docker Compose: Define services in docker-compose.yml and run:
docker-compose up\nWith Kubernetes (Using Minikube):
kubectl.docker swarm init\nUse overlay networks for service communication across nodes.
Using Kubernetes:
Kubernetes: Pods use DNS names or Service endpoints to communicate.
Multiple Machines Communication:
docker-compose.yml file Multiple YAML files for resources Scaling Manual Automated scaling with kubectl scale High Availability Limited Built-in redundancy and self-healing Use Case Simple applications on one machine Complex workloads across clusters"},{"location":"techdives/DistrubutedSystems/DockerAndK8s/#8-additional-considerations","title":"8. Additional Considerations","text":"Kubernetes: Use Persistent Volumes (PV) and Persistent Volume Claims (PVC).
Monitoring and Logging:
Use Prometheus and Grafana in Kubernetes for comprehensive monitoring.
CI/CD Integration:
Docker Compose can also use the build context to pull code directly from GitHub when creating an image. Here\u2019s how:
docker-compose.yml with GitHub Repository as the Build Context:version: '3.8'\nservices:\n app:\n build:\n context: https://github.com/your-username/your-repo.git\n dockerfile: Dockerfile\n ports:\n - \"5000:5000\"\ndocker-compose up --build\nNote: - This method works only if the GitHub repository is public. For private repositories, you\u2019ll need to provide authentication (e.g., via SSH keys or GitHub tokens). - Docker will pull the latest version from the specified GitHub repository and build the image based on the Dockerfile in the repository.
Can I use a Docker Compose file with Kubernetes? Not directly. Kubernetes doesn\u2019t understand Docker Compose syntax, but there are tools like Kompose that can convert Docker Compose files into Kubernetes YAML files.
What happens if I try to run a Docker Compose file inside a Kubernetes cluster? It won\u2019t work. Kubernetes will look at you confused (figuratively), because it expects YAML manifests with its own syntax, not a docker-compose.yml.
Why do Kubernetes YAML files look scarier than Docker Compose files? Kubernetes YAML files are more complex because they handle more advanced scenarios like scaling, networking, and rolling updates, which Docker Compose doesn\u2019t attempt to address.
Do I need to uninstall Docker if I switch to Kubernetes? Nope! Docker is still useful for building and testing images locally, even if you\u2019re deploying to Kubernetes. In fact, Kubernetes can use Docker as a container runtime.
Will Docker containers fight with Kubernetes Pods if they run on the same machine? Nope, they\u2019ll coexist peacefully. Docker containers and Kubernetes Pods can run side by side without conflict. They\u2019re friends, not rivals!
Can I copy-paste my Docker Compose file into Kubernetes and hope it works? Sorry, no shortcuts here. You need to convert the Compose file into Kubernetes resources, either manually or using tools like Kompose.
Is Docker Compose faster than Kubernetes because it has fewer YAML files? Yes, Docker Compose is faster to set up for local development because it\u2019s simpler. But for production-scale orchestration, Kubernetes is much more powerful.
How do I know if my container is happy inside a Kubernetes Pod? Check with this command:
kubectl get pods\nRunning, your container is content. If you see CrashLoopBackOff, it\u2019s definitely not happy! Can I use Kubernetes without the cloud, or will it complain? You can use Minikube or kind (Kubernetes in Docker) to run Kubernetes locally on your machine. No cloud required.
What\u2019s the difference between docker-compose up and kubectl apply -f?
docker-compose up: Starts containers defined in a docker-compose.yml file. kubectl apply -f: Deploys resources (like Pods, Deployments) described in a Kubernetes YAML file to your cluster.Do I still need to learn Docker Swarm if I already know Kubernetes? Not really. Docker Swarm is simpler but not as widely used in production as Kubernetes. Kubernetes has become the de facto standard.
Can a single Pod run multiple Docker Compose services? Yes! A Pod can run multiple containers, similar to how Docker Compose runs multiple services. However, in Kubernetes, these containers should be tightly coupled (e.g., sharing resources).
If Docker Compose is easier, why do people torture themselves with Kubernetes? Kubernetes offers features like scaling, self-healing, and load balancing. It\u2019s overkill for simple projects but essential for large, distributed applications.
Is Kubernetes just a fancy way of saying, \u201cI don\u2019t want to use Docker Compose\u201d? Not exactly. Docker Compose is great for local setups, while Kubernetes is a powerful orchestration tool for running applications across multiple nodes at scale.
What\u2019s the difference between a Pod and a Container? Can I use the words interchangeably? Not quite. A Pod is a wrapper that can contain one or more containers. Pods are the smallest deployable unit in Kubernetes, but a container is just an isolated environment for running applications.
If a container crashes in Kubernetes, does Kubernetes get sad? Nope! Kubernetes will restart the container automatically. That\u2019s part of its self-healing magic.
Will my application break if I use a Docker image from 2015? It might! Older images could have compatibility issues or security vulnerabilities. Use them only if you\u2019re sure they still meet your needs.
Is Kubernetes allergic to Windows, or will it run happily there? Kubernetes supports Windows nodes, but the experience is smoother with Linux. Most people deploy Kubernetes on Linux-based clusters.
Can I use both Docker and Kubernetes at the same time? Or will it cause chaos? Yes, you can use both. Build your containers with Docker, push them to a registry, and deploy them with Kubernetes. No chaos \u2013 just smooth workflows.
Why can\u2019t Docker Compose just learn scaling and take over Kubernetes' job? Docker Compose is intentionally lightweight and simple. Adding Kubernetes-like features would complicate it and defeat its original purpose.
How much YAML is too much YAML? If you start dreaming in YAML, it\u2019s probably too much. But seriously, Kubernetes relies heavily on YAML, so learning to manage it effectively is key.
Can Kubernetes work without YAML files? (Please say yes!) Unfortunately, no. YAML files are essential for defining resources in Kubernetes. You can use Helm charts to simplify it, but YAML is unavoidable.
What happens if I forget to push my Docker image before deploying with Kubernetes? Your deployment will fail because Kubernetes won\u2019t find the image in the registry. Always remember to push!
Can I use kubectl commands on Docker containers? Nope. kubectl is specifically for managing Kubernetes resources. Use docker commands for Docker containers.
Is Kubernetes only for tech wizards, or can normal humans use it too? Normal humans can use it too! The learning curve is steep, but with practice, anyone can master it.
Do I need to sacrifice sleep to understand Kubernetes? Maybe at first. But once you get the hang of it, Kubernetes will become your friend, and sleep will return.
Can a Docker container tell the difference between running on Kubernetes and Docker Compose? Nope! The container itself doesn\u2019t care where it\u2019s running. As long as it gets its dependencies and configuration, it\u2019ll happily run anywhere.
Can I run two Docker Compose files on one machine? Yes, use the -p option to specify different project names for each Compose file.
Can services communicate across multiple machines? Yes, with Docker Swarm or Kubernetes, services can communicate across machines using overlay networks or Kubernetes networking.
Is Docker Compose suitable for production? Not recommended for large-scale production. Use Kubernetes or Docker Swarm instead.
How do I set up Kubernetes on a single machine? Use Minikube to run a local Kubernetes cluster.
What file formats are used by Docker and Kubernetes? Docker uses Dockerfile and docker-compose.yml. Kubernetes uses YAML files for resources like Deployments and Services.
Elasticsearch is a search engine based on Apache Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#elasticsearch-basics-and-fundamentals","title":"ElasticSearch Basics and Fundamentals","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-core-concepts","title":"1. Core Concepts:","text":"Diving into Elasticsearch's core concepts is essential for understanding its architecture and functionality.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#11-documents-and-indices","title":"1.1 Documents and Indices","text":"{\n \"title\": \"Learning Elasticsearch\",\n \"author\": \"John Doe\",\n \"published_date\": \"2023-01-15\"\n}\nPUT /books\nPUT /books/_mapping\n{\n \"properties\": {\n \"title\": { \"type\": \"text\" },\n \"author\": { \"type\": \"keyword\" },\n \"published_date\": { \"type\": \"date\" }\n }\n}\nPUT /books\n{\n \"settings\": {\n \"number_of_shards\": 3,\n \"number_of_replicas\": 1\n }\n}\nPOST /books/_doc/1\n{\n \"title\": \"Learning Elasticsearch\",\n \"author\": \"John Doe\",\n \"published_date\": \"2023-01-15\"\n}\nGET /books/_doc/1\nPOST /books/_doc/1/_update\n{\n \"doc\": { \"author\": \"Jane Doe\" }\n}\nDELETE /books/_doc/1\nAn inverted index is a fundamental data structure in Elasticsearch and other search engines. It optimizes search efficiency by storing a mapping from terms (words) to their locations within documents. Let\u2019s break it down into key components and processes:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#21-core-structure-of-inverted-index","title":"2.1. Core Structure of Inverted Index","text":"For instance, if a dataset contains the documents: - Doc 1: \u201cElasticsearch powers search\u201d - Doc 2: \u201cSearch powers insights\u201d
The inverted index would look like this:
\"Elasticsearch\" -> [Doc 1]\n\"powers\" -> [Doc 1, Doc 2]\n\"search\" -> [Doc 1, Doc 2]\n\"insights\" -> [Doc 2]\nThe process involves several stages: - Tokenization: Splitting text into words or tokens. - Normalization: Making tokens consistent, like converting to lowercase. - Stemming/Lemmatization (optional): Reducing words to their base or root forms. - Indexing: Populating the index with terms and the corresponding document references.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#23-how-searches-work","title":"2.3. How Searches Work","text":"When a user searches for a term, Elasticsearch retrieves the postings list from the inverted index, quickly locating documents containing that term. For multi-term queries, Elasticsearch can intersect postings lists, using logical operations (e.g., AND, OR) to combine or filter results.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#24-optimizations","title":"2.4. Optimizations","text":"Inverted indices are the foundation of Elasticsearch\u2019s speed and relevance in text search. This structure is tailored for high performance in full-text search scenarios, especially when complex queries and filtering are involved.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#3-analyzers","title":"3. Analyzers","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#31-what-is-an-analyzer","title":"3.1. What is an Analyzer?","text":"Creating a custom analyzer involves defining: - A tokenizer (e.g., edge-ngram tokenizer for partial word matches). - A list of token filters to process the tokens (e.g., synonym filters, ASCII folding for diacritical marks).
Example configuration:
{\n \"analysis\": {\n \"analyzer\": {\n \"custom_analyzer\": {\n \"type\": \"custom\",\n \"tokenizer\": \"whitespace\",\n \"filter\": [\"lowercase\", \"stop\", \"synonym\"]\n }\n }\n }\n}\nenglish, french) handle language-specific tokenization and filtering.Analyzers transform raw text into optimized, searchable data, playing a critical role in making Elasticsearch searches accurate and efficient.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#4-elasticsearch-queries","title":"4. ElasticSearch Queries","text":"In Elasticsearch, queries are central to retrieving data. They\u2019re categorized as leaf queries (operating on specific fields) and compound queries (combining multiple queries). Here's a deep dive into each type, with examples to illustrate their functionality:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#41-leaf-queries","title":"4.1 Leaf Queries","text":"These are standalone, field-specific queries (like term and match above) that don\u2019t depend on other queries to function.
Match Query: Used for analyzing full-text fields, capable of handling fuzziness, synonyms, and relevance scoring.
{\n \"query\": {\n \"match\": {\n \"description\": \"elastic search\"\n }\n }\n}\nMatch Phrase Query: Requires terms to appear in the specified order, useful for exact phrase searches.
{\n \"query\": {\n \"match_phrase\": {\n \"description\": \"elastic search\"\n }\n }\n}\n{\n \"query\": {\n \"term\": {\n \"status\": \"active\"\n }\n }\n}\n{\n \"query\": {\n \"terms\": {\n \"status\": [\"active\", \"pending\"]\n }\n }\n}\nCompound queries allow for complex logic by combining multiple queries, enabling fine-grained control over query conditions and relevance.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#421-bool-query","title":"4.2.1. Bool Query","text":"The most flexible compound query, allowing logic-based combinations:
Must Clause: The must clause ensures that all enclosed queries must match. Think of it as an \"AND\" operation.
\"title\" contains \"Elasticsearch\" and \"content\" contains \"query\". {\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"match\": { \"title\": \"Elasticsearch\" } },\n { \"match\": { \"content\": \"query\" } }\n ]\n }\n }\n}\nShould Clause: The should clause is a flexible OR operation. If any query in the should clause matches, the document is considered a match. Adding multiple should clauses increases relevance based on how many match.
tags field are included. {\n \"query\": {\n \"bool\": {\n \"should\": [\n { \"match\": { \"tags\": \"search\" } },\n { \"match\": { \"tags\": \"indexing\" } }\n ],\n \"minimum_should_match\": 1\n }\n }\n}\nMust Not Clause: The must_not clause excludes documents that match any query within it, similar to a \"NOT\" operation.
{\n \"query\": {\n \"bool\": {\n \"must\": { \"match\": { \"title\": \"Elasticsearch\" } },\n \"must_not\": { \"match\": { \"status\": \"archived\" } }\n }\n }\n}\nFilter Clause: The filter clause narrows down the result set without affecting relevance scores. It\u2019s optimized for structured queries and often used to improve query performance.
{\n \"query\": {\n \"bool\": {\n \"must\": { \"match\": { \"title\": \"Elasticsearch\" } },\n \"filter\": { \"term\": { \"status\": \"published\" } }\n }\n }\n}\nCombining multiple clauses:
{\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"match\": { \"title\": \"Elasticsearch\" } }\n ],\n \"should\": [\n { \"match\": { \"category\": \"tutorial\" } },\n { \"match\": { \"category\": \"guide\" } }\n ],\n \"must_not\": [\n { \"term\": { \"status\": \"archived\" } }\n ],\n \"filter\": [\n { \"range\": { \"publish_date\": { \"gte\": \"2023-01-01\" } } }\n ]\n }\n }\n}\nOptimizes for the highest relevance score among multiple queries, often used when querying across similar fields with varied wording. - Example: Searching for the most relevant match between \u201ctitle\u201d and \u201cdescription\u201d fields:
{\n \"query\": {\n \"dis_max\": {\n \"queries\": [\n { \"match\": { \"title\": \"elastic search\" } },\n { \"match\": { \"description\": \"elastic search\" } }\n ]\n }\n }\n}\nElasticsearch provides several geo-specific queries for filtering and scoring documents based on geographic location:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#431-geo-bounding-box-query","title":"4.3.1 Geo Bounding Box Query","text":"Defines a rectangular area by specifying two corner points (top-left and bottom-right). Documents with locations inside this box are matched.
{\n \"query\": {\n \"geo_bounding_box\": {\n \"location\": {\n \"top_left\": {\n \"lat\": 40.73,\n \"lon\": -74.1\n },\n \"bottom_right\": {\n \"lat\": 40.01,\n \"lon\": -71.12\n }\n }\n }\n }\n}\nFinds documents within a certain distance from a point. Useful for proximity searches, like \"find stores within 10 miles.\"
{\n \"query\": {\n \"geo_distance\": {\n \"distance\": \"50km\",\n \"location\": {\n \"lat\": 40.7128,\n \"lon\": -74.0060\n }\n }\n }\n}\nSearches within a polygon defined by a series of latitude and longitude points, allowing for irregular area shapes.
{\n \"query\": {\n \"geo_polygon\": {\n \"location\": {\n \"points\": [\n { \"lat\": 40.73, \"lon\": -74.1 },\n { \"lat\": 40.01, \"lon\": -71.12 },\n { \"lat\": 39.73, \"lon\": -73.1 }\n ]\n }\n }\n }\n}\nThe geo_shape query allows for more complex spatial filtering, using pre-defined shapes like circles, polygons, or lines. This is often used with indexed geometries.
{\n \"query\": {\n \"geo_shape\": {\n \"location\": {\n \"shape\": {\n \"type\": \"circle\",\n \"coordinates\": [-74.1, 40.73],\n \"radius\": \"1000m\"\n },\n \"relation\": \"within\"\n }\n }\n }\n}\nGeo filters are often used in combination with other query types within bool queries, allowing flexible, location-based filtering along with other criteria.
Finds published documents within a specific area and filters out archived content.
{\n \"query\": {\n \"bool\": {\n \"must\": [\n { \"term\": { \"status\": \"published\" } }\n ],\n \"filter\": {\n \"geo_distance\": {\n \"distance\": \"50km\",\n \"location\": {\n \"lat\": 40.7128,\n \"lon\": -74.0060\n }\n }\n },\n \"must_not\": [\n { \"term\": { \"status\": \"archived\" } }\n ]\n }\n }\n}\nThese queries, combined thoughtfully, make Elasticsearch highly adaptable to various search needs.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#5-aggregations","title":"5. Aggregations","text":"Elasticsearch\u2019s aggregation framework is divided into metrics and bucket aggregations. Here\u2019s a deep dive into each, with subtypes and examples.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#51-metrics-aggregations","title":"5.1. Metrics Aggregations","text":"These calculate values from field data, like sums or averages.
Example: Calculate average price across documents.
{\n \"aggs\": {\n \"avg_price\": { \"avg\": { \"field\": \"price\" } }\n }\n}\nSum Aggregation: Totals numeric field values.
Example: Sum of all sales amounts.
{\n \"aggs\": {\n \"total_sales\": { \"sum\": { \"field\": \"amount\" } }\n }\n}\nMin/Max Aggregations: Find minimum or maximum field values.
Example: Find the highest score.
{\n \"aggs\": {\n \"max_score\": { \"max\": { \"field\": \"score\" } }\n }\n}\nStats Aggregation: Generates summary stats (min, max, avg, sum, count).
Example: Summarize views across articles.
{\n \"aggs\": {\n \"article_views\": { \"stats\": { \"field\": \"views\" } }\n }\n}\nCardinality Aggregation: Counts unique values.
Example: Count unique users by ID.
{\n \"aggs\": {\n \"unique_users\": { \"cardinality\": { \"field\": \"user_id\" } }\n }\n}\nPercentiles/Percentile Ranks Aggregations: Show value distribution.
{\n \"aggs\": {\n \"response_percentiles\": {\n \"percentiles\": { \"field\": \"response_time\", \"percents\": [50, 90, 99] }\n }\n }\n}\nThese create groups (buckets) of documents based on field values or criteria. Each bucket can contain documents matching conditions and may contain further sub-aggregations.
Example: Group documents by category.
{\n \"aggs\": {\n \"by_category\": { \"terms\": { \"field\": \"category\" } }\n }\n}\nRange Aggregation: Group documents within defined ranges.
Example: Bucket users by age ranges.
{\n \"aggs\": {\n \"age_ranges\": {\n \"range\": { \"field\": \"age\", \"ranges\": [{ \"to\": 20 }, { \"from\": 20, \"to\": 30 }, { \"from\": 30 }] }\n }\n }\n}\nDate Histogram: Buckets documents based on date intervals (e.g., daily, monthly).
Example: Group events by month.
{\n \"aggs\": {\n \"monthly_events\": { \"date_histogram\": { \"field\": \"date\", \"calendar_interval\": \"month\" } }\n }\n}\nHistogram Aggregation: Creates buckets based on custom numeric ranges.
Example: Histogram of item prices.
{\n \"aggs\": {\n \"price_histogram\": { \"histogram\": { \"field\": \"price\", \"interval\": 10 } }\n }\n}\nFilter Aggregation: Buckets documents matching a specific filter.
Example: Filter for high-value orders.
{\n \"aggs\": {\n \"high_value_orders\": { \"filter\": { \"range\": { \"price\": { \"gt\": 100 } } } }\n }\n}\nFilters Aggregation: Allows multiple filter buckets in a single query.
Example: Separate buckets for high and low prices.
{\n \"aggs\": {\n \"price_buckets\": {\n \"filters\": {\n \"filters\": {\n \"expensive\": { \"range\": { \"price\": { \"gt\": 100 } } },\n \"cheap\": { \"range\": { \"price\": { \"lt\": 50 } } }\n }\n }\n }\n }\n}\nGeohash Grid Aggregation: For geolocation-based bucketing, especially with maps.
{\n \"aggs\": {\n \"geo_buckets\": { \"geohash_grid\": { \"field\": \"location\", \"precision\": 5 } }\n }\n}\nEach aggregation can nest other aggregations, allowing complex analysis structures.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#example-of-a-combined-aggregation","title":"Example of a Combined Aggregation","text":"Calculate the average order amount by city and age range:
{\n \"aggs\": {\n \"by_city\": {\n \"terms\": { \"field\": \"city\" },\n \"aggs\": {\n \"age_ranges\": {\n \"range\": { \"field\": \"age\", \"ranges\": [{ \"to\": 20 }, { \"from\": 20, \"to\": 30 }, { \"from\": 30 }] },\n \"aggs\": {\n \"avg_order_amount\": { \"avg\": { \"field\": \"order_amount\" } }\n }\n }\n }\n }\n }\n}\nWith these aggregations, Elasticsearch becomes a powerful analytics engine, enabling sophisticated data analysis directly within the index.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#6-sorting","title":"6. Sorting","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#61-basic-sorting","title":"6.1. Basic Sorting","text":"_score (relevance-based sorting).sort in the query.{\n \"sort\": [\n { \"price\": { \"order\": \"asc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\nkeyword fields).{\n \"sort\": [\n { \"release_date\": { \"order\": \"desc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\n{\n \"sort\": [\n { \"price\": { \"order\": \"asc\" } },\n { \"rating\": { \"order\": \"desc\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\nnested with a path to sort by nested fields.{\n \"sort\": [\n {\n \"products.price\": {\n \"order\": \"asc\",\n \"nested\": {\n \"path\": \"products\",\n \"filter\": { \"range\": { \"products.price\": { \"gt\": 10 } } }\n }\n }\n }\n ],\n \"query\": { \"match_all\": {} }\n}\ngeo_distance to sort by proximity for geo_point fields.{\n \"sort\": [\n {\n \"_geo_distance\": {\n \"location\": \"40.715, -73.988\",\n \"order\": \"asc\",\n \"unit\": \"km\"\n }\n }\n ],\n \"query\": { \"match_all\": {} }\n}\npainless scripting language.{\n \"sort\": {\n \"_script\": {\n \"type\": \"number\",\n \"script\": {\n \"source\": \"doc['price'].value * params.factor\",\n \"params\": { \"factor\": 1.2 }\n },\n \"order\": \"desc\"\n }\n },\n \"query\": { \"match_all\": {} }\n}\nmissing, setting them as the highest or lowest values.{\n \"sort\": [\n { \"price\": { \"order\": \"asc\", \"missing\": \"_last\" } }\n ],\n \"query\": { \"match_all\": {} }\n}\ncount or custom metrics.{\n \"aggs\": {\n \"top_brands\": {\n \"terms\": {\n \"field\": \"brand.keyword\",\n \"order\": { \"_count\": \"desc\" }\n }\n }\n }\n}\nElasticsearch's relevance scoring is crucial for ranking documents based on their similarity to a query. Here\u2019s an in-depth look:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#71-scoring-mechanism-and-bm25-algorithm","title":"7.1. Scoring Mechanism and BM25 Algorithm","text":"The BM25 (Best Matching 25) algorithm is Elasticsearch\u2019s default relevance scoring algorithm. BM25 improves upon traditional TF-IDF (Term Frequency-Inverse Document Frequency) by adjusting term frequency saturation and document length normalization, providing more nuanced relevance.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#core-components-of-bm25","title":"Core Components of BM25:","text":"The BM25 formula combines these components, with two main parameters: - k1: Controls term frequency saturation (default around 1.2). Higher values give more influence to term frequency. - b: Controls length normalization (default around 0.75). Higher values penalize longer documents more strongly. - BM25 Alogirthm
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#73-understanding-scoring-in-elasticsearch-queries","title":"7.3. Understanding Scoring in Elasticsearch Queries","text":"In Elasticsearch, relevance scores are generated by the \"match\" or \"multi_match\" queries. Each document receives a relevance score, and results are ranked based on these scores. You can inspect scores using the \"explain\": true parameter, which details each document\u2019s score and shows how BM25 factors contribute.
{\n \"query\": {\n \"match\": {\n \"content\": {\n \"query\": \"Elasticsearch relevance scoring\",\n \"boost\": 1.5\n }\n }\n },\n \"explain\": true\n}\ncontent field. The \"boost\" parameter can emphasize this field for relevance, while \"explain\": true helps analyze the scoring breakdown."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#74-improving-relevance-with-advanced-techniques","title":"7.4. Improving Relevance with Advanced Techniques","text":"Relevance scoring with BM25 is foundational to Elasticsearch\u2019s search quality, offering powerful controls for tuning results to your specific needs.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#8-pagination-and-cursors","title":"8. Pagination and Cursors","text":"Pagination in Elasticsearch is essential for handling large result sets efficiently, as it prevents overwhelming the client and server. Elasticsearch offers different methods for pagination, each with specific use cases. Let's break down each method:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#81-basic-pagination-with-from-and-size","title":"8.1. Basic Pagination withfrom and size","text":"from parameter skips a set number of results, while size controls the number of results returned.from increases (since Elasticsearch has to load and sort through all documents before the specified offset).Example:
{\n \"from\": 20,\n \"size\": 10,\n \"query\": { \"match_all\": {} }\n}\nfrom and size by using a cursor.search_after parameter to specify a unique field value (like a timestamp or an ID) from the last document of the previous page.Example:
{\n \"sort\": [ { \"timestamp\": \"asc\" }, { \"id\": \"asc\" } ],\n \"size\": 10,\n \"query\": { \"match_all\": {} },\n \"search_after\": [1627489200, \"XYZ123\"] \n}\nsearch_after takes the values from the timestamp and id fields of the last document on the previous page, ensuring seamless navigation."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#83-scroll-api-for-bulk-data-retrieval","title":"8.3. Scroll API for Bulk Data Retrieval","text":"Example Workflow: - First, initiate a scroll session:
{\n \"size\": 100,\n \"query\": { \"match_all\": {} },\n \"scroll\": \"1m\" \n}\n_scroll_id returned by the initial request to retrieve subsequent pages: {\n \"scroll\": \"1m\",\n \"scroll_id\": \"DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAA...\"\n}\nAfter each scroll request, repeat until the returned results are empty, which indicates that all documents have been retrieved.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#84-point-in-time-pit-for-real-time-pagination-with-consistency","title":"8.4. Point-in-Time (PIT) for Real-time Pagination with Consistency","text":"Example Workflow: - First, initiate a Point-in-Time session:
POST /index_name/_pit?keep_alive=1m\npit_id with search_after for paginated queries: {\n \"size\": 10,\n \"query\": { \"match_all\": {} },\n \"pit\": { \"id\": \"PIT_ID\", \"keep_alive\": \"1m\" },\n \"sort\": [ { \"timestamp\": \"asc\" }, { \"id\": \"asc\" } ],\n \"search_after\": [1627489200, \"XYZ123\"]\n}\nDELETE /_pit\nsearch_after or PIT for deeper, efficient pagination.search_after and PIT handle real-time updates better.search_after and PIT require sorted fields to maintain proper order across paginated results.from & size Simple pagination for small datasets Performance drop for large from values Basic search pages, small datasets search_after Deep pagination without from overhead Requires sorted fields, can\u2019t skip pages Infinite scrolling, data tables with lots of records Scroll API Bulk data export/processing High memory usage, no real-time consistency Data migration, report generation Point-in-Time Consistent real-time pagination Needs frequent re-creation to avoid memory issues Dashboards, applications requiring consistent views Each method serves specific needs, balancing consistency, performance, and real-time capabilities. This setup allows Elasticsearch to handle vast and dynamic datasets while supporting efficient data retrieval.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#9-facets-and-filters","title":"9. Facets and Filters","text":"Faceting creates summaries of data, useful for search result filtering, like categorizing search results by price or brand. Filters, on the other hand, optimize performance by narrowing down documents without affecting scoring.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#91-facets-aggregations","title":"9.1. Facets (Aggregations)","text":"Faceting is a process in Elasticsearch that aggregates search results, providing structured summaries for complex queries. For example, if searching for \"laptops,\" facets can aggregate results by price range, brand, or processor type, allowing users to filter search results dynamically.
Filters enable the narrowing of search results by criteria (e.g., price < $500), improving query efficiency and bypassing relevance scoring. They\u2019re often used to pre-process data before a full-text search and work well with caches, resulting in faster query performance.
bool query, filters can be applied to queries under must or filter clauses. Filters in filter clauses are cached and reusable, significantly boosting speed for repetitive filters (e.g., \"available items only\").In complex search interfaces, facets allow users to drill down through categories, while filters further refine their selections without recalculating scores, ensuring responsive user experiences.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#94-sample-implementations","title":"9.4. Sample Implementations","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#941-creating-facets-aggregations","title":"9.4.1. Creating Facets (Aggregations)","text":"To implement facets, you\u2019ll define bucket aggregations to group and categorize data. For instance, creating a facet for \"price range\" and \"brand\" in a search for laptops:
GET /products/_search\n{\n \"query\": {\n \"match\": { \"description\": \"laptop\" }\n },\n \"aggs\": {\n \"price_ranges\": {\n \"range\": {\n \"field\": \"price\",\n \"ranges\": [\n { \"to\": 500 },\n { \"from\": 500, \"to\": 1000 },\n { \"from\": 1000 }\n ]\n }\n },\n \"brands\": {\n \"terms\": { \"field\": \"brand.keyword\" }\n }\n }\n}\nThis example provides a breakdown of price ranges and a count of each brand, creating flexible filters users can click on to refine results.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#942-using-filters-for-optimized-performance","title":"9.4.2. Using Filters for Optimized Performance","text":"Filters improve performance by narrowing results without scoring. Here\u2019s an example of using a bool query with a filter clause:
GET /products/_search\n{\n \"query\": {\n \"bool\": {\n \"must\": {\n \"match\": { \"description\": \"laptop\" }\n },\n \"filter\": [\n { \"term\": { \"in_stock\": true } },\n { \"range\": { \"price\": { \"lt\": 1000 } } }\n ]\n }\n }\n}\nIn this query, in_stock and price filters optimize search results without affecting scoring.
match should only appear in scoring-related queries (e.g., must). Using term or range filters for precise values (like IDs, prices) reduces computational load.Each node type in an Elasticsearch cluster has specialized roles that allow it to handle different aspects of indexing, searching, and managing data. Let's explore the node types in detail.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#11-master-node","title":"1.1. Master Node","text":"The master node is the cluster\u2019s brain, responsible for the overall management and health of the cluster.
Health Checks: Constantly checks the health of nodes, removing unresponsive nodes to maintain a stable cluster state.
Consensus and Election Process:
Data nodes are the primary nodes for storing data, processing indexing operations, and executing search and aggregation requests.
Also known as the client node, the coordinating node acts as a router for client requests, managing query distribution and response aggregation.
Ingest nodes preprocess data before it is indexed, often using ingest pipelines to transform and enrich data.
Each level in the cluster architecture plays a role in organizing and distributing data efficiently across nodes.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#21-cluster","title":"2.1. Cluster","text":"To deeply understand how request flow and cluster operations work in Elasticsearch, let\u2019s walk through each stage of the process in detail:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#31-receiving-a-request","title":"3.1. Receiving a Request","text":"When a client sends a request to Elasticsearch, it can be either a query (search request) or an indexing (write) request. Here\u2019s how this begins:
Identifying Relevant Shards and Nodes:
Selecting Shards (Primary or Replica):
At this stage, each shard executes the request locally. This process differs slightly between a search request and an indexing request.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#331-query-execution-search-requests","title":"3.3.1. Query Execution (Search Requests)","text":"Query Phase:
Fetch Phase:
Primary Shard Indexing:
Replication to Replica Shards:
After the query or indexing operation completes, the coordinating node consolidates the response:
Sorting and Ranking:
Returning the Response:
The indexing flow includes several key mechanisms that ensure data consistency and durability:
Primary-Replica Synchronization:
Distributed Write-Ahead Log (WAL):
Commit Process:
To visualize this, here\u2019s a simplified flow of the entire process:
Node Failure Handling: If a node fails, Elasticsearch promotes a replica shard to primary, maintaining data consistency and availability.
Cluster State Management:
To thoroughly understand Elasticsearch's storage and retrieval mechanisms, let\u2019s go deep into Lucene's segments, inverted index, and advanced data structures like BKD trees. Lucene, at its core, powers Elasticsearch, giving it the ability to handle and query massive datasets with impressive speed and efficiency.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#41-lucene-and-segments","title":"4.1. Lucene and Segments","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#411-what-is-a-segment","title":"4.1.1. What is a Segment?","text":"A segment in Lucene is a self-contained, immutable collection of documents that forms a subset of a shard. Each segment is essentially a mini-index with its own data structures, including inverted indexes, stored fields, and other data structures to facilitate efficient searching and retrieval.
Example: - Imagine a shard with 100 small segments. Lucene might merge them into fewer, larger segments (say, 10 segments), consolidating their data and removing any \"marked as deleted\" documents.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#413-advantages-of-segments","title":"4.1.3. Advantages of Segments","text":"The inverted index is Lucene\u2019s most fundamental data structure and is the backbone of Elasticsearch\u2019s fast full-text search capabilities.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#421-structure-of-an-inverted-index","title":"4.2.1. Structure of an Inverted Index","text":"The inverted index allows quick lookups by mapping terms to postings lists (lists of documents containing each term).
Example: - Suppose you index the text \"Elasticsearch is scalable search\". The inverted index might look like this:
Term Documents\n------------------------\n\"Elasticsearch\" [1]\n\"is\" [1, 2]\n\"scalable\" [1, 3]\n\"search\" [1]\nApart from the inverted index, Lucene also uses specialized data structures to handle numeric and spatial data.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#431-bkd-trees","title":"4.3.1. BKD Trees","text":"BKD Trees are used in Elasticsearch for indexing and querying numeric, date, and geospatial data, especially for high-cardinality fields (fields with a large number of unique values).
Splitting Mechanism: The tree splits data points along axes at each level, grouping data into smaller ranges or \"blocks.\" Each block holds a set of points within a defined range.
Hierarchical Storage: At each level of the tree, the data is divided along a specific dimension, allowing efficient narrowing of the search space during queries.
geo_point fields, Elasticsearch calculates distances within BKD trees, efficiently handling \"find near\" queries.Example: - Suppose you have a field geo_point representing user locations. A BKD tree indexes these coordinates, allowing Elasticsearch to quickly retrieve points within a bounding box or radius without scanning all documents.
BKD trees are essentially a form of a k-d tree (k-dimensional tree), optimized for indexing and searching over multiple dimensions. Each dimension can represent a distinct numeric field (e.g., latitude, longitude, or timestamp).
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#using-bkd-trees-for-queries","title":"Using BKD Trees for Queries","text":"BKD trees efficiently handle:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-bkd-trees","title":"Advantages of BKD Trees","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#432-doc-values","title":"4.3.2. Doc Values","text":"Doc values enable efficient retrieval of field values for sorting, aggregation, and faceting. Instead of retrieving data from inverted indexes (which are optimized for search), doc values provide a columnar storage format that is ideal for analytical tasks.
On-Disk Storage: Doc values are stored on disk rather than in memory, allowing efficient memory usage and enabling large datasets to be queried without excessive RAM usage.
sum, avg).Geo-point Doc Values: Specifically for geographic points, enabling geo-distance calculations.
Example: - Sorting results by price in a large index of products: - Doc values store price in a single column, which Elasticsearch reads to quickly sort documents without scanning each document individually.
Efficient Sorting and Aggregation: Since doc values are structured columnar data, they allow fast and memory-efficient operations.
Doc values are defined by the field type:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#using-doc-values-in-queries","title":"Using Doc Values in Queries","text":"Doc values are essential for operations such as:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-doc-values","title":"Advantages of Doc Values","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#433-summary-of-lucene-data-structures-in-elasticsearch","title":"4.3.3. Summary of Lucene Data Structures in Elasticsearch","text":"Data Structure Purpose Use Cases Benefits Segments Immutable sub-indices within a shard All document storage Concurrent searches, immutability Inverted Index Maps terms to documents Full-text search Fast term lookups BKD Trees Indexes numeric and multidimensional data Geospatial, timestamp queries Efficient range queries Doc Values Columnar storage for fields Sorting, aggregations Optimized memory usage Point Data Types Indexes geographic points (latitude-longitude) Proximity, bounding box queries Fast geospatial indexing"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#44-advanced-data-structures-in-lucene","title":"4.4. Advanced Data Structures in Lucene","text":"Let's dive into the point data types and spatial indexes used in Elasticsearch, especially focusing on how it handles geospatial data with geo_point fields. We\u2019ll look at how Quadtrees and R-trees work, their role in spatial indexing, and how they support geospatial queries such as bounding box and proximity searches.
Elasticsearch supports geospatial data using the geo_point and geo_shape data types: - geo_point: Stores latitude-longitude pairs for points on a map and is primarily used for proximity searches (e.g., \u201cfind locations within 10km\u201d). - geo_shape: Used for more complex shapes, such as polygons or multipoints, and is suitable for defining geographical areas like cities or lakes.
Geospatial queries include: - Bounding Box Queries: Searches for documents within a specific rectangle defined by coordinates. - Distance Queries: Searches for documents within a specified radius from a point. - Polygon Queries: Searches for documents within or intersecting with a complex polygonal area.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#442-quadtrees-and-r-trees-in-spatial-indexing","title":"4.4.2. Quadtrees and R-trees in Spatial Indexing","text":"Quadtrees and R-trees are tree-based data structures that organize spatial data by dividing the space into hierarchical grids or regions, allowing efficient geospatial query processing.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#quadtrees","title":"Quadtrees","text":"Quadtrees are hierarchical, 2-dimensional spatial indexes that recursively partition space into four quadrants or nodes, making them highly suitable for spatial data like latitude and longitude pairs.
The depth of the tree (how many times it splits) depends on data density, as each quadrant only further divides if it contains more than a specified number of points.
How it Works:
Querying: For a bounding box or radius query, the algorithm quickly eliminates entire quadrants that fall outside the search area, only scanning relevant sub-quadrants, drastically reducing search space.
Use Cases:
Example: Imagine we have a city map with thousands of restaurants, each represented as a point (latitude, longitude). - A quadtree organizes the map into quadrants based on restaurant density. Denser regions are divided further to create sub-quadrants. - To find restaurants within a specific neighborhood, the quadtree quickly filters out distant quadrants, only scanning nearby ones, significantly speeding up search.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-quadtrees","title":"Advantages of Quadtrees","text":"R-trees are another popular spatial data structure used to index multi-dimensional data (e.g., geographic shapes) by grouping nearby objects in bounding rectangles.
Each leaf node in an R-tree contains actual data points or bounding rectangles for objects (e.g., specific areas on a map).
How it Works:
Querying: For a search (e.g., finding locations within a radius), the R-tree filters bounding rectangles that don\u2019t intersect the search area, allowing the query to narrow down results without exhaustive scans.
Use Cases:
Example: Consider a map with various regions, like parks, lakes, and neighborhoods, each represented as a polygon. - An R-tree groups these polygons in bounding rectangles based on location. Polygons that are close to each other fall under the same rectangle. - When searching for parks within a 5km radius, the R-tree discards rectangles outside this range, only exploring relevant areas to find matching polygons.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#advantages-of-r-trees","title":"Advantages of R-trees","text":"While Elasticsearch doesn\u2019t directly expose Quadtrees and R-trees as configurations, Lucene, its underlying search library, utilizes versions of these structures to handle spatial indexing efficiently.
Points (like those in geo_point fields) benefit from this structure, as it speeds up bounding box queries and basic distance queries.
R-trees in Lucene:
geo_shape fields and other complex shapes where simple 2D point indexing isn\u2019t enough.Lucene optimizes these data structures to fit within its segment-based storage, allowing them to scale across multiple indices and segments, handling both large-scale geospatial queries and basic point-based distance queries.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#444-spatial-query-processing-in-elasticsearch","title":"4.4.4. Spatial Query Processing in Elasticsearch","text":"Using these data structures, Elasticsearch processes spatial queries as follows:
For a rectangular region, Elasticsearch leverages Quadtrees to restrict the search space to quadrants that intersect with the bounding box. Points or shapes within these quadrants are retrieved.
Distance Query:
For a proximity search (e.g., finding locations within 5km of a point), the Geo Distance Filter calculates distances from a central point and retrieves points from quadrants or nodes that fall within this radius.
Polygon Query:
These data structures optimize spatial queries in Elasticsearch, allowing it to handle diverse geospatial data efficiently. For example: - Bounding Box Queries are accelerated by Quadtrees, making them ideal for finding all points in a geographic area. - Distance Queries are optimized by both Quadtrees and BKD trees, allowing real-time retrieval of nearby points. - Polygon Queries are handled by R-trees, which efficiently manage irregular shapes and large polygons for accurate intersection checks.
By integrating these structures into Lucene, Elasticsearch supports powerful geospatial capabilities across various applications, including mapping services, logistics, and location-based searches.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#45-how-lucenes-structures-fit-into-elasticsearchs-query-flow","title":"4.5. How Lucene's Structures Fit into Elasticsearch\u2019s Query Flow","text":"As documents are indexed, Lucene tokenizes text fields, stores terms in the inverted index, and creates doc values for fields that require sorting or aggregation.
Segment Creation:
Documents are grouped into segments, with each segment containing its own inverted index, BKD trees, and doc values.
Query Execution:
Lucene\u2019s well-designed structures\u2014segments, inverted indexes, and multidimensional BKD trees\u2014create the foundation for Elasticsearch\u2019s speed and scalability, enabling it to support complex queries and large datasets efficiently.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#5-visual-representation-of-cluster-architecture-hierarchy-flow","title":"5. Visual Representation of Cluster Architecture (Hierarchy Flow)","text":"Elasticsearch Cluster\n\u2514\u2500\u2500 Nodes\n \u251c\u2500\u2500 Master Node\n \u2502 \u251c\u2500\u2500 Manages cluster state\n \u2502 \u2514\u2500\u2500 Handles shard allocation and rebalancing\n \u251c\u2500\u2500 Data Node\n \u2502 \u251c\u2500\u2500 Stores data and handles indexing/searching\n \u2502 \u251c\u2500\u2500 Manages primary and replica shards\n \u2502 \u2514\u2500\u2500 Processes local queries and aggregations\n \u251c\u2500\u2500 Coordinating Node\n \u2502 \u251c\u2500\u2500 Routes client requests to data nodes\n \u2502 \u251c\u2500\u2500 Aggregates responses from data nodes\n \u2502 \u2514\u2500\u2500 Sends final response to the client\n \u2514\u2500\u2500 Ingest Node\n \u251c\u2500\u2500 Processes and transforms data before indexing\n \u2514\u2500\u2500 Enriches data with pipelines\n\nIndex\n\u2514\u2500\u2500 Shards (Primary and Replica)\n \u2514\u2500\u2500 Lucene Index (Each shard is a Lucene index)\n \u251c\u2500\u2500 Segments (Immutable data units in a shard)\n \u2502 \u251c\u2500\u2500 Inverted Index\n \u2502 \u251c\u2500\u2500 Doc Values\n \u2502 \u2514\u2500\u2500 BKD Trees (for numeric & geo fields)\n \u2514\u2500\u2500 Documents (JSON objects representing data records)(within segments)\nDiving into Elasticsearch\u2019s search and other thread pools is crucial to understanding its performance and scalability. These pools are essential for managing various tasks like indexing, searching, and handling incoming requests. Let\u2019s go through these pools from end to end, covering configurations, management, and performance metrics to monitor.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-overview","title":"1. Overview","text":"GET requests for individual documents).Here\u2019s a closer look at each pool, along with common configurations and considerations:
Search Pool
threads: Generally calculated based on available CPU cores. It can be configured as # of processors * 3.queue_size: Defines the maximum number of requests that can be queued before rejecting new ones. Default is 1000.queue_size if you experience high query load, but monitor memory use carefully to avoid OutOfMemory errors.search_pool.active: Shows active threads handling search requests.search_pool.queue: Tracks pending search requests.search_pool.rejected: Indicates rejected requests due to an overloaded pool.Index Pool
threads: Typically set to the number of processors.queue_size: Default is 200. If your indexing rate is high, you may need to increase this.index_pool.active: Active threads indexing data.index_pool.queue: Number of indexing requests waiting in the queue.index_pool.rejected: Tracks rejected indexing requests.Get Pool
threads: Tied to the number of processors, generally around # of processors * 2.queue_size: Default is 1000.get_pool.active: Active threads processing GET requests.get_pool.queue: Queued retrieval requests.get_pool.rejected: Indicates if any requests are being rejected.Bulk Pool
threads: Usually set as # of processors * 2.queue_size: Often set at 50 to limit memory usage.bulk_pool.active: Active threads handling bulk operations.bulk_pool.queue: Queued bulk requests.bulk_pool.rejected: Shows if the bulk pool is rejecting requests due to overload.Management Pool
1 to 5 threads as these tasks are not frequent.management_pool.active: Active management threads.management_pool.queue: Queued management tasks.Snapshot Pool
snapshot_pool.active: Active threads handling snapshots.snapshot_pool.queue: Queued snapshot operations.node_stats.thread_pool.* via monitoring tools (like Kibana or Prometheus) helps assess each pool's workload.active: Indicates active threads. High values suggest a busy node.queue: Shows queued tasks. A consistent rise may indicate that the pool size needs tuning.rejected: Rejected tasks are a clear sign of bottlenecks.# of processors * 3 1000 search_pool.active, search_pool.queue, search_pool.rejected Increase queue_size if many queries are queued; monitor memory usage to prevent OutOfMemory issues. Index Pool Handles indexing requests for documents # of processors 200 index_pool.active, index_pool.queue, index_pool.rejected For high indexing rates, increase queue size and thread count as necessary. Get Pool Retrieves individual documents # of processors * 2 1000 get_pool.active, get_pool.queue, get_pool.rejected Increase queue_size if retrieval requests are high; monitor latency and resource usage. Bulk Pool Processes bulk indexing operations # of processors * 2 50 bulk_pool.active, bulk_pool.queue, bulk_pool.rejected Keep queue_size modest to limit memory use; monitor latency during high bulk loads. Management Pool Manages maintenance tasks like merges 1\u20135 5 management_pool.active, management_pool.queue Generally low usage; monitor only if queue is frequently non-empty, indicating background task delays. Snapshot Pool Handles snapshot creation and restoration 1\u20132 5 snapshot_pool.active, snapshot_pool.queue Schedule snapshots during low-activity periods; adjust resources if snapshots interfere with other tasks. These pools are essential to optimizing Elasticsearch\u2019s handling of diverse workloads. Monitoring and adjusting each pool based on your workload ensures better performance and resource management across the Elasticsearch cluster.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#caching","title":"Caching","text":"In Elasticsearch, caching plays a vital role in speeding up queries by storing frequently accessed data at various levels, minimizing I/O operations and improving response times. Here\u2019s a detailed, end-to-end look at caching in Elasticsearch, from the lowest level to the highest, covering each caching mechanism and its role.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-types-of-caches-in-elasticsearch","title":"1. Types of Caches in Elasticsearch","text":"There are several caching mechanisms in Elasticsearch, each working at a different level:
Each of these caches serves a specific purpose and optimizes a different aspect of query processing.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#2-filesystem-cache-os-level","title":"2. Filesystem Cache (OS-level)","text":"keyword or numeric types.indices.fielddata.cache.size setting. Default eviction strategy is based on the Least Recently Used (LRU) policy.fielddata.memory_size: Shows the amount of memory used by field data.fielddata.evictions: Tracks the number of evictions, which can indicate if your cache size is too small.term queries and range queries, but does not cache full-text queries as they are more complex and vary widely.indices.queries.cache.size setting (default is 10% of the heap).query_cache.memory_size: Memory occupied by the query cache.query_cache.evictions: Counts the number of evictions due to cache full.query_cache.hit_count and query_cache.miss_count: Useful for determining cache effectiveness.request_cache=true and doesn\u2019t cache when track_total_hits is enabled.index.requests.cache.enable (enabled by default). Also LRU-based.request_cache.memory_size: Shows the memory consumed by the request cache.request_cache.evictions: Tracks evictions in the request cache.request_cache.hit_count and request_cache.miss_count: Indicates request cache hit ratio.segment.memory_in_bytes: Memory used for cached segments.segments.count: Number of segments in the index.indices.memory and indices.evictions to track the memory usage and evictions for index metadata.query_cache and request_cache. For frequent aggregations, prioritize field data cache.fscache: The filesystem cache is essential, so configure sufficient memory for the OS to hold index files and segments, avoiding excessive I/O operations.request_cache=true in your queries or adjusting cache policies at the index level.fielddata.memory_size, fielddata.evictions Increase size for high aggregation requirements Query Cache Shard-level Caches individual filter query results LRU-based, configurable query_cache.memory_size, query_cache.evictions, query_cache.hit_count, query_cache.miss_count Monitor hit/miss ratios to determine effectiveness Request Cache Shard-level Caches entire search request results LRU-based, per-index request_cache.memory_size, request_cache.evictions, request_cache.hit_count, request_cache.miss_count Best for aggregations on static data Segment Cache Node-level Caches Lucene index segments and postings Managed by Lucene segment.memory_in_bytes, segments.count Larger heap size improves cache efficiency Indices Cache Node-level Caches index metadata (mappings, settings) LRU-based indices.memory, indices.evictions Adjust based on frequency of metadata updates Each caching layer works in tandem to optimize query speed and efficiency, and monitoring these caches is essential for fine-tuning Elasticsearch performance to meet your specific use cases.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#index-life-management-ilm","title":"Index Life Management - ILM","text":"Elasticsearch\u2019s Hot-Warm-Cold (Hot-Warm-Cold-Frozen) architecture is part of its Index Lifecycle Management (ILM) system, designed to optimize storage and cost-efficiency for managing data that has different usage patterns over time. This architecture allows you to store data on different types of nodes (hot, warm, cold, and frozen) based on data retention needs and access frequency. Here\u2019s an in-depth look at each phase and how to effectively manage data with Elasticsearch\u2019s ILM:
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-purpose-of-hot-warm-cold-architecture","title":"1. Purpose of Hot-Warm-Cold Architecture","text":"The ILM policy defines actions to transition data through different stages based on time or data access patterns:
Here\u2019s an example ILM policy to illustrate phase-based transitions for a log analytics use case:
{\n \"policy\": {\n \"phases\": {\n \"hot\": {\n \"actions\": {\n \"rollover\": {\n \"max_age\": \"7d\",\n \"max_size\": \"50gb\"\n },\n \"set_priority\": {\n \"priority\": 100\n }\n }\n },\n \"warm\": {\n \"min_age\": \"30d\",\n \"actions\": {\n \"allocate\": {\n \"number_of_replicas\": 1\n },\n \"forcemerge\": {\n \"max_num_segments\": 1\n },\n \"set_priority\": {\n \"priority\": 50\n }\n }\n },\n \"cold\": {\n \"min_age\": \"90d\",\n \"actions\": {\n \"allocate\": {\n \"require\": {\n \"data\": \"cold\"\n }\n },\n \"freeze\": {},\n \"set_priority\": {\n \"priority\": 0\n }\n }\n },\n \"delete\": {\n \"min_age\": \"365d\",\n \"actions\": {\n \"delete\": {}\n }\n }\n }\n }\n}\nThis Hot-Warm-Cold architecture in Elasticsearch enables you to balance cost, performance, and data accessibility across various hardware configurations, ensuring that data is always stored cost-effectively without compromising on necessary access patterns.
"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#list-of-rest-apis","title":"List of REST APIs","text":"API Endpoint Purpose Description Typical Latency Use Cases Index APIsPOST /{index}/_doc Index a Document Adds or updates a document in the specified index. Low (~5-10ms for small docs) Real-time data ingestion, document updates. POST /{index}/_bulk Bulk Indexing Allows indexing, updating, or deleting multiple documents in a single request. Low to Medium (~10-50ms) High-volume data ingestion, ETL processes. POST /{index}/_update/{id} Update a Document Partially updates a document by ID in the specified index. Low to Medium (~10-30ms) Updating specific fields in documents. DELETE /{index}/_doc/{id} Delete a Document Deletes a document by ID from the specified index. Low (~5-10ms) Document removal based on unique IDs. Search APIs GET /{index}/_search Search Documents Executes a search query with optional filters, aggregations, and pagination. Medium to High (~10-100ms, based on complexity) Full-text search, structured queries, analytics. POST /{index}/_search/scroll Scroll Search Enables retrieving large datasets by scrolling through search results. Medium to High (~50-200ms) Pagination for large datasets, data exports. DELETE /_search/scroll Clear Scroll Context Clears scroll contexts to free up resources. Low (~5ms) Resource management after scroll search. POST /_msearch Multi-Search Allows execution of multiple search queries in a single request. Medium to High (~20-150ms) Batch querying, dashboard visualizations. POST /{index}/_count Count Documents Counts the documents matching a query without returning full results. Low (~5-20ms) Quick counts of filtered datasets. Aggregation APIs POST /{index}/_search Aggregation Queries Used with the search API to retrieve aggregated data (e.g., histograms, averages). Medium to High (~20-150ms) Analytics, reporting, data summarization. Cluster and Node APIs GET /_cluster/health Cluster Health Returns health information on the cluster, nodes, and indices. Low (~5ms) Monitoring cluster health and node status. GET /_cluster/stats Cluster Statistics Provides statistics on cluster status, node usage, and storage. Low to Medium (~5-20ms) Cluster-wide monitoring and performance analysis. POST /_cluster/reroute Cluster Reroute Manually reroutes shards in the cluster. Medium (~20-50ms) Shard management and rebalancing. GET /_nodes/stats Node Statistics Returns stats for nodes, including CPU, memory, and thread pools. Low to Medium (~5-20ms) Node health monitoring, resource usage analysis. GET /_cat/nodes List Nodes Provides a list of all nodes in the cluster in a human-readable format. Low (~5ms) Node overview, node status. GET /_cat/indices List Indices Lists all indices with metadata on size, health, and document count. Low (~5ms) Index management and monitoring. Index Management APIs PUT /{index} Create Index Creates a new index with specific settings and mappings. Low (~10-20ms) Index setup and schema definition. DELETE /{index} Delete Index Deletes the specified index. Low (~5-10ms) Index removal, data management. PUT /{index}/_settings Update Index Settings Updates settings (e.g., refresh interval) of an index. Low (~10ms) Dynamic adjustments of index settings. POST /{index}/_refresh Refresh Index Refreshes an index to make recent changes searchable. Medium (~10-30ms) Ensures data is available for search in near-real-time. POST /{index}/_forcemerge Force Merge Index Reduces the number of segments in an index to optimize storage. High (~100ms - 1s+) Optimize index storage, improve search speed. Cache and Memory Management POST /{index}/_cache/clear Clear Index Cache Clears the cache for the specified index. Low (~5ms) Cache management for performance tuning. POST /_flush Flush Index Writes all buffered changes to disk. Medium (~10-30ms) Data durability, cache clearing. POST /{index}/_refresh Refresh Makes recent changes to an index visible to search. Medium (~10-30ms) Near real-time updates in the search index. Snapshot and Backup APIs PUT /_snapshot/{repo}/{snapshot} Create Snapshot Creates a snapshot of indices for backup. High (dependent on index size) Data backup, disaster recovery. GET /_snapshot/{repo}/{snapshot} Get Snapshot Status Checks the status of an existing snapshot. Low (~5ms) Monitor snapshot progress, status checks. DELETE /_snapshot/{repo}/{snapshot} Delete Snapshot Deletes an existing snapshot. Medium (~10-20ms) Snapshot lifecycle management, freeing storage. Security and Role Management POST /_security/role/{role} Create/Update Role Creates or updates a security role with specific permissions. Low (~5-10ms) Access control, role-based permissions. POST /_security/user/{user} Create/Update User Creates or updates a user in Elasticsearch. Low (~5-10ms) User management, access permissions. GET /_security/_authenticate Authenticate User Authenticates the current user. Low (~5ms) Session management, authentication checks."},{"location":"techdives/DistrubutedSystems/ElasticSearch/#additional-notes","title":"Additional Notes","text":"indices.fielddata.cache.size based on available heap Field data cache size impacts sorting and aggregation speed, stored in memory. Set to 20-30% of available heap for high-aggregation workloads Query Cache Size 10% of heap by default (adjustable) Caches filter queries to speed up repeated search operations. Increase for clusters with frequent repetitive queries Request Cache Size Enabled per index, LRU-based Cache complete search requests, especially useful for aggregation-heavy queries on static data. Enable on indices with frequent aggregation queries Cluster Health GET /_cluster/health (monitor green, yellow, red status) Regularly monitor cluster health to identify potential issues in shard allocation and node status. Alerts for yellow or red statuses to detect unallocated shards Pending Tasks GET /_cluster/pending_tasks Track the number of pending tasks; delays may signal node or cluster overload. Monitor to ensure tasks are processed promptly CPU Usage per Node Track via node.cpu.percent Monitor CPU load on nodes, especially data and coordinating nodes handling heavy queries. Keep below 80% for balanced performance Search Latency Monitor search.fetch_time and search.query_time Search latency metrics indicate performance bottlenecks; high values can suggest tuning is needed. Target <100ms for interactive queries, <500ms for aggregations Indexing Latency Monitor indexing.index_time Tracks indexing speed; high values indicate indexing bottlenecks. Optimize disk I/O if consistently high GC Pause Time Track jvm.gc.collection_time Excessive GC pause time (>100ms) can degrade performance, especially on data nodes. Keep heap usage <75% to avoid frequent GC pauses Disk Utilization disk.used_percent Ensure disk usage remains within optimal limits to prevent resource contention. Monitor for high usage, keep below 75% Heap Usage per Node jvm.heap_used_percent Monitor heap usage across nodes; values near 100% can trigger frequent GC and degrade performance. Keep below 75% for stable performance Shard Count per Node Shards should not exceed 50-75 per data node Optimal shard count balances memory usage and search latency. Distribute shards evenly to avoid bottlenecks Index Rollover Frequency Based on data ingestion and retention policy Use index rollover for high-ingestion use cases to manage shard size and count. Time-based or size-based rollover (e.g., daily, 10GB) Snapshot Frequency Schedule during off-peak hours Regular snapshots for backup without affecting active workloads. Daily or weekly snapshots for disaster recovery Ingest Node CPU Requirement Optimize for transformation-heavy workloads Ingest nodes need higher CPU for ETL tasks before indexing. ~8 CPUs per ingest node for transformation-heavy clusters Write Thread Pool Size Controlled via thread_pool.write.size Configure thread pool size for write-heavy workloads. Default based on available processors; increase for high-write loads Bulk Thread Pool Size Set via thread_pool.bulk.size Bulk operations often have separate thread pools, useful for high-ingestion clusters. Default based on processors; increase if high bulk ingestion Query Throughput Measure search.thread_pool.queue The search queue size indicates if the search load is too high, leading to delays. Keep queue size low to avoid bottlenecks Bulk Queue Size Monitor bulk.thread_pool.queue Large bulk queue size indicates ingestion pressure; tune for high-ingestion needs. Increase queue size or add ingest nodes for bulk-heavy workloads Network Throughput Monitor network interface utilization High network usage can impact inter-node communication, especially during replication. Ensure sufficient network bandwidth for large clusters Network Latency Track round-trip time between nodes Low-latency network is critical for distributed search and replication. <1ms recommended within data centers, 1-5ms for cross-regions"},{"location":"techdives/DistrubutedSystems/ElasticSearch/#questions","title":"Questions","text":""},{"location":"techdives/DistrubutedSystems/ElasticSearch/#1-sql-or-nosql","title":"1. SQL or NoSQL:","text":"Kafka is a distributed event streaming platform that allows applications to publish, store, and consume streams of data in real-time or batch mode. It is designed to handle continuous streams of events or records by functioning as a distributed commit log, where data is written sequentially and can be read independently by multiple consumers.
Kafka follows a publish-subscribe model where producers write data to topics divided into partitions, and consumers pull data from those partitions at their own pace. Kafka ensures that data flows reliably between producers and consumers, with fault-tolerant replication and durable storage to prevent data loss.
At its heart, Kafka provides three core functionalities: 1. Message Streaming: Enabling systems to send and receive continuous streams of data asynchronously. 2. Durable Storage: Persisting messages on disk, ensuring data is not lost even in case of failures. 3. Distributed Processing: Allowing data to be partitioned and processed across multiple servers for scalability and fault tolerance.
"},{"location":"techdives/DistrubutedSystems/Kafka/#core-components-and-keywords","title":"Core Components and Keywords","text":"Component / Keyword Description Topic A logical channel for data, used to categorize messages. Topics are divided into partitions for parallelism. Partition A segment of a topic that stores messages in a log structure. It ensures parallel processing. Each partition contains messages with offsets to track their position. Producer A client or application that publishes messages to a Kafka topic. Producers can distribute messages across partitions. Consumer A client or application that subscribes to Kafka topics and reads messages from partitions at their own pace. Broker A Kafka server that stores messages, handles requests, and coordinates with other brokers in a Kafka cluster. Kafka Cluster A group of multiple Kafka brokers working together to provide scalability and fault tolerance. Zookeeper / KRaft Zookeeper is used for metadata management and leader election in older Kafka versions. Newer versions replace Zookeeper with KRaft (Kafka Raft) for native metadata management. Offset A unique identifier for each message within a partition, representing its position in the log. Consumers use offsets to track the messages they have processed. Replication Kafka duplicates partitions across multiple brokers to ensure fault tolerance and data availability. Leader Partition The primary replica of a partition that handles all reads and writes for that partition. Other replicas act as followers. Follower Partition A copy of the leader partition that replicates its data. If the leader fails, a follower can take over. Consumer Group A group of consumers sharing the same group ID, ensuring that each partition is consumed by only one member of the group at any given time. In-Sync Replica (ISR) A replica that is up-to-date with the leader partition. Kafka promotes an ISR as the new leader if the current leader fails. Acknowledgments (ACKs) A producer configuration that defines when a message is considered successfully sent (e.g., only after being replicated to all followers). Retention Policy A configuration that determines how long Kafka retains messages before deleting or compacting them. Messages can be removed based on time or size limits. Log Compaction A process that keeps only the latest version of a key within a topic, useful for data clean-up and long-term storage. Controller A designated broker responsible for managing partition leadership and cluster rebalancing. Kafka Streams A lightweight client library used for processing and analyzing data streams directly within Kafka. Kafka Connect A framework for integrating Kafka with external systems by providing connectors for data ingestion and extraction."},{"location":"techdives/DistrubutedSystems/Kafka/#how-kafka-achieves-high-scalability","title":"How Kafka Achieves High Scalability ?","text":"Kafka\u2019s design is fundamentally scalable, allowing it to handle millions of events per second efficiently. It achieves this by leveraging distributed architecture, partitioning, horizontal scaling, and several load balancing strategies. Let\u2019s explore the mechanisms, architecture, and workflows that enable Kafka to scale end-to-end.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-partitioning-the-foundation-of-scalability","title":"1. Partitioning: The Foundation of Scalability","text":"Consumers in a consumer group split the partitions among themselves for parallel processing.
When Kafka detects new brokers, partition leadership rebalances automatically to evenly distribute the load across brokers.
Key-based partitioning: Ensuring messages with the same key always go to the same partition for order guarantees.
Kafka producers also use batching to group multiple messages into a single request, reducing network overhead and increasing throughput.
Compression: Messages can be compressed (e.g., with Gzip or Snappy) to reduce the amount of data transferred over the network.
Kafka also makes use of page cache to keep frequently accessed data in memory, further optimizing performance.
Metadata updates: Ensuring all brokers have consistent metadata.
Dynamic partition leadership reassignment ensures that even when new brokers are added or partitions are migrated, Kafka distributes workload evenly.
Kafka\u2019s design for high availability and fault tolerance centers on replication, leader election, distributed brokers, and self-healing mechanisms. Together, these mechanisms ensure Kafka can handle hardware, software, and network failures, while maintaining data integrity, durability, and continuity.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-partition-replication-foundation-of-ha-and-ft","title":"1. Partition Replication \u2013 Foundation of HA and FT","text":"This automated and rapid leader election ensures Kafka remains operational with minimal interruption.
"},{"location":"techdives/DistrubutedSystems/Kafka/#3-in-sync-replicas-isr-ensuring-data-integrity","title":"3. In-Sync Replicas (ISR) \u2013 Ensuring Data Integrity","text":"acks=all).Kafka provides acknowledgment modes for tuning data durability against performance. These settings control when a message is considered \u201csuccessfully written\u201d:
Kafka employs log compaction and retention to maintain long-term data availability:
A split-brain scenario happens when a broker loses connectivity with others, risking data inconsistency.
"},{"location":"techdives/DistrubutedSystems/Kafka/#kafkas-approach-to-preventing-split-brain","title":"Kafka\u2019s Approach to Preventing Split-Brain:","text":"This quorum-based replication prevents data corruption during network partitions.
"},{"location":"techdives/DistrubutedSystems/Kafka/#9-self-healing-and-automated-recovery","title":"9. Self-Healing and Automated Recovery","text":"Kafka\u2019s self-healing mechanisms enable it to quickly recover from broker or replica failures:
These self-healing features maintain availability and data consistency without requiring manual intervention.
"},{"location":"techdives/DistrubutedSystems/Kafka/#10-multi-datacenter-replication-for-disaster-recovery","title":"10. Multi-Datacenter Replication for Disaster Recovery","text":"Kafka supports multi-datacenter replication for cross-region fault tolerance using tools like Kafka MirrorMaker.
"},{"location":"techdives/DistrubutedSystems/Kafka/#how-multi-cluster-replication-ensures-availability-and-fault-tolerance","title":"How Multi-Cluster Replication Ensures Availability and Fault Tolerance:","text":"Kafka\u2019s producers and consumers have built-in resilience to handle failures gracefully:
3 or 5) High High Ensures multiple copies of data across brokers; if the leader fails, a follower can be promoted, maintaining data availability and durability. In-Sync Replicas (ISR) Use acks=all to ensure ISR sync before acknowledgments Moderate High Guarantees data consistency by ensuring messages are replicated to all ISR replicas before confirming writes, reducing data loss. Leader Election Mechanism Managed by Zookeeper or KRaft for automatic failover High Moderate Automatically promotes a follower to leader when the current leader fails, minimizing downtime. Controller Broker Redundancy provided by re-election if the current controller fails High Moderate Ensures the orchestrator of metadata and rebalancing has a backup, maintaining consistent cluster operations. Distributed Broker Placement Spread partitions across brokers; no two replicas on the same broker High High Reduces the risk of data unavailability and loss by preventing single points of failure. Rebalancing Strategy Configure partition reassignment for balanced broker load High Low Prevents overload on individual brokers, enhancing availability; this has less impact on data durability. Acknowledgment Policy (ACKs) Set acks=all for highest data durability Low High Ensures writes are only confirmed after replication to all ISR replicas, reducing the risk of data loss. Log Compaction Enable for compacted topics to retain latest state Moderate Moderate Retains the latest state of each key, useful for stateful applications; supports recovery but doesn\u2019t guarantee availability. Retention Policies Configure time-based or size-based retention High Low Maintains historical data for consumer recovery, contributing to high availability if consumers fall behind. Client Retry Mechanisms Configure producer and consumer retries High Low Enables producers and consumers to handle temporary broker unavailability, ensuring continuous operation. Consumer Group Rebalancing Set rebalancing policies to avoid bottlenecks High Low Ensures efficient load distribution among consumers, enhancing availability but minimally impacting data durability. Multi-Datacenter Replication Enable with Kafka MirrorMaker or similar tools High High Provides cross-region redundancy for both availability and fault tolerance, especially critical for disaster recovery. Backpressure Handling Use offset tracking and monitoring Moderate High Allows consumers to fall behind producers without causing data loss, enhancing fault tolerance by protecting against backpressure failures. Split-Brain Handling Managed by Zookeeper/KRaft to avoid conflicting writes Low High Prevents data inconsistency by ensuring only one leader exists per partition, critical for consistency in partitioned network conditions. Log Recovery Enable brokers to rebuild from log segments on restart Moderate High Ensures brokers can recover their state after a crash, minimizing data loss and ensuring continuity post-restart. Kafka\u2019s architecture for high availability and fault tolerance ensures the system remains resilient under various failure scenarios. Through partition replication, leader election, dynamic rebalancing, and multi-datacenter replication, Kafka provides a robust infrastructure with no single points of failure and near-zero downtime, making it reliable for critical real-time data streaming and processing applications.
"},{"location":"techdives/DistrubutedSystems/Kafka/#what-makes-kafka-unique","title":"What Makes Kafka Unique ?","text":"Append only Log-Based Architecture and High-Throughput with Low-Latency Design two of Kafka\u2019s core features that make it unique.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-log-based-architecture-the-foundation-of-kafkas-data-model","title":"1. Log-Based Architecture: The Foundation of Kafka\u2019s Data Model","text":"Kafka\u2019s log-based architecture is what makes it fundamentally different from traditional messaging systems. It\u2019s built on the concept of a distributed, partitioned, and immutable log, allowing Kafka to scale, preserve data ordering, and enable consumers to replay data as needed. Here\u2019s a deep dive into what this architecture entails and why it\u2019s special.
"},{"location":"techdives/DistrubutedSystems/Kafka/#how-log-based-architecture-works","title":"How Log-Based Architecture Works","text":"Offsets serve as pointers to each message\u2019s position within the log, making it easy for consumers to track their progress.
Immutable Log Storage:
This immutability provides consistency and durability, as every message has a fixed position within the partition log.
Replayable and Persistent Data:
Efficient Ordering: Kafka maintains strict ordering within each partition since messages are written sequentially. This guarantees that messages within a partition are processed in the order they were produced, which is essential for applications like transaction logs and real-time event processing.
Parallelism and Scalability: By dividing topics into multiple partitions, Kafka enables horizontal scalability. Each partition can be processed independently by separate consumers, allowing for parallel data processing and high throughput across large datasets.
Fault Tolerance and Durability: The log-based structure, combined with replication (having multiple copies of each partition on different brokers), ensures that data remains durable and available even in the event of broker failures.
Kafka\u2019s design is optimized for handling millions of events per second with minimal delay, even under heavy load. This high throughput and low latency are achieved through a combination of disk I/O optimizations, data compression, and efficient network handling. Let\u2019s explore these in detail.
"},{"location":"techdives/DistrubutedSystems/Kafka/#key-components-of-kafkas-high-throughput-low-latency-design","title":"Key Components of Kafka\u2019s High-Throughput, Low-Latency Design","text":"Sequential writes take advantage of modern disks\u2019 ability to handle high-speed sequential I/O, especially in SSDs, allowing Kafka to process large volumes of data quickly.
Page Cache Usage:
For producers writing data, Kafka batches messages in memory before flushing them to disk, improving throughput by reducing the number of disk write operations.
Zero-Copy Data Transfer:
This allows Kafka to handle network I/O with minimal CPU usage, reducing latency and increasing throughput, especially for large messages.
Batching and Compression:
Batching not only increases efficiency but also improves compression rates by reducing network overhead. Kafka supports Gzip, Snappy, and LZ4 compression, reducing data size and thus speeding up data transfer.
Network Optimization for Low Latency:
Here are the organized into individual tables based on their sections. Each table provides a summary of the key settings along with recommended values and explanations.
"},{"location":"techdives/DistrubutedSystems/Kafka/#1-broker-level-settings","title":"1. Broker-Level Settings","text":"Setting Recommended Value Purposereplication.factor 3 Ensures redundancy and fault tolerance, allowing partition recovery if a broker fails. min.insync.replicas 2 Reduces data loss risk by ensuring at least two replicas acknowledge writes. num.partitions 6 (or based on load) Balances throughput and scalability; more partitions allow greater parallel processing. log.retention.hours 168 (7 days) Controls how long messages are retained; suitable for standard processing and replay needs. log.segment.bytes 1 GB Manages the segment size for optimal disk usage and performance in log rolling. log.cleanup.policy delete or compact delete for default; compact for retaining only the latest version of each key. compression.type producer, snappy, or lz4 Saves bandwidth and improves throughput, especially under high data volume conditions."},{"location":"techdives/DistrubutedSystems/Kafka/#2-producer-level-settings","title":"2. Producer-Level Settings","text":"Setting Recommended Value Purpose acks all Ensures that all in-sync replicas acknowledge writes, increasing reliability. retries Integer.MAX_VALUE Handles transient network issues by allowing indefinite retries, preventing message loss. retry.backoff.ms 100 Introduces a pause between retries, avoiding retry flooding and improving stability. enable.idempotence true Prevents duplicate messages by ensuring exactly-once semantics in data delivery. batch.size 32 KB Enhances throughput by accumulating records into batches before sending them. linger.ms 5 Small linger time allows batches to fill, reducing network overhead without delaying sends. compression.type snappy or lz4 Compresses data to reduce payload size, saving bandwidth and reducing transfer time."},{"location":"techdives/DistrubutedSystems/Kafka/#3-consumer-level-settings","title":"3. Consumer-Level Settings","text":"Setting Recommended Value Purpose auto.offset.reset earliest Ensures consumers start reading from the beginning if no offset is committed. max.poll.records 500 Controls the batch size per poll, balancing throughput and processing time. session.timeout.ms 30,000 Provides enough time for consumers to process data without triggering unnecessary rebalances. heartbeat.interval.ms 10,000 Sets the interval for heartbeat checks within the session timeout, reducing rebalance triggers. fetch.min.bytes 1 MB Improves fetch efficiency by waiting for a minimum data size before retrieving. fetch.max.bytes 50 MB Enables large batch sizes for high-throughput consumers, reducing network calls. enable.auto.commit false Disables automatic offset commits, allowing applications to commit only after processing."},{"location":"techdives/DistrubutedSystems/Kafka/#4-cluster-level-settings-for-high-availability-and-fault-tolerance","title":"4. Cluster-Level Settings for High Availability and Fault Tolerance","text":"Setting Recommended Value Purpose min.insync.replicas 2 Ensures at least two replicas must be in sync, providing better durability. controlled.shutdown.enable true Enables controlled shutdown, allowing brokers to gracefully transition leadership. unclean.leader.election.enable false Prevents out-of-sync replicas from being elected as leaders, protecting data consistency. num.network.threads 8 Increases concurrency for network traffic, supporting high-throughput applications. num.io.threads 8 Increases I/O concurrency, allowing efficient data transfer under heavy load. num.replica.fetchers 4 Enhances replication speed by allowing multiple fetcher threads for synchronizing replicas."},{"location":"techdives/DistrubutedSystems/Kafka/#5-zookeeper-settings","title":"5. Zookeeper Settings","text":"Setting Recommended Value Purpose zookeeper.session.timeout.ms 18,000 Prevents frequent Zookeeper disconnections during high loads, stabilizing metadata handling. zookeeper.connection.timeout.ms 6,000 Ensures reliable connections to Zookeeper, reducing the likelihood of leader election issues."},{"location":"techdives/DistrubutedSystems/Kafka/#6-kraft-kafka-raft-settings","title":"6. KRaft (Kafka Raft) Settings","text":"Setting Recommended Value Purpose process.roles broker,controller Defines the roles of Kafka nodes. A KRaft cluster typically has combined broker and controller roles, but can be split if desired. controller.quorum.voters List of controllers Specifies the list of controller nodes in the form nodeID@hostname:port, where each entry represents a voter in the Raft consensus group. controller.listener.names CONTROLLER Designates the listener name for inter-controller communication in the Raft quorum. controller.heartbeat.interval.ms 2,000 Sets the interval between heartbeats for controller nodes, ensuring they stay connected and responsive within the Raft quorum. controller.metrics.sample.window.ms 30,000 Configures the window size for collecting metrics, helping to monitor Raft performance over time. controller.log.dirs /path/to/controller/logs Specifies the directory where controller logs are stored. It\u2019s best to use a dedicated disk for controller logs to avoid I/O contention with brokers. metadata.log.segment.bytes 1 GB Controls the segment size for metadata logs, managing disk usage and log rolling frequency for metadata in KRaft mode. metadata.log.retention.bytes -1 (unlimited) Configures metadata log retention based on disk space, allowing infinite retention by default. Adjust based on available storage. metadata.log.retention.ms 604,800,000 (7 days) Retains metadata for a set duration; typically configured for a week to enable rollback in case of issues. controller.socket.timeout.ms 30,000 Sets the timeout for controller-to-controller connections, ensuring stability during network issues. leader.imbalance.check.interval.seconds 300 (5 minutes) Defines the interval at which the controller checks for leader imbalance, helping to maintain even load distribution across brokers. Reference Links:
https://www.hellointerview.com/learn/system-design/deep-dives/kafka
"},{"location":"techdives/DistrubutedSystems/Redis/","title":"Redis","text":"Redis is an open-source, in-memory data structure store that serves as a database, cache, message broker, and streaming engine. Its versatility and high performance make it a popular choice for various applications. This comprehensive guide delves into Redis's architecture, data structures, commands, deployment strategies, and best practices.
"},{"location":"techdives/DistrubutedSystems/Redis/#1-introduction-to-redis","title":"1. Introduction to Redis","text":"Redis, short for Remote Dictionary Server, is renowned for its speed and flexibility. Operating primarily in memory, it supports various data structures and offers features like replication, persistence, and high availability.
"},{"location":"techdives/DistrubutedSystems/Redis/#2-core-data-structures","title":"2. Core Data Structures","text":""},{"location":"techdives/DistrubutedSystems/Redis/#21-strings","title":"2.1. Strings","text":"Definition: A string in Redis is a binary-safe sequence of bytes. It's the most basic data type in Redis, with a maximum size of 512 MB.
Examples: - Set and get a simple string:
SET key \"Hello, World!\"\nGET key\nSET counter 10\nINCR counter // Result: 11\nINCRBY counter 5 // Result: 16\nUnderlying Data Structure: Dynamic String (SDS - Simple Dynamic String)
Time Complexity: - SET key value: O(1) - GET key: O(1) - INCR key: O(1)
Best Use Cases: - Caching: Store frequently accessed values. - Counters: Track counts for metrics or events. - Session Data: Store serialized JSON for user sessions.
"},{"location":"techdives/DistrubutedSystems/Redis/#22-hashes","title":"2.2. Hashes","text":"Definition: A hash in Redis is a collection of field-value pairs, ideal for representing objects (e.g., user profiles).
Examples: - Creating and managing user data in a hash:
HSET user:1001 name \"Alice\" age 30 city \"New York\"\nHGET user:1001 name // Returns \"Alice\"\nHGETALL user:1001 // Returns all key-value pairs in hash\nUnderlying Data Structure: Hash Table or ZipList (for small hashes)
Time Complexity: - HSET key field value: O(1) (amortized) - HGET key field: O(1) - HGETALL key: O(N) (N being the number of fields in the hash)
Best Use Cases: - Storing Objects: Represent complex entities. - Configuration Settings: Store configurations as a set of key-value pairs.
"},{"location":"techdives/DistrubutedSystems/Redis/#23-lists","title":"2.3. Lists","text":"Definition: A list is an ordered collection of strings, allowing elements to be added at either end, functioning much like a linked list.
Examples: - Using a list to store recent activity:
LPUSH recent_activity \"login\" \"view_profile\" \"logout\"\nLRANGE recent_activity 0 -1 // Fetches all elements in the list\nUnderlying Data Structure: Linked List or QuickList (optimized for performance and memory usage)
Time Complexity: - LPUSH key value: O(1) - LRANGE key start stop: O(S+N) (S being the starting offset and N the number of elements retrieved)
Best Use Cases: - Activity Streams: Store recent actions or logs. - Task Queues: Implement FIFO or LIFO queues.
"},{"location":"techdives/DistrubutedSystems/Redis/#24-sets","title":"2.4. Sets","text":"Definition: Sets are unordered collections of unique strings, ideal for performing set operations.
Examples: - Managing unique tags:
SADD tags \"redis\" \"database\" \"in-memory\"\nSMEMBERS tags // Returns all unique tags\nUnderlying Data Structure: Hash Table or IntSet (for small sets of integers)
Time Complexity: - SADD key value: O(1) - SMEMBERS key: O(N) (N being the number of elements in the set) - SINTER key1 key2 ... keyN: O(N*M) (N being the number of sets and M the smallest set)
Best Use Cases: - Unique Values: Track unique items like IP addresses. - Social Networks: Represent social relationships (e.g., friends, followers).
"},{"location":"techdives/DistrubutedSystems/Redis/#25-sorted-sets","title":"2.5. Sorted Sets","text":"Definition: Similar to sets, but with an associated score that allows elements to be sorted by score.
Examples: - Storing leaderboard data:
ZADD leaderboard 100 \"Alice\" 200 \"Bob\"\nZRANGE leaderboard 0 -1 WITHSCORES\nUnderlying Data Structure: Skip List and Hash Table (for fast access and sorted ordering)
Time Complexity: - ZADD key score member: O(log(N)) - ZRANGE key start stop: O(log(N)+M) (M being the number of elements returned)
Best Use Cases: - Leaderboards: Rank users based on scores. - Event Prioritization: Sort items by priority or timestamp.
"},{"location":"techdives/DistrubutedSystems/Redis/#26-bitmaps","title":"2.6. Bitmaps","text":"Definition: Bitmaps use strings to store and manipulate individual bits, offering efficient binary storage.
Examples: - Tracking user flags:
SETBIT user_flags 5 1 // Sets the 6th bit to 1\nGETBIT user_flags 5 // Returns 1\nUnderlying Data Structure: String (each bit is set or retrieved from the byte representation)
Time Complexity: - SETBIT key offset value: O(1) - GETBIT key offset: O(1) - BITCOUNT key: O(N) (N being the length of the string)
Best Use Cases: - Feature Flags: Toggle features for users. - Activity Tracking: Record binary states like presence or attendance.
"},{"location":"techdives/DistrubutedSystems/Redis/#27-hyperloglogs","title":"2.7. HyperLogLogs","text":"Definition: HyperLogLog is a probabilistic structure for approximating unique element counts.
Examples: - Counting unique visitors:
PFADD visitors \"user1\" \"user2\"\nPFCOUNT visitors // Returns approximate unique count\nUnderlying Data Structure: Sparse and Dense Data Representations (optimized for low memory usage)
Time Complexity: - PFADD key element: O(1) - PFCOUNT key: O(1)
Best Use Cases: - Unique Counting: Approximate counts of unique views or visitors. - Low-Memory Use: Ideal for large datasets with memory constraints.
"},{"location":"techdives/DistrubutedSystems/Redis/#28-streams","title":"2.8. Streams","text":"Definition: A stream is a log-like data structure for managing continuous data flows, supporting consumer groups.
Examples: - Tracking event streams:
XADD mystream * name \"Alice\" action \"login\"\nXREAD COUNT 2 STREAMS mystream 0\nUnderlying Data Structure: Radix Tree (used for efficient storage and traversal of stream entries)
Time Complexity: - XADD key * field value: O(log(N)) (N being the number of items in the stream) - XREAD key start stop: O(log(N)+M) (M being the number of items returned)
Best Use Cases: - Event Sourcing: Track ordered events or logs. - Message Queues: Reliable message distribution with consumer groups.
"},{"location":"techdives/DistrubutedSystems/Redis/#29-geospatial-indexes","title":"2.9. Geospatial Indexes","text":"Definition: Redis provides commands for storing and querying location data with latitude and longitude.
Examples: - Adding and querying locations:
GEOADD cities 13.361389 38.115556 \"Palermo\"\nGEORADIUS cities 15 37 200 km\nUnderlying Data Structure: Geohash with Sorted Sets (uses sorted sets for indexing)
Time Complexity: - GEOADD key longitude latitude member: O(log(N)) - GEORADIUS key longitude latitude radius: O(log(N)+M) (M being the number of results)
Best Use Cases: - Location-Based Services: Search and display nearby locations. - Geofencing: Detect whether users enter specific geographic zones.
"},{"location":"techdives/DistrubutedSystems/Redis/#3-commands-table","title":"3. Commands Table","text":"Data Structure Definition Example Commands Best Use Cases Time Complexity Strings Binary-safe sequences of bytes for text or binary data.SET key value, GET key, INCR key, DECR key, APPEND key value, STRLEN key Caching, counters, session data SET: O(1), GET: O(1), INCR: O(1), APPEND: O(N) Hashes Collection of key-value pairs, suitable for objects. HSET key field value, HGET key field, HGETALL key, HDEL key field, HLEN key Storing objects, configuration settings HSET: O(1), HGET: O(1), HGETALL: O(N), HLEN: O(1) Lists Ordered collection of strings, acts like a linked list. LPUSH key value, RPUSH key value, LPOP key, RPOP key, LRANGE key start stop, LLEN key Activity streams, task queues LPUSH: O(1), LRANGE: O(S+N), LLEN: O(1) Sets Unordered collections of unique strings, optimized for sets. SADD key value, SREM key value, SMEMBERS key, SISMEMBER key value, SUNION key1 key2, SINTER key1 key2 Unique values, social relationships SADD: O(1), SMEMBERS: O(N), SINTER: O(N*M) Sorted Sets Sets with scores, allowing elements to be sorted by score. ZADD key score member, ZRANGE key start stop WITHSCORES, ZREM key member, ZSCORE key member, ZREVRANGE key start stop, ZCOUNT key min max Leaderboards, event prioritization ZADD: O(log(N)), ZRANGE: O(log(N)+M), ZSCORE: O(1) Bitmaps Stores and manipulates bits in a binary-safe string. SETBIT key offset value, GETBIT key offset, BITCOUNT key, BITOP operation destkey key1 key2 Feature flags, activity tracking SETBIT: O(1), GETBIT: O(1), BITCOUNT: O(N) HyperLogLogs Probabilistic structure for approximate unique counts. PFADD key element, PFCOUNT key, PFMERGE destkey sourcekey1 sourcekey2 Unique counting, low-memory usage PFADD: O(1), PFCOUNT: O(1), PFMERGE: O(N) Streams Log-like structure for managing continuous data flows. XADD key * field value, XREAD COUNT n STREAMS key, XGROUP CREATE key group consumer_id, XACK key group message_id, XDEL key message_id, XINFO key, XLEN key, XTRIM key MAXLEN ~ count Event sourcing, message queues XADD: O(log(N)), XREAD: O(log(N)+M), XGROUP: O(1), XACK: O(1) Geospatial Indexes Stores and queries location data with latitude and longitude. GEOADD key longitude latitude member, GEODIST key member1 member2, GEORADIUS key longitude latitude radius m km, GEORADIUSBYMEMBER key member radius m km, GEOHASH key member Location-based services, geofencing GEOADD: O(log(N)), GEORADIUS: O(log(N)+M)"},{"location":"techdives/DistrubutedSystems/Redis/#4-persistence-and-durability","title":"4. Persistence and Durability","text":"Redis operates primarily as an in-memory database, prioritizing speed and low-latency operations. However, it provides two main persistence mechanisms to ensure data durability:
Advantages: Lower I/O overhead, compact file size.
Disadvantages: Risk of data loss between snapshots if Redis crashes.
SET or INCR) received by the server, appending each to a file. This approach provides a more durable form of persistence because every operation is logged in real-time or semi-real-time.Advantages: Better durability, logs every operation for more frequent data persistence.
Disadvantages: Larger file sizes, higher I/O usage.
Choosing Between RDB and AOF: You can use either or both of these methods in combination based on your application needs. For example, using both allows for rapid recovery (RDB) with high durability (AOF).
"},{"location":"techdives/DistrubutedSystems/Redis/#5-replication-and-high-availability","title":"5. Replication and High Availability","text":"Redis supports replication to create replicas of the primary (master) instance, enabling multiple read replicas and providing redundancy.
Master-Replica Replication: A master Redis instance can replicate data to any number of replica instances, which can then serve read-only queries. This setup enhances scalability by spreading read requests across replicas and ensuring data availability in case the primary instance fails.
Redis Sentinel: Redis Sentinel is a high-availability solution that monitors Redis instances. It automatically promotes one of the replicas to the master role if the current master goes down, providing automatic failover and notifications. Sentinel also provides monitoring and notifications.
Best for: - Applications requiring high availability with automatic failover. - Scenarios where read-heavy workloads benefit from scaling reads across multiple replicas.
"},{"location":"techdives/DistrubutedSystems/Redis/#6-clustering-and-scalability","title":"6. Clustering and Scalability","text":"Redis supports sharding through Redis Cluster, which enables data partitioning across multiple nodes, allowing horizontal scalability and distributed storage. Redis Cluster uses hash slots to determine data distribution across nodes, ensuring no single node contains the entire dataset.
Hash Slot Mechanism: Redis Cluster divides the keyspace into 16,384 hash slots, assigning a subset of these slots to each node. Keys are then hashed into these slots, which determines their storage node.
Hash Slot Mapping: Redis Cluster uses 16,384 slots as a way to divide the keyspace. Every key is assigned to a specific slot by applying a hashing function (called CRC16) that maps the key to one of these 16,384 slots.
Multiple Keys per Slot: Each hash slot can contain many keys. This means you can store millions of keys, not just 16,384.
Distribution Across Nodes: In a Redis Cluster, each node is responsible for a subset of the 16,384 slots. For example:
Redis assigns keys to slots, and the slot location determines which node will store the key.
Dynamic Scaling: When more nodes are added to the cluster, the slots are rebalanced, and some slots are reassigned to the new nodes. Redis automatically adjusts the key distribution to accommodate the additional nodes, enabling the cluster to scale horizontally without needing to rewrite keys.
Automatic Failover: Redis Cluster supports failover, ensuring high availability by electing a new primary if a node fails.
Considerations: - Redis Cluster supports most single-key commands, but multi-key operations are restricted unless all keys map to the same slot. - Clustering can increase complexity in handling data operations across nodes but is essential for large datasets needing horizontal scalability.
"},{"location":"techdives/DistrubutedSystems/Redis/#7-security-considerations","title":"7. Security Considerations","text":"While Redis is often deployed in secure, private networks, security remains essential, especially for production environments:
AUTH command. Although not enabled by default, setting up AUTH is recommended for production instances.bind 127.0.0.1) and accessed through trusted networks or VPNs, especially if running on public cloud services. Redis supports setting bind and requirepass configuration directives.Redis has a robust ecosystem with libraries for popular programming languages, making it easy to integrate Redis across platforms:
redis-pyJedis, Lettuceioredis, node-redisgo-redisAdditionally, Redis CLI and Redis Insight are commonly used tools for managing and monitoring Redis instances.
"},{"location":"techdives/DistrubutedSystems/Redis/#9-rediss-single-threaded-nature-and-atomic-operations","title":"9. Redis\u2019s Single-Threaded Nature and Atomic Operations","text":"Redis uses a single-threaded event loop to handle client requests, which makes operations simpler and efficient. This single-threaded model has specific implications:
Atomic Operations: All Redis commands are atomic because Redis processes each command sequentially. Even if multiple clients issue commands simultaneously, Redis processes one command at a time. This ensures that operations like INCR or multi-step transactions are inherently thread-safe and don't require locks.
Efficiency: Since Redis is single-threaded, it avoids the complexity of thread management and the overhead of context switching, which helps maintain its high-speed performance. Redis\u2019s speed is due to efficient data structures, in-memory storage, and minimal CPU usage.
Impact on Use Cases: - In scenarios where atomicity is essential, such as counters or distributed locks, Redis's single-threaded nature provides strong consistency guarantees. - For CPU-bound workloads, Redis may be limited by its single-threaded design. However, since Redis is primarily I/O-bound, it scales well for read-heavy or network-intensive applications.
Benefits of Single-Threaded Execution: - Simplicity in design and implementation, eliminating race conditions. - Predictable performance with guaranteed atomicity.
Drawbacks: - Limited to single-core processing for request handling. For CPU-bound tasks, Redis's single-threading may become a bottleneck, but horizontal scaling (e.g., Redis Cluster) can help distribute the load.
"},{"location":"techdives/DistrubutedSystems/Redis/#10-approximate-requests-per-second-rps","title":"10. Approximate Requests Per Second (RPS)","text":"Operation Type Description Approximate RPS per Instance Notes Simple Reads (GET) Basic read operation for retrieving a single value 100,000 - 150,000 RPS Higher performance achievable on optimized hardware Simple Writes (SET) Basic write operation for setting a single key-value pair 100,000 - 150,000 RPS Slightly reduced if using AOF with always persistence Complex Reads (e.g., ZRANGE) Reads on complex data structures like sorted sets 50,000 - 80,000 RPS Lower due to additional computation and memory access Complex Writes (e.g., ZADD) Writes on complex data structures like sorted sets 50,000 - 80,000 RPS Additional processing to maintain sorted order impacts performance With AOF (Append-Only File) Writes with always mode persistence (AOF) 60,000 - 80,000 RPS Slightly reduced due to disk I/O overhead Snapshotting (RDB) Writes with periodic snapshots (RDB) 80,000 - 100,000 RPS Minimal impact on RPS except during snapshotting periods when CPU/I/O load is higher With Redis Cluster Distributed across multiple nodes Millions of RPS (scales with nodes) Redis Cluster allows horizontal scaling, increasing RPS proportionally with additional nodes"},{"location":"techdives/DistrubutedSystems/Redis/#notes","title":"Notes:","text":"Redis is highly effective as a caching layer, providing extremely low-latency data retrieval that reduces the load on backend databases and improves application performance.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works","title":"How It Works","text":"Configuration settings
Expiration and Eviction: Redis supports configurable expiration for keys, which allows cached data to expire after a specific time. It also supports eviction policies (like LRU or LFU) to automatically remove older or less-used items when memory limits are reached.
Set Up Caching Logic: In your application, implement caching logic where data is first checked in Redis. If the key doesn\u2019t exist in Redis (cache miss), retrieve it from the database, store it in Redis, and serve it to the client.
if not redis.exists(\"user:123\"):\n data = fetch_from_database(\"user:123\")\n redis.setex(\"user:123\", 3600, data) # Cache for 1 hour\nExample Commands:
SETEX key value TTL # Sets a value with an expiration time (TTL)\nGET key # Retrieves value\nDEL key # Deletes a cached key\nRedis is commonly used as a session store for web applications, especially in distributed environments where sharing session data across multiple servers is critical.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_1","title":"How It Works","text":"Set Up Session Key: For each user session, generate a unique session ID and store session data in Redis.
HSET session:abc123 user_id 456 name \"Alice\"\nEXPIRE session:abc123 1800 # Session expires after 30 minutes\nExample Commands:
HSET session:<session_id> field value # Sets session data as a hash\nHGETALL session:<session_id> # Retrieves all session data\nEXPIRE session:<session_id> ttl # Sets expiration for session\nRedis\u2019s support for data structures like HyperLogLogs, sorted sets, and streams enables it to handle real-time analytics, tracking metrics, counts, and trends without requiring a traditional database.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_2","title":"How It Works","text":"HyperLogLog for Unique Visitors:
PFADD visitors:today user123 # Adds a unique user\nPFCOUNT visitors:today # Gets approximate unique visitor count\nSorted Set for Leaderboards:
ZADD leaderboard 1000 user123 # Adds user with score\nZREVRANGE leaderboard 0 10 WITHSCORES # Retrieves top 10 users\nStream for Event Tracking:
XADD events * type \"click\" user \"user123\" # Adds event to stream\nXREAD COUNT 10 STREAMS events 0 # Reads 10 most recent events\nRedis\u2019s publish/subscribe (pub/sub) feature allows it to act as a lightweight message broker, facilitating real-time communication between distributed applications.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_3","title":"How It Works","text":"Set Up Publisher:
PUBLISH channel_name \"New user registered\" # Publishes a message to a channel\nSet Up Subscriber:
SUBSCRIBE channel_name # Subscribes to a channel\nPublish a message:
PUBLISH notifications \"User123 has logged in\"\nSubscribe to channel:
SUBSCRIBE notifications\nRedis provides geospatial commands that make it suitable for applications requiring location-based searches and geofencing, such as ride-sharing or delivery tracking.
Redis uses a geohashing-like approach to handle geospatial data, but it combines it with sorted sets to enable efficient location-based queries, Redis converts latitude and longitude coordinates into a geohash-like value, which is then stored as a score in a sorted set. This encoding allows Redis to store location data compactly and enables efficient proximity queries.
"},{"location":"techdives/DistrubutedSystems/Redis/#how-it-works_4","title":"How It Works","text":"GEOADD to add locations with latitude, longitude, and an associated member (e.g., user ID or landmark).Store Locations:
GEOADD cities 13.361389 38.115556 \"Palermo\"\nGEOADD cities 15.087269 37.502669 \"Catania\"\nRetrieve Locations in Radius:
GEORADIUS cities 15 37 100 km # Finds cities within 100 km of coordinates (15, 37)\nGEORADIUSBYMEMBER cities \"Palermo\" 200 km # Finds locations within 200 km of \"Palermo\"\nAdd a Location:
GEOADD locations 40.7128 -74.0060 \"New York\"\nFind Nearby Locations:
GEORADIUS locations -74.0060 40.7128 50 km\nSETEX, GET, DEL Reduces latency, lowers database load Session Management Store user sessions for distributed web applications. HSET, HGETALL, EXPIRE Fast, centralized session access across servers Real-Time Analytics Track metrics, counts, and trends in real time. PFADD, PFCOUNT, ZADD, XADD, XREAD Provides instant insights, reduces need for dedicated platforms Message Brokering Facilitate real-time communication between services. PUBLISH, SUBSCRIBE Real-time updates, lightweight message broker Geospatial Apps Perform location-based searches and calculations. GEOADD, GEORADIUS, GEORADIUSBYMEMBER Efficient geospatial operations for location-based services"},{"location":"techdives/DistrubutedSystems/Redis/#12-redis-issues","title":"12. Redis Issues","text":"Let's dive deep into some key challenges, such as hot key issues, cache avalanche, cache penetration, cache stampede, and their corresponding solutions.
"},{"location":"techdives/DistrubutedSystems/Redis/#121-hot-key-issue","title":"12.1. Hot Key Issue","text":""},{"location":"techdives/DistrubutedSystems/Redis/#description","title":"Description","text":"A hot key issue occurs when a single key in Redis is accessed extremely frequently, causing uneven load distribution. This can happen in applications where certain data (e.g., a trending topic or popular product) is heavily requested. A hot key can overwhelm the Redis server or specific nodes in a Redis Cluster, leading to latency spikes and reduced performance.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes","title":"Causes","text":"Store multiple copies of the hot key in Redis (e.g., hotkey_1, hotkey_2, hotkey_3). Then, use application logic to randomly pick a replica each time the key is accessed. This distributes the load across multiple keys.
Use Redis Cluster:
In a Redis Cluster, distribute the load by sharding hot keys across nodes. This may not completely eliminate the issue, but it can help mitigate its impact by spreading access across the cluster.
Client-Side Caching:
Implement a local cache on the client side or within the application servers to reduce the frequency of requests to Redis. This technique works well when the data is static or changes infrequently.
Use a Load-Balancing Proxy:
A cache avalanche occurs when many cache keys expire at once, leading to a sudden flood of requests to the backend database as the cache misses accumulate. This can overwhelm the database, causing latency spikes or even outages.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_1","title":"Causes","text":"Set expiration times with a randomized offset (e.g., add a few seconds or minutes randomly) to avoid simultaneous expiry. For example:
ttl = 3600 + random.randint(-300, 300) # 3600 seconds +/- 5 minutes\nCache Pre-Warming:
Preload critical data into Redis before it expires. You can use background jobs to check key expiration and refresh data periodically.
Lazy Loading with Synchronized Locking:
Use a distributed locking mechanism to ensure that only one thread refreshes the data in Redis, while others wait. This can prevent multiple processes from overloading the backend database.
Fallback Graceful Degradation:
Cache penetration happens when requests for non-existent keys repeatedly bypass the cache and go to the backend database. Since these keys don\u2019t exist, they are never cached, resulting in continuous database requests, increasing the load on the database.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_2","title":"Causes","text":"When a request results in a database miss, store a null value in Redis with a short TTL (e.g., 5 minutes). Future requests for the same key will hit Redis instead of the database. Example:
if not redis.exists(\"non_existent_key\"):\n data = fetch_from_database(\"non_existent_key\")\n if data is None:\n redis.setex(\"non_existent_key\", 300, None) # Cache null for 5 minutes\nInput Validation:
Filter out clearly invalid requests before querying Redis or the backend. For instance, if certain key patterns are obviously invalid, ignore them early in the request flow.
Bloom Filter:
A cache stampede occurs when multiple threads or clients attempt to update an expired cache key simultaneously, causing a burst of requests to the backend database. This is similar to a cache avalanche but occurs at the key level rather than across all keys.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_3","title":"Causes","text":"Use a distributed lock (e.g., Redlock) to ensure that only one client refreshes the cache while others wait. This reduces the load on the database:
# Pseudocode for acquiring a lock\nif redis.setnx(\"lock:key\", 1):\n try:\n # Fetch and cache the data\n data = fetch_from_database(\"key\")\n redis.setex(\"key\", 3600, data)\n finally:\n redis.delete(\"lock:key\") # Release the lock\nEarly Re-Caching (Soft Expiration):
Implement soft expiration by setting a short expiration on frequently requested keys and refreshing them asynchronously before they expire. This keeps the data fresh in Redis and avoids a stampede.
Leverage Stale Data:
Redis operates in memory, so it has a limited capacity. When Redis reaches its memory limit, it must evict keys to free up space, potentially removing critical data. Improper eviction policies can lead to cache churn and data inconsistency.
"},{"location":"techdives/DistrubutedSystems/Redis/#causes_4","title":"Causes","text":"Redis offers multiple eviction policies (noeviction, allkeys-lru, volatile-lru, allkeys-lfu, etc.). Choose one that matches your data access patterns. For instance:
Optimize Data Size:
Reduce the memory footprint by optimizing data storage, such as using shorter key names or serializing data efficiently (e.g., storing integers directly rather than as strings).
Monitor and Scale:
Continuously monitor Redis memory usage with tools like Redis CLI or Redis Insights. If memory usage grows, consider horizontal scaling with Redis Cluster.
Use Redis as a Pure Cache:
Redis is designed for fast access, but certain operations can cause high latency, especially when handling large datasets or complex commands like ZRANGE on large sorted sets.
Avoid commands that can block or are computationally expensive. For example, break large list processing into smaller ranges instead of processing the entire list.
Monitor Slow Queries:
Use the Redis Slow Log to identify and optimize slow commands. Redis provides insights into commands that exceed a specified execution time threshold.
Use Sharding or Clustering:
allkeys-lru, allkeys-lfu, etc.). - Compress data (e.g., shorter key names). - Use Redis MEMORY USAGE to check memory footprint of specific keys. INFO memory, MEMORY USAGE CPU Utilization CPU load on the Redis server, indicating overall processing load. - Reduce CPU-intensive commands (ZRANGE on large sets, large LRANGE operations). - Offload tasks to background or batch processing if possible. - Use pipelining for batch operations. System tools (e.g., top), INFO cpu Cache Hit Ratio The ratio of cache hits to total cache requests (ideally close to 1). - Identify hot keys and cache them effectively. - Increase Redis memory if hit ratio is low due to evictions. - Ensure sufficient TTL to avoid frequent cache misses. INFO stats Evicted Keys Number of keys evicted due to memory limits. - Increase available memory if eviction is high. - Choose an appropriate eviction policy (allkeys-lru, volatile-ttl, etc.). - Adjust key TTLs to prevent frequent eviction of important data. INFO memory Connected Clients Number of clients connected to Redis at any given time. - Increase the maxclients configuration if reaching limits. - Use client-side caching to reduce load on Redis. INFO clients, CLIENT LIST Latency (Command Time) Measures the average response time per command in milliseconds. - Avoid using blocking or heavy commands on large data sets. - Distribute large data across a Redis Cluster. - Monitor slow log for commands that exceed expected time. SLOWLOG GET, INFO commandstats Command Rate Rate of commands per second, which affects overall performance. - Spread load across multiple Redis instances if command rate is high. - Use pipelining to reduce round-trips. - Optimize or reduce the frequency of unnecessary commands. INFO stats Key Expirations Number of keys that expire per second. - Add randomized TTLs to prevent cache avalanches. - Pre-warm critical keys to avoid sudden cache misses. - Monitor TTL settings to ensure balanced expiration. INFO stats Replication Lag Delay in data synchronization between master and replica nodes. - Tune repl-backlog-size for better sync reliability. - Monitor network latency and throughput between master and replica. - Use Redis Sentinel for reliable failover. INFO replication, REPLCONF Data Persistence Durability How frequently Redis saves data to disk (AOF/RDB). - Use RDB for infrequent snapshots; use AOF for higher durability. - Tune AOF rewrite frequency (auto-aof-rewrite-percentage). - Adjust RDB save intervals based on data criticality. CONFIG SET save, CONFIG SET appendonly Keyspace Misses Number of attempts to access non-existent keys. - Cache null values temporarily for non-existent keys to reduce misses. - Add input validation to filter invalid requests. - Use Bloom filters for non-existent keys in high-traffic systems. INFO stats, MEMORY USAGE Redis Slow Log Logs slow-running commands that exceed a threshold. - Use SLOWLOG to monitor commands that exceed time limits. - Adjust commands and optimize keys based on slow log findings. - Tune slowlog-log-slower-than to track performance bottlenecks. SLOWLOG GET, SLOWLOG RESET Network Bandwidth Measures bandwidth usage, impacting latency and speed. - Use Redis clustering to reduce network load on a single instance. - Enable pipelining and compression where possible. - Monitor and minimize network latency for high-frequency queries. System tools (e.g., ifconfig), INFO Eviction Policy Determines which keys Redis evicts first when memory limit is reached. - Choose policies based on use case (allkeys-lru, allkeys-lfu for caching, volatile-ttl for expiring keys first). - Regularly review and adjust TTLs for key eviction optimization. CONFIG SET maxmemory-policy, INFO memory Persistence Overhead Memory and CPU impact due to persistence settings (RDB or AOF). - Adjust save intervals or AOF rewriting to reduce persistence load. - Use a combination of AOF and RDB if the application requires high durability with performance. INFO persistence, CONFIG SET save Cluster Slot Utilization Measures how well data is balanced across Redis Cluster slots. - Rebalance slots if certain nodes handle disproportionate load. - Use Redis Cluster sharding to ensure balanced key distribution. - Regularly monitor slots and reshard as needed. CLUSTER INFO, CLUSTER NODES, CLUSTER REBALANCE"},{"location":"techdives/DistrubutedSystems/Redis/#14-best-practices","title":"14. Best Practices","text":"To make the most of Redis:
LRU, LFU, etc.) to manage memory limits. Redis provides a range of policies for automatically removing old keys.Here\u2019s a structured Q&A-style deep dive into Redis to address all of these technical aspects.
"},{"location":"techdives/DistrubutedSystems/Redis/#1-sql-or-nosql-if-nosql-what-type-of-nosql","title":"1. SQL or NoSQL? If NoSQL, what type of NoSQL?","text":"Q: Is Redis an SQL or NoSQL database?
A: Redis is a NoSQL database. Specifically, it is a key-value store that supports various data structures (e.g., strings, hashes, lists, sets, sorted sets, streams, bitmaps, and geospatial indexes).
"},{"location":"techdives/DistrubutedSystems/Redis/#2-type-of-db-supports-polymorphic","title":"2. Type of DB \u2026 Supports Polymorphic?","text":"Q: What type of NoSQL database is Redis, and does it support polymorphism?
A: Redis is a key-value in-memory data store with support for a variety of data structures. Redis does not natively support polymorphic types in the way that document-based NoSQL databases do, but you can achieve some level of polymorphism by encoding data in a structured way (e.g., JSON or hash maps).
"},{"location":"techdives/DistrubutedSystems/Redis/#3-main-feature-db-built-for-and-who-built-it-and-on-what","title":"3. Main Feature, DB Built For, and Who Built It and on What","text":"Q: What was Redis built for, who built it, and what are its main features?
A: Redis was initially created by Salvatore Sanfilippo as a high-performance in-memory database for use cases requiring low-latency, real-time data processing. Redis is built on C, and its main features include in-memory storage, data persistence, flexible data structures, and capabilities for caching, messaging, and real-time analytics.
"},{"location":"techdives/DistrubutedSystems/Redis/#4-olap-or-oltp-does-it-support-acid-or-base","title":"4. OLAP or OLTP? Does it support ACID or BASE?","text":"Q: Is Redis OLAP or OLTP, and does it adhere to ACID or BASE properties?
A: Redis is generally used in OLTP (Online Transaction Processing) scenarios due to its low-latency and high-throughput design. Redis does not natively support full ACID properties but can achieve atomic operations within individual commands due to its single-threaded nature. It follows the BASE (Basically Available, Soft state, Eventual consistency) model.
"},{"location":"techdives/DistrubutedSystems/Redis/#5-cap-theorem-where-does-redis-fall","title":"5. CAP Theorem \u2013 Where does Redis fall?","text":"Q: How does Redis align with the CAP theorem, and what does each part (Consistency, Availability, Partition Tolerance) mean?
A: Redis, especially in a clustered setup, adheres to the CP (Consistency and Partition Tolerance) model of the CAP theorem. In a non-clustered single-instance setup, Redis is highly consistent. However, in a clustered setup, it sacrifices some availability for consistency.
Time Consistency in Redis can be achieved with strict persistence settings and synchronous replication.
"},{"location":"techdives/DistrubutedSystems/Redis/#6-cluster-structure-from-cluster-to-records-the-whole-path","title":"6. Cluster Structure \u2013 From Cluster to Records, the Whole Path","text":"Q: What is the structure of a Redis cluster from clusters down to individual records?
A: A Redis cluster is organized as follows: - Cluster: Composed of multiple nodes. - Nodes: Each node is responsible for a subset of the keyspace, organized into hash slots (16,384 in total). - Shards: Each node represents a shard of the data and can replicate across replicas. - Keys/Records: Each key is hashed to a specific slot, determining the node responsible for storing it.
"},{"location":"techdives/DistrubutedSystems/Redis/#7-the-fundamentals-of-a-cluster-all-building-blocks-from-cluster-to-records","title":"7. The Fundamentals of a Cluster \u2013 All Building Blocks from Cluster to Records","text":"Q: What are the core building blocks of a Redis cluster?
A: Core components include: - Nodes: Independent Redis instances in a cluster. - Hash Slots: Redis divides keys into 16,384 slots for distribution across nodes. - Replication: Each primary node can have replicas to ensure data redundancy. - Partitions (Shards): Each node holds a partition of data for horizontal scalability.
"},{"location":"techdives/DistrubutedSystems/Redis/#8-multi-master-support","title":"8. Multi-Master Support","text":"Q: Does Redis support multi-master configurations?
A: Redis does not support multi-master configurations in its native setup. It uses a single-master architecture per shard to ensure consistency.
"},{"location":"techdives/DistrubutedSystems/Redis/#9-master-slave-relationship-in-data-nodes","title":"9. Master-Slave Relationship in Data Nodes","text":"Q: Does Redis follow a master-slave structure between data nodes?
A: Yes, in a Redis cluster, each data shard has a single master with one or more replicas (slaves) for redundancy. The slaves serve as read-only replicas unless promoted during failover.
"},{"location":"techdives/DistrubutedSystems/Redis/#10-node-structures-in-cluster","title":"10. Node Structures in Cluster","text":"Q: What are the structures of nodes in a Redis cluster?
A: In a Redis cluster, each node is responsible for a subset of hash slots, with a master node serving write requests and one or more replicas serving as failover or read-only instances.
"},{"location":"techdives/DistrubutedSystems/Redis/#11-cluster-scaling-horizontal-and-vertical","title":"11. Cluster Scaling \u2013 Horizontal and Vertical","text":"Q: Does Redis support horizontal and vertical scaling, and which is preferred?
A: Redis supports horizontal scaling (adding more nodes) via sharding in a cluster, which is generally preferred. Vertical scaling (adding more memory/CPU) is also possible but limited by hardware.
"},{"location":"techdives/DistrubutedSystems/Redis/#12-high-availability-explanation","title":"12. High Availability \u2013 Explanation","text":"Q: How does Redis provide high availability?
A: Redis achieves high availability through replication and Redis Sentinel for monitoring and automatic failover. Redis Cluster further enhances availability by automatically promoting replicas if a primary node fails.
"},{"location":"techdives/DistrubutedSystems/Redis/#13-fault-tolerance-explanation","title":"13. Fault Tolerance \u2013 Explanation","text":"Q: What mechanisms does Redis have for fault tolerance?
A: Redis ensures fault tolerance through data replication across replicas, and Sentinel monitors the master nodes to trigger failover in case of node failure.
"},{"location":"techdives/DistrubutedSystems/Redis/#14-replication","title":"14. Replication","text":"Q: How does replication work in Redis?
A: Redis replication is asynchronous by default, with each master node replicating data to one or more replicas. In the event of a master failure, a replica is promoted to master status.
"},{"location":"techdives/DistrubutedSystems/Redis/#15-partitioning-and-sharding","title":"15. Partitioning and Sharding","text":"Q: How does Redis handle partitioning and sharding?
A: Redis uses hash-based partitioning with 16,384 hash slots to distribute data across nodes. Each key is assigned a hash slot, which maps it to a specific node.
"},{"location":"techdives/DistrubutedSystems/Redis/#16-caching-in-depth","title":"16. Caching in Depth","text":"Q: How does Redis perform caching?
A: Redis is an in-memory cache, providing low-latency access with various caching strategies (e.g., TTL, eviction policies like LRU and LFU). It supports key expiration and eviction for memory management.
"},{"location":"techdives/DistrubutedSystems/Redis/#17-storage-type-trees-used-for-storage","title":"17. Storage Type \u2013 Trees Used for Storage","text":"Q: What storage type and structures does Redis use?
A: Redis stores data in memory using simple data structures and does not use B-trees or similar structures. Data is kept in-memory and optionally persisted to disk (AOF/RDB).
"},{"location":"techdives/DistrubutedSystems/Redis/#18-segments-or-page-approach","title":"18. Segments or Page Approach?","text":"Q: Does Redis use a segments approach, page approach, or something else?
A: Redis does not use segments or page-based storage. Data is stored in-memory and is managed directly by the Redis process.
"},{"location":"techdives/DistrubutedSystems/Redis/#19-indexing-how-does-it-work","title":"19. Indexing \u2013 How Does It Work?","text":"Q: How does Redis handle indexing?
A: Redis does not use traditional indexing. Instead, it directly maps keys to hash slots in the cluster, providing O(1) access time to each key.
"},{"location":"techdives/DistrubutedSystems/Redis/#20-routing","title":"20. Routing","text":"Q: How does Redis route requests to the correct node in a cluster?
A: Redis routes requests based on key hashing. The key is hashed to determine its slot, which maps it to a specific node.
"},{"location":"techdives/DistrubutedSystems/Redis/#21-latency-including-write-read-indexing-and-replication-latency","title":"21. Latency \u2013 Including Write, Read, Indexing, and Replication Latency","text":"Q: What are Redis\u2019s latency characteristics?
A: Redis provides sub-millisecond read/write latency under normal conditions. Replication latency is generally low, though network overhead may add some delay.
"},{"location":"techdives/DistrubutedSystems/Redis/#22-versioning","title":"22. Versioning","text":"Q: Does Redis support versioning?
A: Redis does not natively support versioning. Application logic may be required to manage version control if needed.
"},{"location":"techdives/DistrubutedSystems/Redis/#23-locking-and-concurrency","title":"23. Locking and Concurrency","text":"Q: How does Redis handle locking and concurrency?
A: Redis supports distributed locking through the Redlock algorithm for ensuring safe concurrent access across clients.
"},{"location":"techdives/DistrubutedSystems/Redis/#24-write-ahead-logging-wal","title":"24. Write-Ahead Logging (WAL)","text":"Q: Does Redis support WAL?
A: Redis does not use WAL directly. However, the Append-Only File (AOF) is similar, logging each write operation to ensure persistence.
"},{"location":"techdives/DistrubutedSystems/Redis/#25-change-data-capture-cdc-support","title":"25. Change Data Capture (CDC) Support","text":"Q: Does Redis support CDC?
A: Redis does not natively support Change Data Capture. External tools may be needed for real-time data change tracking.
"},{"location":"techdives/DistrubutedSystems/Redis/#26-query-type-and-query-in-depth","title":"26. Query Type and Query in Depth","text":"Q: What types of queries does Redis support?
A: Redis is key-based and supports simple read/write commands without complex query languages. Operations include GET, SET, HGET, ZADD, etc.
Q: Does Redis have query optimizers?
A: Redis does not have traditional query optimizers, as it operates in O(1) for most key-based lookups.
"},{"location":"techdives/DistrubutedSystems/Redis/#28-sql-support","title":"28. SQL Support","text":"Q: Does Redis support SQL?
A: Redis does not natively support SQL. However, RedisJSON or other libraries can provide SQL-like querying capabilities.
"},{"location":"techdives/DistrubutedSystems/Redis/#29-circuit-breakers","title":"29. Circuit Breakers","text":"Q:
Does Redis have built-in circuit breaker support?
A: Redis itself does not implement circuit breakers. This is typically handled at the application or middleware layer.
"},{"location":"techdives/DistrubutedSystems/Redis/#30-data-retention-and-lifecycle-management","title":"30. Data Retention and Lifecycle Management","text":"Q: How does Redis handle data lifecycle and retention?
A: Redis supports TTL on keys, and policies like Least Recently Used (LRU) enable retention management. Redis doesn\u2019t support multi-tier storage.
"},{"location":"techdives/DistrubutedSystems/Redis/#31-other-features","title":"31. Other Features","text":"Q: What other features does Redis offer?
A: Redis supports data structures like streams for event logging, pub/sub for messaging, and geospatial indexing for location-based queries.
"},{"location":"techdives/DistrubutedSystems/Redis/#32-additional-modules","title":"32. Additional Modules","text":"Q: What modules or libraries can be added to Redis?
A: Redis offers modules like RedisJSON (for JSON handling), RedisGraph (for graph data), and RedisBloom (for probabilistic data structures).
"},{"location":"techdives/DistrubutedSystems/Redis/#33-optimization-and-tuning-of-clusters","title":"33. Optimization and Tuning of Clusters","text":"Q: How do you optimize and tune Redis clusters?
A: Key optimizations include appropriate partitioning, replication settings, eviction policies, and monitoring memory/CPU usage.
"},{"location":"techdives/DistrubutedSystems/Redis/#34-backup-and-recovery","title":"34. Backup and Recovery","text":"Q: How does Redis handle backup and recovery?
A: Redis supports RDB snapshots and AOF for persistence. Backups are easily managed via AOF or manual RDB dumps.
"},{"location":"techdives/DistrubutedSystems/Redis/#35-security","title":"35. Security","text":"Q: What are Redis\u2019s security features?
A: Redis supports authentication (AUTH command), SSL/TLS encryption, IP whitelisting, and role-based access control.
"},{"location":"techdives/DistrubutedSystems/Redis/#36-migration","title":"36. Migration","text":"Q: Does Redis support migration tools?
A: Redis offers tools like redis-cli for basic migration, and Redis Enterprise provides more advanced migration capabilities.
"},{"location":"techdives/DistrubutedSystems/Redis/#37-recommended-cluster-setup","title":"37. Recommended Cluster Setup","text":"Q: What\u2019s the recommended Redis cluster setup?
A: Typically, a Redis Cluster setup starts with 3 master nodes (for redundancy) and 3 replicas for high availability, totaling 6 nodes.
"},{"location":"techdives/DistrubutedSystems/Redis/#38-basic-cluster-setup-with-node-numbers-in-distributed-mode","title":"38. Basic Cluster Setup with Node Numbers in Distributed Mode","text":"Q: How does a basic Redis cluster setup look in distributed mode?
A: A minimal Redis Cluster in distributed mode consists of 3 master nodes (handling 5,461 slots each) with 1 replica per master for redundancy.
"},{"location":"techdives/DistrubutedSystems/Redis/#39-segments-approach-or-page-approach-or-others","title":"39. Segments Approach or Page Approach or others","text":"Q: Does Redis use a segments approach, page approach, or another storage approach?
A: Redis does not use a segments or page-based approach as it is an in-memory database. Data is stored directly in memory with no fixed segment or page structure, allowing for rapid access to keys. Redis is optimized for speed, relying on data structures like hash tables and direct in-memory allocation rather than traditional on-disk segment or page methods common in disk-based databases.
"},{"location":"techdives/DistrubutedSystems/S3/","title":"Amazon S3 (Simple Storage Service)","text":""},{"location":"techdives/DistrubutedSystems/S3/#1-introduction","title":"1. Introduction","text":"Amazon S3 is a scalable object storage service offered by Amazon Web Services (AWS). It is designed to store and retrieve any amount of data from anywhere on the web, making it suitable for various use cases, including data backup, archiving, big data analytics, and hosting static websites.
"},{"location":"techdives/DistrubutedSystems/S3/#2-architecture-and-fundamentals","title":"2. Architecture and Fundamentals","text":"S3 offers a variety of storage classes optimized for different use cases, balancing cost and performance:
S3 Standard: For frequently accessed data, providing high durability and availability.
S3 Intelligent-Tiering: Automatically moves data between two access tiers based on changing access patterns, optimizing costs.
S3 Standard-IA (Infrequent Access): Lower-cost storage for infrequently accessed data with retrieval fees.
S3 One Zone-IA: For infrequently accessed data that does not require multiple availability zone redundancy.
S3 Glacier: For archival data, offering retrieval times from minutes to hours.
S3 Glacier Deep Archive: The lowest-cost storage option for rarely accessed data, with retrieval times of 12 hours or more.
Durability: S3 is designed for 99.999999999% (11 nines) durability, ensuring high protection against data loss by automatically storing data across multiple devices in multiple facilities within an AWS Region.
Availability: S3 offers 99.99% availability over a given year, ensuring data accessibility when needed.
Redundancy: Data is redundantly stored across multiple availability zones, providing resilience against failures.
Identity and Access Management (IAM): Provides fine-grained control over who can access S3 resources and what actions they can perform.
Bucket Policies and ACLs: Define permissions at the bucket and object level, allowing public or private access as needed.
Encryption: Supports both server-side and client-side encryption to protect data at rest. SSE options include:
Data Transfer Security: Supports SSL/TLS for data in transit to protect data from interception.
Access Logs: Enable server access logging to track requests for access to your S3 resources.
Versioning: Keeps multiple versions of an object, allowing recovery from accidental deletions or overwrites.
Lifecycle Management: Automatically transitions objects between storage classes or deletes objects based on defined rules.
Event Notifications: Configure notifications for events like object creation or deletion, which can trigger AWS Lambda functions or send messages to Amazon SNS.
Object Tags: Use object tagging for better organization and management, including cost allocation for various projects.
Transfer Acceleration: Speeds up uploads and downloads using Amazon CloudFront's globally distributed edge locations.
Multipart Upload: Enables the uploading of large objects in parts, improving performance and allowing resuming of uploads in case of network failures.
Caching with CloudFront: Integrate with Amazon CloudFront for content delivery, reducing latency and improving performance for global users.
Data Consistency: S3 provides strong read-after-write consistency for PUT and DELETE operations, ensuring that data is immediately available after a successful write.
Backup and Disaster Recovery: Store backups of critical data and applications to ensure business continuity.
Data Lakes: Centralize diverse data sources for analytics and machine learning - OLAP
Media Hosting: Store and serve images, videos, and other media files for web and mobile applications.
Big Data Analytics: Use S3 to store large datasets for processing with services like Amazon Athena, Amazon EMR, or Amazon Redshift.
Static Website Hosting: Serve static websites directly from S3, leveraging its high availability and durability.
Organize Data: Use a consistent naming convention for buckets and objects to improve manageability.
Monitor Usage: Utilize AWS CloudTrail and AWS CloudWatch to monitor S3 usage and access patterns.
Optimize Costs: Regularly review storage classes and utilize lifecycle policies to manage data costs effectively.
Implement Security Best Practices: Regularly audit permissions, enable logging, and implement encryption to secure data.
Data Governance: Implement data governance strategies to comply with legal and regulatory requirements.
Version control is the cornerstone of modern software development, and Git stands as the most widely used version control system in the world. Whether you're a beginner or an experienced if you're a developer understanding Git is crucial for collaborative and individual projects alike. In this article, we'll take a deep dive into Git, covering everything from its basics to its advanced features, to equip you with the knowledge to master it and with a cheat sheet at the end.
"},{"location":"techdives/GeneralConcepts/git/#what-is-git","title":"What is Git ?","text":"Git is a distributed version control system that tracks changes in files, enabling multiple developers to collaborate on a project effectively. Created in 2005 by Linus Torvalds, Git was initially designed for managing the Linux kernel's development. Today, it powers everything from small personal projects to massive enterprise software systems.
Key features
Getting started with Git begins with installing it on your system. Here's how you can set it up based on your operating system:
MacwindowsLinux Use Homebrew to install Gitbrew install git\nDownload Git for Windows from git-scm.com and follow the installer instructions.
Install Git using your distribution's package managersudo apt install git # For Debian/Ubuntu\nsudo yum install git # For CentOS/Red Hat\ngit --version\ngit config --global user.name \"Your Name\"\ngit config --global user.email \"your.email@example.com\"\ngit config --global core.editor \"code\" # Use VSCode or any editor\nTo start tracking changes in a project, initialize a repository
git init\nTo work on an existing project, clone its repository
git clone <repository-url>\nStage Changes: Add files to the staging area
git add <file>\nCommit Changes: Save changes to the repository
git commit -m \"<Write a proper commit message>\"\nView the status of your working directory and staged files
git status\nReview the project's history with
git log\ngit log --oneline # Concise view\nGit's branching system is one of its most powerful features. Branches allow you to work on different features or bug fixes without affecting the main codebase.
"},{"location":"techdives/GeneralConcepts/git/#creating-switching-branches","title":"Creating Switching Branches","text":"Create a new branch
git branch <branch-name>\ngit checkout <branch-name>\ngit switch <branch-name> # New alternative\ngit checkout -b <branch-name>\nTo integrate changes from one branch into another
git checkout main # replace main with custom branch\ngit merge <branch-name>\nIf Git detects conflicting changes, resolve them manually by editing the affected files. Then
git add <file>\ngit commit\nRemote repositories allow teams to collaborate effectively.
"},{"location":"techdives/GeneralConcepts/git/#adding-a-remote","title":"Adding a Remote","text":"Link your local repository to a remote
git remote add origin <repository-url>\nSend your commits to the remote repository
git push origin <branch-name>\nFetch and integrate changes from the remote repository
git pull\nIf needed, you can remove a remote
git remote remove origin\nTemporarily save changes without committing
git stash\ngit stash apply\nApply a specific commit from another branch
git cherry-pick <commit-hash>\nRebase your branch onto another
git rebase <branch-name>\nFix the last commit message or contents
git commit --amend\nGit operates by storing snapshots of your project at each commit, not deltas (differences). The key components of Git's internal storage include:
All data is stored in the .git directory.
Teams often adopt workflows to streamline collaboration. Popular ones include:
main, develop, feature, and release for structured development.main branch, often behind feature flags.git switch <branch-name>\n.gitignore.Below are all the essential Git commands.
Git Cheat Sheet
# Git Commands Cheat Sheet\n\n# Configuration\ngit config --global user.name \"Your Name\" # Set user name\ngit config --global user.email \"your.email@uth.com\" # Set user email\ngit config --global core.editor \"code\" # Set default editor\ngit config --list # View current configuration\n\n# Repository Management\ngit init # Initialize a new repository\ngit clone <repository-url> # Clone an existing repository\ngit remote add origin <url> # Add remote repository\ngit remote -v # View remote repositories\ngit remote remove <name> # Remove a remote\n\n# Staging and Committing\ngit add <file> # Stage specific file\ngit add . # Stage all files\ngit status # Check status of repository\ngit commit -m \"Message\" # Commit with message\ngit commit --amend # Amend the last commit\n\n# Branching\ngit branch # List branches\ngit branch <branch-name> # Create a new branch\ngit checkout <branch-name> # Switch to a branch\ngit checkout -b <branch-name> # Create and Switch to a branch\ngit switch <branch-name> # Modern way to switch branches\ngit branch -d <branch-name> # Delete a branch\ngit branch -D <branch-name> # Force delete a branch\n\n# Merging\ngit merge <branch-name> # Merge a branch into the current branch\n\n# Pulling and Pushing\ngit pull # Fetch and merge from remote repository\ngit pull origin <branch-name> # Pull specific branch\ngit push origin <branch-name> # Push to remote repository\ngit push --all # Push all branches\ngit push --tags # Push tags to remote\n\n# Logs and History\ngit log # View commit history\ngit log --oneline # View concise commit history\ngit log --graph # View graphical commit history\n\n# Undo Changes\ngit reset HEAD <file> # Unstage a file\ngit checkout -- <file> # Discard changes in working directory\ngit revert <commit-hash> # Undo a specific commit (safe)\ngit reset <commit-hash> # Reset to a specific commit (dangerous)\n\n# Stashing\ngit stash # Stash changes\ngit stash list # List stashes\ngit stash apply # Apply the last stash\ngit stash drop # Remove the last stash\ngit stash clear # Clear all stashes\n\n# Rebasing\ngit rebase <branch-name> # Rebase current branch onto another\ngit rebase -i <commit-hash> # Interactive rebase\n\n# Tags\ngit tag <tag-name> # Create a tag\ngit tag -a <tag-name> -m \"Message\" # Create an annotated tag\ngit tag -d <tag-name> # Delete a tag locally\ngit push origin <tag-name> # Push a specific tag\ngit push --tags # Push all tags\n\n# Collaboration\ngit fetch # Fetch updates from remote\ngit pull # Fetch and merge updates\ngit pull origin <branch-name> # Pull specific branch\ngit push # Push changes to remote\ngit push origin <branch-name> # Push specific branch\n\n# Ignoring Files\necho \"filename\" >> .gitignore # Add file to .gitignore\ngit rm --cached <file> # Stop tracking a file\n\n# Viewing Changes\ngit diff # View unstaged changes\ngit diff --staged # View staged changes\ngit diff <commit-hash1> <commit-hash2> # Compare two commits\n\n# Cherry-Picking\ngit cherry-pick <commit-hash> # Apply a specific commit to the current branch\n\n# Aliases\ngit config --global alias.co checkout # Alias for checkout\ngit config --global alias.br branch # Alias for branch\ngit config --global alias.cm commit # Alias for commit\n