Microservices

The Honest Truth: Monoliths Are Often Better

Before discussing microservices, let’s be clear about something the industry glosses over: a well-structured monolith beats a poorly-designed microservices system every time.

Netflix, Uber, and Amazon run microservices because they have thousands of engineers, hundreds of services, and genuine deployment independence requirements. If your team has 10 engineers and one product, microservices will slow you down.

What a monolith gets right:

• Single deployment unit — no distributed coordination
• In-process function calls (nanoseconds) instead of network hops (milliseconds)
• Shared memory — no serialization overhead
• Atomic database transactions across all domains
• Simple debugging — one call stack, one log stream
• No network partition risk within the service boundary

What forces the microservices decision:

• Teams stepping on each other in a shared codebase (organizational scaling)
• Different scaling requirements (auth service vs. video transcoding)
• Different deployment cadences (payment service needs strict change control, recommendation engine deploys 50× per day)
• Fault isolation (a crash in one service shouldn’t take down the entire product)
• Technology heterogeneity (ML inference in Python, API in Go, legacy integration in Java)

The rule: Start with a modular monolith. Extract services when you feel specific pain, not in anticipation of it.

Domain-Driven Design and Bounded Contexts

Domain-Driven Design (DDD), introduced by Eric Evans, gives us the vocabulary for decomposing systems without creating distributed spaghetti.

Bounded Context: A boundary within which a particular domain model applies. “Order” means different things in the warehouse context (picking list, physical location) vs. the billing context (invoice, payment terms) vs. the analytics context (conversion funnel event). Each bounded context owns its model.

E-commerce domain map:

┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
│  Order Context  │   │ Catalog Context  │   │ Payment Context │
│                 │   │                 │   │                 │
│  Order          │   │  Product        │   │  Invoice        │
│  LineItem       │   │  SKU            │   │  Charge         │
│  Fulfillment    │   │  Inventory      │   │  Refund         │
└────────┬────────┘   └────────┬────────┘   └────────┬────────┘
         │                    │                      │
         └──────── Shared ────┘                      │
              Kernel (tiny)             Customer Context
                                       ┌─────────────┐
                                       │  Customer   │
                                       │  Address    │
                                       │  Preference │
                                       └─────────────┘

Context mapping relationships:

• Shared Kernel: Two contexts share a small subset of the domain model (risky, creates coupling)
• Customer/Supplier: Upstream context publishes; downstream consumes (supplier defines the API)
• Conformist: Downstream simply conforms to upstream’s model with no negotiation
• Anti-Corruption Layer (ACL): Downstream translates upstream’s model to its own (the correct approach for legacy integrations)
• Published Language: An explicit, versioned contract (Protobuf, Avro schema) shared between contexts

A microservice should align with exactly one bounded context. If a service spans multiple bounded contexts, it’s either too large or your context boundaries are wrong.

Service Decomposition Patterns

Decompose by Business Capability

Business capabilities are stable, high-level functions the business performs. They map to organizational units.

Business capabilities → Services:

Order Management      → order-service
Inventory Management  → inventory-service
Customer Management   → customer-service
Payment Processing    → payment-service
Shipping & Logistics  → shipping-service
Notifications         → notification-service

Stability: business capabilities rarely change even as implementation changes. This makes them excellent service boundaries.

Decompose by Subdomain

DDD distinguishes subdomains by strategic importance:

• Core subdomain: Your competitive differentiator. Build it, own it, invest heavily. For Uber: dynamic pricing, matching algorithm.
• Supporting subdomain: Necessary but not differentiating. Build it lean. For Uber: driver onboarding, document verification.
• Generic subdomain: Commodity functionality. Buy it. For Uber: email delivery (SendGrid), maps (Google Maps API), payments (Stripe).

Don’t build a microservice for a generic subdomain — buy the SaaS product and integrate at the edge.

The Strangler Fig Pattern

Named after a vine that gradually replaces a host tree. The pattern for migrating a monolith to microservices without a big-bang rewrite:

Phase 1: Route new features to microservices, monolith handles existing

        ┌─────────┐
Client → │  Proxy  │ ──────────────────┐
        └─────────┘                   │
             │                        ▼
             │              ┌──────────────────┐
             └──────────────│    Monolith      │
              (existing)    │  (legacy routes) │
                            └──────────────────┘

Phase 2: Migrate routes one by one

        ┌─────────┐
Client → │  Proxy  │ ──────► new-orders-service
        └─────────┘
             │       ──────► new-payments-service
             │
             └──────────────► Monolith (remaining routes)

Phase 3: Monolith is strangled (empty), decommission it

Key rule: The proxy (API gateway or Nginx) intercepts all traffic. You can migrate routes without client changes, and roll back by updating a routing rule.

Inter-Service Communication

Synchronous: REST vs. gRPC

REST (HTTP/1.1 or HTTP/2):

• Ubiquitous tooling, human-readable (JSON), easy debugging with curl
• Overhead: HTTP header bloat, JSON serialization/deserialization
• Suitable for: external APIs, services with diverse clients, simple CRUD

gRPC (HTTP/2 + Protocol Buffers):

• Binary framing, schema-enforced (no more “what fields does this endpoint return?”)
• Streaming: client-side, server-side, bidirectional
• Generated clients in 11 languages — no hand-written SDK maintenance
• Overhead: requires HTTP/2, less debuggable without tooling

// order.proto — the contract is explicit and versioned
syntax = "proto3";

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (Order);
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc StreamOrderUpdates(GetOrderRequest) returns (stream OrderEvent);
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated LineItem items = 2;
  string shipping_address_id = 3;
}

message Order {
  string order_id = 1;
  string status = 2;
  int64 created_at_unix = 3;
  repeated LineItem items = 4;
}

When to choose gRPC: Internal service-to-service calls in a controlled environment, especially when latency matters, schema enforcement is valuable, or you need streaming. Netflix and Google use it internally almost exclusively.

Asynchronous: Message Queues and Event Streams

Async communication decouples services temporally — the sender doesn’t wait for the receiver. This is the backbone of event-driven microservices.

// Synchronous: order-service calls payment-service and WAITS
async function createOrder(req: CreateOrderRequest): Promise<Order> {
  const order = await orderRepo.save(req);
  // Blocking: if payment-service is down, this throws
  const charge = await paymentService.charge(order.total, req.paymentMethod);
  return order;
}

// Asynchronous: order-service publishes, payment-service consumes when ready
async function createOrder(req: CreateOrderRequest): Promise<Order> {
  const order = await orderRepo.save(req);
  // Non-blocking: publishes and returns immediately
  await eventBus.publish("order.created", {
    orderId: order.id,
    total: order.total,
    paymentMethod: req.paymentMethod,
  });
  return order; // status: PENDING_PAYMENT
}

// In payment-service (separate process, separate deployment)
eventBus.subscribe("order.created", async (event) => {
  await paymentGateway.charge(event.total, event.paymentMethod);
  await eventBus.publish("payment.completed", { orderId: event.orderId });
});

Trade-off matrix:

Service Discovery

Client-Side Discovery

The client queries a service registry (Consul, Eureka) and picks a healthy instance using its own load-balancing logic.

Client → Consul (query: "give me payment-service instances")
       ← [10.0.1.5:8080, 10.0.1.6:8080, 10.0.1.7:8080]
Client → 10.0.1.6:8080 (client-side round-robin)

import Consul from "consul";

const consul = new Consul({ host: "consul.internal" });

async function getServiceInstances(serviceName: string): Promise<string[]> {
  const services = await consul.health.service({
    service: serviceName,
    passing: true, // only healthy instances
  });
  return services.map((s: any) => `${s.Service.Address}:${s.Service.Port}`);
}

class ServiceClient {
  private instances: string[] = [];
  private cursor = 0;

  async call(service: string, path: string, body: unknown) {
    if (this.instances.length === 0) {
      this.instances = await getServiceInstances(service);
    }
    const host = this.instances[this.cursor % this.instances.length];
    this.cursor++;
    return fetch(`http://${host}${path}`, {
      method: "POST",
      body: JSON.stringify(body),
    });
  }
}

Downside: Every client must implement discovery and load balancing. Polyglot environments require library support in every language.

Server-Side Discovery

The client sends the request to a load balancer (AWS ALB, Nginx, HAProxy). The load balancer queries the registry and forwards to a healthy instance.

Client → ALB (DNS: payment-service.internal)
ALB → Consul/ECS/K8s (query healthy instances)
ALB → 10.0.1.6:8080 (transparent to client)

In Kubernetes, this is the default model: a Service object provides a stable DNS name and virtual IP; kube-proxy routes to healthy pods.

The Saga Pattern: Distributed Transactions Without 2PC

Two-phase commit (2PC) is the classic protocol for distributed transactions. It’s also a distributed systems anti-pattern: it’s blocking, it has coordinator as a single point of failure, and it’s fundamentally at odds with network partitions (CAP theorem).

The Saga pattern replaces a distributed ACID transaction with a sequence of local transactions, each publishing events or messages that trigger the next step. On failure, compensating transactions undo the work.

Choreography-Based Saga

Services react to events without a central coordinator. Each service listens for events, does its work, and publishes new events.

// Order Service
eventBus.subscribe("checkout.initiated", async (event) => {
  const order = await orderRepo.create({
    customerId: event.customerId,
    items: event.items,
    status: "PENDING",
  });
  await eventBus.publish("order.created", { orderId: order.id, ...event });
});

// Inventory Service
eventBus.subscribe("order.created", async (event) => {
  try {
    await inventoryRepo.reserve(event.items);
    await eventBus.publish("inventory.reserved", { orderId: event.orderId });
  } catch (e) {
    await eventBus.publish("inventory.reservation.failed", {
      orderId: event.orderId,
      reason: (e as Error).message,
    });
  }
});

// Payment Service
eventBus.subscribe("inventory.reserved", async (event) => {
  try {
    const charge = await paymentGateway.charge(event.total, event.paymentMethod);
    await eventBus.publish("payment.completed", {
      orderId: event.orderId,
      chargeId: charge.id,
    });
  } catch (e) {
    // Trigger compensating transaction
    await eventBus.publish("payment.failed", { orderId: event.orderId });
  }
});

// Inventory Service — compensating transaction
eventBus.subscribe("payment.failed", async (event) => {
  await inventoryRepo.release(event.orderId); // undo reservation
  await eventBus.publish("inventory.released", { orderId: event.orderId });
});

// Order Service — compensating transaction
eventBus.subscribe("inventory.released", async (event) => {
  await orderRepo.updateStatus(event.orderId, "CANCELLED");
});

Saga flow diagram:

checkout.initiated
       │
       ▼
order.created ──────────────────────────────┐
       │                                    │ (failure path)
       ▼                                    │
inventory.reserved ─────────────────┐       │
       │                            │       │
       ▼                            │       ▼
payment.completed           payment.failed
       │                            │
       ▼                            ▼
order.confirmed            inventory.released
                                    │
                                    ▼
                             order.cancelled

Problem with choreography: As the saga grows in complexity, the event flow becomes difficult to reason about. Cyclic event dependencies, “who published this event?”, and debugging distributed failures are all genuine pain points.

Orchestration-Based Saga

A central orchestrator (saga execution controller) explicitly drives the saga, calling each participant and handling failures.

class OrderSagaOrchestrator {
  async execute(checkoutRequest: CheckoutRequest): Promise<SagaResult> {
    const sagaId = generateId();
    const saga = await sagaRepo.create({ id: sagaId, status: "STARTED" });

    // Step 1: Create order
    let order: Order;
    try {
      order = await orderService.createOrder(checkoutRequest);
      await sagaRepo.updateStep(sagaId, "ORDER_CREATED");
    } catch (e) {
      await sagaRepo.updateStatus(sagaId, "FAILED");
      throw e; // Nothing to compensate yet
    }

    // Step 2: Reserve inventory
    let reservation: Reservation;
    try {
      reservation = await inventoryService.reserve(order.items);
      await sagaRepo.updateStep(sagaId, "INVENTORY_RESERVED");
    } catch (e) {
      // Compensate: cancel order
      await orderService.cancelOrder(order.id);
      await sagaRepo.updateStatus(sagaId, "FAILED_COMPENSATED");
      throw e;
    }

    // Step 3: Process payment
    try {
      await paymentService.charge(order.total, checkoutRequest.paymentMethod);
      await sagaRepo.updateStep(sagaId, "PAYMENT_COMPLETED");
    } catch (e) {
      // Compensate in reverse order
      await inventoryService.release(reservation.id);
      await orderService.cancelOrder(order.id);
      await sagaRepo.updateStatus(sagaId, "FAILED_COMPENSATED");
      throw e;
    }

    // Step 4: Confirm order
    await orderService.confirmOrder(order.id);
    await sagaRepo.updateStatus(sagaId, "COMPLETED");
    return { sagaId, orderId: order.id };
  }
}

Orchestration pros: Clear control flow, easy to debug, single place to add new steps, saga state is explicit (can resume after crashes).

Orchestration cons: Orchestrator becomes a point of coupling — it must know about all participant services. Scale the orchestrator separately.

Circuit Breaker Pattern

A circuit breaker prevents cascading failures by short-circuiting calls to a failing service.

States:
  CLOSED  → Normal operation. Failures counted.
  OPEN    → All calls fail immediately (no network calls). Timer running.
  HALF-OPEN → Allow one probe call. If success → CLOSED. If fail → OPEN.

Transitions:
  CLOSED →(failures exceed threshold)→ OPEN
  OPEN →(timeout elapsed)→ HALF-OPEN
  HALF-OPEN →(success)→ CLOSED
  HALF-OPEN →(failure)→ OPEN

type CircuitState = "CLOSED" | "OPEN" | "HALF_OPEN";

class CircuitBreaker {
  private state: CircuitState = "CLOSED";
  private failureCount = 0;
  private lastFailureTime = 0;

  constructor(
    private readonly threshold: number = 5,        // failures before opening
    private readonly timeout: number = 30_000,     // ms before trying again
    private readonly halfOpenProbes: number = 1
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "OPEN") {
      if (Date.now() - this.lastFailureTime > this.timeout) {
        this.state = "HALF_OPEN";
      } else {
        throw new Error("Circuit breaker OPEN — service unavailable");
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  private onSuccess(): void {
    this.failureCount = 0;
    this.state = "CLOSED";
  }

  private onFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.threshold) {
      this.state = "OPEN";
    }
  }
}

// Usage
const paymentCircuit = new CircuitBreaker(5, 30_000);

async function chargeCustomer(amount: number, token: string) {
  return paymentCircuit.call(() => paymentService.charge(amount, token));
}

Bulkhead Pattern

Named after the watertight compartments in a ship — a leak in one compartment doesn’t sink the ship.

In microservices: isolate the thread pools (or connection pools) used for different services. A slow downstream service exhausts only its own pool, not the shared pool.

class BulkheadPool {
  private activeRequests = 0;

  constructor(
    private readonly maxConcurrent: number,
    private readonly name: string
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.activeRequests >= this.maxConcurrent) {
      throw new Error(
        `Bulkhead '${this.name}' full (${this.maxConcurrent} concurrent max)`
      );
    }
    this.activeRequests++;
    try {
      return await fn();
    } finally {
      this.activeRequests--;
    }
  }
}

// Each downstream service gets its own bulkhead
const paymentBulkhead = new BulkheadPool(20, "payment-service");
const inventoryBulkhead = new BulkheadPool(50, "inventory-service");
const notificationBulkhead = new BulkheadPool(10, "notification-service");

When the notification service goes slow, its bulkhead fills up and requests fail fast. Payment and inventory processing are unaffected.

Service Mesh: Istio and Envoy

A service mesh is an infrastructure layer that handles service-to-service communication: mutual TLS, retries, circuit breaking, observability — without any application code changes.

Sidecar proxy pattern: Each service pod runs an Envoy proxy sidecar. All inbound and outbound network traffic is intercepted by the sidecar.

Pod A:                              Pod B:
┌─────────────────────────┐        ┌─────────────────────────┐
│  App Container          │        │  App Container          │
│  (port 8080)            │        │  (port 8080)            │
│         │               │        │         ▲               │
│         ▼               │        │         │               │
│  Envoy Sidecar          │───────▶│  Envoy Sidecar          │
│  (port 15001)           │  mTLS  │  (port 15001)           │
└─────────────────────────┘        └─────────────────────────┘
         ▲                                   │
         │                                   │
         └──────── Istio Control Plane ───────┘
                   (Pilot, Citadel, Galley)

What the service mesh gives you for free:

• mTLS between all services: Zero-trust networking. No service can call another without a valid certificate.
• Automatic retries: Configurable retry policies without touching application code.
• Circuit breaking: Envoy implements outlier detection (consecutive 5xx → eject from load balancing).
• Distributed tracing: Envoy propagates trace headers (B3, W3C TraceContext) automatically.
• Traffic management: Canary deployments, A/B testing, traffic mirroring — all via YAML.

# Istio VirtualService: route 10% of traffic to v2
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service
spec:
  http:
    - route:
        - destination:
            host: payment-service
            subset: v1
          weight: 90
        - destination:
            host: payment-service
            subset: v2
          weight: 10

The cost: Envoy adds ~2–5ms latency per hop, significant CPU overhead (~0.5 vCPU per 1,000 RPS), and a complex control plane to operate. Don’t adopt Istio unless you have a platform team.

Observability: Tracing and Correlation IDs

In a distributed system, a single user request may touch 12 services. When something goes wrong, you need to reconstruct the full request path.

Correlation ID pattern:

import { AsyncLocalStorage } from "async_hooks";

const storage = new AsyncLocalStorage<{ traceId: string; spanId: string }>();

// Express middleware — inject at the edge
app.use((req, res, next) => {
  const traceId = req.headers["x-trace-id"] as string ?? generateTraceId();
  const spanId = generateSpanId();
  storage.run({ traceId, spanId }, next);
});

// Logger — always includes trace context
function log(level: string, message: string, meta?: Record<string, unknown>) {
  const ctx = storage.getStore();
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level,
    message,
    traceId: ctx?.traceId,
    spanId: ctx?.spanId,
    ...meta,
  }));
}

// HTTP client — propagate to downstream
async function callService(url: string, body: unknown) {
  const ctx = storage.getStore();
  return fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "x-trace-id": ctx?.traceId ?? "",
      "x-parent-span-id": ctx?.spanId ?? "",
    },
    body: JSON.stringify(body),
  });
}

The three pillars of observability:

1. Logs: Structured JSON with traceId, spanId, service name, timestamp
2. Metrics: RED (Rate, Errors, Duration) per service, per endpoint
3. Traces: Distributed traces via OpenTelemetry → Jaeger/Zipkin/Tempo

import { trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("order-service");

async function processOrder(orderId: string) {
  return tracer.startActiveSpan("processOrder", async (span) => {
    span.setAttributes({ "order.id": orderId });
    try {
      const result = await doWork(orderId);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (e) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: (e as Error).message });
      span.recordException(e as Error);
      throw e;
    } finally {
      span.end();
    }
  });
}

Health Checks and Graceful Shutdown

import { createServer, IncomingMessage, ServerResponse } from "http";

let isReady = false;
let isLive = true;

// Kubernetes liveness probe: is the process alive?
// Return 503 to trigger pod restart
// Return 200 if healthy

// Kubernetes readiness probe: can the pod accept traffic?
// Return 503 during startup, warm-up, or graceful shutdown
// Return 200 only when fully ready

const healthServer = createServer((req: IncomingMessage, res: ServerResponse) => {
  if (req.url === "/healthz/live") {
    res.writeHead(isLive ? 200 : 503);
    res.end(JSON.stringify({ status: isLive ? "ok" : "dead" }));
  } else if (req.url === "/healthz/ready") {
    res.writeHead(isReady ? 200 : 503);
    res.end(JSON.stringify({ status: isReady ? "ready" : "not-ready" }));
  }
});

// Graceful shutdown: stop accepting new requests, finish existing ones
process.on("SIGTERM", async () => {
  isReady = false; // Stop receiving new traffic from load balancer
  await new Promise((resolve) => setTimeout(resolve, 5_000)); // drain window
  await closeDbConnections();
  await closeMqConnections();
  process.exit(0);
});

// Application startup
async function main() {
  await connectToDatabase();
  await connectToMessageQueue();
  isReady = true; // NOW accept traffic
}

Interview Checklist

Before walking out of a microservices design interview, verify you have covered:

• When NOT to use microservices: Acknowledge the monolith tradeoff upfront — it shows senior judgment
• Decomposition strategy: DDD bounded contexts, by business capability vs. subdomain
• Communication: Sync (gRPC preferred over REST internally) vs. async (Kafka/RabbitMQ) — articulate when to use each
• Service discovery: Client-side (Consul) vs. server-side (K8s Service + kube-proxy)
• Saga pattern: Choreography (simple, hard to debug at scale) vs. Orchestration (explicit control flow, easier to monitor)
• Failure modes: Circuit breaker (Hystrix/Resilience4j pattern), bulkhead, timeout, retry with jitter
• Service mesh: What it solves (mTLS, observability, retries) and what it costs (latency, ops burden)
• Observability: Distributed tracing (OpenTelemetry), correlation IDs, structured logging
• Data ownership: Each service owns its own database. No cross-service JOIN queries. Data consistency via events.
• Migration strategy: Strangler Fig pattern — never big-bang rewrite

The hardest question — data consistency: “If order-service and inventory-service each have their own database, how do you ensure a customer can’t order something out of stock?”

Answer: Saga pattern with compensating transactions. The order is created in PENDING state; inventory is reserved; payment is charged; order is confirmed. At each step, failure triggers compensating transactions in reverse order. The system is eventually consistent, not immediately consistent — and that’s a deliberate design trade-off, not a bug.