I’ve been building backend systems for over a decade, and I’ve seen the same nightmare play out again and again: one downstream service slows down, and within minutes the whole application freezes. Calls pile up, threads exhaust, and the inevitable 503 cascade begins. That’s why I started using the Circuit Breaker pattern. It’s not a new idea—engineers have used it in hardware for over a century—but in software it saves you from the domino effect of distributed failures. Let me walk you through how I implemented it in a TypeScript microservices setup using Opossum and Express.
I was working on an e-commerce checkout system. The order service had to call the payment service and the inventory service. One day during a flash sale, the payment gateway started responding slowly. Within seconds, the order service’s connection pool was exhausted. Then the API gateway started timing out. The result? A complete system meltdown during the busiest hour. That’s when I realized I needed a mechanism that could fail fast and give the struggling service room to recover. The circuit breaker does exactly that.
Think of it like a real electrical breaker. Under normal conditions, the circuit is closed—requests flow through. When failures cross a threshold, the breaker trips open, blocking all further requests immediately. After a cooldown period, it transitions to half-open, allowing a limited number of test requests. If those succeed, it closes again. If not, it stays open. Simple, but powerful.
You’ve probably used libraries like Hystrix in Java or Sentinel in Python. For Node.js, Opossum is the go-to. It’s lightweight, integrates well with Prometheus, and gives you fine control over thresholds. In my project, I placed circuit breakers at the boundaries between the order service and its downstream dependencies—payment and inventory. Each breaker wraps the HTTP call using axios.
Here’s how I set up the payment circuit breaker. I defined options like the failure threshold (e.g., 50% of requests fail in a 10-second window), the reset timeout (30 seconds), and a fallback function that returns a cached or a degraded response.
import CircuitBreaker from 'opossum';
import axios from 'axios';
const paymentBreakerOptions = {
errorThresholdPercentage: 50,
resetTimeout: 30000,
name: 'payment-service',
volumeThreshold: 10,
};
const paymentBreaker = new CircuitBreaker(
async (orderId: string, amount: number) => {
const response = await axios.post('http://localhost:3002/process-payment', {
orderId,
amount,
});
return response.data;
},
paymentBreakerOptions
);I also exposed the breaker’s state for observability. Opossum fires events like ‘open’, ‘close’, and ‘halfOpen’. I used those to push metrics to Prometheus.
import prometheus from 'prom-client';
const breakerState = new prometheus.Gauge({
name: 'circuit_breaker_state',
help: 'State of the circuit breaker: 0=closed, 1=open, 2=half-open',
labelNames: ['service'],
});
paymentBreaker.on('open', () => breakerState.labels('payment').set(1));
paymentBreaker.on('halfOpen', () => breakerState.labels('payment').set(2));
paymentBreaker.on('close', () => breakerState.labels('payment').set(0));Why does the state matter? Because if you can see in real time that the payment breaker is open, you can alert before users complain. I integrated Grafana to show a dashboard with breaker states, request latencies, and fallback counts. That visibility saved my team from guessing.
Now let’s talk about fallbacks. When the breaker is open, you don’t want your user to see a 500. You provide a graceful degradation. For example, I returned a “payment pending” status and queued the order for later processing. That’s much better than a crash.
paymentBreaker.fallback((orderId, amount) => {
logger.warn('Payment breaker open – queuing order', { orderId });
return { status: 'pending', message: 'Payment will be processed later' };
});You might ask: how do I test this without actually bringing down services? I wrote integration tests using a mock server that simulates failures. I used a simple Express server that I could toggle between success and error responses. Then I wrote tests that ramp up failure rates and verify the breaker transitions.
Here’s a test snippet that verifies the breaker opens after three consecutive errors:
import axios from 'axios';
import CircuitBreaker from 'opossum';
const mockFail = () => Promise.reject(new Error('Service unavailable'));
const breaker = new CircuitBreaker(mockFail, {
errorThresholdPercentage: 100,
resetTimeout: 60000,
volumeThreshold: 3,
});
test('breaker opens after 3 failures', async () => {
for (let i = 0; i < 3; i++) {
await breaker.fire().catch(() => {});
}
expect(breaker.opened).toBe(true);
await expect(breaker.fire()).rejects.toThrow('Breaker is open');
});I also added a bulkhead pattern alongside the circuit breaker. Bulkhead isolates resources per dependency. I used a simple semaphore to limit the number of concurrent calls to the payment service. That way, even if the circuit is closed, a slow payment service won’t eat up all the order service’s worker threads.
import Bottleneck from 'bottleneck';
const paymentLimiter = new Bottleneck({
maxConcurrent: 5,
minTime: 200,
});
const circuitWithBulkhead = async (orderId, amount) => {
return paymentLimiter.schedule(() => paymentBreaker.fire(orderId, amount));
};When you combine circuit breaker with bulkhead, you get double protection. The circuit breaker stops calls when failures are high; the bulkhead prevents one dependency from monopolizing resources even during normal operation. I’ve used this combination in production for years, and it prevents the kind of chain reaction that brought down that flash sale.
I remember a specific incident where a Redis cache outage caused the inventory service to call the database directly, doubling response times. The circuit breaker for inventory opened within 30 seconds, and the order service continued serving cached product availability. Users never noticed. That’s the beauty of this pattern: it degrades gracefully.
Now, a word on configuration. Don’t set thresholds too low or too high. Start with errorThresholdPercentage=50 and resetTimeout=30s, then adjust based on real traffic patterns. I log every breaker state change to correlate with other metrics. If the breaker opens too often, your threshold is too sensitive; if it never opens, it’s too lenient.
What about distributed tracing? Circuit breakers work well with OpenTelemetry. I add span attributes indicating whether a request was blocked or executed. That gives me a complete picture of failures across services.
Finally, I monitor the fallback execution count. If you see many fallbacks, you know the downstream service is unhealthy. But if the fallback itself is slow (e.g., hitting the database), you could create a new problem. So I keep fallbacks cheap and possibly cached.
I encourage you to try this setup in a small project. Start with one circuit breaker between your API gateway and a simulated flaky service. Watch the state change in real time. Once you see how a fast failure protects your system, you’ll never go back.
If you found this article useful, please like, share, and leave a comment. I read every one and I’m happy to help you debug your circuit breaker implementation.
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
