Designing the Optimal Wide Area Network for your Enterprise

Designing a wide area network (WAN) is a bit like shopping for a car. It may sound like fun at first; there are so many options to test drive! But at the end of the day, the endless comparison of models, features and pricing (along with our shifting “must-haves” and “nice-to-haves”) – well, it is exhausting. 

When you are designing a wide area network for your enterprise, the options and decisions are different than those you face when car shopping, but just as plentiful and exhausting. There are traditional wired WANs vs. software-defined WANs, and within each type of WAN there are still more options to weigh in regard to circuits, services, application prioritization, and providers.

Then there are considerations about your business needs in terms of topology, resilience, quality of service, bandwidth, and network management considerations (should you work directly with providers or perhaps an aggregator?). And of course, staying within budget might be your top consideration.

Ultimately, there are no one-size-fits-all approaches that make designing a WAN for your enterprise easy; it’s more like horseshoes, you want to be “closest to the pin”. This guide explains the logic behind WAN design decisions, compares the service options available, and walks through a step-by-step hypothetical WAN design.

Here's a quick look at the (hypothetical) optimal WAN we'll be building today - if this diagram is gibberish to you - fear not - simply read on.

Overview of WAN types we evaluate in this post 

There seems to be a sentiment that you have to choose a single WAN type for your network, but most wide area networks use a combination of multiple services. So, before diving into the WAN design considerations, this section reviews the different types of wide area networking services. 

If you’re a networking stud who knows all of the ins and outs of different WAN technologies, you can skip this section. 

Wired Solutions 

Point to Point / Leased Lines

Diagram 1 

What are P2P Leased Lines: This is pretty much what it sounds like, building a WAN by procuring closed-network, Layer 1, direct links between any two end points. Each point in the network becomes both a sender and recipient of data. 

Pros of P2P Leased Lines: Direct, dedicated links are a great way to provide consistent bandwidth and reliability for applications that put a premium on high performance and quality of service.

Cons/Limitations of P2P Leased Lines: If there is a desire to grow your WAN quickly and rapidly respond to change, point-to-point leased lines could prove challenging and expensive to scale. For example, if you want to add “Site 5 and Site 6” to Diagram 1, you’ll need to go through a P2P procurement process to buy more dedicated links and then wait for them to be installed. In a perfect world, every network node in your WAN would have a P2P line to every other node in the network (“any to any” topology), but this would be extremely hard to build and expensive - so rather, P2P links are often used as a portion of a WAN, connecting the nodes where low latency and high reliability are of the utmost importance. We discuss network topology further in the “WAN Design” section. 

Wavelength Services 

What are Wavelength Services? Wavelength services are the same as P2P leased lines on Layer 1, but utilize Dense Wavelength Division Multiplexing (DWDM) technology at Layer 2, which create transmission paths using multiple light wavelengths to send data over a strand of fiber. Wavelength services are typically deployed in a P2P topology, but don’t get that confused with P2P leased lines - read more here: Point to Point Connections vs. Wave Circuits

Pros of Wavelength Services: DWDM and wavelength services are designed specifically to handle data-intensive applications, so they are appropriate if bandwidth is the highest priority. For example, some types of DWDM hardware support up to 192 channels in a single fiber cable, each carrying up to 100 Gbps.

Cons/Limitations of Wavelength Services: Wavelength services may not be widely available in some areas, and are not a good fit for customers that do not have very high bandwidth needs. Additionally, wavelength network quality is greatly affected by the condition of the fiber and bends in the fiber. 

Dark Fiber 

What is Dark Fiber: Dark Fiber is fiber infrastructure that already has been laid out by a telecom service provider, but that the provider has not “lit” or used yet. A provider might strategically plan to set aside some amount of Dark Fiber that it can resell to other carriers, but also to enterprises in need of high bandwidth connections. Companies that buy dark fiber can install their own equipment on it to create dedicated private network routes.

Pros of Dark Fiber: This option gives companies access to a lot of dedicated bandwidth quickly as their needs grow, and allows for optimal privacy, security and reliability.

Cons/Limitations of Dark Fiber: Dark Fiber isn’t for those who don’t want to get their hands dirty in the process of buying fiber, installing equipment and managing their own network facilities and security. If you have a lean and/or low skill IT team, dark fiber is not the right choice for your network. 

MPLS

Diagram 2 

What is MPLS? Multiprotocol label switching (MPLS), as the phrase suggests, is a layer 3 networking technology that applies “labels” to packets which assign classes of service to different types of traffic. These labels serve as indicators of how packets should be prioritized for delivery.  

Pros of MPLS: Using the labeling system on proprietary equipment makes MPLS extremely predictable and secure, resulting in guaranteed high Quality of Service. The service provider manages MPLS end-to-end, and provides service level agreements (SLAs) guaranteeing that performance expectations will be met. In other words, with MPLS, enterprises don’t have to get their hands dirty. Additionally, given MPLS is more of an old school technology, it is widely available from telecom providers. 

Cons/Limitations of MPLS: MPLS can be expensive, and enterprises may not be able to scale their MPLS network as quickly as they would like because they are dependent on using the same service provider at each location in their network. Additionally, MPLS uses a centralized hub-and-spoke architecture that is not well-designed for the increasingly decentralized nature of cloud and SaaS usage, or for rapid expansion. Additionally, reporting capabilities are limited with MPLS. 

Direct Cloud Connectivity 

Diagram 3

What is Direct Cloud Connectivity? Direct cloud connectivity is a specific application of a P2P WAN, riding on a point-to-point network topology that provides users with private, dedicated, Layer 1 connections between their locations and the enterprise’s cloud. Traffic does not travel over the public internet. Enterprises also can get “hosted” direct cloud connectivity from an outside provider that sets up private, direct connections between their data centers and the customer’s cloud. 

Pros of Direct Cloud Connectivity: A private connection means predictability and security compared to the uncertainty of traveling the public internet. Direct cloud connections also have become increasingly available in recent years, so more enterprises can get them exactly where they need them, while having third-parties manage these connections rather than having to manage them on their own.

Cons/Limitations of Direct Cloud Connectivity: Using direct cloud links from one provider may require investment in a Plan B redundancy option in the event of outages. If you have a lot of direct cloud connections from different providers in different markets, tracking and reporting can get complicated.

Software defined solutions

For quick reference here, check out our post on SD-WAN vs. VPN. 

SD-WAN 

Diagram 4 

What is SD-WAN? SD-WAN is a virtual software overlay to a physical underlay network that uses the public internet or rides over an existing private WAN to transport traffic. The software-defined features allow enterprises to monitor and dynamically route traffic and correct connection issues in situations where traffic bottlenecks impact performance. Enterprises can define priority for different applications or traffic types.

Pros of SD-WAN: SD-WAN can greatly simplify self-management of an enterprise network by providing a centralized view with intelligent control and routing options–although businesses can turn to managed SD-WAN offerings if they don’t want to go the DIY route. The overlay also can be used to overcome traffic issues or disruptions that crop up on underlay networks.

Cons/Limitations of SD-WAN: If your SD-WAN utilizes solely (or primarily) the public internet, it can be susceptible to the bandwidth ups and downs that tend to occur when using the internet, and without the insurance policy of private network SLAs. Additionally, some enterprises might be concerned about making the switch to SD-WAN given the migration headache and potential security concerns. The security concern can be overcome with appropriate configuration, and the migration headache can be managed by working with the right SD-WAN partner. 

VPN

What is a VPN? Virtual Private Networks set up encrypted tunnels for users, lending private network security, privacy and reliability to enterprise communications even as they travel over public networks. Check out this post on SD-WAN vs VPN for a deeper dive here. 

Pros of VPNs: In addition to extending the advantages of private networks across public networks, VPN technology is mature, widely available and has earned the trust of IT experts.

Cons/Limitations of VPNs: Bandwidth performance through a VPN can be slower because traffic makes an extra stop at a VPN server, and the encryption and decryption process adds overhead that leaves less overall bandwidth for the traffic. 

Network Backbone Considerations 

We’ve discussed the pros and cons of the most common WAN solution types. But how do they stand up when comparing these solutions head to head?

Networking studs, again, you can skip this section.  

Point to Point vs. Waves

If you are debating traditional P2P lines vs. Waves, know up front that neither technology can claim easy scalability as an advantage due to the P2P topology.

Point-to-point leased lines are the more mature solution and thus more widely available. Providers themselves have decades of experience managing them, and in terms of bandwidth availability and reliability, you get what you pay for. 

But, wavelength services are built to support the modern demands of an increasingly data-intensive world, where more of your applications are likely to revolve around data and especially video content.

Waves also generally are less expensive than point-to-point, primarily due to the fact that once providers have DWDM in their fiber networks, they can enable many wave circuits over a single strand of fiber. However, increasing competition with the former continues to drive down the cost of the latter (how do we know that? Check out Lightyear’s WAN pricing guide which shares aggregated data from actual telecom quotes we’ve procured).

Dark Fiber vs. Point to Point

These two services offer similar advantages–privacy, security, reliability–but where they start to diverge is in who does most of the network management work and bandwidth capabilities. 

Point-to-point means an enterprise can point the finger of blame at the service provider if something goes wrong because it is provider managed. When something goes wrong with Dark Fiber, the finger of blame will point right back at your IT team who manages the fiber. 

Enterprises have to buy and manage their own lightwave networking gear when they invest in Dark Fiber, so they need to be sure they are ready for that investment and the related technical and operational challenges. Additionally, Dark Fiber may not be as widely available as Point-to-Point lines.

On the bandwidth front, if you have lower bandwidth requirements, P2P is right for your network. If your enterprise has massive bandwidth requirements, you should be comparing Waves & Dark Fiber (where the same network management consideration applies). 

MPLS vs SD-WAN

MPLS vs SD-WAN is a hot, and evolving debate. In the 1990s, MPLS introduced the notion of prioritizing different packet classes and ensuring quality of service. MPLS’ quality is still a strength, and it also offers the security of a closed provider network that does not use the public internet.

But flexibility and scalability are becoming more important to many enterprises. They want their traffic to take the most efficient route, along with the ability to reroute around traffic in the event of bottlenecks, and adjust application priority as events and trends create temporary surges in demand. 

SD-WAN is generally less costly and easier to configure than MPLS. It can be self-managed, co-managed or you can leave all the management headaches to a provider, and SD-WAN services increasingly are being packaged with Secure Access Service Edge, Zero Trust and more layers of protection that are making it a more secure mode of networking.

For more information on the MPLS vs SD-WAN debate, check out the Definitive Guide to Transitioning from MPLS to SD-WAN.

MPLS vs. a network of P2P Links

Both technologies are very widely available, and while Point-to-Point is more mature, MPLS has been around for a long time and service providers have plenty of experience managing it. 

Both technologies traditionally have been more expensive than other emerging competitive options, but competition continues to cause price erosion on both Point-to-Point and MPLS. Even so, Point-to-Point may be generally less expensive than MPLS. Neither technology is easy or cost-efficient to scale.

Though it is the newer of the two, MPLS is more architecturally rigid, and not well-suited to a time when users need to connect directly to clouds to use enterprise resources and applications. 

Network Design Considerations 

All of the above gives you a sense of the network backbone and service options available to you as you plan your WAN, and some of the decisions you will need to make (i.e. MPLS vs. SD-WAN). 

Making WAN plans involves much more than the backbone considerations discussed above… Here are a few network design considerations to weigh when selecting your WAN service type(s) and provider(s). 

WAN Design Considerations Summary

• What’s your top priority: low latency, high throughput, or lowest cost? We know you want all three, but if you had to pick one? 

• Which applications do your locations use most heavily and where are they housed? Which clouds do you utilize most heavily? 

• What are your IT team capabilities and time constraints?

• How will each location/node utilize the network in terms of applications and use cases?

Network Topology Considerations

Diagram 5  

In a perfect world, every network node would have dedicated, low latency P2P links in an “any-to-any” topology, but as discussed in the previous sections, that is an extremely hard to build, expensive, and inflexible WAN design. So, topology compromises must be made based on your network priorities. 

How do you want each network node to be connected, if at all? You can make those decisions based on these factors: 

Network Quality Requirements

What are your quality requirements in terms of latency, jitter and packet loss?

Some network nodes will require higher network quality (e.g. your colocation facility or DRaaS data center which have ultra high bandwidth and availability requirements) than others (e.g. your administrative office); use this information to decide what type of WAN service links are built into your network and where.

Additionally, the distance between network nodes and paths that your data takes has an impact on your network’s latency. For these nodes, you should utilize the closest geographic location and most direct path in your topology, if possible. 

Application Prioritization/Cloud Readiness 

What are the key applications and clouds to be used through each of your network nodes? 

In an enterprise WAN, you likely have certain applications where low latency is mission critical - for these locations, you should bake in the most direct topology possible (e.g. a P2P link to your headquarters or a direct cloud connection to your Salesforce instance). You might also choose to colocate in one data center vs another due to its proximity to your mission critical applications. 

You should use your top applications and clouds to help determine the optimal network topology for your WAN. 

Downtime Tolerance/Redundancy Considerations

What is your level of tolerance for downtime at each network node? This will help you  determine if or what kind of network redundancy you need to build into your topology.

Telecom Provider Considerations

You’re probably aware that there’s more than a few things to consider when choosing the type of telecom provider you want to work with when designing, procuring and managing your wide area network. 

When designing and procuring your WAN, here are a few best practices to keep in mind regarding your telecom providers: 

• Carrier diversity  - utilize more than one carrier to avoid widespread outages, however the more providers you work with, the more contracts, portals and sales representatives you’ll be juggling after installation 

• Carrier type - we have written some helpful resources on the different carrier types; check out this post on Telecom providers vs telecom aggregators and this post on Telecom agents  

• On-Net vs Off-Net Providers - utilizing on-net providers (e.g. the providers who already have fiber at your location) is always the economic decision, however sometimes paying for construction by an off-net provider is necessary when designing the optimal WAN 

• Managed vs unmanaged services - Across carriers, their definition of a “managed service” will vary. Be sure to understand what is the carrier’s responsibility and what will fall on your IT team

• Circuit types they sell - some carriers sell solely Type 1 or Type 2 circuits, while others sell a mix 

• Customer support - be sure to vet the reputation of every carrier you choose to work with 

Hypothetical WAN Design Example

Disclaimer! 

First and foremost we must reiterate – if building a network was a “pass/fail” class, it would be easy. However, it's not. There will almost always be more than one WAN design that will “work”.

So here, we’re walking through a hypothetical WAN design. 

Set the Stage

In this example, our hypothetical network includes the following nodes:

• 1 Headquarters (core infrastructure)

• 1 Primary Data Center (core infrastructure)

• 1 Backup Data Center (core infrastructure)

• 4 branch offices 

• Remote workforce 

Here is where this topology is going to end up (the same image shown in the intro to this blog) - as shown in Diagram 6. 

Diagram 6

Step 1: Scope the Core

Determine which sites are critical and part of the “core” infrastructure. 

• In this example we’ve determined that the Corporate Headquarters and the two data centers (primary and backup) are mission critical.  

• Carrier diversity should be used on these links to help eliminate a single carrier network as a single point of failure.  Most carriers have built in redundancy and do a great job of ensuring that their network is resilient.

Analyze the throughput requirements that are needed for applications and users to work efficiently and effectively. 

• For this hypothetical, we’ve determined that the core connections are all latency sensitive and require a very static and reliable environment.

Based on these considerations, three types of connections come to mind:

• Point to Point

• Wavelength Services 

• Dark Fiber

Step 2: Select Providers 

Select your providers. You need to choose providers that are able to provide the throughput and connectivity that is required to meet the objectives laid out in step 1. 

Carrier Diversity: It’s best practice to build in carrier diversity as it adds a significant amount of resiliency to your network, meaning you’re less susceptible to network outages.  For our hypothetical core WAN infrastructure, we selected AT&T, Verizon, and Lumen as the carriers. It has been my experience that carrier diversity adds a significant amount of resiliency, but it also creates several vendors that need to be managed. 

Most data centers have several providers to choose from.  For the main data center, we used AT&T and Verizon for the P2P connections, Cogent’s MPLS, with Cox and Lumen used for the SD-WAN internet connections.  At the secondary data center we used Lumen and Verizon for the P2P connections, Cogent’s MPLS, with AT&T and Comcast on the SD-WAN internet connections.  This is what diversified connectivity looks like to me!  

On-Net: It’s also best practice to first consider the providers that are already “on-net” for your project, meaning no new construction would be required to service your WAN. In this “optimal WAN” core infrastructure example, we are going to end up utilizing five different carriers just at the corporate, core network level, but it’s imperative to understand that there won’t always be X number of carriers ready to service your needs! Determining which providers are “on-net” and able to service your needs with minimal new construction is a very important step in the WAN design process. 

Procurement: Determining which providers are viable for your network (offer the services required, are on-net, etc.) is a project in and of itself. The next step is to actually reach out to the carriers and procure quotes for these services. In order to get the best pricing, you’ll want to run the broadest RFQ possible which is extremely time consuming. 

This is a core benefit of working with Lightyear - our telecom procurement software completely offloads your telecom sourcing headaches and guarantees the best pricing for your services by using data and automation (our customers achieve 20%+ savings when procuring with Lightyear). 

Anyways, here’s a visual of our core infrastructure using dedicated static links via multiple providers - as shown in Diagram 7.  

Diagram 7

These links are normally set up as full duplex and allow traffic to flow freely in both directions and effectively create a ring topology.  The data flow of traffic is represented by the yellow dashed lines - as shown in Diagram 8.

Diagram 8

Step 3: Build Redundancy & Resiliency 

Core Infrastructure: Single Site Outage 

In the event that any one of the three links goes down, the sites will remain connected by routing around the failed link - shown in diagram 9.

Diagram 9  

Ring around the network, pocket full of packets, outage… outage… we all fall down.  

Core Infrastructure: Multi-Site Outage

There could come a time when one of the sites becomes isolated - aka has an outage - aka the stuff of IT Engineers’ nightmares.  An outage could be due to a misconfiguration, or perhaps due to environmental factors like a power outage or overheated HVAC. 

When the site becomes isolated, oftentimes the IT Engineers have no visibility into why or how this happened.  Compound that with instances where the site that’s down is remote (like a data center) and you have a nightmare situation on your hands - as shown in diagram 10. 

Diagram 10

My recommendation as a best practice and an insurance policy against this sort of multi-site outage is installation of a console server at each site.  

Connectivity to the console server would be via a separate carrier - adding a 4th provider into the mix.  There are network requirements that would be best suited for MPLS, and others that would be best suited for VPLS.  Either one would provide dedicated secure access between sites for a console server network.  

For illustration purposes, we selected Cogent for the MPLS/VPLS connectivity between console servers.  There are several other viable carriers that could have been selected for this connectivity - as shown in diagram 11.

Via the separate “out of band” access, IT Engineers would have additional insight and console access to equipment at remote data centers.

Diagram 11

At this point, we have a viable and hardened core network infrastructure! 

Here’s another look at the core with the provider logos removed, just for readability’s sake :) - as shown in Diagram 12.

Diagram 12

Step 4: Connect Branch Offices

Now that we have a hardened core with a significant amount of diversity and redundancy built in, we need to connect our WAN to the branch offices. In this hypothetical scenario, our branch sites are not part of our “core” network infrastructure and are less mission critical. This means that we can afford to connect to these sites via a more flexible and cost effective solution than the P2P/Waves utilized in our core. In this instance, I would recommend an SD-WAN overlay for its flexibility, scalability, and routing capabilities. 

In a perfect world, the SD-WAN’s underlay internet connections will be carrier diverse, just like in our core.  Here, we are utilizing primary and secondary redundant circuits out of each of our core network nodes to connect to the public Internet, cloud services and our branch offices - as shown in Diagram 13.

Diagram 13

Step 5: Connect Remote Workforce

The last design element that we’re going to add to this hypothetical network are the SD-WAN connections for remote sites.  

In our network, remote workers are required to provide their own connection to the public internet (i.e. the best effort, broadband internet in their homes), and a secure connection into the corporate network is enabled by their employer via SD-WAN or VPN. 

The Optimal WAN

We did it - we designed the optimal WAN!  

The graphic below showcases the optimal WAN with carrier diversity information included - see Diagram 14. 

Diagram 14

How to achieve the Optimal WAN

I know that building and deploying networks is an exhausting process - I’ve built and managed thousands of them. I hope that this guide is helpful and provides a framework for you when designing your own wide area network.

If after you consider all the different WAN service options you find yourself unable to draw up an absolutely perfect WAN plan, do not panic. Planning the optimal WAN for your enterprise is an extensive process of gathering information, making compromises and tough decisions - and perfection is nearly impossible to achieve. 

This walk-through covered the networking considerations of WAN design in depth but only scratched the surface regarding the cost implications, procurement, implementation and management of a wide area network. Lightyear’s software automates all of these processes for you - for free. Schedule a demo here to learn more.