My parents and grandparents lived through hard times. They taught me the virtue of thrift: “use it up, wear it out, make it do or do without”; “a penny saved is a penny earned”; “waste not, want not”; “money doesn’t grow on trees”; “take care of the pennies, and the dollars will take care of themselves”; “the bitterness of poor quality outlasts sweetness of low price”. Professionally, that philosophy translates to the term “value engineering”.
So I have something I have to get off my chest. All too often, community broadband deployments cost more than they have to, for reasons that do not create commensurate value. Or, to put it bluntly, money is wasted.
Anybody who’s been following community broadband recognizes the difficulty of funding these vital projects. Municipalities, co-ops and small utilities seeking to build broadband infrastructure must cobble together a financing plan from various sources of money: federal and state “alphabet soup” programs, private grants, municipal bonds, bank loans, taxes, assessments, fees, retained earnings from utility operations, etc. Finding the money requires time and effort. Fund-raising sometimes falls short of project cost estimates, killing or delaying the project. Time to positive cash flow is scrutinized by potential lenders and bond ratings agencies. Funding a broadband project can affect other community priorities. Most important, project cost is ultimately paid for by the public — it’s their money and their trust.
For those reasons, every dollar of capital expense and future operating expenses must be spent wisely. That goes beyond public ethics and fiscal responsibility (about which, unfortunately, the FCC recently needed to remind some folks). It also means having a clear-eyed understanding of what the project is supposed to accomplish, eliminating or postponing anything not critical to that mission, and optimizing for low life-cycle cost with high value.
Some activists see community broadband as a way to create competition, overthrow the incumbent monopolies, enable anyone to create killer applications, unlock hidden creativity, actuate democracy, revolutionize the economy, empower citizens, and save the world. Grandiose visions lead to grandiose projects. Pragmatically, those are not the problem that communities need to solve for themselves. Their problem is concrete and immediate: lack of adequate broadband service, at reasonable prices, with good customer support. This problem, with its familiar litany of symptoms – frustrated citizens, businesses driven away, kids doing their homework in the library parking lot at night, adults unable to telecommute, depressed property values, etc. — is self-evident. Solving it should be the laser-like focus of the business plan, network architecture and budget.
I keep hearing things in community broadband conferences, blogs, and webcasts which make me cringe, knowing that they lead to overly expensive projects:
- “As long as we’re putting in fiber, we might as well put in lots of it in fat cables. Who knows what we might want it for later?” Fiber-rich deployment can be appropriate in the backbone of the network: the middle mile, long haul and between data centers. It doesn’t scale to the extremities: residential and small business access networks. There, the cost of more fiber mounts quickly, for a lot of not-always-obvious reasons. I’ve written extensively on this subject for my clients. And by the way, a looming global supply shortage of optical fiber portends higher prices in the near future. Last mile networks should be designed fiber-lean, with reasonable margin for expected growth, plus a little extra.
- “Every customer should have a dedicated point-to-point fiber, all the way to the switch….” In urban centers, “home-run” architectures are sometimes the lowest cost solution. They are also justified for large customers like office towers, data centers, hospitals, universities and wireless systems. However, for community broadband projects primarily serving residences and small businesses, shared fiber Passive Optical Networks (PONs) are usually the most cost-effective architecture. This is particularly true in rural and semi-rural areas, because of the distances involved. Home-run architectures can dramatically inflate project cost – sometimes by more than 100%!
- “…so they get a first-class experience.” Dedicated fiber does not offer a perceptibly better customer experience than properly engineered shared fiber. The reasons are complex, as I explain in an article that I wrote for Broadband Communities The bumper sticker version is that a fiber’s bandwidth is either being used or wasted at any instant of time. Each customer actually uses only a tiny fraction of the available bandwidth. If that bandwidth is shared, more can be used and less is wasted. With proper engineering, there’s almost always more bandwidth available than is being used at a given time. This kind of resource sharing is how the Internet works, and is the main reason for its success. There’s no reason why the first mile should be any different.
- “…because we’re building an open access network.” Some broadband network plans require “open access”. This means that infrastructure is owned by a Network Provider (NP), and shared by multiple Service Providers (SP), which offer Internet and other services to retail customers. This recognizes that infrastructure tends to be what economists call a “natural monopoly”; capital cost, rights-of-way and market dilution present high barriers to new competitors. Open access brings consumers the benefits of competition for services, which are not natural monopolies. Open access can also simplify community broadband business plans by assigning ownership of lots of headaches to the SPs, which are organized to deal with them.The boundary between the NP and the SP can be drawn in several different ways. Some of them involve dedicating a point-to-point fiber from each customer to an NP-owned interconnection point, to which the customer’s chosen SP connects their own dedicated fiber. These approaches, in addition to suffering from the higher cost of dedicated fiber, present a number of operational problems. For example, when competition incents customers to switch SPs, change orders have to be performed by manually disconnecting and reconnecting fibers. Determining responsibility for problems can be tricky, and often ends up in a finger pointing contest between SP and NP.Instead, I advise interconnection between NP and SP using “virtual open access”. “Virtual Local Area Networks (VLANs)” are a standard Ethernet feature which is integral to common FTTH technologies. The NP supplies VLANs to connect each customer to the SP of their choice. Change orders are performed entirely through software. Virtual open access has significantly lower capital and operating costs, and fewer headaches, than physical open access approaches. My recent article in Lightwave Magazine discusses this in detail. As a bonus, virtual open access makes it easy for a customer to have more than one SP (for example, one for family use and one for a home office).
- “Symmetric bandwidth is an absolute requirement.” Residential and most small business customers consume much more traffic than they generate. This fact is well confirmed by measurements of live networks, and easily explained by application requirements, consumer behavior and typical small business needs. No foreseeable application is likely to change that. In fact, growing mass-market penetration of 4K/UHDTV video will increase the asymmetry.Asymmetric bit rates in some FTTH technologies have a valid engineering rationale, unrelated to sinister industry plots: cost optimization. The reasons have to do with the manufacturing costs of various kinds of lasers, and the way that light travels through fiber. It’s a long story, of interest only to fellow engineering geeks. Other FTTH technologies attain bit rate symmetry either at higher cost, or by unnecessarily restricting the downstream bit rate to match the most cost-effective upstream rate.Asymmetric bit rates do not preclude symmetric services. Many providers use a PON technology called GPON (2.5 Gigabits per Second (Gb/s) in the downstream direction, 1.25 Gb/s upstream) to offer 50/50, 100/100, 500/500 and even 1000/1000 Megabit per second (Mb/s) services. They do this by taking advantage of bandwidth sharing and the asymmetry of user traffic. It works.
- “Gigabit is so cliché. We’re going to offer 10 G”. If bragging rights and one-upping other communities are primary objective, then go for it. However, equipment is significantly more expensive – by multiples of 1.5 to 3 for 10 Gb/s, 5 to 8 for 40 GB/s per fiber. Few customers can utilize 1 Gb/s service, never mind 10 Gb/s. As I explained in my Broadband Communities article, 10 Gb/s doesn’t even necessarily offer 10 times the performance of 1 Gb/s. I recommend marketing a 1 Gb/s flagship service to residential and small-to-medium business customers, while offering large customers 10 Gb/s service by special order.
- “We can’t afford paid experts. Some of our volunteers have science and technology backgrounds. They’ll figure it out.” Community broadband efforts usually start with activists who volunteer their time and know-how. Occasionally, a volunteer might have worked in a planning, engineering or technology role at a telecom or cable company. Or there might be a volunteer with a background like mine, on the equipment vendor side of the telecom industry. Not many community broadband projects are fortunate enough to have those resources. Technical committees tend to consist of information technology (IT) professionals, software developers, engineers and scientists in unrelated disciplines, and technology enthusiasts. While these folks are presumably highly intelligent and knowledgeable in their own field, they lack specialized knowledge of FTTH technologies and FTTH engineering economics. Without guidance from subject matter experts, they are likely to lose their way, and fall into the traps I discuss above. Some technical committees spin their wheels in poorly informed arguments and thus can’t agree on direction, make decisions, determine next steps, or generate actionable plans. Not bringing in appropriate expertise for these projects is “penny wise, pound foolish”.
These themes recur all too often. I hate to think how much money has been wasted because of them. There’s too much misinformation and hyperbole out there, and too often the perfect becomes the enemy of the good.
If you’re organizing a community broadband network, I want to help you make sense of all of this, so you don’t waste your money. Contact me at: firstname.lastname@example.org.