Category Archives: NetAccess Futures

Mind your Money

My parents and grandparents lived through hard times. They taught me the virtue of thrift: “use it up, wear it out, make it do or do without”; “a penny saved is a penny earned”; “waste not, want not”; “money doesn’t grow on trees”; “take care of the pennies, and the dollars will take care of themselves”; “the bitterness of poor quality outlasts sweetness of low price”. Professionally, that philosophy translates to the term “value engineering”.

So I have something I have to get off my chest. All too often, community broadband deployments cost more than they have to, for reasons that do not create commensurate value. Or, to put it bluntly, money is wasted.

Anybody who’s been following community broadband recognizes the difficulty of funding these vital projects. Municipalities, co-ops and small utilities seeking to build broadband infrastructure must cobble together a financing plan from various sources of money: federal and state “alphabet soup” programs, private grants, municipal bonds, bank loans, taxes, assessments, fees, retained earnings from utility operations, etc. Finding the money requires time and effort. Fund-raising sometimes falls short of project cost estimates, killing or delaying the project. Time to positive cash flow is scrutinized by potential lenders and bond ratings agencies. Funding a broadband project can affect other community priorities. Most important, project cost is ultimately paid for by the public — it’s their money and their trust.

For those reasons, every dollar of capital expense and future operating expenses must be spent wisely. That goes beyond public ethics and fiscal responsibility (about which, unfortunately, the FCC recently needed to remind some folks). It also means having a clear-eyed understanding of what the project is supposed to accomplish, eliminating or postponing anything not critical to that mission, and optimizing for low life-cycle cost with high value.

Some activists see community broadband as a way to create competition, overthrow the incumbent monopolies, enable anyone to create killer applications, unlock hidden creativity, actuate democracy, revolutionize the economy, empower citizens, and save the world. Grandiose visions lead to grandiose projects. Pragmatically, those are not the problem that communities need to solve for themselves. Their problem is concrete and immediate: lack of adequate broadband service, at reasonable prices, with good customer support. This problem, with its familiar litany of symptoms – frustrated citizens, businesses driven away, kids doing their homework in the library parking lot at night, adults unable to telecommute, depressed property values, etc. — is self-evident. Solving it should be the laser-like focus of the business plan, network architecture and budget.

I keep hearing things in community broadband conferences, blogs, and webcasts which make me cringe, knowing that they lead to overly expensive projects:

  • “As long as we’re putting in fiber, we might as well put in lots of it in fat cables. Who knows what we might want it for later?” Fiber-rich deployment can be appropriate in the backbone of the network: the middle mile, long haul and between data centers. It doesn’t scale to the extremities: residential and small business access networks. There, the cost of more fiber mounts quickly, for a lot of not-always-obvious reasons. I’ve written extensively on this subject for my clients. And by the way, a looming global supply shortage of optical fiber portends higher prices in the near future. Last mile networks should be designed fiber-lean, with reasonable margin for expected growth, plus a little extra.
  • “Every customer should have a dedicated point-to-point fiber, all the way to the switch….” In urban centers, “home-run” architectures are sometimes the lowest cost solution. They are also justified for large customers like office towers, data centers, hospitals, universities and wireless systems. However, for community broadband projects primarily serving residences and small businesses, shared fiber Passive Optical Networks (PONs) are usually the most cost-effective architecture. This is particularly true in rural and semi-rural areas, because of the distances involved. Home-run architectures can dramatically inflate project cost – sometimes by more than 100%!
  • “…so they get a first-class experience.” Dedicated fiber does not offer a perceptibly better customer experience than properly engineered shared fiber. The reasons are complex, as I explain in an article that I wrote for Broadband Communities The bumper sticker version is that a fiber’s bandwidth is either being used or wasted at any instant of time. Each customer actually uses only a tiny fraction of the available bandwidth. If that bandwidth is shared, more can be used and less is wasted. With proper engineering, there’s almost always more bandwidth available than is being used at a given time. This kind of resource sharing is how the Internet works, and is the main reason for its success. There’s no reason why the first mile should be any different.
  • “…because we’re building an open access network.” Some broadband network plans require “open access”. This means that infrastructure is owned by a Network Provider (NP), and shared by multiple Service Providers (SP), which offer Internet and other services to retail customers. This recognizes that infrastructure tends to be what economists call a “natural monopoly”; capital cost, rights-of-way and market dilution present high barriers to new competitors. Open access brings consumers the benefits of competition for services, which are not natural monopolies. Open access can also simplify community broadband business plans by assigning ownership of lots of headaches to the SPs, which are organized to deal with them.The boundary between the NP and the SP can be drawn in several different ways. Some of them involve dedicating a point-to-point fiber from each customer to an NP-owned interconnection point, to which the customer’s chosen SP connects their own dedicated fiber. These approaches, in addition to suffering from the higher cost of dedicated fiber, present a number of operational problems. For example, when competition incents customers to switch SPs, change orders have to be performed by manually disconnecting and reconnecting fibers. Determining responsibility for problems can be tricky, and often ends up in a finger pointing contest between SP and NP.Instead, I advise interconnection between NP and SP using “virtual open access”. “Virtual Local Area Networks (VLANs)” are a standard Ethernet feature which is integral to common FTTH technologies. The NP supplies VLANs to connect each customer to the SP of their choice. Change orders are performed entirely through software. Virtual open access has significantly lower capital and operating costs, and fewer headaches, than physical open access approaches. My recent article in Lightwave Magazine discusses this in detail. As a bonus, virtual open access makes it easy for a customer to have more than one SP (for example, one for family use and one for a home office).
  • “Symmetric bandwidth is an absolute requirement.” Residential and most small business customers consume much more traffic than they generate. This fact is well confirmed by measurements of live networks, and easily explained by application requirements, consumer behavior and typical small business needs. No foreseeable application is likely to change that. In fact, growing mass-market penetration of 4K/UHDTV video will increase the asymmetry.Asymmetric bit rates in some FTTH technologies have a valid engineering rationale, unrelated to sinister industry plots: cost optimization. The reasons have to do with the manufacturing costs of various kinds of lasers, and the way that light travels through fiber. It’s a long story, of interest only to fellow engineering geeks. Other FTTH technologies attain bit rate symmetry either at higher cost, or by unnecessarily restricting the downstream bit rate to match the most cost-effective upstream rate.Asymmetric bit rates do not preclude symmetric services. Many providers use a PON technology called GPON (2.5 Gigabits per Second (Gb/s) in the downstream direction, 1.25 Gb/s upstream) to offer 50/50, 100/100, 500/500 and even 1000/1000 Megabit per second (Mb/s) services. They do this by taking advantage of bandwidth sharing and the asymmetry of user traffic. It works.
  • “Gigabit is so cliché. We’re going to offer 10 G”. If bragging rights and one-upping other communities are primary objective, then go for it. However, equipment is significantly more expensive – by multiples of 1.5 to 3 for 10 Gb/s, 5 to 8 for 40 GB/s per fiber. Few customers can utilize 1 Gb/s service, never mind 10 Gb/s. As I explained in my Broadband Communities article, 10 Gb/s doesn’t even necessarily offer 10 times the performance of 1 Gb/s. I recommend marketing a 1 Gb/s flagship service to residential and small-to-medium business customers, while offering large customers 10 Gb/s service by special order.
  • “We can’t afford paid experts. Some of our volunteers have science and technology backgrounds. They’ll figure it out.” Community broadband efforts usually start with activists who volunteer their time and know-how. Occasionally, a volunteer might have worked in a planning, engineering or technology role at a telecom or cable company. Or there might be a volunteer with a background like mine, on the equipment vendor side of the telecom industry. Not many community broadband projects are fortunate enough to have those resources. Technical committees tend to consist of information technology (IT) professionals, software developers, engineers and scientists in unrelated disciplines, and technology enthusiasts. While these folks are presumably highly intelligent and knowledgeable in their own field, they lack specialized knowledge of FTTH technologies and FTTH engineering economics. Without guidance from subject matter experts, they are likely to lose their way, and fall into the traps I discuss above. Some technical committees spin their wheels in poorly informed arguments and thus can’t agree on direction, make decisions, determine next steps, or generate actionable plans. Not bringing in appropriate expertise for these projects is “penny wise, pound foolish”.

These themes recur all too often. I hate to think how much money has been wasted because of them. There’s too much misinformation and hyperbole out there, and too often the perfect becomes the enemy of the good.

If you’re organizing a community broadband network, I want to help you make sense of all of this, so you don’t waste your money. Contact me at:




Over the holidays (naturally) some [expletives deleted] inserted some malware into this site.  It redirected viewers to an attack site, no doubt to nefarious ends.   The vulnerability appears to have been in a WordPress 4.0 script.  We’ve updated to WP 4.1 and taken other necessary steps.

Lessons learned:

  • Web sites aren’t “set and forget”.
  • Being a low-value target is not a defense.
  • Security updates don’t happen automagically
  • Don’t trust the tools to set up protections appropriately.
  • Google and are your friends… in the “tough love” sense of friends.
  • My first incident response involved a lot of trial-and-error.  I can’t imagine how a site owner with no CS background could begin to deal with these kinds of problems.

Anyway, all clear.  Maybe someday law enforcement will catch up with these [expletives deleted].

Observations from the Broadband Communities Economic Development Conference

Last week, I attended a conference in Springfield, MA, sponsored by Broadband Communities Magazine, on broadband for economic development. This was the most cross-functional business event I’d attended in a very long time.  There were town planners, activists, civil servants, elected officials, academics, consultants, economists, not-for-profits, educators, healthcare experts, regulators, lobbyists, lawyers, trade groups, analysts, design/build firms, hardware, software and outside plant vendors, among many others.  So in addition to a big stack of business cards, here are a few things I brought back.

What brought all these people together was a problem:  business and residential broadband access is now a necessity,  yet many communities are un-served or under-served by incumbent telecom and cable providers.   Community leaders recognize that since the incumbents cannot make a business case for investment in their communities, the public sector has to step in.  The conversation revolved around “how?” rather than “why?”

Springfield would probably not come up on the typical  conference organizer’s screen.  Yet it is an ideal place to talk about community broadband.  New England is a hive of activity.  All of the states have broadband plans and are in the process of planning broadband networks.   In particular, Springfield is close to WiredWest, a consortium of un-served western Massachusetts towns.  A few western Massachusetts towns outside the consortium are planning their own broadband networks.  A new initiative for broadband deployment in Connecticut was just announced; Springfield is a short drive from Hartford.  Vermont is a straight shot on I-91, and New Hampshire not much further.  And finally, Springfield is the hub of Mass123, the middle-mile network serving the western part of the state.  So there’s lots of good stuff happening within easy driving distance.

My world is centered on the technological  complexity of broadband networks.  So I learned a lot about the many political, regulatory, financial, Federal funding,  legal,  marketing, urban planning, educational and economic development complexities with which the technology must mesh.  I was particularly happy to get a deep dive into the financial analysis toolkit developed by our sponsor.  Attorneys from Baller-Herbst took their audience through a great survey of the legal landscape.

All of this activity is fueled by small piles of Federal cash, plus various State sources.   In particular, the critical middle mile infrastructure was built out mostly with Stimulus money, through the Broadband Technology Opportunity Program (BTOP).  There’s FCC money for educational user under the e-Rate program,  Universal Service Fund money,  Connect America Fund “Experiments” grants.  There are USDA grants.  There are HHS grants.  State grants.  Private philanthropy grants.  And they are all painfully hard to get.   Somehow, the would-be network operator has to cobble together enough of this “free” money to make a financial plan that would pass muster in the (troubled) municipal bond market, or at a bank.  A lot of the folks I met are determined to do that, and came to the conference to figure out how.

I got to meet a lot of people.  Overall, this was an impressive group.  I knew some of them from my past life, some I was meeting in person after first meeting on the web, some I knew by reputation.  I enjoyed a number of deep discussions, on history, economic development, Net Neutrality, and more, as well as the topic at hand.

I did not get to meet very many of my technology peers.  Many of these organizations are running on a shoe-string, or haven’t gotten to the point of figuring how to make their networks work.   Some are drawing on volunteers – local IT professionals,  civil and mechanical engineers, scientists, academics and enthusiasts – to inform the planning and procurement processes.  I heard that they were having fierce internal debates about  topics I’d thought had been settled.   I heard about RFPs that cost more in change orders than the original award. I heard subtle misunderstandings that could lead to poor decisions.   And the most of the technical folks I talked with recognize that broadband is outside their professional experience, or need a second opinion.

Plug:  This is the problem that NetAccess Futures is in business to solve.

As an aside, New Haven, CT is one of the three Connecticut cities that is undertaking a broadband infrastructure project.  It also happens to be the city I grew up in, and my family was deeply rooted in the New Haven community.  It was a privilege to shake hands with Mayor Toni Harp.

I’d like to thank Broadband Communities Magazine for their organization and hospitality.  This is not a typical trade rag.  One pleasant surprise was the degree to which the magazine supports this diverse community.  Plus the editors are multi-talented and exceptionally intelligent and passionate about broadband.  In particular, Steve Ross gave me the better part of an evening for a very enjoyable conversation, and valuable business advice.  For whatever it’s worth, I’ll have an article in their October edition.