I have first hand experience with this, although in Australia. My first job was working for a IT development and solutions shop (this is over 17 years ago) where they won the bid to rollout the internet and network gateways across a large number of Australian government schools.
We were doing similar work for private school at the same time, and the process for the public and private school could not be more different.
For eg. in the tender process, which I was part of, for private school we would use cheaper Taiwanese routers. For the public schools it 'had to be Cisco'. The only time we ever used Cisco, outside of large enterprise clients with 1000+ seats, was with public school tenders. The private school would get a $500 white-label router, the public school s would get a $6-15k Cisco router with additional VPN module costs.
We would charge a higher consultant rate on the Cisco jobs, and would bill 3 days instead of 1. We won the contract because we were 50% cheaper than the other tenders who all wanted to install 3 and 5-series routers. How we got to the point of being able to even tender is another story that involves somebody in our organization sleeping with somebody at the government organization. The other tenderers were accustomed to dividing the work up amongst themselves at inflated prices, they didn't even know who we were and we received a lot of abuse for breaking up their little scheme.
(Edit: a further idea of how this worked, the 3 people in the gov office responsible for tenders all had very nice cars and holiday homes while the rest of the office was working away on below-average wages. You could see what was going on just by looking at the car park)
So a dozen of us roll out hundreds of these routers in public schools and after a month we find that we rolled out the wrong version of IOS, one that was vulnerable to a simple security attack. Instead of forcing us to upgrade all the routers remotely, or out of our own pocket, we instead won another few-million-dollars worth of work to send a person out and apply the upgrade to each router (which took 5 minutes, we charged a full day plus travel).
The routers weren't even being used properly - the topography was net connection -> cisco router -> internal server -> switches. The internal server would do all the DHCP and everything else. These expensive routers were being used as bridges, although they were pitched as having 'forward compatibility' incase the school wanted to implement features such as user accounts (they did, although again they used a custom server, not the router).
When these projects are audited there is nobody who is technically competent enough to make an argument against who would be on the side of ditching or shrinking the projects. Some of the smarter teachers knew what was going on but didn't mind since they got access to fancy equipment (we would create user accounts for them).
My first, and not my last, experience with government bureaucracy and budgets. I would estimate that the private schools got more out of us at a tenth of the cost. Since then I had an even worse experience with the government health department, where 6-figure invoices were written and paid for goods that didn't exist (that department has since been broken up and the subject of a large corruption enquire). No surprise that I became very anti-government size and spending.
Edit: to add, we were so 'disruptive' to the backdoor deals that we were uninvited from conferences, kicked off panels, not invited to the mixer events where gov buyers met providers, a couple of years later we lost our accreditation[0] temporarily until we appealed to the Government Minister. On site we would be locked out of network cabinets, not given IP information for the net connections, etc. Our jobs were made difficult by competitors and others because of our pricing and methods. They couldn't figure out how they didn't get rid of us, because they didn't know that one of our guys had a solid relationship with somebody at the government (that person wanted to clean things up). Usually we wouldn't have been allowed anywhere near these projects and if we did win one we wouldn't be allowed back in for not playing with the system.
[0] 'accreditation' for government tenders needs to die, it is a formal method used to keep honest operators out of what is essentially a cartel. The Australian government is getting better in this regard[1], they now have an open tender website but I believe it still requires some form of accreditation that has a person in a department standing between application and approval.
[1] although not too much better. NBN Co., a government owned company that is building a $40 billion nation-wide fibre network recently suspended tender process because every single bid came in over the expected price:
You get similar stuff at private companies too, though few private companies are large enough for corruption like that to be able to reach the same scale without getting too big to go unnoticed by auditors for long enough.
The number of times I've had potential supplier hint at extra perks or "return favours" is fairly high, and I've not even been in positions of very substantial purchasing power.
It's often not even direct exchanges that are being hinted at either, but "simply" seemingly innocent networking: Buy from the "right" well connected sales guys and make it clear you're "flexible" about requirements, and they know they benefit from exploiting their network to the max helping you get lucrative jobs in big companies where you can spend more with him.
It's often only morals that prevents it from happening - I've been in plenty of situations while where my budgets were not huge, nobody knew enough to second guess my decisions, and I could've easily spent a lot more than I did without anyone batting an eyelid. Especially since just the very act of bloating your budget in many places makes you look more impressive and important and translates into status (and looks good on your CV...)
This was my 2nd-hand impression growing up with a father who worked for a large oil company. The extent to which nobody was really at the wheel in these kinds of negotiations was fascinating. As far as I could tell, many things sort of worked, when they did, because engineers had a cultural desire to "get it right", mostly for their own satisfaction. But the company itself didn't seem to either have an effective way of policing it, or maybe even really an interest. Somewhat as in government, low- and mid-level contracting didn't excite anyone in management, who were focused on issues that really "moved the needle", like securing and overseeing new exploration deals, not how much someone spent on a piece of equipment that wasn't even in the 6-figure range anyway.
One main difference is that public companies are required by law to have independent auditors.
Government departments are not.
In Australia independent audits rely on the Australian National Audit Office, which is profoundly and deliberately underfunded because of its annoying habit of discovering embarrassing incompetence and wastefulness.
Waste in terms of inflated costs is more likely a function of size more than public/private dimension. 20k $ is literally not worth lowering down to get the money in some industries I have witnessed. I have yet seen obvious examples of corruption, and incompetence + incentive is the prevalent explanation: the bigger a project is, the better it is for the people at the top of it obviously, and enough bureaucracy to make it a local optimum.
One typical example I had is not receiving a replacement laptop when working for a bank because replacement hw is too expensive, so I could not work for essentially one month. My consulting rate would have paid for a top of the line laptop in 2 days. But those are of course two different cost centers, so it will never surface.
Sometimes I feel like the only long-term Australian (state) government-employed IT professional reading HN that doesn't regularly witness widespread corruption, incompetence and overspending. Seriously, in almost ten years, the very worst thing I've seen is someone being fired over stealing an old desktop PC instead of disposing of it via proper channels.
I've always worked with people that, even if not cutting-edge geniuses, are generally more than competent. They don't accept dodgy tender because they know the value of what they're buying, and they certainly don't overspend on gold-plated crap because our budgets are always stretched to the limit.
I would genuinely like to work for one of these mythical government departments where money flows like water. In my experience, the budget is always barely enough to fulfil our role and all these stories about profligate government waste are little more than hearsay.
Then again, I've never worked in the Health or Mining industries.
There seems to be something very broken with contracting in the US, which seems more specifically US government operations than government in general. If you look at something where you can compare prices between countries, like per-mile subway construction costs, the U.S. is a huge outlier, with procurement costs much higher than Europe.
If you look at something where you can compare prices between countries, like per-mile subway construction costs, the U.S. is a huge outlier, with procurement costs much higher than Europe.
The higher-than-Europe comparison I had not heard before. The higher than private industry cost of United States government contracting (and contracting by local units of government when they spend federal funds) is dictated by the Davis-Bacon Act,
once we understand that the law's requirement to pay "prevailing wages" means, in that context, to hire union labor even when less expensive nonunion labor is available.
The fact that the government is not competent to spend taxpayers' money wisely seems like a good reason to be anti-government. After hearing enough anecdotal evidence on this topic (which very few people have both the capability and desire to thoroughly investigate in order to properly quantify it, thus the reliance on anecdotal evidence), it starts to appear as if such incompetence is an integral feature of all governments once they reach a certain size.
If you don't like this kind of behaviour you are doing yourself a disservice by describing it in as simple a manner as you have. Having had similar experience it is surprising that they were allowed to tender that offer at all.
I've seen examples of this all over the UK as well, when ever it's a Government tender the quotes were always much higher than they need to be, and they always HAVE to have the most expensive, 'enterprise' equipment, whether they need it or not.
I've never understood it myself. Governments are funded by tax money, and the people involved in these companies that love overcharging the Government pay taxes. Congratulations on ripping yourself off.
> I've never understood it myself. Governments are funded by tax money, and the people involved in these companies that love overcharging the Government pay taxes. Congratulations on ripping yourself off.
Any additional taxes are diluted amongst the general population, while my sales commission is mine and mine alone...
the people involved in these companies that love overcharging the Government pay taxes
One thing I've learned from Private Eye: many of our government contractors don't pay taxes -- or at least not British taxes, or a negligible amount thereof.
Its the level of management and attention to detail your organization can afford.
At some point people will probably need to fully fund and empower an independent audit organization staffed with the kind of detail oriented savants that will start cutting through a lot of fat.
As has been mentioned above, there are many times that people end up doing work only because they are wired to try and do it cheaply/well.
In my experience, the govt. schools themselves have extremely limited IT admin resources available to them to help with this sort of thing. A senior high school I know a little about only has one person doing IT admin, over 1500 students, each with their own email and network login, and is running a combination of Macs and PCs/Windows.
This all stems from the outsourcing mania of the 80s which was driven by the idea you could save money by firing your internal IT people and outsourcing everything. It's been a boondoggle throughout the western world.
Also, the same employer had an extremely low-rent situation going on at the only train station in Sydney I ever witnessed a commuter straight out punch a ticket inspector in the face at. They under paid as well. One learns quickly that in capitalism, it's whatever you can get away with...
I hope you wrote this up detailing the people involved? I guess you it would cause a lot of trouble for you, but if you could do it anonymously, that would go a long way to changing this behavior.
I love that this story starts "sleeping with somebody" and ends with "solid relationship". There's a RomCom script in there somewhere, if the IT thing doesn't work out.
IT Director to Sales Guy: "We just got $20M in grant money for getting broadband across the state. Can you get me some numbers?"
Sales guy to engineering team (partner or internal): "Hey I need lots of boxes. They've got 1300 sites."
Engineering team: "Ok.. what do they need?"
IT Director: "Pretty sure my network guy says everything has to have redundant power supplies and at least 1 ethernet connection. To do a survey for each site would take over a year due to bureaucracy, and I've got 3 months on this grant"
Engineer to Sales guy: "Ok I built out those routers. Do they really need redundant power supplies everywhere? 3900's seem big."
Sales guy: "Ya, that's what they said. Anyways this came out below budget. Thanks!"
IT Director: "Looks to be under budget, meets our needs, thanks!"
.. meanwhile IT management/engineers aren't involved. Somewhere, someone didn't slow this project down to do due diligence. That or somewhere buried in some document is a requirement for redundant power supplies, but that sounds less likely the case.
> Somewhere, someone didn't slow this project down to do due diligence
In it a nutshell.
If anything, there should have been multiple flags raised.
1) A ball park figure should have been estimated on what this should cost, broken down by equipment type.
2) That should have made it clear that switches would be fine for multiple use cases, and the overall estimate should have dropped. *
3) Back and forth with provider as haggling over components proceeds. Massive attention to detail would be shown here.
4) First quote received from Cisco. Due diligence proceeds, issues raised. Customers says NO, just to buy negotiation room. (fine they have a grant, maybe not so aggressive)
5) Updated quote, go back to 4 if issues.
6) Quote acceptance. Now monitoriing of execution.
There should have been many opportunities for flags to have been raised, even before the quote was received. And most definitely after it would have crossed the ball park figure.
* It looks like the clause for redundant power was what tripped it up. Someone said we need X, and then somehow it became the TRUTH for all routers/switches.
So its entirely possible (if not likely) that Cisco said "based on your constraints, these are the only components which match your requirements", and the Government bought it.
That is also a valid point - if the way your system is set up, it would cost more than the difference to send someone out for the survey to verify a cheaper model is enough, just throw in the higher model. Though in this particular case, it's clearly not what happened.
What's worse than overpaying millions for a few routers? Overpaying billions.
At the federal level, specifically to prevent such things from occurring, there's this huge byzantine procurement process. A process that is literally worth tens of billions of dollars to game.
So if you have product foo and want to see a zillion copies of it to the feds instead of some freeware or cheaper solution? 1) Convince somebody on the inside of the procurement system that yours is the best product, 2) they write up the specification so that no other products will qualify (many times just copying directly from your brochure) even though it's technically an "open" bid, and 3) have somebody with lots of procurement knowledge help guide the paperwork through the system. At the end of the day, it's all just paperwork, no matter how much money is involved.
And that's just what I've observed in IT. I'm guessing IT is the worst, since all the products somewhat look alike. But I'm not sure. Quite frankly, its way too depressing to think about much.
BTW, the best way to do #1 is simply hire people retiring out of the procurement system. Yep, there are laws against direct hires, so you hire somebody from DoD to help shepherd a Commerce Department contract, or a DHS procurement expert to help with a DoD job, and so forth. If you do this correctly, the poor schmucks left handling the paperwork will be so happy that you can offer extensive support in making sure everything is done correctly that this is another huge plus in favor of your getting the bid.
The trouble is that you actually do need some of this flexibility in the procurement process. If your department needs Photoshop, you don't want some goofy non-technical auditor telling you that he googled around and thinks GIMP will do just fine. Or you want one particular contractor because just last year they did an amazing job on something that was pretty much exactly what you need.
I'd like to know how much of that $5 million savings would have been burned up in the capacity studies. My guess is a lot.
Given the choice between a fixed cost and an unbounded cost (being the capacity study referenced throughout the article) I'm not surprised they went with the fixed cost.
I'm not saying WV didn't get swindled, but I can sure see how this might have happened:
WV Rep: Do we really need these $20k routers for all our locations?
Cisco: We really couldn't say what your exact needs are without a proper study.
WV Rep: What will that cost?
Cisco: It's really hard to say. We'll have to visit all of your locations and speak with the IT Manager there. We'll have to measure average and peak load. Of course, we'll want to plan for future growth so we're not at this same point in two years...
The real crime is that they didn't open the RFP to multiple bidders. A little bit of competition can go a long way.
- How many users do you have?
- What is the total size of the community you serve?
- Do you need new equipment?
That would have quickly revealed that many of them had so few users that these routers were total overkill, and had so few potential users that even 100% simultaneous usage in some cases would not max out the capacity, and that some of them did not have a need for new equipment.
I agree with you that multiple bidders would be essential. But a tiny little bit of due diligence and investigation on the behalf of WV would have gone a long way.
That said, sometimes this is what backfires too: I've seen a government contract recently where a lot of extra work was carried out to meet the requirements of a "security consultant" that refused to sign off on a system because of "issues" that were clearly generated only to ensure that he could pad his list of "problems" he identified to help justify his fees.
Wouldn't the central procurement already know the max capacity of the lines (DSL, fibre, whatever) that had been installed, already know what routers/switches were in use and already know the populations being served? That's pretty basic audit information that I'd expect could be pulled up from a central database lickity-split.
Eyeballing those figures should be all you'd need - but yes if there was no central record of past procurements an email survey (or better yet online using a survey system) would apparently be pretty quick.
I doubt the one-room library open three days a week has an IT manager. But yes, obviously they know where the equipment needs to be so somebody must be able to make an educated guess.
Actually, the real crime is that they don't sue Cisco to be allowed to exchange the original routers for new ones with appropriate capacity along with a refund in the price difference, with Cisco eating the depreciation and the legal fees.
The grant implementers should also be fired for cause, to wit, incompetence).
Capacity study? Hahaha Like they're going to go to the schools and someone there will "know" how much internet they need. They can't even go to the schools and get a good estimate of how many students they expect to be attending in 10 years, almost all the schools in the US are reactive to that sort of thing, they build new ones when the old ones are either so broken down that they can't fix them or they are over capacity. If you go to the schools you get one of two things: 1) a tiny internet straw that 2000 kids share and it's useless or 2) an OC48 to a tier one peer that 163 kids and teachers use for email.
There is something more fundamental, there were probably schools that had nothing close to reasonable internet access. (Likely a lot of them were in places where they can't get great internet at home either.) The masses were fed up with it, especially as they thought their kids were being left behind so they beat on their representatives and they just got a blob of cash to fix it, costs be damned. You think it was going to make the parents happy when they announce that it's going to take 7 years to get DSL to their child's school?
It's a monster of our own creation. No idea how you solve it... the people that doll out the money have no idea how to do the work or how to estimate if an estimate is good or not; they're just too far away it and the public's memory is too short term.
They probably did open an RFP to multiple bidders.. That's just game theory though, you think the bidders want to leave money on the table? Every where I've seen government bids produced, they increase the costs.
Oh, the FCC E-Rate program. So much corruption, so much waste. There were people who would purchase something for a school with the most % of kids on free or reduced price lunch, since that was used to set the "FCC match" percentage. So, if 90% of the kids were on that plan, we'd only pay 10% of the actual cost.
So, they'd order something for the school with the highest percentage and would then park that equipment there for a year (to fulfill the requirement that it must be used there). A year later, they'd move it somewhere else.
Probably going against the grain here, but what about the operations staff who are charged with supporting these 1,164 routers? What about warranties and parts replacements? There is a price for standardizing on a piece of hardware at such a large scale. When you standardize, sure there is going to be hardware overkill in some places, but in the big picture this is just a price of doing business at this scale. What is the price of service calls out to this location when the hardware is down? You have a tool-chain that supports this hardware, you have experts supporting the network (security, patching, etc), and you have contracts in place to make sure it is operational.
If you have 5 office buildings with 50-500-500-500-2000 people in them (respectively), then standardising is probably a good idea since now you can use the same company to fix problems in all the builds and not have to take special care with the small building.
However. If you have 30 builds with 5 people in each. Sending a new, preconfigure, router by fedex is so much cheaper.
That is how my phone company treats me. If my home router dies, or has an odd problem that they can't figure. They just send me a new router by mail and I switch it out with the old one and send that one back.
Nothing wrong with standardizing, so standardize on a $100-$500 router instead of a $20,000 router. The point is that some guy made a killing selling gear only a place with 10,000 users needs.
I am one of those charged with maintaining this equipment. Spread throughout our state, there are 8 Education Services Agencies that manage their region's networks. I personally installed ~115 of these routers...
What's the price?
Not a lot.
We are state employees, and as such, are paid state employee wages. I am compensated for mileage, but beyond that, there is no charge to the individual districts. Part of the cost of these routers was a 4 year Cisco Smart Care plan which replaces the hardware within 24 hours at no additional cost. What that really means is, they ship it to me and it's my responsibility to replace and return the failed part. Much of what I do is remote configuration changes and very little on-site time.
Thanks! Nice job at sowing seeds of doubt when faced with with clearly damning evidence of incompetence. You clearly have a brilliant future in government procurement.
Next task: make the case standardizing on Ferrari 599s for all government vehicles. All of you arguments will hold in that cases as well.
""Everybody had to comply with this spreadsheet," he said. "Every campus had two border routers, two cores, and two server farm switches. All the vendors had to propose exactly the same solution" based on the average number of servers deployed at each CSU campus. "All of this is based on exactly the same data to all of the vendors. It's exactly the same formula for all of the vendors.""
I am related to an ex-sales exec at Cisco UK. They are a bunch of shysters who will desperately oversell anything to anyone. The person in question was responsible for selling such kit to managers who didn't even know what it was but they apparently needed it. You know the sort who populate senior positions in the public sector. They are rife in the healthcare and council sectors in the UK. Incompetent morons waiting to be milked for our cash.
The asshat is now selling VoIP and video conferencing solutions to medium sized businesses (via their executives) which is the next cash cow he can milk.
Readit News
We were doing similar work for private school at the same time, and the process for the public and private school could not be more different.
For eg. in the tender process, which I was part of, for private school we would use cheaper Taiwanese routers. For the public schools it 'had to be Cisco'. The only time we ever used Cisco, outside of large enterprise clients with 1000+ seats, was with public school tenders. The private school would get a $500 white-label router, the public school s would get a $6-15k Cisco router with additional VPN module costs.
We would charge a higher consultant rate on the Cisco jobs, and would bill 3 days instead of 1. We won the contract because we were 50% cheaper than the other tenders who all wanted to install 3 and 5-series routers. How we got to the point of being able to even tender is another story that involves somebody in our organization sleeping with somebody at the government organization. The other tenderers were accustomed to dividing the work up amongst themselves at inflated prices, they didn't even know who we were and we received a lot of abuse for breaking up their little scheme.
(Edit: a further idea of how this worked, the 3 people in the gov office responsible for tenders all had very nice cars and holiday homes while the rest of the office was working away on below-average wages. You could see what was going on just by looking at the car park)
So a dozen of us roll out hundreds of these routers in public schools and after a month we find that we rolled out the wrong version of IOS, one that was vulnerable to a simple security attack. Instead of forcing us to upgrade all the routers remotely, or out of our own pocket, we instead won another few-million-dollars worth of work to send a person out and apply the upgrade to each router (which took 5 minutes, we charged a full day plus travel).
The routers weren't even being used properly - the topography was net connection -> cisco router -> internal server -> switches. The internal server would do all the DHCP and everything else. These expensive routers were being used as bridges, although they were pitched as having 'forward compatibility' incase the school wanted to implement features such as user accounts (they did, although again they used a custom server, not the router).
When these projects are audited there is nobody who is technically competent enough to make an argument against who would be on the side of ditching or shrinking the projects. Some of the smarter teachers knew what was going on but didn't mind since they got access to fancy equipment (we would create user accounts for them).
My first, and not my last, experience with government bureaucracy and budgets. I would estimate that the private schools got more out of us at a tenth of the cost. Since then I had an even worse experience with the government health department, where 6-figure invoices were written and paid for goods that didn't exist (that department has since been broken up and the subject of a large corruption enquire). No surprise that I became very anti-government size and spending.
Edit: to add, we were so 'disruptive' to the backdoor deals that we were uninvited from conferences, kicked off panels, not invited to the mixer events where gov buyers met providers, a couple of years later we lost our accreditation[0] temporarily until we appealed to the Government Minister. On site we would be locked out of network cabinets, not given IP information for the net connections, etc. Our jobs were made difficult by competitors and others because of our pricing and methods. They couldn't figure out how they didn't get rid of us, because they didn't know that one of our guys had a solid relationship with somebody at the government (that person wanted to clean things up). Usually we wouldn't have been allowed anywhere near these projects and if we did win one we wouldn't be allowed back in for not playing with the system.
[0] 'accreditation' for government tenders needs to die, it is a formal method used to keep honest operators out of what is essentially a cartel. The Australian government is getting better in this regard[1], they now have an open tender website but I believe it still requires some form of accreditation that has a person in a department standing between application and approval.
[1] although not too much better. NBN Co., a government owned company that is building a $40 billion nation-wide fibre network recently suspended tender process because every single bid came in over the expected price:
http://www.theaustralian.com.au/business/in-depth/high-price...
The number of times I've had potential supplier hint at extra perks or "return favours" is fairly high, and I've not even been in positions of very substantial purchasing power.
It's often not even direct exchanges that are being hinted at either, but "simply" seemingly innocent networking: Buy from the "right" well connected sales guys and make it clear you're "flexible" about requirements, and they know they benefit from exploiting their network to the max helping you get lucrative jobs in big companies where you can spend more with him.
It's often only morals that prevents it from happening - I've been in plenty of situations while where my budgets were not huge, nobody knew enough to second guess my decisions, and I could've easily spent a lot more than I did without anyone batting an eyelid. Especially since just the very act of bloating your budget in many places makes you look more impressive and important and translates into status (and looks good on your CV...)
Government departments are not.
In Australia independent audits rely on the Australian National Audit Office, which is profoundly and deliberately underfunded because of its annoying habit of discovering embarrassing incompetence and wastefulness.
One typical example I had is not receiving a replacement laptop when working for a bank because replacement hw is too expensive, so I could not work for essentially one month. My consulting rate would have paid for a top of the line laptop in 2 days. But those are of course two different cost centers, so it will never surface.
I've always worked with people that, even if not cutting-edge geniuses, are generally more than competent. They don't accept dodgy tender because they know the value of what they're buying, and they certainly don't overspend on gold-plated crap because our budgets are always stretched to the limit.
I would genuinely like to work for one of these mythical government departments where money flows like water. In my experience, the budget is always barely enough to fulfil our role and all these stories about profligate government waste are little more than hearsay.
Then again, I've never worked in the Health or Mining industries.
The higher-than-Europe comparison I had not heard before. The higher than private industry cost of United States government contracting (and contracting by local units of government when they spend federal funds) is dictated by the Davis-Bacon Act,
http://www.dol.gov/whd/govcontracts/dbra.htm
once we understand that the law's requirement to pay "prevailing wages" means, in that context, to hire union labor even when less expensive nonunion labor is available.
I've never understood it myself. Governments are funded by tax money, and the people involved in these companies that love overcharging the Government pay taxes. Congratulations on ripping yourself off.
Any additional taxes are diluted amongst the general population, while my sales commission is mine and mine alone...
One thing I've learned from Private Eye: many of our government contractors don't pay taxes -- or at least not British taxes, or a negligible amount thereof.
At some point people will probably need to fully fund and empower an independent audit organization staffed with the kind of detail oriented savants that will start cutting through a lot of fat.
As has been mentioned above, there are many times that people end up doing work only because they are wired to try and do it cheaply/well.
The rest of the time no one is at the helm.
Aren't there any avenues to get people fired over this? Talk to a investigative journalist maybe?
IT Director to Sales Guy: "We just got $20M in grant money for getting broadband across the state. Can you get me some numbers?"
Sales guy to engineering team (partner or internal): "Hey I need lots of boxes. They've got 1300 sites."
Engineering team: "Ok.. what do they need?"
IT Director: "Pretty sure my network guy says everything has to have redundant power supplies and at least 1 ethernet connection. To do a survey for each site would take over a year due to bureaucracy, and I've got 3 months on this grant"
Engineer to Sales guy: "Ok I built out those routers. Do they really need redundant power supplies everywhere? 3900's seem big."
Sales guy: "Ya, that's what they said. Anyways this came out below budget. Thanks!"
IT Director: "Looks to be under budget, meets our needs, thanks!"
.. meanwhile IT management/engineers aren't involved. Somewhere, someone didn't slow this project down to do due diligence. That or somewhere buried in some document is a requirement for redundant power supplies, but that sounds less likely the case.
In it a nutshell.
If anything, there should have been multiple flags raised.
1) A ball park figure should have been estimated on what this should cost, broken down by equipment type.
2) That should have made it clear that switches would be fine for multiple use cases, and the overall estimate should have dropped. *
3) Back and forth with provider as haggling over components proceeds. Massive attention to detail would be shown here.
4) First quote received from Cisco. Due diligence proceeds, issues raised. Customers says NO, just to buy negotiation room. (fine they have a grant, maybe not so aggressive)
5) Updated quote, go back to 4 if issues.
6) Quote acceptance. Now monitoriing of execution.
There should have been many opportunities for flags to have been raised, even before the quote was received. And most definitely after it would have crossed the ball park figure.
* It looks like the clause for redundant power was what tripped it up. Someone said we need X, and then somehow it became the TRUTH for all routers/switches.
So its entirely possible (if not likely) that Cisco said "based on your constraints, these are the only components which match your requirements", and the Government bought it.
Which is face-palm/head-desk territory.
At the federal level, specifically to prevent such things from occurring, there's this huge byzantine procurement process. A process that is literally worth tens of billions of dollars to game.
So if you have product foo and want to see a zillion copies of it to the feds instead of some freeware or cheaper solution? 1) Convince somebody on the inside of the procurement system that yours is the best product, 2) they write up the specification so that no other products will qualify (many times just copying directly from your brochure) even though it's technically an "open" bid, and 3) have somebody with lots of procurement knowledge help guide the paperwork through the system. At the end of the day, it's all just paperwork, no matter how much money is involved.
And that's just what I've observed in IT. I'm guessing IT is the worst, since all the products somewhat look alike. But I'm not sure. Quite frankly, its way too depressing to think about much.
BTW, the best way to do #1 is simply hire people retiring out of the procurement system. Yep, there are laws against direct hires, so you hire somebody from DoD to help shepherd a Commerce Department contract, or a DHS procurement expert to help with a DoD job, and so forth. If you do this correctly, the poor schmucks left handling the paperwork will be so happy that you can offer extensive support in making sure everything is done correctly that this is another huge plus in favor of your getting the bid.
Given the choice between a fixed cost and an unbounded cost (being the capacity study referenced throughout the article) I'm not surprised they went with the fixed cost.
I'm not saying WV didn't get swindled, but I can sure see how this might have happened:
WV Rep: Do we really need these $20k routers for all our locations?
Cisco: We really couldn't say what your exact needs are without a proper study.
WV Rep: What will that cost?
Cisco: It's really hard to say. We'll have to visit all of your locations and speak with the IT Manager there. We'll have to measure average and peak load. Of course, we'll want to plan for future growth so we're not at this same point in two years...
The real crime is that they didn't open the RFP to multiple bidders. A little bit of competition can go a long way.
E-mail all the IT managers, and ask them:
- How many users do you have? - What is the total size of the community you serve? - Do you need new equipment?
That would have quickly revealed that many of them had so few users that these routers were total overkill, and had so few potential users that even 100% simultaneous usage in some cases would not max out the capacity, and that some of them did not have a need for new equipment.
I agree with you that multiple bidders would be essential. But a tiny little bit of due diligence and investigation on the behalf of WV would have gone a long way.
That said, sometimes this is what backfires too: I've seen a government contract recently where a lot of extra work was carried out to meet the requirements of a "security consultant" that refused to sign off on a system because of "issues" that were clearly generated only to ensure that he could pad his list of "problems" he identified to help justify his fees.
Eyeballing those figures should be all you'd need - but yes if there was no central record of past procurements an email survey (or better yet online using a survey system) would apparently be pretty quick.
The grant implementers should also be fired for cause, to wit, incompetence).
There is something more fundamental, there were probably schools that had nothing close to reasonable internet access. (Likely a lot of them were in places where they can't get great internet at home either.) The masses were fed up with it, especially as they thought their kids were being left behind so they beat on their representatives and they just got a blob of cash to fix it, costs be damned. You think it was going to make the parents happy when they announce that it's going to take 7 years to get DSL to their child's school?
It's a monster of our own creation. No idea how you solve it... the people that doll out the money have no idea how to do the work or how to estimate if an estimate is good or not; they're just too far away it and the public's memory is too short term.
They probably did open an RFP to multiple bidders.. That's just game theory though, you think the bidders want to leave money on the table? Every where I've seen government bids produced, they increase the costs.
So, they'd order something for the school with the highest percentage and would then park that equipment there for a year (to fulfill the requirement that it must be used there). A year later, they'd move it somewhere else.
I wrote about this particular router instance last year and linked it with my own FCC story from the year before that. http://rachelbythebay.com/w/2012/05/08/router/
If you have 5 office buildings with 50-500-500-500-2000 people in them (respectively), then standardising is probably a good idea since now you can use the same company to fix problems in all the builds and not have to take special care with the small building.
However. If you have 30 builds with 5 people in each. Sending a new, preconfigure, router by fedex is so much cheaper.
That is how my phone company treats me. If my home router dies, or has an odd problem that they can't figure. They just send me a new router by mail and I switch it out with the old one and send that one back.
What's the price? Not a lot.
We are state employees, and as such, are paid state employee wages. I am compensated for mileage, but beyond that, there is no charge to the individual districts. Part of the cost of these routers was a 4 year Cisco Smart Care plan which replaces the hardware within 24 hours at no additional cost. What that really means is, they ship it to me and it's my responsibility to replace and return the failed part. Much of what I do is remote configuration changes and very little on-site time.
Next task: make the case standardizing on Ferrari 599s for all government vehicles. All of you arguments will hold in that cases as well.
http://en.wikipedia.org/wiki/Ferrari_599_GTB_Fiorano
http://www.networkworld.com/news/2012/102512-cisco-csu-26371...
""Everybody had to comply with this spreadsheet," he said. "Every campus had two border routers, two cores, and two server farm switches. All the vendors had to propose exactly the same solution" based on the average number of servers deployed at each CSU campus. "All of this is based on exactly the same data to all of the vendors. It's exactly the same formula for all of the vendors.""
I am related to an ex-sales exec at Cisco UK. They are a bunch of shysters who will desperately oversell anything to anyone. The person in question was responsible for selling such kit to managers who didn't even know what it was but they apparently needed it. You know the sort who populate senior positions in the public sector. They are rife in the healthcare and council sectors in the UK. Incompetent morons waiting to be milked for our cash.
The asshat is now selling VoIP and video conferencing solutions to medium sized businesses (via their executives) which is the next cash cow he can milk.