​

If you’re planning to roll out a large-scale IT project, you might want to pay heed to some lessons learned from the FBI’s troubled Virtual Case File (VCF) software project.

Part of a technology-modernization project known as Trilogy, VCF was intended to allow FBI personnel to quickly and efficiently share records, a daunting project given that the Bureau has more than one billion pages of information in its archives. But after years of development and more than $100 million spent, the project may have to be scrapped.

The project has been plagued with problems, according to congressional testimony. For starters, it was poorly defined, concluded Inspector General Glenn A. Fine in his report on the issue. Additionally, it suffered from regularly evolving design requirements, an overly ambitious implementation plan, and employee turnover.

In defending the effort to Congress, FBI Director Robert Mueller put a positive spin on changing requirements: “As the FBI’s mission evolved over the past several years, so did our technological needs.”

But that translated into wasted time and effort for SAIC, the contractor hired to create the program, according to Arnold Punaro, SAIC executive vice president and general manager. Punaro told Congress that the original contract was for “development of a Web front-end to the existing legacy applications used to manage case information.”

Only after 9-11 did the FBI realize it needed to improve those legacy systems as well, and the SAIC contract was revised. “Thus, the FBI shelved six months of work that no longer fit the post-9-11 world, and directed SAIC [to] take on a much more ambitious, high-risk project,” said Punaro.

The FBI’s original plan was to deploy VCF all at once, a strategy known as “flash cutover.” This approach was dropped last summer in favor of a less ambitious plan that would allow the software to be phased in over time.

That was a change for the better, says Scott Larson, managing director of cybercrime consultancy Stroz Friedberg, LLC, and a former Supervisory Special Agent with the FBI’s Computer Intrusion Squad. “I think the turnkey solution was a mistake with an organization of 28,000 users. You have to do it in stages, and you need a good testbed,” he says.

Punaro said that SAIC had told the FBI that flash cutover “was a high-risk strategy,” though he admitted that they should have made clearer that it “was too ambitious.”

He also stated that it was “a fundamental error” to agree to undertake an ambitious project whose requirements were so murky. This led to some mistakes by SAIC. With multiple teams working on a short deadline, coding standards were not always enforced, which resulted in “less-than-uniform code,” he said.

Another problem was obsolescence. The FBI noted in its comments on the IG’s report that “technological innovation has overtaken our original vision” for the case-file system, and Mueller testified that he is mulling a recommendation from an outside consultancy that they discard VCF altogether and start over with commercial off-the-shelf products.

The private sector can avoid similar problems by having “effective program managers who understand the consultant side of the business so that they can more effectively manage the whole process” of a large-scale IT project, Larson says.