SPATHE Systems LLC

DOCUMENT FOR PUBLIC RELEASE
The decision issued on the date below was subject to a GAO Protective Order. This redacted version has been approved for public release.

Decision

Matter of: SPATHE Systems LLC

File: B-420463.2

Date: June 13, 2022

DIGEST

Protest challenging the agency’s evaluation of the protester’s technical quotation is denied where the record shows that the evaluation was consistent with the terms of the solicitation.

SPATHE Systems LLC, of Tampa, Florida, protests the issuance of task orders to Omni Federal, of Gainesville, Virginia, F9 Teams, Inc., of Seattle, Washington, and Raft, LLC, of Reston, Virginia, under request for quotations (RFQ) No. FA8307-20-G-0050, issued by the Department of the Air Force to support the Air Force Lifecycle Management Center, Detachment 12 (Kessel Run) mission. The protester contends that the agency unreasonably evaluated SPATHE’s quotation.

We deny the protest.

BACKGROUND

The Air Force issued the RFQ to Platform One Basic Ordering Agreements (BOA) Software DevSecOps[1] Services small business contractors, in accordance with Federal Acquisition Regulation (FAR) section 16.703, BOAs. Agency Report (AR), Tab 4, RFQ at 1-2. The RFQ sought support services for Kessel Run, a detachment of the Air Force Life Cycle Management Center that “leverages industry best practices and products to rapidly deliver capabilities while posturing for future and potentially disruptive information technology (IT).” AR, Tab 6, RFQ amend. 1, Performance Work Statement (PWS) at 4. The PWS describes Kessel Run as “require[ing] vendors that can accurately forecast technological changes and attract the cutting edge skill sets, known and unknown, to ensure the continued dominance of the Global [Air Operation Centers (AOC)][[2]] network.” AR, Tab 6, RFQ, amend. 1, PWS at 5.

The RFQ advised vendors that the Air Force intended to issue three or more task orders, but reserved the right to issue fewer than three orders. AR, Tab 8, RFQ amend. 1, Evaluation Factors for Award at 2. The agency’s selection would be a best-value tradeoff between two factors--technical and price. Id. at 3. The technical factor contained the following three subfactors: take-home challenge and solution presentation; leadership oral presentation; and technical approach/written assessment. Id. The first two subfactors--take-home challenge and solution presentation, and leadership oral presentation--were of equal importance and were more important than the third factor--technical approach/written assessment. Id. The technical factor was significantly more important than the price factor, but price would contribute significantly to the selection decision. Id. Quotations were to be evaluated for their “completeness, feasibility, and credibility,” and the agency would gauge the vendors’ “approach and understanding of the [PWS], supporting documentation, and any other additional criteria listed below.” Id.

Take-Home Challenge and Solution Presentation Subfactor

At issue in this protest is the Air Force’s evaluation of SPATHE’s quotation under the first technical subfactor--take-home challenge and solution presentation.[3] The Air Force explains that the take-home challenge was designed to require vendors to demonstrate their capability and functionality by completing a sample problem. Vendors were to develop an aircraft tracking application, submit the source code, and record a programming video of the team conducting a portion of the application development. The second portion of the take-home challenge was an oral presentation of the vendor’s aircraft tracking application. Contracting Officer’s Statement (COS) at 5, citing AR Tab 7, RFQ amend. 1, Instructions for Quote Preparation at 5-7.

More specifically, the take-home challenge required vendors to “build out the frontend and backend components of an aircraft tracking application for the Federal Aviation Administration (FAA).” AR, Tab 9, RFQ amend. 1, Take-Home Challenge at 1. Vendors were to build out the project with the functionality outlined in several “Expected Behavior[s]” of the aircraft tracking simulation--centered on the display of both “current” and “historic” flight data--and to “document [their] code as if it were going into production.” Id. at 1-2. To meet the expected behaviors for displaying current flight data, the application was to show the user a map on which to view flight data, which was limited to “a 100-mile radius around Boston Logan International Airport.” Id. The simulation was also to “[p]lace an icon at the location of the aircraft facing the direction of the aircrafts path” and “[u]pdate the position of each aircraft when new data is received.” Id. The take-home challenge contained three expected behaviors for the display of historic flight data; the simulation was required to permit the user “to click a current flight,” “[h]ide other current flights,” and to “[s]how the historical path of the flight selected.”[4] Id.

The solicitation explained that the purpose of the take-home challenge was to “gauge the vendors’ technical competency and ability to adhere to key Kessel Run software development principles and/or methodologies.”[5] AR, Tab 8, RFQ amend. 1, Evaluation Factors for Award at 3. Vendors were to provide “supporting documentation” with their solutions. Id. The take-home challenge team, comprised of three engineers, was expected to “demonstrate elite full stack software engineers[’] ability to adhere to modern software development principles such as Extreme Programming (XP) practices and Test-Driven Development (TDD).” Id. The purpose of the aircraft tracking application presentation was to evaluate the vendor’s “technical competency.” Id. at 4.

The RFQ advised that quotations would be rated under the take-home challenge and solution presentation subfactor as excellent, good, acceptable, or unacceptable. Id. An unacceptable quotation was one where the “[v]endor presentation does not meet requirements or expectations” and where the “[n]egative aspects within the collaboration far outweigh any positive aspects.” Id. In assessing vendor performance on the take- home challenge, the Air Force would consider, among other things, whether the technical solution demonstrated: completeness (whether all features were completed); correctness (whether the functionality acted in sensible, thought-out ways); maintainability (whether the project was written in a clean, maintainable way); testing (whether the system had been adequately tested); performance (whether the system was responsive and displayed new data in a timely manner); and system architecture (how the components of the assignment interacted with each other). Id.

The Air Force distributed the take-home challenge to all potential vendors on August 24, 2021, and offerors had seven calendar days to submit their response by the RFQ closing date of August 31. See AR, Tab 7, RFQ amend. 1, Instructions for Quote Preparation at 4, 6.

Evaluation and Award Decision

The three task order recipients and SPATHE were among 10 firms to submit quotations. AR, Tab 17, Technical Evaluation at 2. The Air Force determined that SPATHE’s “Take Home Challenge and Solution Presentation, overall, did not indicate a technical competency and ability to adhere to key Kessel Run software development principles and/or methodologies.” Id. at 37. The “comprehensive qualitative assessment of the evaluation team’s observations” found that “the negative aspects [of SPATHE’s quotation] far outweighed any positive aspects for [the take-home challenge and solution presentation subfactor],” and the Air Force rated SPATHE’s quotation under this subfactor unacceptable.[6] Id. That unacceptable rating rendered SPATHE’s quotation ineligible for award. See AR, Tab 18, Decision Document at 9. The agency issued task orders to Omni Federal, F9 Teams, and Raft, id. at 13, and SPATHE’s protest followed.[7]

DISCUSSION

SPATHE asserts numerous challenges to the agency’s evaluation of the protester’s quotation as unacceptable under the take-home challenge and solution presentation subfactor. In this context and as stated above, in assessing vendor performance on the take-home challenge, the Air Force would consider, among other things, whether the technical solution demonstrated: completeness; correctness; maintainability; testing; performance; and system architecture. As explained in greater detail below, the protester takes issue with several of the evaluators’ negative findings under the subfactor.[8] We discuss, and deny, the protester’s allegations below.[9]

In reviewing protests of awards in task order competitions, we do not reevaluate quotations but examine the record to determine whether the evaluations and source selection decision were reasonable and consistent with the solicitation’s evaluation criteria and applicable procurement laws and regulations. 22nd Century Techs., Inc., B‑417478.3, B-417478.4, Feb. 24, 2020, 2020 CPD ¶ 74 at 5. A protester’s disagreement with the agency’s judgment regarding the evaluation of proposals or quotations, without more, is not sufficient to establish that the agency acted unreasonably. Id.

Completeness

First, the protester challenges a negative finding about the completeness of its take-home challenge solution. The RFQ required a vendor to demonstrate that its “application is able to be fully instantiated[[10]] using Docker Compose.” AR, Tab 9, RFQ amend. 1, Take-Home Challenge at 1. The agency noted that SPATHE’s quotation “did not follow the instructions within the take-home challenge document requiring that the entire application be fully instantiated using Docker Compose.” AR, Tab 17, Technical Evaluation at 31. In other words, the agency explained, the instructions provided by SPATHE require “an additional step to build and run the application separately.” Id. The Air Force concluded that the protester’s failure to follow the instructions “demonstrates an inability to present a viable solution to the Take-Home Challenge with regard[] to completeness and demonstrates a lack of understanding of the requirements.” Id.

The protester does not dispute the agency’s finding that SPATHE’s application was not fully instantiated. See Comments at 16. Rather, SPATHE argues that the requirement for full instantiation using Docker Compose was not included in the tasks or expected behaviors of the take-home challenge, and that a reasonable vendor would not have understood the requirement would be evaluated as “an important element of the Completeness or Correctness criteria.” Comments at 16.

As discussed above, the take-home challenge’s “Expected Behavior[s]” described the user experience of the application--principally, how current and historic flight data would be displayed. See AR, Tab 9, RFQ amend. 1, Take-Home Challenge at 1-2. Other portions of the take-home challenge contained additional requirements. Importantly, the take-home challenge, under “Brief,” required an application to be fully instantiated using Docker Compose. AR, Tab 17, RFQ amend. 1, Take-Home Challenge at 1. While the protester did not believe “the Docker Compose Up listed in the Brief section of the Take Home Challenge was an important element” of the evaluation criteria, Comments at 16, there can be no reasonable dispute that the Brief section included an explicit application requirement and that SPATHE’s application failed to meet that requirement. The protester’s apparent surprise that the Air Force considered this requirement an “important element” of the evaluation in no way defeats the reasonableness of the agency’s finding.

Correctness

The protester also challenges the agency’s negative findings under the correctness criterion, which, as noted above, the RFQ defined as whether the functionality acted in sensible, thought-out ways. AR, Tab 8, RFQ amend. 1, Evaluation Factors for Award at 4. The agency assessed SPATHE’s quotation three negative aspects under correctness. See AR, Tab 17, Technical Evaluation at 32-33. The first negative aspect was that SPATHE’s application displayed flight data outside the 100‑mile radius required around the airport. Id. at 32. The agency noted that “numerous flights” were as far as 135 miles from the airport. Id. SPATHE argues that, because it used “a square bounding box with a 100-mile radius,” flights at the corners of the box “will have a distance up to 141.42 miles from center.” Protest at 13. According to SPATHE, the agency should have viewed this increased radius as a positive. Id. The requirement, however, was to “[l]imit your flight data to a 100-mile radius around the airport.” AR, Tab 9, RFQ amend. 1, Take-Home Challenge at 2. SPATHE’s application failed to satisfy that requirement and the quotation was thus reasonably assessed this negative aspect.

The agency assessed SPATHE’s quotation a second negative aspect because the rectangular boundary around the airport made it difficult for the Air Force to understand what was within the 100-mile radius. AR, Tab 17, Technical Evaluation at 32. SPATHE contends that the “RFQ does not make a circle a requirement, only a radius,” and that, “[i]n mathematics, the term radius does not require a circle.” Protest at 12. The Air Force “argues that a reasonable person would interpret the term ‘radius’ to pertain to a circle or sphere, which is how the term is used in classical geometry.” COS at 18. We agree. The only reasonable interpretation of the requirement to “[l]imit your flight data to a 100-mile radius around the airport” was that the agency expected a circular display of flight data. The Air Force reasonably assessed SPATHE’s quotation a negative aspect because the protester’s rectangular display failed to satisfy the RFQ requirement and made it difficult to tell which flights were within 100 miles of the airport.

The agency assessed SPATHE’s quotation a third and final negative aspect under correctness because the icons depicting flights “remain visible to users indefinitely at the most recent valid aircraft location.” AR, Tab 17, Technical Evaluation at 33. Over time, the agency found that the map became “cluttered,” making it difficult to discern which icons represented current flights. Id. SPATHE argues that its solution is a “sensible and thought-out way” because, “while there was no requirement for how long planes should remain visible after landing or exiting, there was a requirement to ‘Display historic flight data.’” Protest at 14. The Air Force contends that “the indefinite accumulation of plane icons” was not sensible or thought-out because individual icons were lost in the clutter of landed planes and made it difficult “to discern which icons represent[ed] current flights.” COS at 21-22, citing AR, Tab 17, Technical Evaluation at 32-33. While the protester disagrees with the agency’s evaluation, SPATHE has not demonstrated that the assessment of this negative aspect was unreasonable.

Maintainability

Next, SPATHE challenges negative findings assessed by the evaluators under the maintainability criterion. Under this criterion, the Air Force considered whether the project was written in a clean and maintainable way. The agency assessed SPATHE’s quotation the first of three negative aspects because the “Government did not observe any documentation regarding architectural or design decisions that [the protester] made while developing the application.” AR, Tab 17, Technical Evaluation at 33. Without this information, the Air Force concluded that “a new team maintaining this code would find it difficult to rationalize the trade-offs of architectural and design decisions that [the protester] made when the product was developed.” Id. SPATHE contends that the “Air Force did not provide any documentation requirements in the RFQ, and the RFQ did not define what would constitute comprehensive or well executed [documentation].” Protest at 15. Thus, the protester argues that “finding a lack of documentation as a weakness is inconsistent with the evaluation criteria.” Protest at 15.

The intervenor asserts that the RFQ did, in fact, require documentation. Intervenor’s Comments at 7. As Raft notes, the take-home challenge required vendors to “document [their] code as if it were going into production.” Id., quoting AR, Tab 9, RFQ amend. 1, Take-Home Challenge at 2. Moreover, the evaluation factors for award advised vendors that their quotations would be evaluated on “supporting documentation.” AR, Tab 8, RFQ amend. 1, Evaluation Factors for Award at 3. The instructions for quotation preparation required vendors to submit “source code with documentation.” AR, Tab 7, RFQ amend. 1, Instructions for Quote Preparation at 6. There is no merit to SPATHE’s assertion that the RFQ did not contain a documentation requirement; such a requirement was set out at multiple places in the RFQ. Accordingly, the assessment of this negative aspect was reasonable.

SPATHE also contests a second negative aspect that the agency assessed SPATHE’s quotation under maintainability. The agency assigned this second negative aspect because SPATHE chose to create its own OpenSky REST client[11]--instead of using the client provided within the documentation of OpenSky Network--without providing any documentation for doing so. AR, Tab 17, Technical Evaluation at 34. The protester asserts that the “RFQ did not require the production of any documentation besides an architecture diagram and an application description,” and that SPATHE did not provide documentation for the use of its own OpenSky client because “this is not the best practice in private code bases.” Protest at 16.

As discussed above, the RFQ required documentation of the vendor’s solution. SPATHE disagrees with the agency’s assessment of this negative aspect without providing a legal or factual rationale for finding the Air Force’s evaluation unreasonable. As such, SPATHE’s argument does not provide us with a basis to sustain its protest.[12]

Testing

The protester also challenges a negative finding assessed by the agency under the testing criterion, which provided for consideration of whether the system had been adequately tested. The Air Force assessed SPATHE’s quotation a negative aspect because the protester’s “submission had minimal integration testing, two integration tests which focused on how the application interacted with OpenSky, and no end-to-end testing.” AR, Tab 17, Technical Evaluation at 35. The protester concedes that it did not propose end-to-end tests, but regardless notes that its “Independent Evaluation [conducted by the protester’s consultant] found that the lack of end-to-end testing was acceptable.” Comments at 20.

SPATHE also argues that its application team wrote database integration tests. Comments at 20, citing exh. 1, Independent Evaluation at 18. Based, in part, on the drafting of these database integration tests, SPATHE explains at length, with reference to its independent evaluation, why the protester views the agency’s evaluation as unreasonable. See id. Nonetheless, while the agency’s evaluation recognized that SPATHE’s solution provided for some integration testing, the agency found it insufficient. Moreover, the protester acknowledges that its solution lacked end-to-end testing. On this record, we see no basis to question the reasonableness of the negative aspect assessed the protester’s quotation for “minimal” integration testing and no end-to-end testing.

Performance

Next, SPATHE takes issue with negative findings assessed by the evaluators under the performance criterion. The RFQ defined performance as whether the system was responsive and displayed new data in a timely manner. AR, Tab 8, RFQ amend. 1, Evaluation Factors for Award at 4. The Air Force “observed that Spathe’s submission” had an issue with “ever-growing data sets.” AR, Tab 17, Technical Evaluation at 35. Based on both a review of the code and observing the application “for many hours,” the evaluators found that the last entry for every flight “remained valid in perpetuity.” Id. The agency stated that the increase in payload size will degrade the performance to a point where data cannot be transmitted consistently. Id. As a result, the agency assessed the protester’s quotation a negative aspect. Id.

The protester asserts that the “Take Home Challenge Document, however, did not include the removal of flights no longer flying in the list of Tasks or Expected Behavior.” Comments at 18, citing AR, Tab 9, Take-Home Challenge at 2. First, SPATHE’s own solution suggests that it intended to remove inactive flights because SPATHE concedes that it “incorrectly suggested,” during its solution presentation, “that the flights are removed [in] under three minutes.” Comments at 18-19.

Second, the Air Force observed a degradation in performance which the agency attributed to the system’s failure to remove inactive flights.[13] COS at 36. SPATHE’s solution did not operate as the protester had intended, and the system’s flaw, the Air Force concluded, would degrade performance. Thus, the assessment of this negative aspect to SPATHE’s quotation was reasonable.[14]

System Architecture

In its comments on the agency report, SPATHE objected to the agency’s assessment of a negative finding pertaining to its system architecture. The RFQ provided that under the system architecture criterion, the agency would consider how the components of the assignment interacted with each other. The agency assessed SPATHE’s quotation a negative aspect under this criterion because the “application’s system architecture does not comply with the single responsibility principle.” AR, Tab 17, Technical Evaluation at 36. SPATHE contends that the agency used an incorrect definition of single responsibility principle in its evaluation of system architecture. Comments at 20.

The protester was on notice at the time of its debriefing that the Air Force assessed SPATHE’s quotation this negative aspect. See AR, Tab 21, Technical Evaluation--Debriefing at 6 (discussing why “Spathe’s application’s system architecture does not comply with the single responsibility principle”). Accordingly, this challenge to the evaluation, raised for the first time in the protester’s comments, is dismissed as untimely. 4 C.F.R. § 21.2(a)(1); Warrior Serv. Co., B-417574, Aug. 19, 2019, 2019 CPD ¶ 298 at 5 n.7.

Solution Presentation

Finally, the protester challenges two negative findings pertaining to its solution presentation. The Air Force assessed SPATHE’s solution with a negative aspect because SPATHE made two incorrect claims regarding its project during the presentation: that the application adhered to the single responsibility principle; and that the application removed flights after three minutes (the same erroneous representation discussed above). AR, Tab 17, Technical Evaluation at 37. In addition, the solution presentation included a programming video requirement. Tab 7, RFQ amend. 1, Instructions for Quote Preparation at 5-7. In evaluating the video, the agency also assessed the protester’s quotation a negative aspect under that requirement because “the type of work each developer performed appeared to align with their preference of technology,” and because SPATHE’s “proposed take home challenge team did not demonstrate elite full stack software engineers’ ability to adhere to modern software development principles.” Id.

The protester alleges that the application’s failure to adhere to a single responsibility principle was considered elsewhere in the evaluation and cannot be double-counted under the solution presentation criteria. Protest at 26. Regarding the erroneous statement about the removal of flights from the display after three minutes, as noted above, SPATHE conceded that its solution presentation suggested that flights would be removed from the display. See Comments at 18-19. The protester also took issue with the agency’s double counting of its concerns that the software developers only demonstrated capability with their preferred software and that they failed to demonstrate the capabilities of an elite stack software engineer. According to the protester, these are essentially the same issue. Protest at 27. The agency substantively responded to these challenges to the reasonableness of the evaluation. See COS at 39-44. SPATHE did not address the agency’s defense of its evaluation, except to assert that its inaccurate statement regarding the removal of flights “should not have been assessed as a negative or given significant weight under the Solution Presentation portion of the evaluation because it was not a requirement.” Comments at 19. On this record, we have no basis to question the reasonableness of the assessment of these negative aspects.

In sum, we find that the Air Force reasonably assessed numerous negative aspects to SPATHE’s quotation. We further find that, based on those negative aspects, the agency reasonably rated the protester’s quotation unacceptable under the take-home challenge and solution presentation subfactor. As a result, the agency properly found SPATHE’s quotation ineligible for award.

The protest is denied.

Edda Emmanuelli Perez
General Counsel

 

---