More bugs discovered with Leaving Cert code during review

The Department of Education is standing on its Leaving Cert calculated grade system after more errors were discovered following an external review of the code that caused thousands of students to receive incorrect grades.

A review by a third-party Educational Testing Services (ETS) contractor released Saturday night uncovered further problems within the code used by the department to standardize results.

Since then, each of these issues has been resolved, according to Norma Foley, the Minister of Education and department officials.

Earlier tonight, the department confirmed that 6,100 students – roughly one in ten – received lower grades than they deserved due to bugs in the system.

Students also received higher grades than they should due to system bugs, but the department has not released these figures.

Students who received a higher grade will not have this taken away, according to Ms. Foley.

Mrs. Foley again apologized to the students for the “exceptionally difficult year” they faced. “And I’m sorry that this last week has created more uncertainty for you,” he said.

“When we found errors in the code, I decided to seek independent expert oversight for the sake of certainty, especially for students.”

“I am pleased that we are now able to provide students whose scores were lower than their corrected results should have been today and that this period of uncertainty is over for all students.”

Following the discovery of two bugs in the code used to standardize the calculated scores of Leaving Cert students, the US-based ETS was engaged to examine the corrections made by the department and Polymetrika, the Canadian company that implemented the code.

These errors were related to the junior cycle data used, which was intended to be based on the core subjects Irish, English and mathematics, and combined with the two best supplementary subjects for the students.

Instead, coding error combined them with the students’ two weakest supplementary subjects.

Another bug found that the CSPE was meant to be removed from these scores, but the subject had remained included.

Details of these errors were first made public last Wednesday, although Norma Foley and the department had learned of the errors the week before.

ETS was commissioned to examine whether the code that carried out the standardization process now worked correctly after making corrections. He found that both errors had been corrected by the department and Polymetrika.

However, ETS also found a new bug that affected students who did not sit through all three core subjects in Junior Cycle. This would apply primarily to students with special educational needs.

In these cases, the system was intended to use the average national junior cycle score in the missing subject of the group of students who obtained their certificate of completion in 2017, 2018, 2019 and 2020.

Instead, he chose that student’s next best subject for inclusion.

All three errors are contained in the same coding section and had an impact on student scores.

They have since been resolved and the code is working correctly, according to the department.

Polymetrika was asked to modify this code element, and after doing so, ETS rechecked the code to make sure there were no more bugs.

At this time, a larger discrepancy was discovered, although it did not make a significant difference in student scores.

According to the report published by ETS, the algorithm used to treat “extreme / outlier” students at each school does not exactly match a mathematical process outlined by the department.

“Although the optimization criteria is closely related to what is described in the Report, it is not exactly equivalent [sic]. ”

ETS has now signed that the code revision is now correct and that the second issue had no “significant impact” on the results, that is, no students received a lower grade as a result.

ETS also concluded that, in addition to the two problems identified, the “other parts” of the complex algorithm it reviewed match the procedures outlined by the Department of Education and are “producing scores according to the methodology described in the Report. [sic]. “

The report compiled by ETS released this Saturday also notes that due to the “very short period of time,” a full audit of the entire coding was not carried out, rather the coding was sampled.

“By sampling the coding, the Department [of Education] and ETS agree that certain coding areas should be prioritized within the available time frame. ”

Areas were prioritized and ETS was given access to the full coding and databases used to run the standardization process, it says.