Noting that this week's train is happening over US Thanksgiving. Due to the holiday, if we run into a blocker we need the US cabal to resolve, then we may need to abandon this week's train and pick it up with 1.44.0-wmf.6 next week.

I hope this will be unlikely, we're run the train over the week of Thanksgiving for the past two years and all went well.

Change #1097581 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.44.0-wmf.5] Branch commit for wmf/1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1097581

Change #1097581 merged by jenkins-bot:

[mediawiki/core@wmf/1.44.0-wmf.5] Branch commit for wmf/1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1097581

I was looking at the roadmap linked above

which seem to suggest that wmf.5 will not be deployed. While the comment above

In T375664#10353431, @thcipriani wrote:

Noting that this week's train is happening over US Thanksgiving. Due to the holiday, if we run into a blocker we need the US cabal to resolve, then we may need to abandon this week's train and pick it up with 1.44.0-wmf.6 next week.

is more optimistic that there will be wmf.5.

Unfortunately, the extension Math has 5 commits (T380184#10355795) that should be included in wmf.5. I tried to cherry pick the first commit, but it failed, and I don't know how to create patches for the other 4 nor I'm available in todays backport window. I would greatly appreciate if someone can merge the fixes into wmf.5.

In T380184#10355795, @Physikerwelt wrote:

... It would be good if the patches (or most of them)

• https://gerrit.wikimedia.org/r/1097393
• https://gerrit.wikimedia.org/r/1097465
• https://gerrit.wikimedia.org/r/1097553
• https://gerrit.wikimedia.org/r/1097555
• https://gerrit.wikimedia.org/r/1097557

ended up in the same train (T375664) as

• https://gerrit.wikimedia.org/r/1092348

Otherwise, we will see quite some <merror tags....

I don't know what is up with the listed roadmap. https://wikitech.wikimedia.org/wiki/Deployments/Yearly_calendar/2024 shows it as normal since we have decided we would deploy as usual despite thanksgiving in the USA ( https://wikitech.wikimedia.org/w/index.php?title=Deployments/Yearly_calendar/2024&diff=prev&oldid=2241893 ). wmf.5 got removed from the file on October 30th: https://www.mediawiki.org/w/index.php?title=MediaWiki_1.44/Roadmap&diff=prev&oldid=6820400 , and we forgot to add it back.

!log Fast forwarded mediawiki/extensions/Math wmf/1.44.0-wmf.5 branch from 0458ea16 to d5076423. That is to apply hotfixes that merged after the branch has been cut # T375664 | T380184

We have an ongoing issue with toolforge T380844 which breaks the MediaWiki train, notably our automatic jobs that processes the branch.

On IRC @jnuche wrote:

I checked the error from the nightly patches check job, it seems to be failing due to the same failure that made the presync fail. We are > getting sporadic failures from the train-blockers app:

$ while :; do curl -s https://train-blockers.toolforge.org/api.php | jq -r '.current.version'; sleep 1; done
1.44.0-wmf.5
parse error: Invalid numeric literal at line 1, column 4
parse error: Invalid numeric literal at line 1, column 4
1.44.0-wmf.5

The Branch cut test patches Jenkins job passed :)

Change #1097980 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] testwikis to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1097980

Change #1097980 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1097980

Mentioned in SAL (#wikimedia-operations) [2024-11-26T10:47:37Z] <hashar@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.5 refs T375664

Mentioned in SAL (#wikimedia-operations) [2024-11-26T11:25:08Z] <hashar@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.5 refs T375664

Mentioned in SAL (#wikimedia-operations) [2024-11-26T11:28:52Z] <hashar@deploy2002> Started scap sync-world: testwikis to 1.44.0-wmf.5 refs T375664

Mentioned in SAL (#wikimedia-operations) [2024-11-26T11:54:44Z] <hashar@deploy2002> Finished scap sync-world: testwikis to 1.44.0-wmf.5 refs T375664 (duration: 25m 52s)

Change #1098011 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group0 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098011

Change #1098011 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098011

Mentioned in SAL (#wikimedia-operations) [2024-11-26T12:26:12Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.5 refs T375664

Summary

I have confirmed yesterday scap clean --delete-gerrit-branch properly delete old wmf branches (solving T303828). I have adjusted the timer that runs it automatically on Tuesday ( 7d301cb368631b2d22451b22c6e21658a35243cf ), and thanks to Reuven that got deployed before it starts. I have confirmed this morning it ran properly.

Ahead of running the train, I went to start tackling a similar task for release branches (T380841). The dry run scripts looked fine but upon running targetting REL1_19 that ended up deleting the branch and tagging the HEAD of the repo instead of the old commit. I thus went to restore those branch which delayed my attention.

Jaime Nuche mentioned the nightly patches check job failed (release Jenkins: Branch cut test patches job), investigated and found out it was due to https://train-blockers.toolforge.org/ not being available.

The overnight presync failed as well:

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://train-blockers.toolforge.org/api.php

The root cause was T380844.

syncing

The manual sync took a while since the presync step did not run. The images had to be build, all the data transferred and it is rather lengthy. Eventually a canary check failed and I had it immediately aborted cause I previously pressed ⏎ in the terminal, that got read by scap and validated the default to abort the deployment. I gotta complete https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/468 to fix that. That first run took ~ 38 minutes.

I started a second run at 11:25 and immediately aborted it because a mediawiki-config change got merged during the window (Gerrit 1062430 - config: remove eventbus instrumentation setting for T363587). I have thus aborted the train, reverted the patch (Gerrit 1097995).

I then did a third run to promote testwikis and it logged a 25 minutes duration.

Amir had a backport pending which automatically triggered, that took 15 minutes more. After that I have promoted the wikis to group 0 which took ~ 12 minutes.

Change #1098471 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group1 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098471

Change #1098471 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098471

Mentioned in SAL (#wikimedia-operations) [2024-11-27T09:44:59Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.5 refs T375664

Change #1098887 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] group2 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098887

Change #1098887 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 to 1.44.0-wmf.5

https://gerrit.wikimedia.org/r/1098887

Mentioned in SAL (#wikimedia-operations) [2024-11-28T09:22:19Z] <hashar@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.5 refs T375664