Wikipedia back up – recovery from power outage
Wikipedia is now up again after several hours down time. I found the Wikipedia/Wikitech Server Admin Log which provides some insights about what happened.It seems that they had a major power failure. Even though they seemed to have gotten power back fairly quickly, did it take a lot more time to get all the servers up and running properly again.See here the excerpt from from Server Admin Log. Please note that the Date and Times are GMT Time.April 10 04:26 jeluf: ixia, thistle, lomaria, db1 have broken replication settings, webster has database page corruption. Taking db2 out of rotation to create copies from it. 04:20 jeluf: mounted /home on all DB servers 04:03 brion: ran mass-correction of bad-timestamped entries on enwiki (1529 revision records) 03:05 brion: srv71-srv79 had wrong clock, apparently set to local time instead of UTC. 01:45 brion: irc feeds online. had to rescue udprec from kate’s old home dir 01:38 brion: taking thistle and db1 out of rotation; broken replication. 01:32 brion: turning read_only off on adler. seems to be set to go on always on boot. 01:28 brion: things look mostly good; tried to take site read/write but someone has put adler into read-only? examining 01:23 brion: got fs-squids on the right ip. seems to work now. 01:20 brion: had to start lighty on amane 01:18 brion: trying to get fileserver squids+lvs up. (avicenna as lvs master) 01:10 brion: run-icpagent.sh didn’t take previously; seems to have helped now 01:04 brion: trying to add 10.0.5.5 on dalembert[…]
Not logged in