Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: Migrate to new network storage on MacStadium #33155

Closed
toothrot opened this issue Jul 17, 2019 · 7 comments
Closed

x/build: Migrate to new network storage on MacStadium #33155

toothrot opened this issue Jul 17, 2019 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Unreleased

Comments

@toothrot
Copy link
Contributor

MacStadium is deprecating the old storage SAN for a new NFS based storage. We should be able to automate moving the data over in govc, then updating makemac to use the new vSphere datastore name.
@toothrot toothrot added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Jul 17, 2019
@toothrot toothrot added this to the Unreleased milestone Jul 17, 2019
@toothrot toothrot self-assigned this Jul 17, 2019
@toothrot
Copy link
Contributor Author

The new NFS-based storage in the new MacStadium datacenter is now available. I've been copying the vm base images (osx_14_frozen, osx_12_frozen, and so on) to the new storage GGLGLN-A-001-STV1, adding a _nfs suffix: osx_14_frozen_nfs.

Once these are up, we should be able to make a change to makemac.go to reference the new image names and location.

We'll probably want to roll this out on a night or a weekend to test. We should be able to verify by manually starting a VM for each image with the same command that makemac uses.

The final step will be to move the bastion VM box as well. I haven't yet done enough research to determine the necessary steps there yet, but vSphere's snapshotting functionality seems solid.

@gopherbot
Copy link

Change https://golang.org/cl/188219 mentions this issue: cmd/makemac: use new MacStadium datastore

@toothrot
Copy link
Contributor Author

I was able to move the dns host / bastion / makemac vm to the new storage with vSphere's live migration tool.

I cloned all the base VMs to the new storage, and ensured that a snapshot ran on each one for use by cmd/makemac/makemac.go#CreateMac. I tested starting a VM, and verified that it successfully connected to the coordinator.

After https://golang.org/cl/188219, we should be able to restart makemac and see if the new VMs it creates (which it does regularly) work correctly.

Once we've restarted all VMs on the new storage, we can unmount the old datastore in vSphere and allow it to be decommissioned.

gopherbot pushed a commit to golang/build that referenced this issue Aug 6, 2019
@toothrot
cmd/makemac: use new MacStadium datastore
a2aaac0 
MacStadium is migrating off the current datastore. As part of the
transition, we copied our base images to the new storage, adding _nfs as
the suffix. The new images were snapshotted to allow for copy on write.

New base image names were suffixed with _nfs to accommodate the
migration. The new storage device was named GGLGLN-A-001-STV1 by
MacStadium.

Updated vmware-notes.txt to include manual instructions for a test run
of this latest change. The VM started and successfully connected to the
coordinator.

Updates golang/go#33155

Change-Id: I69976ebc6faa4f95a0aefd4d2e4c8d6a5bcf8a0f
Reviewed-on: https://go-review.googlesource.com/c/build/+/188219
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@toothrot
Copy link
Contributor Author

toothrot commented Aug 6, 2019

A new version of makemac has been deployed, using the new datastore. All running VMs are now on the new datastore.

I'm working on deprovisioning the old datastore, as our hosting provider has given us a surprise deadline of August 9th.

@toothrot
Copy link
Contributor Author

toothrot commented Aug 6, 2019

The old datastore (Pure1-1) is unmounted. Our hosting provider is working to remove it.

@andybons
Copy link
Member

Has this work been completed?

@toothrot
Copy link
Contributor Author

toothrot commented Aug 23, 2019 •
edited

Yes!

The migration has been completed, and the old storage has been decommissioned. There are still some references to it in vSphere that I cannot figure out how to get rid of, but it does not seem to be causing any issue.

codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@toothrot @codebien
cmd/makemac: use new MacStadium datastore
62951e7 
MacStadium is migrating off the current datastore. As part of the
transition, we copied our base images to the new storage, adding _nfs as
the suffix. The new images were snapshotted to allow for copy on write.

New base image names were suffixed with _nfs to accommodate the
migration. The new storage device was named GGLGLN-A-001-STV1 by
MacStadium.

Updated vmware-notes.txt to include manual instructions for a test run
of this latest change. The VM started and successfully connected to the
coordinator.

Updates golang/go#33155

Change-Id: I69976ebc6faa4f95a0aefd4d2e4c8d6a5bcf8a0f
Reviewed-on: https://go-review.googlesource.com/c/build/+/188219
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@golang golang locked and limited conversation to collaborators Aug 22, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@toothrot @andybons @gopherbot