In general we want to strip subdir components, rather than appending
`..`s.
Reviewed by: lwhsu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D54373
GCE images are required by Google to include their source code; we do
this by extracting {src,ports}.txz into the images, from the (legacy)
distribution sets.
Make sure those distribution sets actually exist.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53567
Building VMs as non-root requires no-QEMU code paths (installing
packages from outside the VM image rather than inside it) and vice
versa; we have a check for broken combinations.
Unfortunately that check was breaking
make -C src/usr.sbin/pkg NO_ROOT=YES -V PKGCONFBRANCH
because that code reaches into src/release to determine the branch
name (which is then used to determine which /etc/pkg/FreeBSD.conf to
install).
Wrap the no-root/no-qemu check in an .if to only run when we've
asked for VM and/or CLOUD building to be enabled.
Reviewed by: ivy
MFC after: 5 minutes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53486
VM and cloud images are now built as packaged base systems by default,
matching the default for installation media.
Setting -DNOPKGBASE allows building as non-pkgbase systems.
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D51483
There are still some minor breakage with NO_ROOT (aka WITHOUT_QEMU)
builds but turning all the cloudware on unconditionally will allow
for more testing of the build process concurrently with the final
bug fixes landing.
Discussed with: markj
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
Provide variables, PKG_REPOS_DIR and PKG_REPO_NAME, to allow the user to
configure the source package repository. Configure pkg to emit added
files to a metalog which gets concatenated with the main metalog
immediately before we build the root filesystem.
Reviewed by: cperciva, emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D52453
For compatibility with code written before we added ZFS images, we
create compatibility hardlinks from non-FS-labelled image filenames
to the ${VMFS} (typically UFS) images. This fails if the image
didn't build, and can take down the entire release build if we're
not careful.
Fixes: 611c1457ff ("release: Add set -e to abort upon failure")
Replace "mk-vmimage.sh" with "mk-vmimage.sh || true" so that a single
VM image failing doesn't result in the entire release build dying.
Fixes: 611c1457ff ("release: Add set -e to abort upon failure")
We need to pass this to cloudware builds, not just VM builds.
MFC after: 4 days
Fixes: 81ca663642 ("release: Pass PKG_INSTALL_EPOCH to vmimage.subr")
Sponsored by: Amazon
Starting in 2015 I have published "AMI Builder AMIs" for FreeBSD/EC2:
These boot into a memory disk, extract a "clean" copy of FreeBSD onto
the root disk, mount it at /mnt, and allow the user to SSH in to make
customizations before creating a new AMI from the "running" instance
(in fact, from the FreeBSD installation which is not running but is
mounted on /mnt).
This provides a much cleaner mechanism for building customized FreeBSD
AMIs than the traditional Linux approach of "launch an EC2 instance,
SSH in and configure it, then try to wipe logs and credentials before
creating an AMI"; and it's easier than building a customized AMI ab
initio by modifying the FreeBSD release-building code.
This commit brings that functionality into the FreeBSD src tree and
into the collection of images built by the release engineering team:
The EC2 "BUILDER" flavour AMI is essentially a "SMALL" flavour AMI with
a compressed "BASE" flavour disk image, plus an init script which
juggles disks around (rerooting into a memory disk and extracting the
"BASE" image onto disk).
Polished by: bz, emaste
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D49930
This value, if not already set, comes from the timestamp of the most
recent git commit (which is now also available in src/release code as
GITEPOCH) or 0 if git is not installed.
This should allow /var/db/pkg/local.sqlite to be reproducible in VM
images which have packages installed (e.g. cloudware).
Reviewed by: emaste, bapt
MFC after: 5 days
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D49760
To support common shell patterns (test -n and -z) we want WITHOUT_QEMU
to be set and not empty in mk-vmimage.sh if it is defined at all in the
Makefile.
Fix a case I missed previously.
Reported by: cperciva
Fixes: e6e5cd297a ("release/vm: fix setting NO_ROOT and WITHOUT_QEMU")
Sponsored by: DARPA, AFRL
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D49666
To support common shell patterns (test -n and -z) we want NO_ROOT and
WITHOUT_QEMU to be set and not empty in mk-vmimage.sh if they are set at
all in the Makefile.
Actually do this rather than making them always set (which propogates to
submakes run by mk-vmimage.sh and unconditionally enables NO_ROOT with
undesirable side effects such as installing a /METALOG in vm images).
Reported by: cperciva
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D49544
Without this, rebuilding vm images will fail due to the .xz file
existing.
Reviewed by: emaste
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D49321
For images that don't require QEMU, support NO_ROOT. This entails:
- Passing NO_ROOT down to mk-vmimage.sh (which sets it for
installworld, etc)
- Handling etcupdate bootstrapping
- Adding assorted config file METALOG entries
- Running makefs in the right directory and adding -D for dups
The main gap in basic NO_ROOT support is package installation. Each
image type must also be updated to add METALOG entries for any files it
adds.
Reviewed by: emaste
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D49320
This option disables trying to build a qemu-<target>-static and simply
doesn't do any of the operations that rely on it. This disables package
installation which is required by most, but not all image types.
Disable all CLOUDWARE variants that require pkg when WITHOUT_QEMU is set
to avoid broken images.
Reviewed by: emaste
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D49310
- requires base tar & flua, qemu-tools & curl from ports
- set ORACLE_PAR_URL to upload to local file:/// dir or cloud buckets
Reviewed by: emaste
Approved by: cperciva
Differential Revision: https://reviews.freebsd.org/D48382
Sponsored by: SkunkWerks, GmbH
We publish three sets of VM images on download.freebsd.org:
* Generic VM images
* BASIC-CLOUDINIT VM images
* BASIC-CI VM images
Of these, the first had names like
FreeBSD-14.2-RELEASE-amd64-ufs.raw.xz,
the second had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2.xz,
and the third had names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI.ufs.raw.xz
but were listed in the CHECKSUM files with names like
FreeBSD-14.2-RELEASE-amd64-BASIC-CI-ufs.raw.xz
Standardize these to consistently use a hyphen before the filesystem
type rather than a period.
Note: On FreeBSD 14 we ship images without the -${FS} component in
their names; these are hardlinks to the -ufs images for compatibility
purposes. On FreeBSD 13 we only have UFS images and don't include the
filesystem in the name.
MFC after: 1 week
Reported by: jmg
Sponsored by: Amazon
- use raw image disk type and enable zfs, this yields smaller
images for upload after using native qcow2 + zstd compression
Reviewed by: lwhsu, emaste
Differential Revision: https://reviews.freebsd.org/D47055
MFC after: 3 days
Approved by: emaste
This allows future releng tooling to use OCI for the industry
standard Open Container Initiative tooling, reducing potential
for confusion.
Approved by: cperciva
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D46975
MFC after: 3 days
Sponsored by: SkunkWerks, GmbH
These are the same as the standard "base" images except:
* They don't have kernel or world debug symbols,
* They don't have FreeBSD tests,
* They don't have 32-bit libraries,
* They don't have LLDB,
* They don't have the Amazon SSM Agent pre-installed,
* They don't default to installing the awscli at first boot.
This reduces the amount of disk space in use when the EC2 instance
finishes booting from ~5 GB to ~1 GB.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D46509
Due to issues with the riscv64 toolchain, some binaries end up
significantly larger on riscv64 than they should be. This results
in riscv64 VM images -- and at present *only* riscv64 images -- not
fitting within the default 5 GB filesystem size.
Bump the default size for riscv64 to 6 GB until the toolchain issues
can be resolved.
MFC after: 1 week
Sponsored by: Amazon
VM, BASIC-CI, and BASIC-CLOUDWARE images are published with the
filesystem type (UFS or ZFS) in the file name. For backwards
compatibility we have published the UFS images with no filesystem
in the filename (aka with the same file names as we used before we
supported building ZFS VM images); those are going away now.
This will not be MFCed.
While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- which are published via the FTP mirrors. We have
been handling these using a manual and error-prone process of copying
them from the internal build directories into the FTP staging tree,
but they should be handled using the "make install ftp-stage"
mechanism which is used for all the other published images.
This commit 1. Adds a new 'make cloud-install' target which is run as
part of 'make install' and installs BASIC-CI and BASIC-CLOUDINIT images,
and 2. Extends 'make ftp-stage' in Makefile.mirrors to copy those bits
into the FTP staging area.
As a side benefit, this will also make BASIC-CI and BASIC-CLOUDINIT
images available for arm64; until now they have only been available
for amd64.
MFC after: 1 week
This breaks the weekly snapshot builds. We'll try again once Warner has
some time to debug the qemu-user-static-devel port.
This reverts commit a04ecddfc7.
For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.
Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).
MFC after: 5 days
For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.
Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.
MFC after: 1 week
With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.
Fixes: 97bd53ef4d "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747
Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.
MFC after: 5 days
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 1 week
Reviewed by: Allanjude
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44747
The framework do not support multiple format images, so this last minute
addition of qcow2 format was clearly not a good idea.
while here fix a typo
Reported by: cperciva
Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.
2 formats available: qcow2 and raw
This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack
Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369
Let Azure use the image directly built by the release engineering team.
Reviewed by: emaste
Approved by: cperciva (re)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41906
Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2 AMIs available.
The emulator-portinstall target now unconditionally ensures that qemu
is installed; but is only invoked if needed (aka. when cross building
VM images).
MFC After: 3 days
MFC With: 97bd53ef4d ("Fix duplicate rc.conf files")
Two bugs in Makefile.vm resulted in disk images being "built" multiple
times, resulting in lines added to /etc/rc.conf being duplicated:
1. The vm-image target reused the same "staging tree" directory for all
of its builds (multiple disk image types and multiple filesystem types).
2. The cw-type-flavour-fs target depends on emulator-portinstall, which
did not have a 'touch ${.TARGET}' and thus re-ran every time -- and
caused the cw-type-flavour-fs target to be re-run. This was triggered
by release builds running `make cloudware-release` (creating the disk
images) followed by `make ec2amis` (which re-created the disk images
prior to uploading them).
MFC After: 1 week
Sponsored by: https://www.patreon.com/cperciva
Known issues:
1. The ec2-user user is created with a homedir of /usr/home/ec2-user
instead of /home/ec2-user; this appears to be a bug in cloud-init's
FreeBSD support.
2. Cloud-init configures IPv4 networking but not IPv6 networking.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Relnotes: yes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41793
Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names now include the word "base".
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41791
There are two "Vagrant" images right now: "Vagrant Image for VirtualBox"
and "Vagrant Image for VMWare". Rather than listing these separately in
a CLOUDWARE list, place "VAGRANT" into a CLOUDWARE_TYPES list and then
use a VAGRANT_FLAVOURS variable to identify the two versions. Add make
logic to allow defaults (in this case, image format and filesystem) to
be specified once for VAGRANT and inherited by both flavours.
This will make future work to add flavoured EC2 images simpler.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41790
The cw*-package targets were introduced in February 2015 as part of
adding support for building GCE images; but GCE support was reworked
in June 2015 and the cw*-package targets were in fact never used.
Remove them.
The *_DISK variables were introduced in February 2015 as part of
adding the cloudware-install target; this was removed in May 2016 as
the cloudware images are published via the respective cloud systems
and not published as disk images via the FreeBSD FTP site. As such,
the *_DISK variables are not unused; remove them.
releng/14.0 candidate.
Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41789
The pre-existing "ec2ami" target builds and uploads a single AMI
(with filesystem determined by ${VMFS}) as before; a new "ec2amis"
target does both UFS and ZFS.
Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41343
New ${CW}_FSLIST settings control the list of filesystem types with which
each cloudware image will be built; currently these are all set to "ufs",
i.e. no change from previous.
The cloudware images have their filesystem type as part of their file
name; for backwards compatibilty the ${VMFS} image is linked to the
previously used file name. This compatibility can be removed once all
the cloudware uploading/publishing code has been updated to use the new
image names (possibly more than one of them).
Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41341
