libvirt kvm 환경에서의 cloud-init 간단 사용법.
쓰고보니 별로 간단하지 않더라…
1. image download
wget https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1708.qcow2.xz xz -d CentOS-7-x86_64-GenericCloud-1708.qcow2.xz |
2. cloud-init version 확인.
guestfish -a CentOS-7-x86_64-GenericCloud-1708.qcow2 run list-filesystems mount /dev/sda1 / ls /usr/share/licenses/cloud-init-* |
현재 centos7의 cloud image의 cloud-init 버전은 0.7.9 이다. 그럼 여기에 맞는 cloud-init의 manual을 참조해야 한다.
http://cloudinit.readthedocs.io/en/0.7.9/
3. datasource nocloud
http://cloudinit.readthedocs.io/en/0.7.9/topics/datasources/nocloud.html
기본적으로 user-data는 단순히 사용자 데이터이고 meta-data는 EC2 메타 데이터 서비스에서 찾을 수있는 것을 나타내는 yaml 형식의 file이다.
vfat 또는 iso9660 파일 시스템의 파일을 통해 로컬 VM 부트에 meta-data 및 user-data를 제공 할 수 있다. 파일 시스템 볼륨 레이블은 cidata 여야한다. 즉, floppy disk나 iso image가 가능하고, volume label(이름)이 cidata여야 한다는 뜻이다.
이러한 user-data 및 meta-data 파일은 해댱 filesystem(floppy disk, iso image)의 root에 위치 해야 한다.
/meta-data /user-data |
4. 설정.
http://cloudinit.readthedocs.io/en/0.7.9/index.html
cloud init에는 boot stage를 5단계로 나눠 놨다. 순서대로 실행 됨.
1. Generator cloud-init을 enable시켜 주는 역할. disable 하려면 /etc/cloud/cloud-init.disabled file을 만들어주거나 관련서비스 4개를 disable 시켜주면 됨.
2. Local local datasource(floppy, iso image)를 사용할 때 씀. cloud-init-local.service
3. Network cloud-init.service cloud_init_module section
4. Config cloud-config.service cloud_config_module section
5. Final cloud-final.service cloud_final_module section
아래는 centos7 cloud image안에 있는 cloud-init config file 내용이다.
기본 user와 sysinfo 2개의 section이 있고 booting시에 원하는 명령어를 실행 하려면 ,cloud_init_modules,cloud_config_modules, cloud_final_modules section에 알맞는 module을 넣으면 된다.
이걸 수정 하려면 guestfish에서 vi를 사용하면 된다.
[root@youngju3-test1 mnt]# cat /etc/cloud/cloud.cfg
users:
- default
disable_root: 1
ssh_pwauth: 0
mount_default_fields: [~, ~, 'auto', 'defaults,nofail', '0', '2']
resize_rootfs_tmp: /dev
ssh_deletekeys: 0
ssh_genkeytypes: ~
syslog_fix_perms: ~
cloud_init_modules:
- migrator
- bootcmd
- write-files
- growpart
- resizefs
- set_hostname
- update_hostname
- update_etc_hosts
- rsyslog
- users-groups
- ssh
cloud_config_modules:
- mounts
- locale
- set-passwords
- rh_subscription
- yum-add-repo
- package-update-upgrade-install
- timezone
- puppet
- chef
- salt-minion
- mcollective
- disable-ec2-metadata
- runcmd
cloud_final_modules:
- rightscale_userdata
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change
system_info:
default_user:
name: centos
lock_passwd: true
gecos: Cloud User
groups: [wheel, adm, systemd-journal]
sudo: ["ALL=(ALL) NOPASSWD:ALL"]
shell: /bin/bash
distro: rhel
paths:
cloud_dir: /var/lib/cloud
templates_dir: /etc/cloud/templates
ssh_svcname: sshd
# vim:syntax=yaml
|
meta-data와 user-data 내용을 보면…
뭐가 문젠지 meta-data에서 network config가 잘 안된다. 그래서 user-data에서 network: {config: disabled}로 설정하고 bootcmd에서 원하는 설정을 넣어 줬다. shell script에 자신이 있으면 cmd에서 거의 모든것을 할 수 있다. cmd에서는 힘들지만…원하는 binary까지도 넣어 줄 수가 있다.(manual 참조)
[root@youngju3-test1 mnt]# cat meta-data
instance-id: youngju3-test1
hostname: youngju3-test1
local-hostname: youngju3-test1
[root@youngju3-test1 mnt]# cat user-data
#cloud-config
password: root123
chpasswd:
list: |
root:root123
expire: False
ssh_pwauth: True
network: {config: disabled}
# Hostname management
preserve_hostname: False
hostname: youngju3-test1
fqdn: youngju3-test1
bootcmd:
- set -x; echo '#user-data/bootcmd:' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; sed -i -e '/^BOOTPROTO/ s/dhcp/static/g' -e '/PERSISTENT_DHCLIENT/d' /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'IPADDR=10.11.11.211' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'NETMASK=255.255.255.0' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'GATEWAY=10.11.11.1' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'DNS1=8.8.8.8' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'ONBOOT="yes"' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'NM_CONTROLLED=no' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- ifdown eth0
- ifup eth0
# Run command when finished with it
runcmd:
- 'systemctl disable NetworkManager'
- 'timedatectl set-timezone Asia/Seoul'
- 'sed -i "/server .* iburst/d" /etc/chrony.conf'
- 'sed -i "/^# Please consider.*$/a\server time.bora.net iburst" /etc/chrony.conf'
- 'chmod o-rwx /usr/bin/su'
- 'sed -i "s/^PASS_MAX_DAYS.*$/PASS_MAX_DAYS 90/" /etc/login.defs'
- 'sed -i "s/^PASS_MIN_LEN.*$/PASS_MIN_LEN 9/" /etc/login.defs'
- 'yum install bash-completion vim *bin/netstat *bin/route -y'
- 'touch /etc/cloud/cloud-init.disabled'
# - 'systemctl disable cloud-init cloud-init-local cloud-config cloud-final'
# - 'sed -i "/#PermitRootLogin yes/a\PermitRootLogin no" /etc/ssh/sshd_config'
# Configure where output will go
output:
all: ">> /var/log/cloud-init.log"
# Install my public ssh key to the first user-defined user configured
# in cloud.cfg in the template (which is centos for CentOS cloud images)
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2wgI9na26l99n/Hk3S1cnYkI5W5H4k3v39TvHxpxlv60Xp5mtrEhSlbS5pjgvG574dGV+mfdi3d4cA/59KFjiRawkrBP3K93hIvunFHt0U3QBgRgexZd/ApE7Pe3aE
7TVPWs8liCzPTEjm9ZaqgxS0ZaZlTTHMFxNowKPKSQ32tslwPHbnm7QqmRgjZQdS0D9LFpRIpDz2hzBvRLc/HGMHzQ7R+zwcUKc7lx4I+9A9NRhnOKpRV1C52Avk2/eFcisLJfywDTZj0l2j8iRUUDAY4OW2xs/zHrsxK2fJ2
CBrPXZ1XV8Zkqc1FBRgkK313Bp3HxYY/vJ1xk7sk4C+HoF krd@free.style.ted
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC89+KTru1TdaKbrSBdTSJMIdW0wHL6lXHwvNkk1W77ofr/Ys/qI7PBz56IBUOMKhk/Gy1z02RqsG4i4SLH6+Oq5GZOGXG4dNhfQcfKq9ouTyJ2fwqBr/49MB+kAxyZk
CFrjffPe/VWVPYQHxT5SItqLX7e4gLRNFUNR2el1vkOyk4b20PiVZQpJS4R6CuQ0mktwKsMqEu/lb9weIqgRFn9gw4GyWXnA5cSFGting5dbXUvrrlpDIXZ9WDJFKzONbdgw7AuxUF0DPLyj5STMjgZTAhANlUbdkorxwAMJx
5Al6KGtypwdzgEndXD2Z9CdEsV7ZZLfnEhpIw7JArzxo2F root@cloud-test02.osci.kr
disable_root: 0
|
5. vm instance 생성.
vm에 넣어 줄 iso file을 만들자. 이 때 중요한것이 volid 를 반드시 cidata로 넣어야 한다. file 이름도 user-data, meta-data로 지정되어 만들어져 있어야 한다.
# genisoimage -output test.iso -volid cidata -joliet -r user-data meta-data |
booting 시킬 boot image도 만들어 준다.
# qemu-img -b CentOS-7-x86_64-GenericCloud-1708.qcow2 -f qcow2 test.qcow2 50G |
이제 vm을 만들어 주자.
virt-install --import \ --name testvm \ --ram 1024 \ --vcpus 2 \ --os-type linux \ --os-variant rhel7 \ --cpu host-passthrough \ --disk test.qcow2,format=qcow2,bus=virtio \ --disk test.iso,device=cdrom \ --network network=youngju-net-10,model=virtio \ --graphics none \ --hvm \ --virt-type kvm \ --noautoconsole |
vm instance 확인.
# virsh console testvm |
위 설정대로 했으면 root password가 root123으로 되어 있을 것이다.
로그인해서 설정 확인을 해보자.
자 이제 이 일련의 과정을 자동화 시켜보자.
[root@kvm2 createGuestVM]# cat vars/guest.yml
---
#vm_name:
image_file_path: /youngju/vms/
#network_ipaddr: 10.11.11.251
# HW
#cpu: 2
#mem: 8192
disk_size: 50
[root@kvm2 createGuestVM]# cat vars/default.yml
---
# KVM Hypervisor info
host: 192.168.92.1
virt_type: kvm
virt_hypervisor: hvm
# Default network config for intance
network:
bridge: br0
net_name1: youngju-net-10
net_name2: youngju-net-20
net_name3: youngju-net-30
net_name4: youngju-net-40
net_name5: youngju-net-50
interface: eth0
netmask: 255.255.255.0
gateway: 10.11.11.1 # for default user(centos for CentOS, cloud-user for RHEL)
password: root123 os:
type: linux variant: rhel7
disk:
cloud_init: cloud-init.iso
# RHEL Cloud Image
# cloud_image: /root/createGuestVM/images/rhel-guest-image-7.3-35.x86_64.qcow2
# CentOS Cloud Image
cloud_image: /youngju/vms/CentOS-7-x86_64-GenericCloud-1708.qcow2
# Default ssh key
ssh_key1: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2wgI9na26l99n/Hk3S1cnYkI5W5H4k3v39TvHxpxlv60Xp5mtrEhSlbS5pjgvG574dGV+mfdi3d4cA/59KFjiRawkrBP3K93hIvunFHt0U3QBgRgexZd/ApE
7Pe3aE7TVPWs8liCzPTEjm9ZaqgxS0ZaZlTTHMFxNowKPKSQ32tslwPHbnm7QqmRgjZQdS0D9LFpRIpDz2hzBvRLc/HGMHzQ7R+zwcUKc7lx4I+9A9NRhnOKpRV1C52Avk2/eFcisLJfywDTZj0l2j8iRUUDAY4OW2xs/zHrs
xK2fJ2CBrPXZ1XV8Zkqc1FBRgkK313Bp3HxYY/vJ1xk7sk4C+HoF krd@free.style.ted
ssh_key2: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC89+KTru1TdaKbrSBdTSJMIdW0wHL6lXHwvNkk1W77ofr/Ys/qI7PBz56IBUOMKhk/Gy1z02RqsG4i4SLH6+Oq5GZOGXG4dNhfQcfKq9ouTyJ2fwqBr/49MB+
kAxyZkCFrjffPe/VWVPYQHxT5SItqLX7e4gLRNFUNR2el1vkOyk4b20PiVZQpJS4R6CuQ0mktwKsMqEu/lb9weIqgRFn9gw4GyWXnA5cSFGting5dbXUvrrlpDIXZ9WDJFKzONbdgw7AuxUF0DPLyj5STMjgZTAhANlUbdkor
xwAMJx5Al6KGtypwdzgEndXD2Z9CdEsV7ZZLfnEhpIw7JArzxo2F root@cloud-test02.osci.kr
|
[root@kvm2 createGuestVM]# cat templates/meta-data.j2 [20/3651]
instance-id: {{vm_name}}
hostname: {{vm_name}}
local-hostname: {{vm_name}}
[root@kvm2 createGuestVM]# cat templates/user-data.j2
#cloud-config
password: {{password}}
chpasswd:
list: |
root:root123
expire: False
ssh_pwauth: True
network: {config: disabled}
# Hostname management
preserve_hostname: False
hostname: {{vm_name}}
fqdn: {{vm_name}}
bootcmd:
- set -x; echo '#user-data/bootcmd:' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; sed -i -e '/^BOOTPROTO/ s/dhcp/static/g' -e '/PERSISTENT_DHCLIENT/d' /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'IPADDR={{network_ipaddr}}' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'NETMASK={{network.netmask}}' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'GATEWAY={{network.gateway}}' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'DNS1=8.8.8.8' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'ONBOOT="yes"' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- set -x; echo 'NM_CONTROLLED=no' >> /etc/sysconfig/network-scripts/ifcfg-eth0
- ifdown eth0
- ifup eth0
# Run command when finished with it
runcmd:
- 'systemctl disable NetworkManager'
- 'timedatectl set-timezone Asia/Seoul'
- 'sed -i "/server .* iburst/d" /etc/chrony.conf'
- 'sed -i "/^# Please consider.*$/a\server time.bora.net iburst" /etc/chrony.conf'
- 'chmod o-rwx /usr/bin/su'
- 'sed -i "s/^PASS_MAX_DAYS.*$/PASS_MAX_DAYS 90/" /etc/login.defs'
- 'sed -i "s/^PASS_MIN_LEN.*$/PASS_MIN_LEN 9/" /etc/login.defs'
- 'yum install bash-completion vim *bin/netstat *bin/route -y'
- 'touch /etc/cloud/cloud-init.disabled'
# Configure where output will go
output:
all: ">> /var/log/cloud-init.log"
# Install my public ssh key to the first user-defined user configured
# in cloud.cfg in the template (which is centos for CentOS cloud images)
ssh_authorized_keys:
- {{ssh_key1}}
- {{ssh_key2}}
disable_root: 0
|
[root@kvm2 createGuestVM]# cat virt-guest-multiple-nic.yaml
---
- name: manage libvirt guests
user: root
hosts: vms
vars_files:
- vars/default.yml
- vars/guest.yml
tasks:
- name: start libvirtd
service: name=libvirtd state=started enabled=yes
register: libvirtd
- name: create directory
file: path={{ image_file_path }} state=directory mode=0755
- name: wait for libvirtd to get up
pause: seconds=30
when: libvirtd.changed
- name: get list of vms
virt: command=list_vms
register: virt_vms
- name: create cloud-init data directory
file: path=~/cloud-init/{{ vm_name }} state=directory mode=0777
- name: create user-data
template: src=templates/user-data.j2 dest=~/cloud-init/{{ vm_name }}/user-data
- name: create meta-data
template: src=templates/meta-data.j2 dest=~/cloud-init/{{ vm_name }}/meta-data
- name : create cloud-init iso
shell: /bin/bash -c 'genisoimage -output {{ image_file_path }}/{{ vm_name }}-{{ disk.cloud_init }} -volid cidata -joliet -r ~/cloud-init/{{ vm_name }}/user-data ~/cloud-init/{{ vm_name }}/meta-data'
# - name: copy image
# command: cp -a {{ disk.cloud_image }} {{ image_file_path }}/{{ vm_name }}.qcow2
# - name: resize image
# shell: qemu-img resize {{ image_file_path }}/{{ vm_name }}.qcow2 +{{ disk_size }}G
# when: disk_size
- name: backing image create
shell: qemu-img create -b {{ disk.cloud_image }} {{ image_file_path }}/{{ vm_name }}.qcow2 -f qcow2 {{ disk_size }}G
when: disk_size
- name: create vm
command: virt-install --import
--name {{ vm_name }}
--ram {{ mem }}
--vcpus {{ cpu }}
--os-type {{ os.type }}
--os-variant {{ os.variant }}
--cpu host-passthrough
--disk {{ image_file_path }}/{{ vm_name }}.qcow2,format=qcow2,bus=virtio
--disk {{ image_file_path }}/{{ vm_name }}-{{ disk.cloud_init }},device=cdrom
--network network={{ network.net_name1 }},model=virtio
--network network={{ network.net_name2 }},model=virtio
--network network={{ network.net_name3 }},model=virtio
--network network={{ network.net_name4 }},model=virtio
--network network={{ network.net_name5 }},model=virtio
--graphics none
--{{virt_hypervisor}}
--virt-type {{ virt_type }}
--noautoconsole
#--network bridge={{ network.bridge }},model=virtio
when: vm_name not in virt_vms.list_vms
with_items: guests
- name: get guest info
virt: command=info
register: virt_info
- name: make sure all vms are running
virt: name={{ vm_name }} command=start
when: virt_info[vm_name]['state'] != 'running'
with_items: guests
|
[root@kvm2 createGuestVM]# cat hosts-kolla-v2 [node] 192.168.92.1 [vms] youngju2-con1 ansible_host="192.168.92.1" cpu=4 mem=8192 network_ipaddr="10.11.11.11" vm_name="youngju2-con1" youngju2-con2 ansible_host="192.168.92.1" cpu=4 mem=8192 network_ipaddr="10.11.11.12" vm_name="youngju2-con2" youngju2-con3 ansible_host="192.168.92.1" cpu=4 mem=8192 network_ipaddr="10.11.11.13" vm_name="youngju2-con3" youngju2-ceph1 ansible_host="192.168.92.1" cpu=4 mem=4096 network_ipaddr="10.11.11.31" vm_name="youngju2-ceph1" youngju2-ceph2 ansible_host="192.168.92.1" cpu=4 mem=4096 network_ipaddr="10.11.11.32" vm_name="youngju2-ceph2" youngju2-ceph3 ansible_host="192.168.92.1" cpu=4 mem=4096 network_ipaddr="10.11.11.33" vm_name="youngju2-ceph3" youngju2-com1 ansible_host="192.168.92.1" cpu=4 mem=4096 network_ipaddr="10.11.11.21" vm_name="youngju2-com1" youngju2-com2 ansible_host="192.168.92.1" cpu=4 mem=4096 network_ipaddr="10.11.11.22" vm_name="youngju2-com2" youngju2-deploy ansible_host="192.168.92.1" cpu=8 mem=8192 network_ipaddr="10.11.11.2" vm_name="youngju2-deploy" |
[root@kvm2 createGuestVM]# ansible-playbook -i hosts-kolla-v2 virt-guest-multiple-nic.yaml ... PLAY RECAP ************************************************************************************************************************************************************** youngju2-ceph1 : ok=11 changed=5 unreachable=0 failed=0 youngju2-ceph2 : ok=11 changed=5 unreachable=0 failed=0 youngju2-ceph3 : ok=11 changed=5 unreachable=0 failed=0 youngju2-com1 : ok=11 changed=5 unreachable=0 failed=0 youngju2-com2 : ok=11 changed=5 unreachable=0 failed=0 youngju2-con1 : ok=11 changed=5 unreachable=0 failed=0 youngju2-con2 : ok=11 changed=5 unreachable=0 failed=0 youngju2-con3 : ok=11 changed=5 unreachable=0 failed=0 youngju2-deploy : ok=11 changed=5 unreachable=0 failed=0 |
잘된다. ㅎㅎ
youngju blog 