Skip to main content

Posts for year 2022 (old posts, page 1)

ami

create AMI

aws ec2 create-image --description 'backup ami of test server' --instance-id i-xxxx --name 'backup ami of test server' --no-reboot
aws ec2 deregister-image --image-id ami-xxxx 
aws ec2 describe-snapshots --owner-id xxxx --query 'Snapshots[?contains(Description, `ami-xxxx`)]'

create an instance with AMI

aws ec2 run-instances \
--image-id ami-xxxx \
--instance-type t4g.nano \
--key-name testkey \
--security-group-ids sg-xxxx \
--subnet-id subnet-xxxx \
--credit-specification 'CpuCredits=standard' \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=copied_instance}]' \
--associate-public-ip-address

delete AMI

aws ec2 deregister-image --image-id ami-xxxx 
aws ec2 describe-snapshots --owner-id xxxx --query 'Snapshots[?contains(Description, `ami-xxxx`)].SnapshotId'
aws ec2 delete-snapshot --snapshot-id snap-xxxx

ebs

create an EBS volume and attach to an EC2 instance

aws ec2 create-volume --availability-zone ap-northeast-1a --volume-type gp3 --size 1 --encrypted --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=test_volume}]'
aws ec2 describe-volumes --volume-id vol-xxxx
aws ec2 attach-volume --device /dev/xvdb --instance-id i-xxxx --volume-id vol-xxxx

make partition and make filesystem

$ lsblk
$ sudo parted /dev/nvme1n1 print
$ sudo parted /dev/nvme1n1 mklabel gpt
$ sudo parted /dev/nvme1n1 mkpart home ext4 1MB 100%

$ sudo mkfs -t ext4 /dev/nvme1n1p1
$ sudo tune2fs -L homefs /dev/nvme1n1p1
$ ls -l /dev/disk/by-label/

edit /etc/fstab and reboot

$ sudo mount /dev/nvme1n1p1 /mnt
$ sudo cp -pri /home/ubuntu /mnt
$ sudo cp -pri /etc/fstab /etc/fstab.000
$ sudo vi /etc/fstab
$ diff /etc/fstab /etc/fstab.000
3d2

aws ec2 reboot-instances --instance-ids i-xxxx --dry-run

extend disk size

aws ec2 modify-volume --volume-id vol-xxxx --size 2

extend partition and filesystem

$ lsblk
$ sudo growpart /dev/nvme1n1 1

$ df -hT /home
$ sudo resize2fs /dev/nvme1n1p1

create snapshot

aws ec2 describe-snapshots --owner-id xxxx
aws ec2 create-snapshot --volume-id vol-xxxx --description 'test snapshot of homefs' --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=homefs}]'
aws ec2 describe-snapshots --owner-id xxxx --filters 'Name=volume-id,Values=vol-xxxx'

create new EBS volume from snapshot

aws ec2 create-volume --availability-zone ap-northeast-1a --snapshot-id snap-xxxx --volume-type gp3 --encrypted --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=test_volume}]'
aws ec2 describe-volumes --volume-id vol-yyyy

attach new volume and detach old volume

aws ec2 attach-volume --device /dev/xvdc --instance-id i-xxxx --volume-id vol-yyyy
aws ec2 stop-instances --instance-ids i-xxxx
aws ec2 detach-volume --no-force --instance-id i-xxxx --volume-id vol-xxxx
aws ec2 start-instances --instance-ids i-xxxx

delete volume

aws ec2 delete-volume --volume-id vol-yyyy
aws ec2 describe-volumes

delete snapshot

aws ec2 describe-snapshots --owner-id xxxx
aws ec2 delete-snapshot --snapshot-id snap-xxxx

let's encrrypt

certbot certonly \
--dry-run \
-d www.example.net \
-m yourname@example.net \
--preferred-challenges dns-01  \
--server https://acme-v02.api.letsencrypt.org/directory \
--manual \
--manual-auth-hook /home/user/work/letsencrypt/dns01-auth.sh \
--manual-cleanup-hook /home/user/work/letsencrypt/dns01-clean.sh \
--post-hook /home/user/work/letsencrypt/post-hook.sh \
--work-dir /home/user/work/letsencrypt/work \
--logs-dir /home/user/work/letsencrypt/logs \
--config-dir /home/user/work/letsencrypt/conf
  • When you run certbot in non-root user, you have to specify --work-dir, --logs-dir, and --config-dir options. These directories have to be writable with your user.
  • You can only publish new certificate file via certbot. Your new certificate file will pushed under config-dir directory. Afterward, you can deploy it with your deploy tool which you like.
  • The dns-01 challenge authentication only needs DNS validation and don't need to access via 80/tcp nor web server installation on your server. When you use dns-01 challenge, you can use your script to update your dns resource to --manual-auth-hook (for authentication) and --manual-cleanup-hook (for cleanup entry).
    CERTBOT_DOMAIN varaible is used to show domain name which you want to use. CERTBOT_VALIDATION vaibale is used to show validation code.

sample script which create validtion entry for aws route53

sample script which delete validtion entry for aws route53

certbot renew \
--dry-run \
--post-hook /home/user/work/letsencrypt/post-hook.sh \
--work-dir /home/user/work/letsencrypt/work \
--logs-dir /home/user/work/letsencrypt/logs \
--config-dir /home/user/work/letsencrypt/conf
  • Once you get your new certificate, you have to update your certificate periodically.

network time security

Secure NTP with NTS

Chrony supports nts since version 4.0. I tried to build nts server and client.

server configuration

example of chrony 4.2 on ubuntu 22.04 LTS

server time.facebook.com iburst maxpoll 11
server time.google.com   iburst maxpoll 11
server time.apple.com    iburst maxpoll 11

ntsserverkey /etc/chrony/key.pem
ntsservercert /etc/chrony/crt.pem

key.pem and crt.pem are openssl private key and certificate(with intermediate ca certificate) file pair. The certificate needs to include your fqdn of your nts server.

When I put key and crt files in a sub directory of /etc/ssl, below error occured.

Could not set credentials : Error while reading file.

In /var/log/syslog, I found audit log of apparmor which shows it denied for chronyd to open the key file. When I put them in /etc/chrony, I successed to run it.

Before test from client, you have to open not only 123/udp for NTP but also 4460/tcp for NTS-KE.

$ chronyd -Q -t 8 'server mynts.example.com iburst nts'
$ sudo chronyc serverstats

client configuration

example of chrony 4.0 on Debian 11

server mynts.example.com iburst nts
server ptbtime1.ptb.de   iburst nts
server nts.time.nl       iburst nts
server nts.ntp.se        iburst nts

ntstrustedcerts /etc/chrony/cacert.crt

When you use selfsigned CA to make your certificate in the nts server, you have to show your own ca certificate file to chrony.conf in ntstrustedcerts.

wireguard on raspberryrpi

I tried to install wireguard with pivpn on 64bit Raspberry pi OS (buster, 4B), but it failed

$ cat << END > /tmp/pivpn_options.conf                                                                                                                                           
IPv4dev=xx
install_user=pi
VPN=wireguard
pivpnNET=10.x.x.0
subnetClass=24
ALLOWED_IPS="10.x.x.0/24, 192.168.y.0/24"
pivpnMTU=1420
pivpnPORT=51820
pivpnDNS1=1.1.1.1
pivpnHOST=myname.example.com
pivpnPERSISTENTKEEPALIVE=25
UNATTUPG=1
END

$ curl -L https://install.pivpn.io > /tmp/pivpn_install.sh
$ chmod +x /tmp/pivpn_install.sh
$ /tmp/pivpn_install.sh --noipv6 --unattended /tmp/pivpn_options.conf                                                                                                     


:: wireguard is not a supported VPN protocol on arm64 Debian, only 'openvpn' is

I was not able to select wireguard in intereractive interface. It select openvpn automatically.

$ /tmp/pivpn_install.sh --noipv6

Though I successed it on 32bit Raspberry pi OS (bullseye, 3B+) It needs NAPT settings to your pi to make a connection from outside wireguard peer.

I'd like to try it on bookwarm 32bit/64bit when it will be released on the next version of the pi :)

WebARENA Indigo API

WebARENA Indigo is one of the cheapest VPS services in Japan. It can be controlled via Rest-API for some functions.

create API key and API secret key

Before you use the Rest-API, you have to access controle panel and create API key and API secret key.

  1. access Indigo ダッシュボード
  2. click API鍵の管理 on the left side menu then select API鍵
  3. click API鍵の作成 on the top of right side
  4. write down API鍵 and API秘密鍵

If you'd like to do so, set them as environment variable

$ cat << END > ~/.webarena_secret
> export IndigoApiKey=xxxx
> export IndigoApiPrivateKey=xxxx
> END
$ chmod 600 ~/.webarena_secret
$ cat << END >> ~/.bashrc
> if [ -f ~/.webarena_secret ]; then
>   . ~/.webarena_secret
> fi
> END

create API token

After create API key and API secret key, you can create API token. You need this API token to call other Rest-APIs.

IndigoToken=$(curl -s -X POST \
  https://api.customer.jp/oauth/v1/accesstokens\
  -H 'Content-Type: application/json' \
  -d '{
    "grantType": "client_credentials",
    "clientId": "'${IndigoApiKey}'",
    "clientSecret": "'${IndigoApiPrivateKey}'",
    "code": ""
}' | jq -r .accessToken)

The output of this API includes "expiresIn": "3599". I have not confirmed but it would expire in about 3600 seconds.

register ssh key

ssh key will be installed in ~/.ssh/authorized_keys of login user when the instance will be created. You can register your own ssh public key instead of create it.

curl -s -X POST \
  https://api.customer.jp/webarenaIndigo/v1/vm/sshkey \
  -H "Authorization: Bearer ${IndigoToken}" \
  -d '{
   "sshName": "testkey",
   "sshKey": "ssh-rsa xxxx"
}'

When I tested to upload "ed25519" public key but I got error message like below. It seems they don't allow ed25519 keys.

{"success":false,"errorMessage":"Invalid SSH key.","errorCode":"I10034"}

retrieve information to create instance

list ssh keys

curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/vm/sshkey \
  -H "Authorization: Bearer ${IndigoToken}" | jq .

list regions

instanceTypeId=$(curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/vm/instancetypes \
  -H "Authorization: Bearer ${IndigoToken}" | jq '.instanceTypes[] | select(.name == "instance") | .id')
curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/vm/getregion?instanceTypeId=${instanceTypeId} \
  -H "Authorization: Bearer ${IndigoToken}" | jq .

list os

curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/vm/oslist?instanceTypeId=${instanceTypeId} \
  -H "Authorization: Bearer ${IndigoToken}" | jq .

list instance plans

curl -s -X GET \
  'https://api.customer.jp/webarenaIndigo/v1/vm/getinstancespec?instanceTypeId='${instanceTypeId}'&osId=xx' \
  -H "Authorization: Bearer ${IndigoToken}" | jq .

create instance

curl -s -X POST \
  https://api.customer.jp/webarenaIndigo/v1/vm/createinstance \
  -H "Authorization: Bearer ${IndigoToken}" \
  -d '{
  "sshKeyId": xxxx,
  "regionId": x,
  "osId": xx,
  "instancePlan": x,
  "instanceName": "xxxx"
}'

list instances

curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/vm/getinstancelist \
  -H "Authorization: Bearer ${IndigoToken}"

When you create an instance, "instancestatus" will be "OS installation In Progress". After a while it will become "Stopped". It is a time to start your instance.

curl -s -X POST \
  https://api.customer.jp/webarenaIndigo/v1/vm/instance/statusupdate \
  -H "Authorization: Bearer ${IndigoToken}" \
  -d '{"instanceId":"xxxx","status":"start"}'

After you start your instance, "instancestatus" will become "Running"

firewall

The rule is a allow list. When you create a firewall rule and apply it to your instance, others will be denied.

create firewall rule

curl -s -X POST \
  https://api.customer.jp/webarenaIndigo/v1/nw/createfirewall \
  -H "Authorization: Bearer ${IndigoToken}" \
  -d '{
    "name":"xxxx",
    "inbound":[
        {"type":"Custom","protocol":"TCP","port":"22","source":"x.x.x.x"}
    ],
    "instances":["xxxx"]
}'

Above rule only allow ssh connection from x.x.x.x. All other inbound connection will be denied. Since there are no outbound rules, all outbound connection will be allowed

list firewall

curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/nw/getfirewalllist \
  -H "Authorization: Bearer ${IndigoToken}"

describe a firewall rule

curl -s -X GET \
  https://api.customer.jp/webarenaIndigo/v1/nw/gettemplate/xxxx \
  -H "Authorization: Bearer ${IndigoToken}"

update firewall

curl -s -X PUT \
  https://api.customer.jp/webarenaIndigo/v1/nw/updatefirewall \
  -H "Authorization: Bearer ${IndigoToken}" \
  -d '{
    "templateid":"xxxx",
    "name":"xxxx",
    "inbound":[
        {"type":"Custom","protocol":"UDP","port":"123","source":"0.0.0.0"},
        {"type":"Custom","protocol":"TCP","port":"22","source":"x.x.x.x"}
    ],
    "instances":["xxxx"]
}'

snapshot

  1. stop instance
  2. create/restore snapshot
  3. start instance

When start instance after restore from snapshot, the boot process may stack at GRUB. You can confirm the situation at QEMU console.

How to access the console:

  1. access Indigo ダッシュボード
  2. click instance management on the left side menu then select instance.
  3. click select on the right side of the target instance and select access.
  4. click "start console"
  5. If it has stacked at GRUB, click force stop of the left side. After stop, start it again.

copy partition of sd card

dump partition of original sd card

sudo parted /dev/sdx print
sudo dd if=/dev/sdx1 of=tablet_sdx1.img bs=4M

restore to new sd card

sudo parted /dev/sdx print
sudo dd if=tablet_sdx1.img of=/dev/sdx1 bs=4M

confirm the size of filesystem

sudo mount /dev/sdb1 /media/tmp
df
sudo umount /media/tmp

extend the filesystem (if needed)

sudo apt install --no-install-recommends fatresize
sudo fatresize -i /dev/sdb1
max=$(expr $(sudo fatresize -i /dev/sdb1 | awk -F: '$1~/Max size/{print $2}') / 1024)
sudo fatresize -s ${max}k /dev/sdb1
sudo fatresize -i /dev/sdb1

confirm the size of filesystem

sudo mount /dev/sdb1 /media/tmp
df