无忧启动论坛

标题: 利用onedrive加packerpebuilder真正实现本地网络统一装机 [打印本页]
作者: minlearn    时间: 2020-7-2 19:27
标题: 利用onedrive加packerpebuilder真正实现本地网络统一装机
本帖最后由 minlearn 于 2020-7-3 16:08 编辑

本文关键字：bash line string to array, sed 按匹配次数替换，sed替换指定次数的某字符串,bash变量替换，即时赋新值，bash 数组下标变量

在《利用installnet制作一个云装机packerpe》1,2中我们谈到了packerpebuilder.sh用于云主机装机的用处，它的原理是产生一个基于d-i的pe，然后这个pe会使用当初封装的dd url去网络下载镜像并dd到硬件，在《利用od做站和nas》我们谈到了其与终端作pcmate,mobilemate的那些方面（这个系列只讲了做站，即配合client pc as page server mate的方面—想象一下，在客户端我们可以用servering pages，只是拥有了网盘backend的page server app才真正使远程和本地成为一对mates，产生本地远程合一的mate app，未来我们还会讲配合client pc作离线下载，网盘转存等mateable的方面），今天，我们将用onedrve结合packerpebuilder实现本地也能像云主机一样装机，使远程成为本地装机app，实际上这个思路自packerpebuilder一开始就有了，只是一直没有找到合适能用的网盘。直到对onedrive的直链有了研究之后，才有了本文。

不废话了

前置改动

把tmpmirror也消除了。调整了一些注释，如整合checkdeps和selectmirror为prepare prerequisites，selectmirror经过重构变成select1stvalidmirrorfrom3():

  1. function Select1stValidMirrorFrom3(){

  3.   [ $# -ge 1 ] || exit 1

  5.   declare -A MirrorTocheck
  6.   MirrorTocheck=(["Debian0"]="" ["Debian1"]="" ["Debian2"]="")
  7.   
  8.   echo "$1" |sed 's/\ //g' |grep -q '^http://\|^https://\|^ftp://' && MirrorTocheck[Debian0]=$(echo "$1" |sed 's/\ //g');
  9.   echo "$2" |sed 's/\ //g' |grep -q '^http://\|^https://\|^ftp://' && MirrorTocheck[Debian1]=$(echo "$2" |sed 's/\ //g');
  10.   echo "$3" |sed 's/\ //g' |grep -q '^http://\|^https://\|^ftp://' && MirrorTocheck[Debian2]=$(echo "$3" |sed 's/\ //g');

  12.   for mirror in $(echo "${!MirrorTocheck[@]}" |sed 's/\ /\n/g' |sort -n |grep "^Debian")
  13.     do
  14.       CurMirror="${MirrorTocheck[$mirror]}"

  16.       [ -n "$CurMirror" ] || continue

  18.       # CheckPass1='0';
  19.       # DistsList="$(wget --no-check-certificate -qO- "$CurMirror/dists/" |grep -o 'href=.*/"' |cut -d'"' -f2 |sed '/-\|old\|Debian\|experimental\|stable\|test\|sid\|devel/d' |grep '^[^/]' |sed -n '1h;1!H;$g;s/\n//g;s/\//\;/g;$p')";
  20.       # for DIST in `echo "$DistsList" |sed 's/;/\n/g'`
  21.         # do
  22.           # [[ "$DIST" == "jessie" ]] && CheckPass1='1' && break;
  23.         # done
  24.       # [[ "$CheckPass1" == '0' ]] && {
  25.         # echo -ne '\njessie not find in $CurMirror/dists/, Please check it! \n\n'
  26.         # bash $0 error;
  27.         # exit 1;
  28.       # }

  30.       CheckPass2=0
  31.       ImageFile="SUB_MIRROR/dists/jessie/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz"
  32.       [ -n "$ImageFile" ] || exit 1
  33.       URL=`echo "$ImageFile" |sed "s#SUB_MIRROR#${CurMirror}#g"`
  34.       wget --no-check-certificate --spider --timeout=3 -o /dev/null "$URL"
  35.       [ $? -eq 0 ] && CheckPass2=1 && echo "$CurMirror" && break
  36.     done

  38.     [[ $CheckPass2 == 0 ]] && {
  39.       echo -ne "\033[31mError! \033[0minitrd.gz not find in $CurMirror/jessie/main/installer-amd64/current/images/netboot/debian-installer/amd64/! \n";
  40.       bash $0 error;
  41.       exit 1;
  42.     }
  43. }

  45. MIRROR=$(Select1stValidMirrorFrom3 'http://httpredir.debian.org/debian' 'http://xxx' 'http://http://archive.debian.org/debian')
  46. [ -n "$MIRROR" ] && echo -en "Select Mirror ......:" && echo -en "[\033[32m ${MIRROR} \033[0m]\n" || exit 1
复制代码


主体改动

整合prepare parepare dist files包括downloading basic kernel and rootfs files（将它提前，逻辑更合理。）和downloading repo pkgs files，，以及接下来的PrepareDDessentials(其原来内部下载deb的逻辑整合到与下载full udeb一起)，，并将它们都变成可复用的函数和函数调用buildrepo()和PrepareDDessentials().

prepare parepare dist files与prepare others是并列的：前三者是大资源文件，后三者是小参数文件，将二者中间延时变量变成2s，各内部延时3s（内部还去掉了细节方面，肯定情况下的一些echo输出，改为直接exit 1，改为由主要的几句话来echo，界面输出更清），共5s

wget要调用ssl client才能tls certificate已完善，buildrepo()更强大，支持sed "s/(+|)/-/g"处理链接中的+号和号(tcb上的onemanager不支持这类特殊符号)，和更强大更逻辑清楚的拉取安装deb pkgs支持:

  1. IncPkgrepo='1'

  3. declare -A OPTPKGS
  4. OPTPKGS=(
  5.   ["libc1"]="pool/main/g/glibc/libc6_2.28-10_amd64.deb"
  6.   ["fmtlibc"]="xz"
  7.   ["binlibc"]=""

  9.   ["common1"]="pool/main/g/gnutls28/libgnutls30_3.6.7-4+deb10u3_amd64.deb"
  10.   ["common2"]="pool/main/p/p11-kit/libp11-kit0_0.23.15-2_amd64.deb"
  11.   ["common3"]="pool/main/libt/libtasn1-6/libtasn1-6_4.13-3_amd64.deb"
  12.   ["common4"]="pool/main/n/nettle/libnettle6_3.4.1-1_amd64.deb"
  13.   ["common5"]="pool/main/n/nettle/libhogweed4_3.4.1-1_amd64.deb"
  14.   ["common6"]="pool/main/g/gmp/libgmp10_6.1.2+dfsg-4_amd64.deb"
  15.   ["fmtcommon"]="xz"
  16.   ["bincommon"]=""

  18.   ["busybox1"]="pool/main/b/busybox/busybox_1.30.1-4_amd64.deb"
  19.   ["fmtbusybox"]="xz"
  20.   ["binbusybox"]="bin/busybox"

  22.   ["wgetssl1"]="pool/main/libi/libidn2/libidn2-0_2.0.5-1+deb10u1_amd64.deb"
  23.   ["wgetssl2"]="pool/main/libp/libpsl/libpsl5_0.20.2-2_amd64.deb"
  24.   ["wgetssl3"]="pool/main/p/pcre2/libpcre2-8-0_10.32-5_amd64.deb"
  25.   ["wgetssl4"]="pool/main/u/util-linux/libuuid1_2.33.1-0.1_amd64.deb"
  26.   ["wgetssl5"]="pool/main/z/zlib/zlib1g_1.2.11.dfsg-1_amd64.deb"
  27.   ["wgetssl6"]="pool/main/o/openssl/libssl1.0.0_1.0.1t-1+deb8u8_amd64.deb"
  28.   ["wgetssl7"]="pool/main/o/openssl/openssl_1.0.1t-1+deb8u8_amd64.deb"
  29.   ["wgetssl8"]="pool/main/w/wget/wget_1.20.1-1.1_amd64.deb"
  30.   ["fmtwgetssl"]="xz"
  31.   ["binwgetssl"]="usr/bin/wget"

  33.   ["webfs1"]="pool/main/libf/libffi/libffi6_3.2.1-9_amd64.deb"
  34.   ["webfs2"]="pool/main/m/mime-support/mime-support_3.62_all.deb"
  35.   ["webfs3"]="pool/main/libu/libunistring/libunistring2_0.9.10-1_amd64.deb"
  36.   ["webfs4"]="pool/main/w/webfs/webfs_1.21-ds1-12_amd64.deb"
  37.   ["fmtwebfs"]="xz"
  38.   ["binwebfs"]=""
  39. )

  41. function buildrepo(){

  43.   if [[ "$IncPkgrepo" == '1' ]]; then

  45.     echo -e "Downloading full udebs pkg files..... [\033[32m ${MIRROR}/dists/jessie/main/debian-installer/binary-amd64/Packages.gz \033[0m]\n"

  47.     repodir='/tmp/boot/var/log/debian'
  48.     mkdir -p $repodir

  50.     udeburl=".*pool\/main\(.*\)udeb.*"
  51.     wget --no-check-certificate -qO- "$MIRROR/dists/jessie/main/debian-installer/binary-amd64/Packages.gz" |gunzip -dc|sed "/$udeburl/!d"|sed "s/Filename: //g"|while read line
  52.     do
  53.       path=${line%/*}
  54.       mkdir -p $repodir/$path
  55.       file=${line##*/}
  56.       wget --no-check-certificate -qO $repodir/$path/$(echo $file|sed "s/\(+\|~\)/-/g") $MIRROR/$line
  57.     done

  59.     mkdir -p $repodir/dists/jessie/main/binary-amd64/
  60.     mkdir -p $repodir/dists/jessie/main/debian-installer/binary-amd64/
  61.     mkdir -p $repodir/dists/jessie/main/installer-amd64/current/images/

  63.     wget --no-check-certificate -qO $repodir/dists/jessie/Release $MIRROR/dists/jessie/Release
  64.     wget --no-check-certificate -qO $repodir/dists/jessie/main/binary-amd64/Release $MIRROR/dists/jessie/main/binary-amd64/Release
  65.     wget --no-check-certificate -qO $repodir/dists/jessie/main/debian-installer/binary-amd64/Release $MIRROR/dists/jessie/main/debian-installer/binary-amd64/Release
  66.     wget --no-check-certificate -qO $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz $MIRROR/dists/jessie/main/debian-installer/binary-amd64/Packages.gz; \
  67.     orisize=$(cat $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz | wc -c); \
  68.     orimd5=$(md5sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }'); \
  69.     orisha1=$(sha1sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }'); \
  70.     orisha256=$(sha256sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }'); \
  71.     gunzip -c $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz > $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages; \
  72.     rm -rf $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz; \
  73.     sed -i "s/\(+\|~\)/-/g" $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages; \
  74.     gzip -c $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages > $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz; \
  75.     rm -rf $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages; \
  76.     cursize=$(cat $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz | wc -c); \
  77.     curmd5=$(md5sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }'); \
  78.     cursha1=$(sha1sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }'); \
  79.     cursha256=$(sha256sum $repodir/dists/jessie/main/debian-installer/binary-amd64/Packages.gz| awk '{ print $1 }')

  81.     toreplace="main\/debian-installer\/binary-amd64\/Packages.gz"
  82.     linenoarray=($(grep -n $toreplace $repodir/dists/jessie/Release |cut -f1 -d:))

  84.     sed -i ${linenoarray[0]}s/$orimd5/$curmd5/ $repodir/dists/jessie/Release
  85.     sed -i ${linenoarray[0]}s/$orisize/$cursize/ $repodir/dists/jessie/Release
  86.     sed -i ${linenoarray[1]}s/$orisha1/$cursha1/ $repodir/dists/jessie/Release
  87.     sed -i ${linenoarray[1]}s/$orisize/$cursize/ $repodir/dists/jessie/Release
  88.     sed -i ${linenoarray[2]}s/$orisha256/$cursha256/ $repodir/dists/jessie/Release
  89.     sed -i ${linenoarray[2]}s/$orisize/$cursize/ $repodir/dists/jessie/Release

  91.     wget --no-check-certificate -qO $repodir/dists/jessie/main/installer-amd64/current/images/udeb.list $MIRROR/dists/jessie/main/installer-amd64/current/images/udeb.list

  93.     chmod -R 0644 $repodir/
  94.   fi

  96.   echo -e "Downloading optional deb pkg files...... [\033[32m ${MIRROR}/dists/jessie/main/binary-amd64/Packages.gz \033[0m]\n";

  98.   for pkg in `echo "$1" |sed 's/,/\n/g'`
  99.     do
  100.    
  101.       [[ -n "${OPTPKGS[$pkg"1"]}" ]] && {
  102.         for subpkg in $(echo "${!OPTPKGS[@]}" |sed 's/\ /\n/g' |sort -n |grep "^$pkg")
  103.           do
  104.             cursubpkgfile="${OPTPKGS[$subpkg]}"
  105.             [ -n "$cursubpkgfile" ] || continue

  107.             cursubpkgfilepath=${cursubpkgfile%/*}
  108.             mkdir -p $repodir/$cursubpkgfilepath
  109.             cursubpkgfilename=${cursubpkgfile##*/}
  110.             cursubpkgfilename2=$(echo $cursubpkgfilename|sed "s/\(+\|~\)/-/g")

  112.             wget --no-check-certificate -qO $repodir/$cursubpkgfilepath/$cursubpkgfilename2 $MIRROR/$cursubpkgfile; \
  113.             [[ "${OPTPKGS["fmt"$pkg]}" == "tar" ]] && ar x $repodir/$cursubpkgfilepath/$cursubpkgfilename2 data.tar.gz && tar xzf data.tar.gz && rm -rf data.tar.gz
  114.             [[ "${OPTPKGS["fmt"$pkg]}" == "xz" ]] && ar x $repodir/$cursubpkgfilepath/$cursubpkgfilename2 data.tar.xz && xz -d data.tar.xz && tar xf data.tar && rm -rf data.tar

  116.           done
  117.             [[ -n "${OPTPKGS["bin"$pkg]}" ]] && mv -f /tmp/boot/${OPTPKGS["bin"$pkg]} /tmp/boot/${OPTPKGS["bin"$pkg]}2
  118.             # [[ ! -f  /tmp/boot/${OPTPKGS["bin"$pkg]}2 ]] && echo 'Error! $1 SUPPORT ERROR.' && exit 1;
  119.       }

  121.     done

  123. }
  124. buildrepo libc,common,busybox,wgetssl;
复制代码


PrepareDDessentials()也更强大，支持sharepoint和office365个人的302跳转风格，强化《利用installnet制作一个云装机packerpe》2中关于仅支持office365style相关方面功能 — 其实sharepointstyle和office365 style也可自动公判断，但是我不想折腾了。

  1. UNZIP=''
  2. DDURL=''
  3. OFFICE365STYLE='0'
  4. SHAREPOINTSTYLE='1'

  6. function PrepareDDessentials(){

  8.   if [[ -n "$tmpDDURL" ]]; then
  9.     echo "$tmpDDURL" |grep -q '^http://\|^ftp://\|^https://';
  10.     [[ $? -ne '0' ]] && echo 'No valid URL in the DD argument,Only support http://, ftp:// and https:// !' && exit 1;

  12.     IMGHEADER="$(curl -Is "$tmpDDURL")";
  13.     IMGTYPE="$(echo "$IMGHEADER" | grep -E -o '200|302')" || IMGTYPE='0';

  15.     # [[ "$IMGTYPE" -ne '0' ]] && echo 'not a raw,tar,gunzip or 301/302 ref file, exit ... !' && exit 1 || {

  17.       [[ "$IMGTYPE" == '200' ]] && IMGHEADERCHECKPASS2="$(echo "$IMGHEADER" |grep -E -o 'raw|qcow2|gzip|x-gzip')" && {
  18.         [[ "$IMGTYPECHECKPASS2" == 'raw' ]] && UNZIP='0' && DDURL="$tmpDDURL" && sleep 3s && echo -en "[\033[32m raw \033[0m]";
  19.         [[ "$IMGTYPECHECKPASS2" == 'qcow2' ]] && UNZIP='0' && DDURL="$tmpDDURL" && sleep 3s && echo -en "[\033[32m raw \033[0m]";
  20.         [[ "$IMGTYPECHECKPASS2" == 'gzip' ]] && UNZIP='1' && DDURL="$tmpDDURL" && sleep 3s && echo -en "[\033[32m gzip \033[0m]";
  21.         [[ "$IMGTYPECHECKPASS2" == 'x-gzip' ]] && UNZIP='1' && DDURL="$tmpDDURL" && sleep 3s && echo -en "[\033[32m x-gzip \033[0m]";
  22.         [[ "$IMGTYPECHECKPASS2" == 'gunzip' ]] && UNZIP='2' && DDURL="$tmpDDURL" && sleep 3s && echo -en "[\033[32m gunzip \033[0m]";
  23.       }

  25.       [[ "$IMGTYPE" == '302' && "$OFFICE365STYLE" == '1' ]] && { \

  27.         IMGHEADERCHECKPASS2="$(echo "$IMGHEADER" |grep 'Location: http'|sed 's/Location: //g')" && IMGHEADERCHECKPASS2=${IMGHEADERCHECKPASS2%


复制代码

使用方法

中途提示备份,会给你30s上传/tmp/boot/var/log/debian仓库到onedrive或其它服务器创建镜像，。

  1.   [[ "$tmpINSTANTWITHOUTVNC" == '0' ]] &&  echo "finished, auto reboot after 30s...(if needed, you can press ctrl c to interrupt to bak the repodir:$repodir, then manually reboot to continue)";sleep 30s
复制代码

preseed中的mirrorhost换成你的od上传地址。

然后就然后了。。。

未来，我们要往packerpebuilder中集成不死booter。这样装机永远都不会因为抹了第一个硬盘哭了。




作者: minlearn    时间: 2020-7-2 19:57
2011ajie 发表于 2020-7-2 19:52
写了很多字，感谢分享！

谢谢鼓励
作者: minlearn    时间: 2020-7-2 20:01
pebuilder_od.sh.zip (8.85 KB, 下载次数: 5)


作者: minlearn    时间: 2020-7-2 20:02
谁做成win下安装到ntldr/ntmgr的pe
作者: 老婆驾到    时间: 2020-7-3 11:16
看不懂但是感觉很牛逼！
作者: minlearn    时间: 2020-7-3 16:51
老婆驾到 发表于 2020-7-3 11:16
看不懂但是感觉很牛逼！

就这样？

本来我可以搞成一个成品的pe，又怕你们看得太明白。哈
作者: MC阿虎    时间: 2020-7-3 22:19
感谢分享
作者: 老婆驾到    时间: 2020-7-4 10:31
minlearn 发表于 2020-7-3 16:51
就这样？

本来我可以搞成一个成品的pe，又怕你们看得太明白。哈

哈哈！我基本小白，所以你懂的。
作者: minlearn    时间: 2020-7-4 19:28
老婆驾到 发表于 2020-7-4 10:31
哈哈！我基本小白，所以你懂的。

搞个互联onedrive，再等我做出不死bootloader，配上这个脚本，装机简直不要太省事。
作者: 老婆驾到    时间: 2020-7-5 10:31
minlearn 发表于 2020-7-4 19:28
搞个互联onedrive，再等我做出不死bootloader，配上这个脚本，装机简直不要太省事。

阔以阔以！很强势，很牛逼！
作者: dmx2020    时间: 2020-7-7 23:53
感谢分享！
作者: 铿锵玫瑰    时间: 2020-8-7 08:07
太高大上，看不懂。
作者: 双响炮    时间: 2020-8-7 11:24
想法真的很好，支持！
作者: tss3428    时间: 2020-8-8 13:41
了不起，我只能傻傻地看。
作者: minlearn    时间: 2020-8-8 16:20

作者: 2011goodluckwxl    时间: 2020-10-9 01:15
希望有一天我也能成为这样的大神
作者: dawnstar    时间: 2020-11-6 10:37
赞一个，希望可以早日实现。
作者: dawnstar    时间: 2020-11-7 22:10
还要多久才可以实现？太期待了。



欢迎光临 无忧启动论坛 (http://bbs.c3.wuyou.net/) Powered by Discuz! X3.3