shell-not-just-on-work

用于整理工作和平时的脚本。每天更新(暂定连续更新30天)。

详情

A 运维
B 业务
D 开发
😋 自我觉得不错

列表

A-001 😋 SSH批量无密钥登录Expect脚本 - ssh-copy-id.expect

搞几台测试的机器:

docker run -d cu.eshore.cn/library/java:jdk8 /usr/sbin/sshd -D
docker run -d cu.eshore.cn/library/java:jdk8 /usr/sbin/sshd -D
docker run -d cu.eshore.cn/library/java:jdk8 /usr/sbin/sshd -D

结合for使用:

$ HOSTS=` docker ps | head -4 | grep -v IMAGE | awk '{print $1}' | xargs -I{} docker exec {} ifconfig | grep '10.' | awk -F' |:' '{print $(NF-6)}' `
$ for h in $HOSTS ; do 
  ./ssh-copy-id.expect $h 'root';
done

结合while使用:

$ docker ps | head -4 | grep -v IMAGE | awk '{print $1}' | xargs -I{} docker exec {} ifconfig | grep '10.' | awk -F' |:' '{print $(NF-6)}' | while read h ; do  
  ./ssh-copy-id.expect $h 'root' </dev/null ; 
done

A-002 新机器初始化配置脚本 - new-datanode-init.sh

首先新机器的密码PASSWD和Puppet服务器地址PUPPETSERVER两个环境变量；然后执行命令:

$ ./new-datanode-init.sh hadoop-slaver{200..300}

脚本做了如下事情：

SSH无密钥登录；
拷贝yum repo；
安装puppet-agent、mcollective；
修正hostname；
配置启动mcollective。

B-003 使用SparkSQL结合多种数据来源导出数据 - extract-activeresource-domain.sh

A-004 批量修改bond的模式

A-005 😋 使用Puppet批量修改用户密码

A-006 😋 XML转CSV - format.xslt

$ xsltproc format.xslt hadoop-2.6.3/etc/hadoop/core-site.xml

A-007 类似 kubectl exec 的功能 - pod_bash

$ pod_bash CONTAIN_NAME NAMESPACE

A-008 K8S下测试环境集群快速搭建 - deploy-cluster-on-k8s.sh simple-hadoop.yaml

A-009 😋 不重启Docker的情况下通过国内加速下载Docker镜像（for centos6） - docker-download-mirror

A-010 😋 突破堡垒机 - 堡垒机LogonScript.vbs, 需要了解本地端口转发、远程端口转发。

A-011 Scala生产小测试(生产调用java接口还是挺麻烦的-- 💥 可以做Sockt5代理，直接运行scala连接接口还是挺方便的)

$ scala -cp 'lib/common/*:lib/core/*'

然后结合 : paste 和 CTRL+D 以及 : load file.scala(不能含package) 进行块操作。根据你的业务写代码，就可以调用接口了。想怎么调用就怎么调用。

A-012 Shell并发N执行 - .bash_multi 有很多其他的工具可以控制并发N的执行：xargs, mco puppet, pssh

举个例子：

source .bash_multi
WORK_MULTI_THREADS=1
threadpool_start; // 初始化N线程

for line in `cat $WORK_LIST` ;  do
  function handle(){ change it }
  function doit(){ 
    handle "$@" ;  
    threadpool_release ; // 释放
  }
  
  threadpool_require; // 获取一个线程
  
  if [ $WORK_MULTI_THREADS -le 1 ] ; then 
    doit $line
  else 
    doit $line & 
  fi
done

threadpool_destory; // 结束，清理

B-002 定时（结合crontab）根据情况修改分析数据库配置表 - dynamic_access_conf.sh

@deprecated D-001 😋 本地Cygwin命令打开当前路径 - cygexplorer

@deprecated D-002 快速打开Eclipse中选中的文件（夹）所在目录 - explorer.launch

A-013 windows本地跑Zookeeper - win-zkServers.bat

A-014 😋 shell实现数组乱序

cat ABC | shuf
cat ABC | sort -R

D-003 RMI远程调试

首先用SecureCRT连接到生产环境做个Socks5代理（RMI单纯本地端口转发是不行的）。
本地Java配置代理：-DsocksProxyHost=127.0.0.1 -DsocksProxyPort=5555, 然后本地代码RMI的地址指定为生产对应服务的地址。

A-015 内容实现 A - B

首先把A、B的内容保存到两个文件，然后使用diff来处理：

diff -u A B | grep '^-' | grep '\\lib' | sed 's/^-/-libraryjars /'

做项目jar混淆处理时，injars和libraryjars不能重。挺方便的一种方式，尽管不是很严谨，初步筛选出来，再执行报错一个个的处理就好了。

A-016 😋 使用SSH Multiplexing加速SSH

NOTE: 加上ControlPersist后，可以用来实现脚本远程ssh连接只输入一次密码的效果！

启用SSH Multiplexing：

$ vi .ssh/config
Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h:%p
    ControlPersist 10m

效果：

[hadoop@hadoop-master1 ~]$ time ssh hadoop-master2 hostname
hadoop-master2

real    0m0.176s
user    0m0.022s
sys     0m0.005s
[hadoop@hadoop-master1 ~]$ time ssh hadoop-master2 hostname
hadoop-master2

real    0m0.030s
user    0m0.002s
sys     0m0.004s
[hadoop@hadoop-master1 ~]$ ssh -O check hadoop-master2
Master running (pid=26923)
[hadoop@hadoop-master1 ~]$ ps 26923
  PID TTY      STAT   TIME COMMAND
26923 ?        SNs    0:00 ssh: /tmp/hadoop@hadoop-master2:22 [mux]
[hadoop@hadoop-master1 ~]$ ssh -O exit hadoop-master2

A-017 Shell数值运算

A-018 Test常见用法

A-019 优化git-log输出

$ git log --pretty='format:%h %ar%x09%s' -4
357ce9d 2 weeks ago     format code
d58845d 2 weeks ago     fix wrong email
5ba6be9 2 weeks ago     instance alert config dao
db3ad71 2 weeks ago     fix wrong log

D-004 Maven脚手架生成加速

mvn archetype:generate命令时，加上-DarchetypeCatalog=local

A-020 下载远程服务器上的GIT项目

# 设置SOCKS5代理
$ cat ~/.ssh/config
Host hadoop-*
  User hadoop
  ProxyCommand nc -v -x 127.0.0.1:5555 %h %p

# 无密钥登录
$ chmod 600 ~/.ssh/config
$ ssh-copy-id hadoop@hadoop-slaver388

# CLONE
$ git clone hadoop@hadoop-slaver388:~/playbook

A-021 tomcat远程调试

修改startup.sh的最后一行(JDPA的环境变量用默认的也行)启动，然后本地启动eclipse/idea启动远程调试连接对应的服务器端口就行了：

set JPDA_TRANSPORT=dt_socket
set JPDA_ADDRESS=8000
set JPDA_SUSPEND=y

call "%EXECUTABLE%" jpda start %CMD_LINE_ARGS%

A-022 SSH端口转发

ssh -L
ssh -D
ssh -R

supervisor + autossh

A-023 格式化/处理JSON

curl xxxx|python -m json.tool

curl xxxx|jq .

A-024 大文件拆分

split -b 512m nexus-0906.tar.gz nexus-0906-split

A-025 Less跳到第N行

If the file is open you can type:

* 100g to go to the 100th line.
* 50p to go to 50% into the file.
* 100P to go to the line containing 100th byte.

You can use these from terminal by adding + in front of them: less +100g bigfile.txt

you can type $ to go to the last line

D-005 抓取数据的一些片段

D-0006 scala启动程序

https://github.com/winse/spark-examples/blob/master/src/main/scala/com/github/winse/spark/HelloMetrics.scala

A-026 Windows-Ubuntu explorer

#!/bin/sh

WHERE=$( readlink -f $PWD | sed -e 's#/mnt/\([a-zA-Z0-9]*\)#\1:#' -e 's#/#\\#g' )

explorer.exe "$WHERE"

then on the shell windows, execute command explorer.

winse@DESKTOP-ADH7K1Q:~/git/pinpoint$ explorer

A-027 虚拟机磁盘整理

设置 - 硬盘 - (右边的选项)磁盘整理|压缩

A-028 Eclipse标识的部分代码不格式化

Java > Code Style > Formatter > Edit > Off/On Tags

A-029 视频转换

mkv to mp4

PS E:\迅雷下载> E:\local\ffmpeg\bin\ffmpeg.exe -i '.\大闹天宫(修复版).DVD高清国语中字.mkv' -c copy dltg.mp4

for f in *.mkv; do ffmpeg -i "$f" -c copy "${f%.mkv}.mp4"; done
for /R %%f IN (*.mkv) DO ffmpeg -i "%%f" -c copy "%%~nf.mp4"

D-004 调试的时刻不修改代码输出信息

通过条件断点增加输出：System.out.println(cursor);false

D-005 仅检出git仓库子目录

repo="https://github.com/winse/helloworld.git"
branch=hello
subfolder=HelloScalaMapReduce

mkdir $subfolder
cd $subfolder
git init
git remote add -f origin $repo
git config core.sparsecheckout true
: 指定需要检出的子目录/文件
echo HelloScalaMapReduce > .git/info/sparse-checkout
git pull origin $branch

A-030 scp如何跨过中转主机直接传输文件？

scp -o ProxyCommand='ssh -q root@jump -W %h:%p' -r bigendian@search1:~/search/lib ~/search/

rsync -a -e 'ssh -A A ssh -A B' /tmp/a/ C:/tmp/b/

scp -o ProxyCommand="ssh $jump_host nc $host 22" $local_path host:$destination_path

A-031 程序启动后再切到后台

simple

To start a command in background, do command & or nohup command & (if you want to log out and still have it running)
To stop the running command, C-z (ctrl-Z), to put the stopped command in background do bg. With fg you bring it back into the foreground, jobs tells you what is stopped or in background. kill %k kills job number k, fg %k (or plain %k) brings it to the foreground. You can start stopped job k in the background by bg %k (or simply %k &).

boot

Option 1: nohup The best way is often the simplest. nohup long-running-command & It was made specifically for this, it even logs stdout to nohup.log.
Option 2: bg + disown

ctrl+z
bg
disown -h

Name		Name	Last commit message	Last commit date
Latest commit History 78 Commits
batch-word-save-as-pdf		batch-word-save-as-pdf
deprecated		deprecated
.bash_multi		.bash_multi
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
THINKING.md		THINKING.md
deploy-cluster-on-k8s.sh		deploy-cluster-on-k8s.sh
download_m3u8.sh		download_m3u8.sh
dynamic_access_conf.sh		dynamic_access_conf.sh
extract-activeresource-domain.sh		extract-activeresource-domain.sh
format.xslt		format.xslt
new-datanode-init.sh		new-datanode-init.sh
pod_bash		pod_bash
simple-hadoop.yaml		simple-hadoop.yaml
ssh-copy-id.expect		ssh-copy-id.expect
win-zkServers.bat		win-zkServers.bat
堡垒机LogonScript.vbs		堡垒机LogonScript.vbs

License

winse/shell-not-just-on-work

Folders and files

Latest commit

History

Repository files navigation