-
Notifications
You must be signed in to change notification settings - Fork 1
/
atom.xml
468 lines (257 loc) · 660 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Shaun's Space</title>
<subtitle>求知！ 视界！ 未来！ ↖(^ω^)↗</subtitle>
<link href="http://cniter.github.io/atom.xml" rel="self"/>
<link href="http://cniter.github.io/"/>
<updated>2024-06-20T16:53:01.477Z</updated>
<id>http://cniter.github.io/</id>
<author>
<name>Shaun</name>
</author>
<generator uri="https://hexo.io/">Hexo</generator>
<entry>
<title>HTTP 超时浅见</title>
<link href="http://cniter.github.io/posts/36b343ff.html"/>
<id>http://cniter.github.io/posts/36b343ff.html</id>
<published>2024-05-12T02:26:32.000Z</published>
<updated>2024-06-20T16:53:01.477Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 最近业务调用方反馈接收到服务连接中断的错误（python requests 请求抛出异常 <code>raise ConnectionError(err, request=request) \n ConnectionError: ('Connection aborted.', BadStatusLine("''",))</code>），但从 golang 服务日志中看，服务应该是正常处理完成并返回了，且抛出异常的时间也基本和服务返回数据的时间一致，即表明在服务响应返回数据的那一刻，请求方同时抛出异常。</p><p> 这个问题很奇怪，起初拿到一个 case 还无法稳定复现，最初怀疑是网络抖动问题，但后续一直会偶发性出现，直到拿到了一个能稳定复现的 case，深入跟踪排查后才发现与网络问题无关，是服务端框架应用设置不合理的问题。</p><span id="more"></span><h2 id="问题篇">问题篇</h2><p> 从网上搜索 <code>python ConnectionError: ('Connection aborted.')</code>，错误种类非常多，有网络问题，服务端问题（关闭连接，拒绝服务，响应错误等），客户端关闭连接，超时设置不合理，请求参数/协议错误等等，但若带上 <code>BadStatusLine("''",)</code> ，错误就相对比较明确了（<a href="https://stackoverflow.com/questions/47196100/badstatusline-error-in-using-python-requests">BadStatusLine Error in using Python, Requests</a>，<a href="https://stackoverflow.com/questions/33174804/python-requests-getting-connection-aborted-badstatusline-error">Python Requests getting ('Connection aborted.', BadStatusLine("''",)) error</a>），主要是由于收到了一个空响应（header/body），空响应可以明确是服务端返回的问题，一般可能有以下几个原因：1. 服务端反爬；2. 服务端超时（比如 nginx 默认 60s 超时）；3. 网络错误。</p><p> 由于是内部服务，所以反爬策略是没有的，而反馈的 case 都带有明显的特征（请求数据量大，处理耗时长），没有网络抖动那种随机性，所以应该也不是网络问题，剩下的只能是超时问题，由于业务方在前置策略上已经识别该 case 数据量大，所以不经过 nginx 网关，直连服务请求，所以也不会有 nginx 超时问题，只能是服务端自己超时。于是直接在代码中查找 timeout 关键字，发现在服务启动时设置了 ReadTimeout 和 WriteTimeout，进一步深挖之后，才对 go 服务的超时有了浅显的认识。</p><h2 id="超时篇">超时篇</h2><p>参考资料：1. <a href="https://cloud.tencent.com/developer/article/1836274">你真的了解 timeout 吗？</a>，2. <a href="https://mp.weixin.qq.com/s?__biz=MzkxNTU5MjE0MQ==&mid=2247492773&idx=1&sn=972e67c2536225e6f01b70c52a08aee6&source=41#wechat_redirect">i/o timeout ， 希望你不要踩到这个net/http包的坑</a>，3. <a href="https://zhuanlan.zhihu.com/p/551557249">net/http完全超时手册</a>。</p><p> 由于 <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP">HTTP</a> 协议规范并未提及超时标准，而为保证服务稳定性，一般的 HTTP 服务请求都会设置超时时间，各 HTTP 服务端/客户端对于超时的理解大同小异，而这次的问题又起源与 go 服务，所以以 go 为例，分析一下超时。</p><h3 id="客户端超时">客户端超时</h3><figure><img src="https://pic4.zhimg.com/80/v2-a66e1905b916028ef5aa2ee41cc9a673_1440w.webp" alt="http.Client.Timeout" /><figcaption aria-hidden="true">http.Client.Timeout</figcaption></figure><p> 客户端超时，即 GET/POST 请求超时，这个很好理解，就是客户端发送请求到客户端接收到服务器返回数据的时间，算是开发的一般性常识，控制参数一般也特别简单，就是一个 timeout，当然 go 服务客户端支持设置更精细化的超时时间，一般也没啥必要。当客户端感知到超时时，会正常发起 TCP 断开连接的“四次挥手”过程。</p><h3 id="服务端超时">服务端超时</h3><figure><img src="https://pic2.zhimg.com/80/v2-8e520795ca5a552fba7ebffd301f6b95_1440w.png" alt="http.Server Timeouts" /><figcaption aria-hidden="true">http.Server Timeouts</figcaption></figure><p> 服务端超时，这才是引发问题的根本原因，go 服务端的超时，主要有两个参数，ReadTimeout 和 WriteTimeout，从上图可以看出，ReadTimeout 主要是设置服务端接收请求到读取客户端请求数据的时间（读请求的时间），WriteTimeout 是服务端处理请求数据以及返回数据的时间（写响应的时间）。GoFrame 框架的 ReadTimeout 默认值是 60s，在请求数据正常的情况下 ReadTimeout 也不可能超时，这次的问题主要出在 WriteTimeout，GoFrame 的默认值是 0s，代表不控制超时，但之前的开发者也同样设置为了 60s，导致服务端在处理大量数据时，发生了超时现象。</p><p> 更深挖之后，才发现 WriteTimeout 的诡异之处，当 WriteTimeout 发生之后，服务端不会即时返回超时消息，而是需要等服务端真正处理完之后，返回数据时，才会返回一个空数据，即使服务端正常写入返回数据，但都会强制为空数据返回，导致请求客户端报错。这种表现，看起来就像是 WriteTimeout 不仅没有起到应有的作用，在错误设置的情况下，还会起到反作用，使服务响应错误。WriteTimeout 无法即时生效的问题，也同样有其他人反馈了：1. <a href="https://adam-p.ca/blog/2022/01/golang-http-server-timeouts/">Diving into Go's HTTP server timeouts</a>；2. <a href="https://github.com/golang/go/issues/59602">net/http: Request context is not canceled when <code>Server.WriteTimeout</code> is reached</a>。可能是网上反馈的人多了，go 官方推出了一个 <a href="https://github.com/golang/go/blob/master/src/net/http/server.go#L3571">TimeoutHandler</a>，通过这个设置服务端超时，即可即时返回超时消息。仿照官方的 TimeoutHandler ，即可在 GoFrame 框架中也实现自己的超时中间件。</p><p> 至于 WriteTimeout 为啥不起作用，个人猜测主要原因在于 go 服务每接收到一个请求，都是另开一个协程进行处理，而 <a href="https://geektutu.com/post/hpg-timeout-goroutine.html#3-%E5%BC%BA%E5%88%B6-kill-goroutine-%E5%8F%AF%E8%83%BD%E5%90%97%EF%BC%9F">goroutine 无法被强制 kill，只能自己退出</a>，通常是要等到 goroutine 正常处理完之后才能返回数据，WriteTimeout 只是先强制写一个空数据占位，返回还是得等 goroutine 正常处理完。</p><p> 所以正常的 go 服务，在使用类似于 TimeoutHandler 中间件的时候，也最好让 goroutine 尽可能快的退出，一种简单的方法是：1. 设置请求的 context 为 context.WithTimeout；2. 分步处理数据，每一步开始前都先检查请求传入的 context 是否已经超时；3. 若已经超时，则直接 return，不进行下一步处理，快速退出 goroutine。</p><h2 id="后记">后记</h2><p> 这次问题排查，碰到的最大障碍在于，前几次反馈的 case 难以复现，客户端请求报错和服务器返回的时间一致也不会让人往超时的角度去想，在拿到一个能稳定复现的 case 之后，才死马当活马医，先调一下超时参数试试。</p><p> 关于 go 服务超时的文章，其实之前也看过，但没碰到具体问题，名词也就仅仅只是名词，很难理解背后的含义和其中的坑点，实践才能出真知 ╮(~▽~)╭。</p><h2 id="附录">附录</h2><h3 id="长连接超时">长连接超时</h3><p> 关于超时问题，也曾看到过有人碰到一个长链接服务的问题，现象是这样的：后端服务宕机之后，客户端可能需要很久才会感知到，原因在于 tcp 的超时重传机制，在 linux 中，默认会重传 tcp_retries2=15 次（即 16 次才会断开连接），而 TCP 最大超时时间为 TCP_RTO_MAX=2min，最小超时时间为 TCP_RTO_MIN=200ms。即在 linux 中，一个典型的 TCP 超时重传表现为：</p><table><thead><tr class="header"><th>重传次数</th><th>发送时间</th><th>超时时间</th></tr></thead><tbody><tr class="odd"><td>-1（原始数据发送）</td><td>0s</td><td>0.2s</td></tr><tr class="even"><td>0 （第 0 次重传）</td><td>0.2s</td><td>0.2s</td></tr><tr class="odd"><td>1</td><td>0.4s</td><td>0.4s</td></tr><tr class="even"><td>2</td><td>0.8s</td><td>0.8s</td></tr><tr class="odd"><td>3</td><td>1.6s</td><td>1.6s</td></tr><tr class="even"><td>4</td><td>3.2s</td><td>3.2s</td></tr><tr class="odd"><td>5</td><td>6.4s</td><td>6.4s</td></tr><tr class="even"><td>6</td><td>12.8s</td><td>12.8s</td></tr><tr class="odd"><td>7</td><td>25.6s</td><td>25.6s</td></tr><tr class="even"><td>8</td><td>51.2s</td><td>51.2s</td></tr><tr class="odd"><td>9</td><td>102.4s</td><td>102.4s</td></tr><tr class="even"><td>10</td><td>204.8s</td><td>120s</td></tr><tr class="odd"><td>11</td><td>324.8s</td><td>120s</td></tr><tr class="even"><td>12</td><td>444.8s</td><td>120s</td></tr><tr class="odd"><td>13</td><td>564.8s</td><td>120s</td></tr><tr class="even"><td>14</td><td>684.8s</td><td>120s</td></tr><tr class="odd"><td>15</td><td>804.8s</td><td>120s</td></tr><tr class="even"><td>断开连接</td><td>924.8s（≈15min）</td><td></td></tr></tbody></table><p>所以客户端需要在 15 分钟之后才能感知到服务端不可用，如此，仅靠 TCP 自身的超时机制，很难发现服务端是否宕机/不可用，长链接不释放，进而可能导致客户端不可用且无感知，所以在长链接服务中，需要有其他的手段来保障服务稳定/可用性（eg：心跳探活）。</p><h3 id="服务端-context-canceled">服务端 context canceled</h3><p><em>Refer to: <a href="https://learnku.com/articles/63884">context canceled，谁是罪魁祸首</a></em></p><p> 从官方的 net/http 包中可以知道，go 服务在接收请求时，会同时生成一个协程监控连接状态，当发现连接有问题（eg：客户端设置请求超时主动断开）时，会将该请求对应的 context cancel 掉，这时服务端如果再继续使用该 context 时，就会报错「context canceled」。当然，如果服务端发生错误，也同样会导致请求对应的 context cancel 掉。</p><p> 服务端主动 cancel context 的好处在于可以快速释放资源，避免无效的请求继续执行（当然也得业务代码上主动去感知 context 是否 cancel，从而及时退出）；坏处在于，<strong>如果服务端需要上报这个请求发生的错误（一般在后置中间件中进行错误上报），这个时候上报错误的请求需要另外生成一个新的 context</strong>，绝不能直接使用现有的 context，因为已有的这个 context 已经 cancel 掉了，继续使用会导致上报错误的请求发送失败，达不到上报的目的。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 最近业务调用方反馈接收到服务连接中断的错误（python requests 请求抛出异常 <code>raise ConnectionError(err, request=request) \n ConnectionError: ('Connection aborted.', BadStatusLine("''",))</code>），但从 golang 服务日志中看，服务应该是正常处理完成并返回了，且抛出异常的时间也基本和服务返回数据的时间一致，即表明在服务响应返回数据的那一刻，请求方同时抛出异常。</p>
<p> 这个问题很奇怪，起初拿到一个 case 还无法稳定复现，最初怀疑是网络抖动问题，但后续一直会偶发性出现，直到拿到了一个能稳定复现的 case，深入跟踪排查后才发现与网络问题无关，是服务端框架应用设置不合理的问题。</p></summary>
<category term="Problems" scheme="http://cniter.github.io/categories/Problems/"/>
<category term="golang" scheme="http://cniter.github.io/tags/golang/"/>
<category term="http" scheme="http://cniter.github.io/tags/http/"/>
</entry>
<entry>
<title>关于中学的学习方法</title>
<link href="http://cniter.github.io/posts/6be57d7f.html"/>
<id>http://cniter.github.io/posts/6be57d7f.html</id>
<published>2024-03-03T04:26:15.000Z</published>
<updated>2024-03-03T05:07:59.542Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 前些日子，小叔说堂弟的学习有点不太能跟上了，让 Shaun 和堂弟聊聊，回想十几年前， 父亲也是这样找堂哥的，仍记得那年的寒暑假，算是 Shaun 进步最快的一年，也是奠定 Shaun 后续学习方法的一年，现在轮到小叔来找 Shaun ，虽说不能当面聊，指导效果会大打折扣，而且当年堂哥教的具体方法也早已忘记，转化为自己的思想和方法，所以 Shaun 也只能把自己的东西说给堂弟，也算是某种意义上的传承。</p><span id="more"></span><h2 id="序篇">序篇</h2><p> Shaun 一直认为学习是有天赋，这种天赋体现在学习某一方面的事特别长记性，看个几眼就能完全记在脑海里，还能灵活变通记得的东西。同时，学习也需要方法的，在天赋不够的情况下，有个好的学习方法也能事半功倍。最后，学习是需要积累的，所谓的积累，就是增长见识，多练习，就中学而言，积累就是多做不同的题，同一类但举一反三的变题，在积累的足够多的情况下，考场上同样的题至少都是见过的，没有太多的心理压力，自然会好解一些。</p><p> 当然中学的学习毕竟是通过考试来验证结果的，而这个结果才是最重要的（也算是一种唯结果论，不过现实如此，社会如此，没人能逃过，只以成败论英雄，唯一需要注意的是英雄很多时候是有保质期的，扯远了 😅），所以应试技巧也很重要，考试是一个在一定时间内如何得分最多的任务，即使是所有的题都能解，但超时了也没用，更何况大部分人只能解一部分，所以对于这种任务，最好是先快速扫一下卷子，心里先有个数（大概都是些啥题），后面再按部就班的的做，性价比低（要花费大量时间，得分又低）的后面再解。当然在绝对的实力面前，所谓的应试技巧都是虚幻，打铁还是得自身硬，应试本质上是一个熟练的事，需要大量的练习，简而言之就是多刷题 🤪。</p><p> 闲话说完了，下面就是正文了，由于 Shaun 是理科生，仅记录 Shaun 还能记得的当年理科六门学科的学习经验。</p><h2 id="正篇">正篇</h2><h3 id="语文">语文</h3><p> 语文一直是 Shaun 的弱项，不过从 Shaun 现在的经验再回过头去看语文，感觉语文考验的更多是对人生和社会的一种感悟，这种感悟不仅仅只是对于自身的体验，也是对别人人生经历和当时社会的一种体会。在学生时代，大部分人受限于家庭和外部环境因素，自身体验很少有丰富的，只能多体会别人的人生，别人的人生只能依赖多看书（小说传记历史都可以），最重要的是在读的时候能有自己的一些思考，假如自己在别人的处境下会是一种什么心态，会有什么行动，一些好的文章，作者为什么会那样描写，遣词造句。当然语文也有直接需要记忆的，字词拼音，古诗文这种，就全靠记忆背诵了。</p><h3 id="数学">数学</h3><p> 中学数学最重要的两个分支就是代数和几何，以及介于两者之间的解析几何，于是也有了数学中最重要的思想——数形结合，抽象的数字有了形状，就不再那么枯燥。熟练使用函数图像以及对应的特点，数学及格就没啥问题了，至于几何，立体空间想象力不够的情况下，也可以加坐标系当解析几何计算了，不过就是时间花的多些。</p><p> 导数算是函数中最核心的概念（导数以及对应的微分也会在高等数学中贯穿始终），函数导数的几何意义就是对应点切线的斜率，当在实际的物理场景下，导数也有其实际意义，比如路程关于时间的导数就是速度，速度关于时间的导数就是加速度。</p><p> 数列可以认为是一种纯数字游戏，虽然通项公式或者递推式可以认为是某种函数，但数列本质还是数字自身的规律，这种更多的是经验和一种直觉，发现不了就是不能发现，无从下手也无法计算。</p><p> 集合和数理逻辑，不等式，极值，推理与证明，对应的反证法。概率与排列组合，这类问题熟记公式，太难的问题，不会就是不会了 🙃。</p><p> 向量计算，数形结合完美的体现，中学物理的利器，三角函数，向量的内外积，单位向量的意义，这些东西还是只能在练习中画图理解。向量这个数学工具的美，也只能在实际应用中体会，角度，投影，面积，距离（点点/点线/点面距离），坐标变换（旋转/平移/缩放）等等。</p><h3 id="英语">英语</h3><p> 英语也是 Shaun 不太在行的，尤其是现在回想 Shaun 整个中学，英语及格的次数都屈指可数，初中英语最后一次考试能及格还是靠初三下死命的记单词，而高中英语也是到高三才能稳定的及格，原因也是单词和语法记少了，更重要的原因是对死记硬背很是反感，甚至由于这个原因还和高二的英语老师对着干，一上英语课 Shaun 就直接出去了，后来还好高三换了个英语老师，给 Shaun 稍微开了一段时间的小灶，就是让 Shaun 每天写篇英语作文，然后针对这篇作文进行指导批改，这种方式很适合 Shaun ，从此也算是踏上了英语及格之路， Shaun 现在依然很感激高三的英语老师。至于英语听力，这个没办法，只能靠多听，以 Shaun 现在的经验看来，每天都有一定的时间处在英语环境下，确实能提高听力水平，多听的频率很重要，不然过一段时间就没那种感觉了。</p><h3 id="物理">物理</h3><p> 尤记得高一的物理也有很多次没及格，后来在堂哥的指导下，物理好歹也算是入门了，每次考个 80 分都还算轻松。目前还能记得堂哥教的物理学习方式就是手推公式，当然手推公式同样能应用到数学和化学上。所谓的手推公式就是利用一些基础的公式推导出一个复杂的公式，或者是两个复杂的公司来回推导，能够熟练的手推公式，圆周运动和电磁场问题公式层面的问题就能比较清楚了。至于受力分析，支撑力与面垂直，摩擦力与面平行，杆提供支撑力或许也有拉力，绳只提供拉力，可以假设圆周运动的离心力真实存在，与向心力平衡。至于能量守恒和动量守恒，这个只能多刷题了。</p><p> 物理是和数学强绑定的一门学科，数学不行，物理不可能会好的，所以要学好物理得先学好数学。</p><h3 id="化学">化学</h3><p> Shaun 算是有一定天赋的，看几遍书上的内容，就基本上都能记住了，不管是无机还是有机化学实验也基本都很清晰会有啥现象，每个元素的性质当时也都能记得，以至于看到一些常见的物质大概就能知道会有啥反应。不过还记得当时对于化学方程式配平， Shaun 还只能靠眼睛看，没啥方法，后来堂哥教了个得失电子法，同时针对性的做了大量的题，让 Shaun 领先全班一个学期熟练使用这个方法，在配平这类问题上基本没怎么丢过分。在刷题的过程中，也可以活用一些书上没见过的公式，曾经有次看到一个理想气体状态方程的公式，发现用这个公式可以很轻松的解释一些化学平衡的移动问题。化学在 Shaun 这里没怎么太刷过题，感觉就靠多看书了，熟记元素和物质的物理化学性质。</p><h3 id="生物">生物</h3><p> 生物感觉没太多好说的，就实验而言和化学有点像，但需要记忆的东西更多，最需要计算的题也就是染色体概率和群落数量估计问题了，不过就算不会算，丢分也不多。</p><h2 id="总结">总结</h2><p> 刷题是一种很有效的应试技巧，国内的大部分考试都能通过刷题解决，如果解决不了，那就多刷几遍，针对性的刷题会更有效果。</p><p> 死记硬背也是一种方式，但能活学活用更重要，在使用中记忆会更好，理科有个很重要的思想就是推理，大部分结论或公式都能通过一些简单前提或公式推导出来，可以试试自己推导一些常用的公式（关于推导，数学科普领域有本书叫「天才引导的历程」可以看看），注重平时的练习，不要怕麻烦，熟才能生巧。</p><p> 至于错题本，得看收集错题的方式，最好是一类题收集在一起，每种解题方式各收集一个经典的题型，后续有时间就翻翻回顾下，就 Shaun 个人的经验，记得很杂的错题本，往往起不到应有的效果，针对性的学习很重要，需要注意的是错题集不要做成了难题/怪题集。</p><p> 独立思考，本意是指不要人云亦云，需要有自己的思考和看法（这本应该是每个人的必备技能，但没有的人确实不少）。在学习领域，特指在寻求问题答案的过程中，一定先得有个自己的思考过程，苦思不得的问题会更深刻，同时思考的过程也是自己串通知识点的过程，更容易知道自己的盲区。</p><p> 因材施教，同样也因人学习，每个人在不同的学习环境下学习效率是不同的，有些人需要被人催促，需要更有压力一点才能学的好，而有些人更主动一些，在宽松的环境下学习更有效果。而目前的学校都是填鸭式教育，一视同仁，虽说每个学校的教学风格不太一样，但不一定适合学校内的每个学生，所以需要找到适合自己的方式。</p><h2 id="后记">后记</h2><p> 回顾整个高中生涯，对 Shaun 影响最大的其实还是堂哥和高三的英语老师，当时的班主任虽然对 Shaun 也很好，但对 Shaun 的学习和做事方式影响就没那么大，只记得当时班主任常说的一句话——读书是能改变命运的。对于大部分人，读书确实是最可行的出路，其他的路不确定性会更多，虽说读书需要一定的天赋，但国内应试教育的本质注定了努力刷题是能弥补这一部分天赋的，当然，如果有个人能在刷题的路上再稍微指导一下，会走很多弯路，也更容易找到适合自己的学习和思考方式。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 前些日子，小叔说堂弟的学习有点不太能跟上了，让 Shaun 和堂弟聊聊，回想十几年前， 父亲也是这样找堂哥的，仍记得那年的寒暑假，算是 Shaun 进步最快的一年，也是奠定 Shaun 后续学习方法的一年，现在轮到小叔来找 Shaun ，虽说不能当面聊，指导效果会大打折扣，而且当年堂哥教的具体方法也早已忘记，转化为自己的思想和方法，所以 Shaun 也只能把自己的东西说给堂弟，也算是某种意义上的传承。</p></summary>
<category term="Life" scheme="http://cniter.github.io/categories/Life/"/>
<category term="record" scheme="http://cniter.github.io/tags/record/"/>
</entry>
<entry>
<title>VNSWRR 算法浅解</title>
<link href="http://cniter.github.io/posts/619020f7.html"/>
<id>http://cniter.github.io/posts/619020f7.html</id>
<published>2024-02-07T14:31:58.000Z</published>
<updated>2024-10-14T16:45:05.956Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 最近偶然在公司内网看到一篇文章「负载均衡算法vnswrr改进——从指定位置生成调度序列」。正好 Shaun 一直觉得调度类算法很有意思，就认真看了下，顺便写下自己的一些理解。</p><span id="more"></span><h2 id="预备篇">预备篇</h2><p> 通俗来讲负载均衡解决的是「在避免机器过载的前提下，多个请求如何分发到多台机器上」的问题，本质上是一个分布式任务调度的问题，在机器性能相同的情况下，最简单的策略就是轮询，多个机器依次轮流处理请求。Nginx 官方的 SWRR 算法解决的是「在机器性能不同的情况下，如何使请求分布更均匀，更平滑，避免短时间大量请求造成局部热点」的问题。</p><h2 id="swrr篇">SWRR篇</h2><p> 在 SWRR 算法中，有两个权重，一个是初始实际权重（effective weight， ew），一个是算法迭代过程中的当前权重（current weight，cw），在负载均衡过程中，每次请求分发都选择当前权重最大的机器，同时更新每台机器的当前权重，当前权重更新策略如下：</p><ol start="0" type="1"><li><p>若设定 n 台机器各自的初始权重为 <span class="math inline">\((ew_1,ew_2,...,ew_n)\)</span>，同时 <span class="math inline">\(ew_1 \le ew_2 \le ... \le ew_n\)</span> ，且 <span class="math inline">\(W_{total}=\sum_{i=1}^n ew_i\)</span> ；</p></li><li><p>第一个请求来时，n 台机器各自的当前权重 <span class="math inline">\(cw_i=ew_i, 1 \le i \le n\)</span> ，由于此时 <span class="math inline">\(cw_{max}=\max(cw_i)=cw_n\)</span> ，则请求分发给第 n 台机器处理，同时更新机器各自的当前权重 <span class="math inline">\(cw_1=cw_1+ew_1, cw_2=cw_2+ew_2,...,cw_{n-1}=cw_{n-1}+ew_{n-1},cw_n=cw_n+ew_n-W_{total}\)</span>，记为 <span class="math inline">\((2*ew_1,2*ew_2,...,2*ew_{n-1},2*ew_n-W_{total})\)</span> ；</p></li><li><p>第二个请求来时，此时 n 台机器的各自权重为 <span class="math inline">\((2*ew_1,2*ew_2,...,2*ew_{n-1},2*ew_n-W_{total})\)</span> ，选取权重值对应的机器进行处理，假设为第 n-1 台，则更新后权重为 <span class="math inline">\((3*ew_1,3*ew_2,...,3*ew_{n-1}-W_{total},3*ew_n-W_{total})\)</span> ；</p></li><li><p>第 <span class="math inline">\(W_{total}\)</span> 个请求来时，此时 n 台机器的各自权重应该为 <span class="math display">\[(W_{total}*ew_1-m_1*W_{total},W_{total}*ew_2-m_2*W_{total},...,W_{total}*ew_{n-1}-m_{n-1}*W_{total},W_{total}*ew_n-m_n*W_{total}) \\\text{s.t.} \quad \sum_{i=1}^n m_i=W_{total}-1 \\\quad 0 <= m_i <= ew_i\]</span> 由于每次调度都是权重最大值减权重和，重新分配权重后权重和无变化，所以理论上此时除第 k 台机器外，每台机器的权重都为 0，第 k 台机器的权重为 <span class="math inline">\(W_{total}\)</span> ，所以这次调度处理之后，每台机器的权重又会重新回到初始权重。</p></li></ol><h2 id="vnswrr-篇">VNSWRR 篇</h2><p> VNSWRR 算法是阿里针对 Nginx 官方的 SWRR 算法实际运行中对于部分场景下（瞬时流量大，权重更新等）均衡效果不太理想的改进算法，其最大的改进点在于预生成调度序列，以空间换时间减少调度时间，同时在权重更新后随机选取调度序列的起点，使初次请求就调度在不同的机器上，减少高权重机器的局部热点问题。具体流程如下：</p><ol type="1"><li>首先使用 SWRR 算法生成前 n 个调度序列；</li><li>再随机选取一个位置作为调度起点，后续的请求依次从调度序列中选取；</li><li>若调度序列用完，则继续用 SWRR 算法生成后 n 个调度序列；</li><li>如此循环，直到调度序列的长度为 <span class="math inline">\(W_{total}\)</span>，即一个周期内的全部调度序列，用完后，从头开始调度即可；</li><li>若有权重更新，则从 1 开始重新生成调度序列；</li></ol><h2 id="正文">正文</h2><p> 从上面的逻辑中，可看出 SWRR 算法调度序列是以 <span class="math inline">\(W_{total}\)</span> 为周期的一个循环序列，只需要知道一个周期内的调度序列，就可以推算出后续的调度机器（除非权重有变更或者有机器增删）。计算一个周期内的调度序列也比较简单，取当前调度权重中最大值对应机器，同时更新每台机器的当前权重，作为下次调度的权重，简而言之，就是从上次调度结果推出下次调度结果，是一个递推式。那有没有办法不从上次结果推下次结果，直接计算当前的调度结果，简化 VNSWRR 的第一步每次都从头开始预生成前 n 个调度序列，直接从任意位置开始生成调度序列，内网中这篇文章就给出了一个看似“可行的”解决方案，直接计算第 q 个请求的调度结果，具体方案如下：</p><p>在 SWRR 算法中，第 q 个请求时，全部机器的当前权重序列应该为 <span class="math display">\[(q*ew_1-m_1*W_{total},q*ew_2-m_2*W_{total},...,q*ew_{n-1}-m_{n-1}*W_{total},q*ew_n-m_n*W_{total}) \\\text{s.t.} \quad \sum_{i=1}^n m_i=q-1 \\\quad 0 <= m_i <= ew_i\]</span> 即权重序列中共减去了 <span class="math inline">\(q-1\)</span> 个 <span class="math inline">\(W_{total}\)</span> ，平均上 <span class="math inline">\(m_i=ew_i/W_{total}*(q-1)\)</span>，区分 <span class="math inline">\(m_i\)</span> 的整数部分 <span class="math inline">\(mz_i\)</span> 和小数部分 <span class="math inline">\(mx_i\)</span>，<span class="math inline">\(\sum_{i=1}^n m z_i\)</span> 代表减去的 <span class="math inline">\(W_{total}\)</span> 个数，计算差值 <span class="math inline">\(d=q-1-\sum_{i=1}^n mz_i\)</span>，即还剩 d 个 <span class="math inline">\(W_{total}\)</span> 待减，对小数部分 <span class="math inline">\(mx_i\)</span> 从大到小排序，取前 d 个对应的机器再减 <span class="math inline">\(W_{total}\)</span>，即可得到第 q 个请求时的当前权重序列，取最大权重对应的机器即为调度结果，后续调度结果可通过递推式得出。</p><hr /><p> 初次看到这个方案的时候，就想动手实现一下，因为思路也比较清晰简单，实现完之后，简单测试一下，也确实没啥问题，后面再深度测试了一下，就发现该方案确实有点小小的问题，在大部分情况下，该方案确实能得到很正确的结果，但还是存在一些错误结果，就因为有少量错误结果，所以<strong>该方案不要在生产环境下应用</strong>。该方案错在了将 <span class="math inline">\(q*ew_i\)</span> 看成最后一个整体进行处理排序，忽略了分步执行结果，导致小部分场景下的错误排序结果，进而生成错误调度权重，调度错误。</p><p> 现在再回到初始问题「如何生成 SWRR 算法中指定轮次的调度结果？」，抽象来看，该问题是个数学问题「如何从数列的递推式计算数列求通项公式」， 但 SWRR 的递推式相对复杂，中间还有取最大值这个不稳定变量，实际很难得到通项公式，直接计算指定调度解果，Shaun 问了 ChatGPT，也自己想了很久，搜了很久，但都没有答案，内网中的这个方案算是最接近的一个答案。</p><h2 id="后记">后记</h2><p> 在内网中看到这个方案的思路很有意思，将整数和小数部分拆开，再单独对小数部分排序，所以就自己测试了一下，顺便学习了下负载均衡 SWRR 算法，虽然问题依旧还在，但总归是有点收获。</p><h2 id="附录">附录</h2><p> 附代码：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PYTHON"><div class="code-copy"></div><figure class="highlight hljs python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> random</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">ouput_schedule</span>(<span class="params">rs_arr, schedule_num</span>):</span></span><br><span class="line"> all_rs_weight_str = <span class="string">";\t"</span>.join([<span class="string">"rs:%s,cw:%s"</span> % (rs[<span class="string">"rs_name"</span>], rs[<span class="string">"cw"</span>]) <span class="keyword">for</span> rs <span class="keyword">in</span> rs_arr])</span><br><span class="line"> schedule_rs = <span class="built_in">max</span>(rs_arr, key=<span class="keyword">lambda</span> x:x[<span class="string">"cw"</span>])</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"%s:\t%s\t===>\trs:%s,cw:%s"</span> % (schedule_num, all_rs_weight_str, schedule_rs[<span class="string">"rs_name"</span>], schedule_rs[<span class="string">"cw"</span>]))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> schedule_rs</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">swrr</span>(<span class="params">rs_arr, weight_total</span>):</span></span><br><span class="line"> schedule_rs = rs_arr[<span class="number">0</span>]</span><br><span class="line"> max_weight = schedule_rs[<span class="string">"cw"</span>]</span><br><span class="line"> <span class="keyword">for</span> rs <span class="keyword">in</span> rs_arr:</span><br><span class="line"> <span class="keyword">if</span> rs[<span class="string">"cw"</span>] > max_weight:</span><br><span class="line"> schedule_rs = rs</span><br><span class="line"> max_weight = rs[<span class="string">"cw"</span>]</span><br><span class="line"></span><br><span class="line"> rs[<span class="string">"cw"</span>] += rs[<span class="string">"ew"</span>]</span><br><span class="line"> </span><br><span class="line"> schedule_rs[<span class="string">"cw"</span>] -= weight_total</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> schedule_rs</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">swrr_test</span>():</span></span><br><span class="line"> real_servers = [{<span class="string">"rs_name"</span>: <span class="built_in">chr</span>(i+<span class="number">64</span>), <span class="string">"ew"</span>: i, <span class="string">"cw"</span>: i} <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">6</span>)]</span><br><span class="line"> weight_total = <span class="built_in">sum</span>([rs[<span class="string">"ew"</span>] <span class="keyword">for</span> rs <span class="keyword">in</span> real_servers])</span><br><span class="line"> schedule_count = weight_total</span><br><span class="line"> swrr_seq = []</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, schedule_count+<span class="number">1</span>):</span><br><span class="line"> ouput_schedule(real_servers, i)</span><br><span class="line"> schedule_rs = swrr(real_servers, weight_total)</span><br><span class="line"></span><br><span class="line"> swrr_seq.append(schedule_rs[<span class="string">"rs_name"</span>])</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">print</span>(swrr_seq)</span><br><span class="line"></span><br><span class="line"><span class="comment"># swrr_test()</span></span><br><span class="line"><span class="comment"># print("---------")</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">swrr_n</span>(<span class="params">rs_arr, weight_total, schedule_num</span>):</span></span><br><span class="line"> ms = [(rs[<span class="string">"ew"</span>] / <span class="built_in">float</span>(weight_total)) * (schedule_num-<span class="number">1</span>) <span class="keyword">for</span> rs <span class="keyword">in</span> rs_arr]</span><br><span class="line"> mzs = [<span class="built_in">int</span>(m) <span class="keyword">for</span> m <span class="keyword">in</span> ms]</span><br><span class="line"> mxs = [(i, m-<span class="built_in">int</span>(m)) <span class="keyword">for</span> i, m <span class="keyword">in</span> <span class="built_in">enumerate</span>(ms)]</span><br><span class="line"> mxs = <span class="built_in">sorted</span>(mxs, key=<span class="keyword">lambda</span> x:x[<span class="number">1</span>], reverse=<span class="literal">True</span>)</span><br><span class="line"> <span class="keyword">for</span> i, rs <span class="keyword">in</span> <span class="built_in">enumerate</span>(rs_arr):</span><br><span class="line"> rs[<span class="string">"cw"</span>] = schedule_num * rs[<span class="string">"ew"</span>]</span><br><span class="line"> rs[<span class="string">"cw"</span>] -= mzs[i] * weight_total</span><br><span class="line"></span><br><span class="line"> d = (schedule_num-<span class="number">1</span>) - <span class="built_in">sum</span>(mzs)</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(d):</span><br><span class="line"> rs_arr[mxs[i][<span class="number">0</span>]][<span class="string">"cw"</span>] -= weight_total</span><br><span class="line"></span><br><span class="line"> schedule_rs = ouput_schedule(rs_arr, schedule_num)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> schedule_rs</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">swrr_n_test</span>():</span></span><br><span class="line"> real_servers = [{<span class="string">"rs_name"</span>: <span class="built_in">chr</span>(i+<span class="number">64</span>), <span class="string">"ew"</span>: i, <span class="string">"cw"</span>: i} <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">6</span>)]</span><br><span class="line"> weight_total = <span class="built_in">sum</span>([rs[<span class="string">"ew"</span>] <span class="keyword">for</span> rs <span class="keyword">in</span> real_servers])</span><br><span class="line"></span><br><span class="line"> schedule_rs_seq = []</span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, weight_total+<span class="number">1</span>):</span><br><span class="line"> schedule_rs = swrr_n(real_servers, weight_total, i)</span><br><span class="line"></span><br><span class="line"> schedule_rs_seq.append(schedule_rs[<span class="string">"rs_name"</span>])</span><br><span class="line"> <span class="comment"># swrr_n(real_servers, weight_total, 9) # err schedule rs</span></span><br><span class="line"> <span class="built_in">print</span>(schedule_rs_seq)</span><br><span class="line"></span><br><span class="line"><span class="comment"># swrr_n_test()</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">vnswrr_preschedule</span>(<span class="params">rs_arr, weight_total, N, schedule_rs_seq</span>):</span></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, N+<span class="number">1</span>):</span><br><span class="line"> schedule_rs = swrr(rs_arr, weight_total)</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(schedule_rs_seq) >= weight_total:</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> schedule_rs_seq.append(schedule_rs)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">vnswrr</span>(<span class="params">rs_arr, rs_count, weight_total, prev_schedule_idx, schedule_rs_seq</span>):</span></span><br><span class="line"> N = <span class="built_in">min</span>(rs_count, weight_total)</span><br><span class="line"> </span><br><span class="line"> schedule_idx = prev_schedule_idx + <span class="number">1</span></span><br><span class="line"> schedule_idx %= weight_total</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> schedule_idx >= <span class="built_in">len</span>(schedule_rs_seq)-<span class="number">1</span>:</span><br><span class="line"> vnswrr_preschedule(rs_arr, weight_total, N, schedule_rs_seq)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">return</span> schedule_idx</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">vnswrr_test</span>():</span></span><br><span class="line"> all_schedule_rs_seq = []</span><br><span class="line"> real_servers = [{<span class="string">"rs_name"</span>: <span class="built_in">chr</span>(i+<span class="number">64</span>), <span class="string">"ew"</span>: i, <span class="string">"cw"</span>: i} <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">6</span>)]</span><br><span class="line"> rs_count = <span class="built_in">len</span>(real_servers)</span><br><span class="line"> weight_total = <span class="built_in">sum</span>([rs[<span class="string">"ew"</span>] <span class="keyword">for</span> rs <span class="keyword">in</span> real_servers])</span><br><span class="line"></span><br><span class="line"> N = <span class="built_in">min</span>(rs_count, weight_total)</span><br><span class="line"> schedule_rs_seq = []</span><br><span class="line"> <span class="comment"># 预生成调度序列</span></span><br><span class="line"> vnswrr_preschedule(real_servers, weight_total, N, schedule_rs_seq)</span><br><span class="line"> <span class="comment"># 随机取调度结果</span></span><br><span class="line"> prev_schedule_idx = random.randint(<span class="number">0</span>, N-<span class="number">1</span>)-<span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">2</span>*weight_total+<span class="number">1</span>):</span><br><span class="line"> schedule_idx = vnswrr(real_servers, rs_count, weight_total, prev_schedule_idx, schedule_rs_seq)</span><br><span class="line"> all_schedule_rs_seq.append(schedule_rs_seq[schedule_idx][<span class="string">"rs_name"</span>])</span><br><span class="line"> prev_schedule_idx = schedule_idx</span><br><span class="line"> </span><br><span class="line"> <span class="built_in">print</span>([rs[<span class="string">"rs_name"</span>] <span class="keyword">for</span> rs <span class="keyword">in</span> schedule_rs_seq])</span><br><span class="line"> <span class="built_in">print</span>(all_schedule_rs_seq)</span><br><span class="line"></span><br><span class="line">vnswrr_test()</span><br></pre></td></tr></table></figure></div><h2 id="参考资料">参考资料</h2><p>1、<a href="https://developer.aliyun.com/article/708538">QPS 提升60%，揭秘阿里巴巴轻量级开源 Web 服务器 Tengine 负载均衡算法</a></p><p>2、<a href="https://claude-ray.com/2019/08/10/nginx-swrr/">Nginx SWRR 算法解读</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 最近偶然在公司内网看到一篇文章「负载均衡算法vnswrr改进——从指定位置生成调度序列」。正好 Shaun 一直觉得调度类算法很有意思，就认真看了下，顺便写下自己的一些理解。</p></summary>
<category term="Mathematics" scheme="http://cniter.github.io/categories/Mathematics/"/>
<category term="algorithm" scheme="http://cniter.github.io/tags/algorithm/"/>
</entry>
<entry>
<title>记一次资源不释放的问题</title>
<link href="http://cniter.github.io/posts/2eb69b33.html"/>
<id>http://cniter.github.io/posts/2eb69b33.html</id>
<published>2023-05-01T14:16:58.000Z</published>
<updated>2024-06-20T17:07:25.994Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 最近发现一个 GoFrame 服务即使空载 CPU 使用率也很高，每次接受请求后资源没有被释放，一直累积，直到达到报警阈值，人工介入重启服务，于是压测排查了一下。</p><span id="more"></span><h2 id="问题篇">问题篇</h2><p> 先新增代码启动 go 自带的 pprof 服务器：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="GO"><div class="code-copy"></div><figure class="highlight hljs go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">"net/http"</span></span><br><span class="line">_ <span class="string">"net/http/pprof"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">Pprof</span><span class="params">(pprof_port <span class="keyword">string</span>)</span></span> {</span><br><span class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">(pprof_port <span class="keyword">string</span>)</span></span> {</span><br><span class="line">http.ListenAndServe(<span class="string">"0.0.0.0:"</span>+pprof_port, <span class="literal">nil</span>)</span><br><span class="line">}(pprof_port)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p>压测以及 profile 命令：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 压测命令</span></span><br><span class="line">wrk -t8 -c1000 -d60s --latency --timeout 10s -s post_script.lua http://host:[srv_port]/post</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> profile 整体分析</span></span><br><span class="line">go tool pprof -http=:8081 http://host:[pprof_port]/debug/pprof/profile?seconds=30</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看函数堆栈调用</span></span><br><span class="line">curl http://host:[pprof_port]/debug/pprof/trace?seconds=30 > ./pprof/trace01</span><br><span class="line">go tool trace -http=:8081 ./pprof/trace01</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看内存堆栈</span></span><br><span class="line">go tool pprof -http=:8081 http://host:[pprof_port]/debug/pprof/heap?seconds=30</span><br></pre></td></tr></table></figure></div><p> 在压测 30 次后，即使服务空载 CPU 也被打满了，查看服务此时的 profile，发现 goroutine 的数目到了百万级别，查看 cpu 堆栈发现集中调用在 gtimer 上，但遍寻服务代码，没有直接用到 GoFrame 的定时器，问题出在哪也还是没想太明白。吃完饭后偶然灵光一现，既然 CPU 看不出啥，那再看看内存，查看内存发现，内存对象最多的是 glog.Logger，看代码也正好有对应的对象，可算是找到问题真正的元凶了。</p><p> log 对象一般都是全生命周期的，不主动销毁就会一直伴随着服务运行，所以 log 对象一般都是程序启动时初始化一次，后续调用，都是用这一个对象实例。而这次这个问题就是因为在代码中用 glog 记录了数据库执行日志，每次请求都会重新生成一个 glog 对象，又没有主动释放造成的。</p><p> 知道问题的真正所在，解决问题就相对很简单了，只在程序启动时初始化一个 glog 对象，后续打印日志就用这一个实例，其实更好的方式是生产环境不打印数据库日志，毕竟影响性能。</p><h2 id="后记">后记</h2><p> CPU 资源的占用往往伴随着内存资源的占用，当从调用堆栈以及线程资源上看不出问题的时候，可以转过头来看看内存堆栈，毕竟内存堆栈更能指示有问题的对象出在哪，知道内存对象是谁，也相当于提供了排查问题代码的方向。</p><h2 id="附录">附录</h2><p> 在排查过程中发现 goroutine 数目异常的高，于是想限制一下 goroutine 数目，在网上搜索的时候发现当用容器部署 go 服务时，go 默认最大的 goroutine 数目为宿主机 cpu 核数，而不是容器的 cpu 核数，从而并发时 goroutine 数目可能比容器 cpu 核数高很多，造成资源争抢，导致并发性能下降，可以通过设置环境变量 <code>GOMAXPROCS</code> 指定 goroutine 最大数目，也可以使用 <code>go.uber.org/automaxprocs</code> 库自动修正最大核数为容器 cpu 核数。</p><p>自适应设置 GOMAXPROCS 上下限代码：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="GO"><div class="code-copy"></div><figure class="highlight hljs go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line">_ <span class="string">"go.uber.org/automaxprocs"</span></span><br><span class="line"></span><br><span class="line"><span class="string">"runtime"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line">procsNum := runtime.GOMAXPROCS(<span class="number">-1</span>)</span><br><span class="line"><span class="keyword">if</span> procsNum < <span class="number">4</span> {</span><br><span class="line">procsNum = <span class="number">4</span></span><br><span class="line">} <span class="keyword">else</span> <span class="keyword">if</span> procsNum > <span class="number">16</span> {</span><br><span class="line">procsNum = <span class="number">16</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">runtime.GOMAXPROCS(procsNum)</span><br><span class="line"></span><br><span class="line"><span class="comment">// todo something...</span></span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><h3 id="python-内存泄漏排查">python 内存泄漏排查</h3><p><strong><em>※注：python 的默认参数是全局变量，若默认参数为一个引用类型（eg：字典对象），且函数中会对该参数进行写操作，就极有可能发生内存泄漏，所以 python 默认参数最好是值类型。</em></strong></p><p>方法一是线上程序直接排查，通过 pyrasite 和 guppy 直接对应 python 程序：</p><blockquote><p>step1：绑定 python 程序 pid，开启 pyrasite shell 窗口，执行 <code>pyrasite-shell <pid></code>；</p><p>step2：使用 guppy 查看 python 程序内存情况，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PYTHON"><div class="code-copy"></div><figure class="highlight hljs python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">>>> </span><span class="keyword">from</span> guppy <span class="keyword">import</span> hpy</span><br><span class="line"><span class="meta">>>> </span>h = hpy()</span><br><span class="line"><span class="meta">>>> </span>h.heap()</span><br></pre></td></tr></table></figure></div><p>step3：间隔一定时间后，再次使用 <code>h.heap()</code>，对比两次内存变化</p></blockquote><p>该方法一般只能粗略查看内存泄露的数据对象，可能无法精确定位到指定位置，这时需要用方法二，手动插入代码查看程序运行日志：</p><blockquote><p>Python标准库的gc、sys模块提供了检测的能力</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PYTHON"><div class="code-copy"></div><figure class="highlight hljs python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gc</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line">gc.get_objects() <span class="comment"># 返回一个收集器所跟踪的所有对象的列表</span></span><br><span class="line">gc.get_referrers(*objs) <span class="comment"># 返回直接引用任意一个 ojbs 的对象列表</span></span><br><span class="line">sys.getsizeof() <span class="comment"># 返回对象的大小（以字节为单位）。只计算直接分配给对象的内存消耗，不计算它所引用的对象的内存消耗。</span></span><br></pre></td></tr></table></figure></div><p>基于这些函数，先把进程中所有的对象引用拿到，得到对象大小，然后从大到小排序，打印出来，代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PYTHON"><div class="code-copy"></div><figure class="highlight hljs python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> gc</span><br><span class="line"><span class="keyword">import</span> sys</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">show_memory</span>():</span></span><br><span class="line"> <span class="built_in">print</span>(<span class="string">"*"</span> * <span class="number">60</span>)</span><br><span class="line"> objects_list = []</span><br><span class="line"> <span class="keyword">for</span> obj <span class="keyword">in</span> gc.get_objects():</span><br><span class="line"> size = sys.getsizeof(obj)</span><br><span class="line"> objects_list.append((obj, size))</span><br><span class="line"> <span class="keyword">for</span> obj, size <span class="keyword">in</span> <span class="built_in">sorted</span>(objects_list, key=<span class="keyword">lambda</span> x: x[<span class="number">1</span>], reverse=<span class="literal">True</span>)[:<span class="number">10</span>]:</span><br><span class="line"> <span class="built_in">print</span>(<span class="string">f"OBJ: <span class="subst">{<span class="built_in">id</span>(obj)}</span>, TYPE: <span class="subst">{<span class="built_in">type</span>(obj)}</span> SIZE: <span class="subst">{size/<span class="number">1024</span>/<span class="number">1024</span>:<span class="number">.2</span>f}</span>MB <span class="subst">{<span class="built_in">str</span>(obj)[:<span class="number">100</span>]}</span>"</span>)</span><br></pre></td></tr></table></figure></div><p>找到内存占用稳定增长的对象，调用 <code>gc.get_referrers(*objs)</code>，查看该对象的引用信息，即可快速定位泄漏位置</p></blockquote><p>该方法更加灵活精确，不好的地方是有侵入性，需要修改代码后重新上线，同时获取这些信息并打印，对性能有一定的影响，排查完之后，需要将该段代码下线。</p><h4 id="参考资料">参考资料</h4><p>1、<a href="https://blog.csdn.net/lonevenn/article/details/120075646">python内存泄露问题定位：附带解决pyrasite timed out</a></p><p>2、<a href="https://blog.qminghe.com/post/2022/01/17/python-memory-leak-oom-resolve">技术 · 一次Python程序内存泄露故障的排查过程</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 最近发现一个 GoFrame 服务即使空载 CPU 使用率也很高，每次接受请求后资源没有被释放，一直累积，直到达到报警阈值，人工介入重启服务，于是压测排查了一下。</p></summary>
<category term="Problems" scheme="http://cniter.github.io/categories/Problems/"/>
<category term="golang" scheme="http://cniter.github.io/tags/golang/"/>
</entry>
<entry>
<title>社畜三年，风雨兼程</title>
<link href="http://cniter.github.io/posts/dbbe01e5.html"/>
<id>http://cniter.github.io/posts/dbbe01e5.html</id>
<published>2023-04-02T02:06:28.000Z</published>
<updated>2023-04-02T13:55:48.939Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 财富和幸福算是绝大部分人的毕生追求，所以在读这本书时，更容易让人有一种思想上的共鸣，但一个人的成功总是独一无二的，需要依靠天时地利人和，正所谓，学我者生，似我者死，可以学习借鉴成功者的一些思想，但不要想着沿着成功者的路继续走下去就能成功。 ——读『纳瓦尔宝典』</p><span id="more"></span><h2 id="感想篇">感想篇</h2><p> 财富并不代表金钱或者地位，而是某种可以自然增长的东西，是一种追求共赢的东西，是一种可以长期存在的东西。最好的投资总是学习，成本低且有效，但接受新知识总是需要消耗更多的能量，而人总是想尽量减少能量的消耗，所以很多人都沉迷于快餐文化中，不需要过多的思考，又消磨了时光，一举两得 (￣ε(#￣)☆╰╮(￣▽￣///)。</p><p> 人生总有选择，也总要承担选择所带来的后果。选择永远比努力重要，光凭运气去选择，或许能瞎猫碰上死耗子，但狗屎运总归会用完，下一次就不好说了。如何选择是需要去学习的，有从自身过去的经验去学习，有从别人身上去学习，更多的是依靠平时积累的各种信息，这就是努力了，努力或许有结果，也或许没有，但不努力，一定没结果。努力的目的是为了有更多的选择，在有更好的选择时能及时判断，不要单纯的为了结果而努力，很有可能得不偿失。付出和回报看对人对事吧，对自己的付出当然总是会有回报的，对事情付出同样会有，但对其他人付出，就别太指望一定要有啥回报，可以设个底线，在没有任何回报的情况下，能付出多少。</p><p> 选择也从来不是一竿子买卖，局部最优也不代表全局最优，人生的选择更是如此，十年前的当时最优放在现在看可能选错了，放在十年后再看可能又选对了，时代在发展，当然倘若十年前真选对了，可能就不需要十年后再回过头来看了 :P。绝对最优这个词在人生道路上就不存在，只能说是相对更优，这个更优不是选项之间的对比，而是选项与自身的对比，哪个选项最能提升自己就选哪个就行，长远来看，投资自己不一定能飞黄腾达，但总能有口饭吃。投机是选择，稳扎稳打也是选择，本身无优劣，回报和风险总是共存的，能承受的起就行，别光见贼吃肉，不见贼挨打。</p><p> 获取金钱的方式有很多，最快的方式都写在刑法里 ๑乛◡乛๑，而创造财富的方式也有很多，最直接的是去资本市场做投资，也可以想方设法积累自己的名气（不管是好是坏），然后走流量变现。创造财富都有其风险，就算是大航海时代，不也一大批人沉在海底，打工人虽然是在给别人创造财富，但对自身而言，也算是一种资本的原始积累，而且是相对最稳定的一种，对于技术人来说，技术经验也算是一种时间杠杆，所以边打工，边在资本市场中学习，也不失为一种创造财富的方式。</p><p> 至于幸福，「哈佛大学公开课:幸福课」中指出幸福是不可持续的，永远幸福是不可能的。幸福的人都是相似的，不幸的人各有各的不幸，穷人的不幸比富人更不幸，借用一句伪科学的话：所有的不幸的都是对现状的一种不满足感，满足是一种幸福，适应也是一种幸福，总而言之就是看开，释放。「银河系漫游指南」中宇宙终极问题的答案是 42，或许本身就是无意义的事，为啥一定要有意义，无聊或许才是人生的常态主题。当欲望得到满足，或许能得到短暂的幸福，但随之而来的空虚感也得忍受，更关键的在于去选择下一件事，绝大部分人总还是要为了生计奔波，吃饱了撑得就容易胡思乱想，忙点啥，哪怕随便干啥，都能体会到闲暇时的幸福 :P。</p><p> 曾经在 TED 上看到不要把梦想告诉别人，告诉了就很难实现，因为能得到的只有嘲笑，以 Shaun 个人经验来说是没错的，梦想还是放在心里比较好，或许等哪天实现了再放出来会更加畅快，有且仅有自己做的梦才是真正的梦想，这才是人最珍爱的东西，就像「来自深渊:烈日的黄金乡」中真正的挚爱是不会让任何人知道的，哪怕是自己最后的传承者。「把时间当作朋友」中有这样一句话，“来自外部的恐惧在于过分在意外界的评价”，Shaun 本身不是个太在意别人看法的人，在意别人的评价，无疑会让自己活得更累，尤其是大部分人的评价没有丁点儿建设性，只是一种优越感作祟，当然，自己真有问题而不自知也是不行的。很喜欢暗杠「童话镇plus」里唱的“很多人一辈子忙忙碌碌不会懂得：有个被嘲笑的梦想万一有天实现了呢？”，人活着，总得有个念想。奔波一世，虽说是为了这一日三餐，但能留下点脚印，哪怕是些许痕迹，好像也还不错。</p><p> 老子云：知人者智，自知者明；邓宁-克鲁格效应也指出认知有四大境界：不知道自己不知道，知道自己不知道，知道自己知道，不知道自己知道；人贵有自知，可惜不知为知者有之，过而不改者有之，好为人师者有之。人之患在好为人师，道不可轻传，薪尽火传在很多时候也只是妄想，自以为是，人以为非，自以为厉害不算厉害，别人认为厉害才是真厉害。自知，知道自己几斤几两，自尊自爱，力所能及，道阻且长，持续学习，徐徐图之。</p><p> 常言道：种一棵树最好的时机是十年前，其次是现在。做事情从来不嫌晚，就怕不行动，人人都会做梦，有些梦不切实际，有些梦有迹可循，空想无用，再周密的计划也比不过实际执行，计划赶不上变化，诚然，试错有成本，但这成本也同样是经验，按部就班的迭代执行，总会走向通往梦想之路，不过大部分人可能在路上就迷失了，甚至有些人走不上路，能真正到达终点的人，终究只是少数。</p><h2 id="工作篇">工作篇</h2><p> 一位长者曾经说过：一个人的命运啊，当然要靠自我奋斗，但是也要考虑到历史的进程。工作不是事业，对于大部分的人目标都只是简单的钱而已。三年期的目标算是达成了，五年期的目标的应该是完不成了，不过总归要尽力，大环境如此，个人的努力显得格外苍白。在时代的洪流之下，众人皆是蝼蚁，按部就班，做好自己该做的就行，至于回报，还是得看外面的环境调整预期。</p><p> Shaun 这三年来，从小公司跳到大公司，体会最深的就是，就单纯的论做事而言，大公司和小公司没啥区别，能接触到的人也就是自己组内和上下游的几十个人而已，唯一的区别可能也就是履历背景了，但这个也只能在跳槽时才能体现其作用，当然或许有人说人的能力强弱不同，但就 Shaun 的感觉来看，不是不同，而是术业有专攻，不同的公司对人的侧重点要求不同，大公司小公司都有工作能力很强的，不过都跑路了 ¯\_(ツ)_/¯，不过有一点是真的有区别的，就是数据量，这是实打实的区别，不过这更多的是对经验的要求，对人做事的本质能力并没有更高的要求，还是用原来的思维方式尝试新的方案和实验，真正难的问题还是交给学术界去解决吧。</p><p> 无论大小公司，拥抱变化或许才是永恒的话题。这种变化，不只是裁员（<em>只要是补偿没问题的裁员，若被裁，那便被裁，找个吃饭的地方，总还是不难的</em>），更多的是体现在做事方面。打工人常听人说，事情是做不完的，没必要这么加班加点的干，没必要加班干是真的，但事情是会做完的，只是做完了又会有新的事过来，事情做完又两层含义：一层是事情可以告一段落了，做到这到此为止了，不要再投入大量人力做这个事，只是日常维护；另一层是这个事情不做了，现有人力要么换岗要么走人。对于公司上层来说，更关心的事是做完这个事之后下一个事做什么，如果没有下一个事，那留下这么多人就没必要了，体现到下层大头兵上，就是要么被裁，要么换个事情做，至于换个的事情可能是类似的事，也可能是全新的事，这就对人的学习能力或知识迁移能力有一定的要求。从上层的角度来说基本上很难让一个大头兵一直持续深入的干某件事，浪费人力，投入和产出完全不成比例，所以持续学习，拥抱变化才是真理。就 Shaun 个人的经验而言，基本上每半年换个新的活儿，有的活能照之前的经验完全迁移，有的活完全没接触过，得重新开始学，不过好在大多数的事都还能胜任，差强人意。</p><p> 换活，有的是开辟新业务，有的是填坑，开辟新业务一般都算是好活，从 0 到 1，事情不一定轻松，但收益可是实打实的，而且越前期的收益越容易拿，除非新业务黄了，不然光啃老都能啃几年，这是真好活。至于填坑，就看坑是什么样的了，若是核心的坑，那当然是极好的，怕就怕是陈年老坑，还需要不断维护更新，经受无数人的东西还有去维护，也是个烂坑；另一种坑是，前人写了系统原型，需要沿着这个原型继续开发，可惜原型一般只是实现核心功能，更可怕的是还是个初版，代码也是一团糟的那种，这样的坑接收恐怕是只能重写了，但收益一般也就不好说了。</p><p> 收益，是企业最核心的两个字，没有收益，一切都是免谈，收益有显而易见的，比如节省成本，开拓市场，也有不明显的，比如为以后节省人力，为将来快速解决问题。显式的收益很容易感知，也很好评估，隐式的收益就基本无法感知，将来的事谁能说的清，况且将来还有没有这事都很难说，员工的人力成本也容易忽视，这些收益就很难评估，也就基本不可能定为上下一心的 kpi，当下面的 kpi 不是上面的 kpi，那下面做的一切工作都是白干，虽然对自身可能有益，但也只能面试时体现了。每离职一个核心人员，至少需要调派 3 个人来填补空缺，为啥企业宁愿后面补充，而不是提前安排一个人来分担工作，主要原因在于企业在赌核心人员不会轻易离职，也不会安排太多人手干维护性质的工作，但当熟手离职，再要去维护的代价就很大了，后来者一边填坑，一边在心里骂街，就这样一轮一轮的后来者，一轮一轮的骂街，屎山也就这样炼成了，直到推倒重来，形成上下一心的 kpi，又开始新一轮的屎山炼成。</p><p> 提问从来不是件丢人的事，只要自己尝试过解决且保持应有的礼貌，毕竟人非生而知之者，不知道才是常态。很多人喜欢闷头干活，遇到问题只会冥思苦想，独立思考固然是一种品质，但思考也需要讲究基本法，思考也分交流前中后，当完全没有交流，就开始想解决方案，这是呆子，没有任何背景，能想出好的方案，那可能是真正的天才，那也没必要和打工人混在一起了。软件系统的问题更多的可能是人的问题，遇到问题，最好的是先了解背景原因后果，了解一下来龙去脉，再和有经验的人交流讨论一下，可以怎么解决，最后再决定具体做法，怎么执行，真梳理完了之后，执行一般是一件比较简单的事了。遇事多沟通，想开点，对人愚钝，对事精明。</p><p> 如无必要，勿增实体，漂亮话人人都会说，可真在具体实施中，一般都是怎么快怎么来，优先增加一个实体再说，而不是去想有没有必要；当然也有的会过度放大这个事，明明增加这个实体能给各方都便利很多，从成本考虑就是不增，孰不知在另一方面反而是增加了成本。软件开发行业中，有很多定律法则，但人是活的，定律是死的，每个人有每个人的理解，在不同的业务场景下，定律的体现都有不同，一般情况下，围绕定律走不会出太大的问题，但有时也需要灵活变通，就像设计模式一样，不是完全一层不变的，有那层意思在就行，Shaun 认为唯一恒定的定律就是「简单」，不管是设计还是开发，越简约，越直观，就越稳定，越正确。</p><p> 年龄歧视在职场中是确实存在的，国内职场普遍认为，当到一定年龄的时候，就应该到一定的职级，到一定的职级就不会再在一线干活了，这就导致了在招干活的人，就不会招年龄大的，这真的就很畸形，职级高就不会也不需要在一线干活，就目前国内职场的环境，技术从来不是主流的上升手段，所谓的技术路线是不存在的，不需要真正的架构师，文档架构师更受欢迎，说的好才能活的好。做完和做好从来都是两回事，但相比较而言，做完更重要些，毕竟做的好与不好，很难量化评估，所以更多的要求是能不能做出来，不关心好与坏，有东西出来这更重要，而一般都能做出来，所以不重视技术人员，而更重视管理人员，这就是互联网 35 岁失业的本质，就算 35 岁以上的技术人员能做的更好，但 ROI 不成正比。管理者掌握是组织技能，执行者掌握的是生存技能，当经济环境下行时，生存技能有优势，当经济环境上行时，组织技能更有优势。</p><p> 跟对人，做对事，简简单单六个字，做起来可太难了，首先怎么定义对，在合适的时间做合适的事，这个合适怎么把握。曾以为只要努力做事，就能得到应有的回报，可在人类世界，老实做事有回报的都上新闻了。做好事不如做对事，但不知道事情做的对不对的时候，也只能选择做好，至少这是对个人能力的要求，少留点骂名。当然，善战者无赫赫之功，做的好了，也没法突出个人的重要性，甚至给上面的感觉是有你没你没区别，只要还在这个位置上，能力就体现不了，相反，经常出问题的人，得一堆人围着转，这看起来就很重要 ๑乛◡乛๑，如果上面再分配不公，能力强的人自然就加速走了，某种程度上的死海效应，劣币驱逐良币。</p><p> 工作也是讲方法论的，做事的出发点和方向错了，即使能解决一部分问题，但解不干净，最后还得推到重来，所以项目启动前的分析非常重要，一定得先抓头部问题，当然穿插一些能快速解决的问题也行，其次就是明确哪些问题一定要解，哪些问题可以不解，最后就是一定要留下文档纪要，这是工作量的体现，也是未来追溯问题的依据。汇报也一般可分为五步：背景，需求和目标，解决方案和排期，进展和依赖项，问题和风险。</p><p> 工作自由，有两种境界，一种是选择做什么的自由，另一种是选择不做什么的自由。大部分或许能选择做什么，但却无法拒绝一些事，有些事情只能被动接受，但有些却是可选的，不要一直被牵着走，事有轻重缓急，要有自己的认知，学会拒绝，哪怕是上级的需求，现如今找个养家糊口的工作还是不难的，此处不留爷自有留爷处。了解并学习职场中一些常用的话术，分辨并挑出真正对自身有益的，别把别人太当回事，别把自己不当回事，事实是事实，话术是话术，主人翁意识是每个领导都希望下属有的，但解释权归领导所有，越上层就越不可能落到实处。尽心做事，尽力做好每一件接手的活，无论喜欢与否，真不想做，就不要接，既然做了，就尽职尽责，算是在工作中积德了，败人品的事还是尽量不要做。放宽心态，大方待人，不要把工作当生活，可以享受工作，但更应该享受生活。勇于分享，分享讨论可能不见得是件好事，但绝不是坏事，分享同样也是一种总结。灵感是易逝的，当有灵感时，就尽快行动起来，优秀的产品需要时间来打磨，对结果需要有更多的耐心。</p><p> 在 22 年一片开源节流的浪潮中，人人自危。「浪潮之巅」中，科技的发展史就是一批企业的兴亡史，或因为自身的原因，尾大不掉，或因为更上层的力量，没有哪家企业能一直辉煌下去，打工人能做的，只有选条赛道，厚积薄发，尽量少换或不换行业，剩下的也就只有听天由命了，毕竟将来的事谁也说不好，潮起潮落，又有谁能一直屹立浪潮之巅 ╮(╯▽╰)╭。</p><h2 id="生活篇">生活篇</h2><p> 这三年，最大的主题就是“新冠”，新冠时代，每个人都是历史的见证者，这次疫情，足以在人类历史上留下浓墨重彩的一笔。这期间，也能真正在现实里体会一把魔幻现实主义，有大规模封城的，有叫嚣着他的软肋是儿子的，也有恶意返乡有家不能回的。天赋🧑权，疫赋🐶权，肉食者的政策总是不食人间疾苦。又或许是上面的政策出发点是好的，但奈何下面的执行者大部分是一帮饭桶蛀虫，能站在布衣角度去落实政策的又有多少。解封之后，对 Shaun 而言，最大快人心的就是不用再看看门大爷的嘴脸。黑色的眼睛可能是黑夜给的，也可能是白天给的，有人用它寻找光明，也有人用它寻找黑暗，西游记里描述的狮驼国从历史角度看，也不是什么神话传说，虽说如今的时代论惨烈规模没那么大，但苦难并没有完全消失，太阳底下也没有新鲜事。</p><p> 「动物庄园」里有句话，“所有动物生来平等，但有些动物比其他动物更平等”，疫情三年，对这句话体会更加深刻，有的人生在罗马，也有的人生来就是骡马，人人平等的乌托邦世界或许只是个伪命题。连科学也只是为政治服务的工具而已，社会的科技发展也从不依赖于上层阶级的想法，往往只是下层某个灵光一现的思路，人类社会有个很奇怪的现象就是，本来一个人在下层成果迭出，当跃迁到上层时，思维就好像僵化了，深层原因可能有很多，但至少表面现象是这样。普罗大宗也很容易受到各种言论的影响，尤其是某些专业人士的公开发言，所以完全的言论自由也意味着完全混乱，需要管控，但也有知情权，不然就像一氧化二氢实验一样，隐瞒一面，重点宣传另一面，就很容易受到别有用心的一些误导。</p><p> 未经他人苦，莫劝他人善，这三年，在网上见过太多的牛鬼蛇神，一把键盘走天下，自以为站在大义之上，实际上只是些吃着人血馒头既蠢又坏的看客，以自己幸福的生活站在道德制高点上去肆意抨击他人，并因此而洋洋得意。诚然无能是最大的罪恶，哀其不幸，怒其不争，但匹夫之怒，也能血溅五步，自救者天救，自助者天助，自弃者天弃。当感叹世事无常的时候，都可以去看看「活着」，有的人觉得蝼蚁尚且偷生，好死不如赖活着，也有的人觉得宁为玉碎不为瓦全，读书毕竟是个很私人的事，一千个人心中有一千个哈姆雷特。</p><p> 生存还是生活，这是一个永恒的话题。苏轼曾感叹道，寄蜉蝣于天地，渺沧海之一粟，人这一辈子，说长也长，年轻的时候往前看，感叹还要这样过几十年，说短也短，回首往事，转瞬即逝。也曾踌躇满志，誓要走出山村，而今跨长江越黄河，问一句，有必要吗？行路难！行路难！多歧路，今安在？生活的方式有很多种，或许平淡才是真，人活一世，顺心而已。22 年，偶然发现北京有很多的户外徒步组织，Shaun 也参加了好几次活动，感觉确实很有意思。有的人把徒步当成一种极限运动，挑战自我，也有的人把徒步作为一种休闲运动，纯纯放松心情，每个人徒步的目的都各不相同，重点在于量力而行，对大自然有敬畏之心。周末去外面走走，听风观景阅人看故事，虽然身体上并不轻松，但能极大的消除一周心理上的劳累感。寄情于山水，逍遥于世间，打工人花点小钱，就能有心灵上的放松，整挺好。</p><p> 至于对象，就感觉自己一直很佛系，或许得等到真正成为大魔法师的那一天，才会转变心态。爱情，Shaun 是从来不奢望的，爱情是咋样，相信一千个人心中有一千个哈姆雷特，古今中外有无数的作品的描述了作者心中认为的模样，共同点在于都有风花雪月，嘻嘻哈哈，哭哭啼啼，而生活，好像优秀的作品不多，一个可能的原因是爱情短暂，更有戏剧性，作品也能更有张力，而生活一般时间跨度较长，再惊天动地的事在时间的长河里或许也只是一朵小水花，看起来很平淡，贾科长或许是个很善于观察生活的人，22 年的「隐入尘烟」或许也描绘了生活的一部分，平平淡淡才是真，哪有那么多跌宕起伏，哪有那么多真善美。</p><p> 『紫阳』中有句话，「男女之情并不深奥，感情的发生有两种诱因，一是源于阴阳交合本能的驱使，以阴阳交合为目的。还有一种是喜欢对方身上的优良本格，愿意与之长相厮守。这两种诱因都可以引发情感，没有高下清浊之分，两种诱因也往往彼此掺杂，很难明确区分。这两者唯一的不同就是后者更容易被世人传颂赞美，但后人传颂和赞美的其实也并不是情感本身，而是少数人身上的优良品格」。爱情或许值得被赞美，但更多的是恋爱过程中经历的事，爱情或许从来都是想象中的产物。「三体」中有描述，大部分人的爱情对象也只是存在于自己的想象之中。他们所爱的并不是现实中的 ta，而只是想象中的 ta，现实中的 ta 只是他们创造梦中情人的一个模板，他们迟早会发现梦中情人与模板之间的差异，如果适应这种差异他们就会走到一起，无法适应就分开，就这么简单。两个人真正在一起的时候，往往只看到对方坏的一面，分开的时候，回忆时却想着好的一面，人生就是这么反复无常。「疑犯追踪」有这样一句台词：有一天你嫁给了自己的灵魂伴侣，然后眼睁睁看着他们变为另一个人，一旦深爱这人曾经的样子，便无法接受他们改变后的样子。最爱的人或许只存在于想象中，毕竟，随环境变化最大的也是人。</p><p> 有人说，两个人在一起就是为了分担风险，当一个人能够足够承担风险的时候，或许就不需要两个人。「在云端」中有讨论过结婚的意义：可以有依靠的人？但能真白头偕老的人又有多少，计划永远赶不上变化，就算能碰到自己的理想型，但你大概率不会是 ta 的理想型，况且人都是善变的；不会孤独终老？如今的时代养儿防老风险也不小，比父母更有能力的孩子基本不可能待在父母身边；或许正如男主后期的思想转变，结婚的意义或许就是找个能分享倾听理解陪伴的人。Shaun 理想的两人关系应该是一种战友之上的关系，相互信任理解尊重，虽有小分歧，但大目标一致。人与人之间的相处哪要那么多心思，顺其自然，求同存异，自尊自爱，人待以诚，待人以诚。庸人为了忘却烦恼，一般以智者不入爱河来自我安慰。人啊，简单又复杂，简单在合心意，复杂在人心难测，所以不需要强求，顺心不香吗？</p><p> 回想起以前高中的时候，就有同学对 Shaun 说：“你这看样子以后就是要靠相亲找对象的”，现在想来，那同学看人真准。有人说相亲是让一个不懂人的去搞定一个难懂的人，其实哪有什么不懂或难懂，有的只是一群彷徨的可怜人罢了，很少有人能确切的明白自己真正想要的是什么。数学中有个最优停止理论，得出了一个 1/e 的数值。通俗来说就是在苏格拉底的麦穗故事里前 1/3 的路程，什么都不摘，只是用来估计自己的预期，在后 2/3 的路程里，一旦有接近甚至超越的预期的麦穗出现，立即摘下。满足 1/e 概率的前提之下是麦穗大小均匀分布，可惜现实世界很少有均匀分布，所以这个东西对于相亲虽说有一定的借鉴作用，但还是不要太盲从。相亲本来就带着强烈的不信任感，而信任的建立一般又是个长期过程，在不信任甚至有些防备的状态下进行交流，自然很难发现别人的闪光点，更多的是在不经意间暴露出的缺点（相对而言是缺点，毕竟善于发现美的人不多，挑刺的人会更多些），从这点而言，相亲和面试差不多了，更重要的是遇上对的人，幸运值很重要。</p><p> 相亲的交流无非就两种方式，一种是直截了当，开始就问对方想找个啥样的；另一种迂回战术，从工作学习生活爱好方面按流程开始话题，虽然很平淡，但是一般也没太大的问题。交流是相互的，就光一个人问，这不是聊天，而是面试。当然如果感觉聊不到一块儿，确实话不投机半句多，就尽早结束；当犹豫要不要开启话题的时候，就可以长痛不如短痛了，当断不断，反受其乱。出来相亲的人都抱着不同的目的，有的人是被迫的，有的人只是想接触一下外人，有的人是想给别人一个机会，也有的人确实很实诚的想找个相守一生的对象。不同的目的造就了不同的人，有的人就是想出来玩玩，享受那种若即若离的朦胧感，Shaun 一贯的态度是抱着一颗平常心，观其行而知其心，得之我幸，失之我命，求而不得，何必强求，弃得失心，方可自在，若不成，那便不成，也不需要刨根问底，在没有结果的事情面前，真相都显得没有任何意义。</p><p> 很多人对别人的看法在第一次交流的时候，就基本已经确定了，先入为主的思想根深蒂固，孰不知交流本身可以算是一件很随意的事，不同的交流方式，不同的语言文字，在不同的人生背景下，都有不同的含义，除非带着很强的目的性，非常明确的知道这次交流的主题是啥，不过这就和工作中开会没啥区别了。生活中的交流还是随意些比较好，一般都是想到啥就说啥，不用费脑子，巴适。</p><h2 id="理财篇">理财篇</h2><p> 关于理财的言论有很多，在经济上行的时候，网络上流传的话是，你不理财财不理你，经济下行的时候，说的更多的就是，你一理财财离开你。理财，其实是一个很私人的事，分享交流可以，但安利就算了，赚钱的时候只会觉得自己眼光不错，亏钱发泄情绪的时候就正好能找到一个出气口。理财的手段也有很多，最稳妥的当然是不理财，直接把现金都放家里，毕竟在这个银行都会爆雷的时代，钱放银行都不太保险。有人说，乱世买黄金，盛世买古董，灾荒屯余粮，这话说的也有一定的道理，毕竟黄金是整个人类文明的硬通货，钱或许不一定是钱，但黄金总还能代表钱，古董更多的是一种精神文明象征，当衣食无忧的时候，自然也会想有点更高层次的追求，当衣不蔽体食不果腹的时候，自然填饱肚子才是最高优先级。对应到国内市场，当银行爆雷的消息传来，黄金大涨，这三年疫情，食品消费行业涨，其他都跌，市场总是跟随人的需求变化而变化，不管是看衰还是看好，总有人能从中发现商机，毕竟资本总是会从一个地方转移到另一个地方，除非整个市场崩了。</p><p> 理财本质上是一个买与卖的哲学，就算是存银行，为方便取用，就用活期，暂时不用，就用定期，这也是存款收益的一种买卖权衡，至于市场里的交易，就是更直接的买卖了。每个人都想从这种买卖中获益，不管是买方还是卖方，但是这种买卖不是简单的零和，有可能是正和，有时甚至会是负和，主要看买卖的东西的是啥。有些人一次两次的交易都获利了，就想着下次能不能获利更多，但随着等待的时间拉长，收益都变成亏损了，最后只能忍痛割肉。不管是盈利还是亏损，都有其背后的逻辑，不长记性，迟早哪天会栽沟里。在「赌城风云中」有类似这样一句台词：让人去赌场，并一直待在赌场，堵的越多总会输的越多，赌场总是赢家。赌场赢的关键在于如何让赌徒一直待在赌场，就算出去，也依然会回到赌场，股票投资市场虽然和赌场差别很大，但本质上有些地方是相通的，对于普通人，一直待在股市中，却不去学习证券金融市场运转规则，不去了解国家政策导向，迟早有翻车的一天，虽然对于整个市场来说不一定有赢家，但赌徒肯定会是输家。</p><p> 有人说，人只能赚到自己认知范围内的钱，但没人说，会亏损到什么程度。所有的亏损都来自于赚钱的欲望，越想赚钱可能亏损就越大，对于理财，止损线和止盈线同等重要，这两条线每个人心里都应该有自己的尺子，当然也得根据当前的市场环境和自我认知进行动态调整。见好就收，见衰则退，说起来容易，但做起来何其艰难，早早退出的懊恼，越陷越深的后悔，这些事情只有亲身体验过，痛过之后长了记性，才能有自己的认知，光凭运气挣的钱，指不定哪天就会连本带息的还回去。亏损是个无底线的事情，短时间大亏，很多人可能就直接退场不玩了，怕就怕在钝刀子割肉，割几天又喂点好的养几天，亏麻了又还有点希望，碰到这种情况更加需要慎重，需要费心费力收集更多的信息做决策，所以与其在后期劳心劳力，不如前期就先做好各项调查准备工作，而且做事的心态也不一样，每一笔投资都应该有充分的理由，不然不投会更好。</p><p> 在这个人人都想赚快钱的时代，国内市场都是很浮躁的，都堵的是自己不是最后一批入场的。很多人也知道快和慢都是相对的，投机倒把拼运气是快，但总归不能长久，除非完成资本的原始积累之后立即转型，不然总是要还回去的。在股市里遨游，不亏就是赚，能保住不亏，稳扎稳打，从长时间维度上来讲，不一定就会比“快”方法慢。Shaun 这三年的投资收益不能说一点没有，只能说和存银行差不多，相当于是玩了三年，也还算能接受，就当是白嫖了些股市经验吧 :P。这三年可以说是入市即巅峰了，经历过连续几个月的万亿交易量，也经历过上证 2800 以下，算是过了一波小牛熊，不说掌握了多少金融相关的专业知识，但一些常识性的经验还是积累了一些。Shaun 总结的经验主要有：1、万亿成交量的市场，需要慎入；2、当不确定甚至不知道买啥的时候，就不要动；3、当一个热点被广泛讨论的时候，可以考虑退出了；4、尽量不要去买大股东在减持的股票，买家不如卖家精；5、两会前一个月的市场，慎入，两会期间可以酌情考虑入场；6、10 月份可以考虑一波白酒，年后卖掉；7、最重要的是，拿来投资的钱一定得是可预见的未来三年内不会动的钱。</p><h2 id="后记">后记</h2><p> 难得写万字长篇，这次看完『纳瓦尔宝典』，又兼之正好工作三年（拖延症拖到快四年了 😅），算是跨过了人生的一道小槛，所以难免会有些想记下来的一些东西，也算是总结一下这三年来的工作生活思考。或许下一篇万字长文是十年回顾，也或许没有，毕竟自古苦难多诗文，天降大任于人，天不降大任于人，经历一样，唯一的区别在于是不是天命人。道阻且长，且行且顾。</p><div style="text-align:center; font-family: Allura, Consolas, Helvetica, Tahoma, Arial, Microsoft YaHei, 微软雅黑, SimSun, 宋体, Heiti, 黑体, sans-serif; font-size:1.3em; color:#4094c3; font-weight:700; margin:.5em auto;">22 年获得技能：<strong><em>接活达人</em></strong><br />22 年获得成就：<strong><em>三年已到</em></strong></div>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 财富和幸福算是绝大部分人的毕生追求，所以在读这本书时，更容易让人有一种思想上的共鸣，但一个人的成功总是独一无二的，需要依靠天时地利人和，正所谓，学我者生，似我者死，可以学习借鉴成功者的一些思想，但不要想着沿着成功者的路继续走下去就能成功。 ——读『纳瓦尔宝典』</p></summary>
<category term="Life" scheme="http://cniter.github.io/categories/Life/"/>
<category term="record" scheme="http://cniter.github.io/tags/record/"/>
</entry>
<entry>
<title>2021 年小结</title>
<link href="http://cniter.github.io/posts/8e1d1e1d.html"/>
<id>http://cniter.github.io/posts/8e1d1e1d.html</id>
<published>2022-02-20T01:46:12.000Z</published>
<updated>2022-04-10T03:07:27.919Z</updated>
<content type="html"><![CDATA[<p> 纵观宇宙史，生物史，人之一生，不过沧海一粟，弹指灰飞，若有重来，何必重来。人生一字，莫过于拼，为私欲者有之，为利他者有之，为后代者有之，为权利者有之，为名声者有之，为理想者有之，。。。不拼之人，难存于世，众生皆苦，苦中作乐。 ——鲁迅没说过文集</p><span id="more"></span><h2 id="前言">前言</h2><p> 21 年，工作上第一阶段的目标算是提前半年完成了，非常感谢前领导的赏识，至于生活上第一阶段的目标感觉还是遥遥无期。</p><h2 id="工作篇">工作篇</h2><p> 21 年，同样一直在学习，感觉全年都在用新事物完成工作，从学习 Scala，Go 到 OSS，K8S，再到熟悉 macOS，Vim。用这些新学的东西从 0 到 1 完成了一个半项目，一个项目是地图切片系统，将 GIS 数据以 S2 网格的形式进行重新分组管理，这个系统算是优化到了 Shaun 能优化的极致，内存和性能之间达成的 trade-off，单机版可以最大程度的利用多核 CPU，集群版同样可以充分发挥多台机器的作用，这个项目算是 Shaun 花大力气做的第二个项目了，同样的满意与自豪，希望能继续发光发热。至于那半个项目，只能说是开了个头，算是 shp 数据的版本管理系统，支持正常的 CRUD，空间查询以及分析能力，初步的属性和几何信息版本管理，这个项目没有做完，算是留下了一点小遗憾。不过 22 年再回过头去看，继续做下去的话，会碰到很多难点，有些问题，对 21 年的 Shaun 来说可能是无解的，甚至可能是导致项目做不下去的关键问题。</p><p> 21 年，人生中第一次跳槽，要说原因，可能也就是想换个环境，接触不同的人，当然也有一部分钱的原因，更重要的还是想出来看看，看看其他的一些流程方案，加快自己的成长速度，正如 Shaun 在学生时代说的，换个环境能使人成长的更快。确实，跳槽了之后能明显感觉到自己做事的一些变化，每个环境对人的要求是不一样的，不谈孰高孰低，只是不同的方面而已，综合这些方面，才能更好的应对后续碰到的一些困难以及有更好的发展前途。</p><p> 21 年，工作上最大的收获不是做了多少项目，学了多少新技术，更不是跳槽涨了多少薪，而是跳槽后心态和做事方面的一些转变。以前虽然嘴上说着社畜社畜，但总还是一种学生心态，年轻气盛（年轻人不气盛还叫年轻人吗 ๑乛◡乛๑ ），做事钻牛角尖，只想尽最大的努力做好一件事，一心多用就比较烦躁，有时也大手大脚的，还好是 toB 的行业，有足够的时间来打磨和优化，也是真的感谢前领导的赏识和放任。跳槽之后，感觉自己做事的心态一下就放开了。在新公司，学到了一个新词语——确定性。</p><p> 向上管理又同时不唯上，却是不简单，保障确定性就是一种比较好的做法，所谓的确定性就是能够完全把控一件自己负责的事。确定性，说到底也就是数据，有些什么事，分别是什么，分别有多少，工作量多少，计划排期，现在的进度，成果如何，剩余情况，预期情况，风险情况，牵扯的上下游安排，碰到的问题与困难，可能的解决方案。并不是说一定要有阶段性的成果才算确定性，每次汇报，都能把上面这些问题说清楚，也是一种确定性，能够确定这些东西也是自己能力的一种体现。领导关心的也是这些数据，向上负责，同时也是对工作负责，对自己负责。</p><h2 id="生活篇">生活篇</h2><p> 对目前的 Shaun 来说，生活和工作基本没啥区别，工作在 Coding，生活有时也会 Coding，唯一的区别在于，工作是为了生存，生活是为了兴趣。生活算是工作之余的放松，所以关于生活能写的确实不多。</p><p> 21 年，虽然疫情还在持续，但常年待在租房里还是有一些出去玩的冲动，遂去了一趟西湖，人确实很多，风景也没有让人耳目一新的感觉，有点名不副实了，还没有旁边的龙井村好玩，杭州的交通也是一场不太美好的出行体验。出去玩，主要是为了散心，这个目的算是达到了。</p><p> 21 年，本来想去一趟黄山的，但由于自己懒得动，还是没去成，这不得不说是一种遗憾了，换了个城市，再想提起勇气去，就不知道是猴年马月了。换城市这件事，Shaun 也认真思考过，代价确实比较大，或许将来有一天，Shaun 会因为这个决定后悔，当然也或许不会，Shaun 一贯的认知就是有钱在哪都舒服，没钱在哪都难受，最终还是 follow my heart，决定趁着年轻，多出去看看，毅然决然的走出过去两年多舒适的工作和生活环境，来到这个陌生的环境重新开始，这件事，算是为平淡的生活增加了些许起伏。</p><p> 刚来到新城市，虽然觉得一切都比较新鲜，但还是被新城市恶劣的天气环境给搞的很不爽，不过还算运气不错，只看了一家就找到了 Shaun 还算满意的房子，新城市的房租确实要高一些，而且中介费居然要一个月的房租，这着实是有些高。新城市的防疫政策对底层打工人没有丁点儿人文关怀，部分小区的看门大爷是真大爷，就像菜鸟程序员写的低级 robot，逻辑写的死死的。防疫软件也是垃圾中的战斗机，纳税人的血汗钱也不知道有多少进了个人口袋。</p><p> 理财方面也开始接触一些更专业的知识，国内金融从业资格考试主要有四个：证券从业资格考试，基金从业资格考试，银行从业资格考试，期货从业资格考试，都有对应的统编教材，并不是说要一定通过这几个考试，而是可以从这几门考试中，对国内金融市场有一定的认识，不是完全的小白（<em>真要参加考试的话可以看看这个 <a href="https://www.zhihu.com/question/27618901/answer/269242670">证券资格证考试要准备多久？</a> ，刷题的话就找个 app 就行，<strong>刷题学习法</strong> 😅</em>），一般看完前两个从业资格的备考资料就差不多了，有个基础的认识，后期的交易策略或计划就只能根据个人的情况慢慢摸索了。至于 21 年的理财成果就不是很好了，把 20 年赚的又亏回去了，主要是出于 20 年的乐观心态，觉得互联网还能再涨点，就一直没卖，没想到 21 年的国家政策对互联网这么不友好，想要挣钱，还是得跟着政策走 ¯\_(ツ)_/¯。</p><h2 id="总结">总结</h2><p> 前路漫漫，不问对错，不求利弊。每个公司做事的风格是不一样的，这种差异性才是需要学习的地方，也是能让人快速成长的基础。年纪越大，越觉得人生若只如初见是一种奢侈。独行备艰难，莫忘守初心。</p><div style="text-align:center; font-family: Allura, Consolas, Helvetica, Tahoma, Arial, Microsoft YaHei, 微软雅黑, SimSun, 宋体, Heiti, 黑体, sans-serif; font-size:1.3em; color:#4094c3; font-weight:700; margin:.5em auto;">21 年获得技能：<strong><em>学无止境</em></strong><br />21 年获得成就：<strong><em>重新开始</em></strong></div>]]></content>
<summary type="html"><p> 纵观宇宙史，生物史，人之一生，不过沧海一粟，弹指灰飞，若有重来，何必重来。人生一字，莫过于拼，为私欲者有之，为利他者有之，为后代者有之，为权利者有之，为名声者有之，为理想者有之，。。。不拼之人，难存于世，众生皆苦，苦中作乐。 ——鲁迅没说过文集</p></summary>
<category term="Life" scheme="http://cniter.github.io/categories/Life/"/>
<category term="record" scheme="http://cniter.github.io/tags/record/"/>
</entry>
<entry>
<title>M1 个人配置</title>
<link href="http://cniter.github.io/posts/4b1f50ff.html"/>
<id>http://cniter.github.io/posts/4b1f50ff.html</id>
<published>2022-02-06T02:21:28.000Z</published>
<updated>2025-01-06T16:50:45.411Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 记录一下 Shaun 个人的 Mac 装机配置。</p><span id="more"></span><h2 id="必备">必备</h2><p><strong><a href="https://github.com/davidwernhart/AlDente-Charge-Limiter">AlDente</a></strong>：Mac 电池健康保护神器，默认 80% 就行，想充满就设置为 100%，需要<em>禁掉自带的优化电池充电</em>。</p><p><strong><a href="https://github.com/objective-see/LuLu">LuLu</a></strong>：防火墙，控制应用联网权限。</p><p><a href="https://github.com/waydabber/BetterDisplay"><strong>BetterDisplay</strong></a>：使外接显示器更清晰，需设置与笔记本同宽高比/同分辨率的 Dummy 以及将 Dummy 屏幕镜像到外接显示器。</p><p><a href="https://maczip.cn/"><strong>MacZip</strong></a>：解压缩。</p><h2 id="mac-黑魔法">Mac 黑魔法</h2><p> 有时 Mac 系统抽风，部分设置在界面上无法修改，需要通过终端命令强制修改。现记录部分命令：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 允许安装任何来源的 app</span></span><br><span class="line">sudo spctl --master-disable</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置时区为中国标准时间</span></span><br><span class="line">sudo systemsetup -settimezone Asia/Shanghai</span><br></pre></td></tr></table></figure></div><hr /><h2 id="iterm2">iTerm2</h2><p> 下载安装 <a href="https://iterm2.com/">iTerm2</a>，默认 shell 就是 zsh，所以不需要安装。</p><p> <em>※注：推荐用 Mac 系统自带的终端安装 Oh My Zsh 和 Powerlevel10k，之后才使用 iTerm2</em>。</p><p> 安装 <a href="https://github.com/ohmyzsh/ohmyzsh#basic-installation">Oh My Zsh</a>，github 上的命令在国内可能无法顺利执行，先 clone 下来，手动执行 <code>sh tools/install.sh</code>。</p><p> 安装 <a href="https://github.com/romkatv/powerlevel10k/#oh-my-zsh">Powerlevel10k</a> 之前，先安装 <a href="https://www.nerdfonts.com/font-downloads">nerd font 字体</a>，Shaun 个人还是比价喜欢 Fira Code 字体，所以就选择下载 Fira Code Nerd Font 字体，只需要安装 <code>Fira Code Retina Nerd Font Complete.ttf</code> 即可。设置 iTerm2 字体为 FiraCode Nerd Font。</p><p> 随后开始安装 Powerlevel10k，安装完之后重启 iTerm2，会有 Powerlevel10k 的配置提问，依次回答（有推荐按推荐）完成即可配置好 Powerlevel10k，若后续想修改配置，可直接编辑 <code>~/.p10k.zsh</code> 文件或使用 <code>p10k configure</code> 命令重新回答配置提问。最后在 zsh 的配置文件 <code>~/.zshrc</code> 中设置 <code>ZSH_THEME=powerlevel10k/powerlevel10k</code>。</p><p> 推荐安装 zsh 插件 <a href="https://github.com/zsh-users/zsh-syntax-highlighting">zsh-syntax-highlighting</a> 和 <a href="https://github.com/zsh-users/zsh-autosuggestions">zsh-autosuggestions</a>，在执行完</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting</span><br><span class="line"></span><br><span class="line">git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions</span><br></pre></td></tr></table></figure></div><p>后修改 ~/.zshrc 的 plugins 值，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">plugins=( </span><br><span class="line"> git</span><br><span class="line"> zsh-syntax-highlighting</span><br><span class="line"> zsh-autosuggestions</span><br><span class="line"> # other plugins...</span><br><span class="line">)</span><br></pre></td></tr></table></figure></div><h2 id="vi">Vi</h2><p> Vi 使用 <a href="https://spacevim.org/cn/"><strong>SpaceVim</strong></a>，Mac 中如果无法使用 vim 命令，需要先安装 Vim。</p><h2 id="vscode">VSCode</h2><p> VSCode 同样需要设置终端字体为 <code>FiraCode Nerd Font</code>，在终端中进入 Downloads 目录执行 <code>mv Visual\ Studio\ Code.app /Applications</code> 命令，将 VSCode 放进 应用程序 中，再执行 <code>sudo ln -s "/Applications/Visual Studio Code.app/Contents/Resources/app/bin/code" /usr/local/bin/code</code>，之后可在终端使用命令（<code>code .</code>）直接打开 VSCode。若无法自动更新，需执行：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo chown -R $USER ~/Library/Caches/com.microsoft.VSCode.ShipIt</span><br><span class="line">xattr -dr com.apple.quarantine /Applications/Visual\ Studio\ Code.app</span><br></pre></td></tr></table></figure></div><h3 id="vsc-插件">VSC 插件</h3><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode.hexeditor">Hex Editor</a>：编辑二进制文件，可替换和新增字节；</li><li><a href="https://marketplace.visualstudio.com/items?itemName=ryu1kn.partial-diff">Partial Diff</a>：文本比较差分，支持选中的文本和剪贴板内容比较；</li><li><a href="https://marketplace.visualstudio.com/items?itemName=josee9988.minifyall">MinifyAll</a>：代码压缩，支持大部分常见格式（xml，json，html 等）；</li></ul><h2 id="homebrew">Homebrew</h2><p><strong>20241112 更新：</strong></p><p>可一键直接 <a href="https://brew.idayer.com/">安装 Homebrew</a>，<a href="http://mirrors.ustc.edu.cn/help/brew.git.html">中科大源</a>，<a href="https://mirror.tuna.tsinghua.edu.cn/help/homebrew/">清华大学源</a></p><p><strong><em>以下命令无需执行。</em></strong></p><hr /><p> 直接执行：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">/bin/bash -c "$(curl -fsSL https://cdn.jsdelivr.net/gh/ineo6/homebrew-install/install.sh)"</span><br><span class="line"></span><br><span class="line">echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile</span><br><span class="line">eval "$(/opt/homebrew/bin/brew shellenv)"</span><br></pre></td></tr></table></figure></div><p>安装完成后先检查目录 <code>/opt/homebrew/Library/Taps/homebrew/homebrew-cask</code> 是否存在，若不存在，则执行：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /opt/homebrew/Library/Taps/homebrew/</span><br><span class="line">git clone https://mirrors.ustc.edu.cn/homebrew-cask.git</span><br></pre></td></tr></table></figure></div><p> 最后设置中科大源：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">git -C "$(brew --repo)" remote set-url origin https://mirrors.ustc.edu.cn/brew.git</span><br><span class="line">git -C "$(brew --repo homebrew/core)" remote set-url origin https://mirrors.ustc.edu.cn/homebrew-core.git</span><br><span class="line">git -C "$(brew --repo homebrew/cask)" remote set-url origin https://mirrors.ustc.edu.cn/homebrew-cask.git</span><br><span class="line">brew update</span><br><span class="line"></span><br><span class="line">echo 'export HOMEBREW_BOTTLE_DOMAIN=https://mirrors.ustc.edu.cn/homebrew-bottles/bottles' >> ~/.zprofile</span><br><span class="line">source ~/.zprofile</span><br></pre></td></tr></table></figure></div><h2 id="aria2">Aria2</h2><p> 直接使用命令 <code>brew install aria2</code> 安装，生成配置文件：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">cd ~</span><br><span class="line">mkdir .aria2</span><br><span class="line">cd .aria2</span><br><span class="line">touch aria2.conf</span><br></pre></td></tr></table></figure></div><p> 打开 Finder，通过 Shift+Cmd+G 进入路径：~/.aria2/，编辑文件 <code>aria2.conf</code>，添加以下内容：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line">#用户名</span><br><span class="line">#rpc-user=user</span><br><span class="line">#密码</span><br><span class="line">#rpc-passwd=passwd</span><br><span class="line">#上面的认证方式不建议使用,建议使用下面的token方式</span><br><span class="line">#设置加密的密钥</span><br><span class="line">#rpc-secret=token</span><br><span class="line">#允许rpc</span><br><span class="line">enable-rpc=true</span><br><span class="line">#允许所有来源, web界面跨域权限需要</span><br><span class="line">rpc-allow-origin-all=true</span><br><span class="line">#允许外部访问，false的话只监听本地端口</span><br><span class="line">rpc-listen-all=true</span><br><span class="line">#RPC端口, 仅当默认端口被占用时修改</span><br><span class="line">#rpc-listen-port=6800</span><br><span class="line">#最大同时下载数(任务数), 路由建议值: 3</span><br><span class="line">max-concurrent-downloads=5</span><br><span class="line">#断点续传</span><br><span class="line">continue=true</span><br><span class="line">#同服务器连接数</span><br><span class="line">max-connection-per-server=5</span><br><span class="line">#最小文件分片大小, 下载线程数上限取决于能分出多少片, 对于小文件重要</span><br><span class="line">min-split-size=10M</span><br><span class="line">#单文件最大线程数, 路由建议值: 5</span><br><span class="line">split=10</span><br><span class="line">#下载速度限制</span><br><span class="line">max-overall-download-limit=0</span><br><span class="line">#单文件速度限制</span><br><span class="line">max-download-limit=0</span><br><span class="line">#上传速度限制</span><br><span class="line">max-overall-upload-limit=0</span><br><span class="line">#单文件速度限制</span><br><span class="line">max-upload-limit=0</span><br><span class="line">#断开速度过慢的连接</span><br><span class="line">#lowest-speed-limit=0</span><br><span class="line">#验证用，需要1.16.1之后的release版本</span><br><span class="line">#referer=*</span><br><span class="line">#文件保存路径, 默认为当前启动位置</span><br><span class="line">dir=/Users/yuanxu/Downloads</span><br><span class="line">#文件缓存, 使用内置的文件缓存, 如果你不相信Linux内核文件缓存和磁盘内置缓存时使用, 需要1.16及以上版本</span><br><span class="line">#disk-cache=0</span><br><span class="line">#另一种Linux文件缓存方式, 使用前确保您使用的内核支持此选项, 需要1.15及以上版本(?)</span><br><span class="line">#enable-mmap=true</span><br><span class="line">#文件预分配, 能有效降低文件碎片, 提高磁盘性能. 缺点是预分配时间较长</span><br><span class="line">#所需时间 none < falloc ? trunc << prealloc, falloc和trunc需要文件系统和内核支持</span><br><span class="line">file-allocation=prealloc</span><br><span class="line">bt-tracker=udp://tracker.opentrackr.org:1337/announce,udp://open.tracker.cl:1337/announce,udp://9.rarbg.com:2810/announce,udp://tracker.openbittorrent.com:6969/announce,udp://exodus.desync.com:6969/announce,udp://www.torrent.eu.org:451/announce,udp://vibe.sleepyinternetfun.xyz:1738/announce,udp://tracker1.bt.moack.co.kr:80/announce,udp://tracker.zerobytes.xyz:1337/announce,udp://tracker.torrent.eu.org:451/announce,udp://tracker.theoks.net:6969/announce,udp://tracker.srv00.com:6969/announce,udp://tracker.pomf.se:80/announce,udp://tracker.ololosh.space:6969/announce,udp://tracker.monitorit4.me:6969/announce,udp://tracker.moeking.me:6969/announce,udp://tracker.lelux.fi:6969/announce,udp://tracker.leech.ie:1337/announce,udp://tracker.jordan.im:6969/announce,udp://tracker.blacksparrowmedia.net:6969/announce</span><br><span class="line"></span><br></pre></td></tr></table></figure></div><p>最后的 bt-tracker 可以从 <a href="https://github.com/ngosang/trackerslist">trackerslist</a> 获取，只用最好的 20 个即可（trackers_best (20 trackers) => <a href="https://raw.githubusercontent.com/ngosang/trackerslist/master/trackers_best.txt">link</a> / <a href="https://ngosang.github.io/trackerslist/trackers_best.txt">mirror</a> / <a href="https://cdn.jsdelivr.net/gh/ngosang/trackerslist@master/trackers_best.txt">mirror 2</a>）。</p><p> 接着启动 aria2：<code>aria2c --conf-path="/Users/xxx/.aria2/aria2.conf" -D</code> （xxx 为电脑用户名），在 ~/.zshrc 中加入</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">alias start-aria2='aria2c --conf-path="/Users/xxx/.aria2/aria2.conf" -D'</span><br><span class="line">start-aria2</span><br></pre></td></tr></table></figure></div><p>将 start-aria2c 作为启动 aria2 的命令别名，顺便开机自启。</p><p> 最后从 <a href="http://aria2.baisheng999.com/">Aria2中文网</a> 安装 Chrome 插件，打开 aria2 的 WebUI 界面。</p><h2 id="expect">expect</h2><p> 经常需要使用 ssh 远程登陆堡垒机再到远程服务器，输密码选机器都很麻烦，可以用 expect 写些脚本，自动填充密码和机器，一键直接进到远程服务器。首先安装 expect：<code>brew install expect</code>。在 /usr/local/bin 目录中新建脚本：<code>sudo vi mysl.sh</code>，填充相应内容：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/usr/bin/expect -f</span></span><br><span class="line"></span><br><span class="line"><span class="built_in">set</span> USER [用户名]</span><br><span class="line"><span class="built_in">set</span> PWD [密码]</span><br><span class="line"><span class="built_in">set</span> TERMSERVIP [堡垒机服务器ip]</span><br><span class="line"></span><br><span class="line"><span class="comment"># 全部的远程服务器（[remote_server_name] 需要修改为对应的服务器名</span></span><br><span class="line"><span class="built_in">set</span> RS1 [remote_server_name]</span><br><span class="line"><span class="built_in">set</span> RS2 [remote_server_name]</span><br><span class="line"></span><br><span class="line"><span class="comment"># help 命令，查看所有需要登录的远程服务器</span></span><br><span class="line"><span class="keyword">if</span> {[lindex <span class="variable">$argv</span> 0] == <span class="string">"help"</span>} {</span><br><span class="line"> puts <span class="string">"1: <span class="variable">$RS1</span> [说明]"</span></span><br><span class="line"> puts <span class="string">"2: <span class="variable">$RS2</span> [说明]"</span></span><br><span class="line"> send <span class="string">"exit\r"</span></span><br><span class="line"> <span class="built_in">exit</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># ===== 脚本正文 =====</span></span><br><span class="line"><span class="comment"># 默认登陆远程服务器1</span></span><br><span class="line"><span class="built_in">set</span> RS <span class="variable">$RS1</span></span><br><span class="line"><span class="built_in">set</span> timeout 10</span><br><span class="line"></span><br><span class="line"><span class="comment"># 输入命令 1，则登陆第一台服务器</span></span><br><span class="line"><span class="keyword">if</span> {[lindex <span class="variable">$argv</span> 0] == <span class="string">"1"</span>} {</span><br><span class="line"> <span class="built_in">set</span> RS <span class="variable">$RS1</span></span><br><span class="line">}</span><br><span class="line"><span class="keyword">if</span> {[lindex <span class="variable">$argv</span> 0] == <span class="string">"2"</span>} {</span><br><span class="line"> <span class="built_in">set</span> RS <span class="variable">$RS2</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">spawn ssh <span class="variable">${USER}</span>@<span class="variable">${TERMSERVIP}</span> -p 22</span><br><span class="line">expect {</span><br><span class="line"> <span class="string">"yes/no"</span> { send <span class="string">"yes\r"</span>; exp_continue; }</span><br><span class="line"> <span class="string">"*assword*"</span> { send <span class="string">"<span class="variable">$PWD</span>\n"</span>}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># 选择几号跳板机</span></span><br><span class="line">expect <span class="string">"*num*"</span> { send <span class="string">"0\n"</span> }</span><br><span class="line"></span><br><span class="line"><span class="comment"># 登陆远程服务器</span></span><br><span class="line">expect <span class="string">"<span class="variable">${USER}</span>@"</span> { send <span class="string">"ssh <span class="variable">$RS</span>\n"</span> }</span><br><span class="line"></span><br><span class="line"><span class="comment"># 退出 expect（保持在远程服务器终端</span></span><br><span class="line">interact</span><br><span class="line"></span><br><span class="line"><span class="comment"># 退出 expect（回到本地终端</span></span><br><span class="line"><span class="comment"># expect eof </span></span><br></pre></td></tr></table></figure></div><p>为新建的脚本增加可执行权限：<code>sudo chmod 777 mysl.sh</code>，之后可直接使用 <code>mysl.sh 1</code> 登录到对应的远程服务器。</p><h2 id="lrzsz">lrzsz</h2><p> 与 FTP 和 NFS 相比，使用 lrzsz 与远程 linux 服务器做文件上传和下载是最简单的，在 iTerm2 中使用 <code>rz</code> 和 <code>sz</code> 命令进行上传和下载文件需要一定的配置。<strong><em>※注</em></strong>： <em>使用 expect 自动登录的远程环境可能无法使用 sz rz 命令</em>。</p><p> 首先安装 lrzsz：<code>brew install lrzsz</code>。再跳转目录：<code>cd /usr/local/bin</code>，新建文件：<code>sudo vi iterm2-recv-zmodem.sh</code>，添加内容：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="comment"># Author: Matt Mastracci ([email protected])</span></span><br><span class="line"><span class="comment"># AppleScript from http://stackoverflow.com/questions/4309087/cancel-button-on-osascript-in-a-bash-script</span></span><br><span class="line"><span class="comment"># licensed under cc-wiki with attribution required </span></span><br><span class="line"><span class="comment"># Remainder of script public domain</span></span><br><span class="line"></span><br><span class="line">osascript -e <span class="string">'tell application "iTerm2" to version'</span> > /dev/null 2>&1 && NAME=iTerm2 || NAME=iTerm</span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$NAME</span> = <span class="string">"iTerm"</span> ]]; <span class="keyword">then</span></span><br><span class="line"> FILE=`osascript -e <span class="string">'tell application "iTerm" to activate'</span> -e <span class="string">'tell application "iTerm" to set thefile to choose folder with prompt "Choose a folder to place received files in"'</span> -e <span class="string">"do shell script (\"echo \"&(quoted form of POSIX path of thefile as Unicode text)&\"\")"</span>`</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"> FILE=`osascript -e <span class="string">'tell application "iTerm2" to activate'</span> -e <span class="string">'tell application "iTerm2" to set thefile to choose folder with prompt "Choose a folder to place received files in"'</span> -e <span class="string">"do shell script (\"echo \"&(quoted form of POSIX path of thefile as Unicode text)&\"\")"</span>`</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$FILE</span> = <span class="string">""</span> ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> Cancelled.</span><br><span class="line"> <span class="comment"># Send ZModem cancel</span></span><br><span class="line"> <span class="built_in">echo</span> -e \\x18\\x18\\x18\\x18\\x18</span><br><span class="line"> sleep 1</span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span> \<span class="comment"># Cancelled transfer</span></span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"> <span class="built_in">cd</span> <span class="string">"<span class="variable">$FILE</span>"</span></span><br><span class="line"> /usr/<span class="built_in">local</span>/bin/rz -E -e -b</span><br><span class="line"> sleep 1</span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span> \<span class="comment"># Sent \-\> $FILE</span></span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure></div><p>再新建文件：<code>sudo vi iterm2-send-zmodem.sh</code>，添加内容：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="comment"># Author: Matt Mastracci ([email protected])</span></span><br><span class="line"><span class="comment"># AppleScript from http://stackoverflow.com/questions/4309087/cancel-button-on-osascript-in-a-bash-script</span></span><br><span class="line"><span class="comment"># licensed under cc-wiki with attribution required </span></span><br><span class="line"><span class="comment"># Remainder of script public domain</span></span><br><span class="line"></span><br><span class="line">osascript -e <span class="string">'tell application "iTerm2" to version'</span> > /dev/null 2>&1 && NAME=iTerm2 || NAME=iTerm</span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$NAME</span> = <span class="string">"iTerm"</span> ]]; <span class="keyword">then</span></span><br><span class="line"> FILE=`osascript -e <span class="string">'tell application "iTerm" to activate'</span> -e <span class="string">'tell application "iTerm" to set thefile to choose file with prompt "Choose a file to send"'</span> -e <span class="string">"do shell script (\"echo \"&(quoted form of POSIX path of thefile as Unicode text)&\"\")"</span>`</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"> FILE=`osascript -e <span class="string">'tell application "iTerm2" to activate'</span> -e <span class="string">'tell application "iTerm2" to set thefile to choose file with prompt "Choose a file to send"'</span> -e <span class="string">"do shell script (\"echo \"&(quoted form of POSIX path of thefile as Unicode text)&\"\")"</span>`</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"><span class="keyword">if</span> [[ <span class="variable">$FILE</span> = <span class="string">""</span> ]]; <span class="keyword">then</span></span><br><span class="line"> <span class="built_in">echo</span> Cancelled.</span><br><span class="line"> <span class="comment"># Send ZModem cancel</span></span><br><span class="line"> <span class="built_in">echo</span> -e \\x18\\x18\\x18\\x18\\x18</span><br><span class="line"> sleep 1</span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span> \<span class="comment"># Cancelled transfer</span></span><br><span class="line"><span class="keyword">else</span></span><br><span class="line"> /usr/<span class="built_in">local</span>/bin/sz <span class="string">"<span class="variable">$FILE</span>"</span> -e -b</span><br><span class="line"> sleep 1</span><br><span class="line"> <span class="built_in">echo</span></span><br><span class="line"> <span class="built_in">echo</span> \<span class="comment"># Received $FILE</span></span><br><span class="line"><span class="keyword">fi</span> </span><br></pre></td></tr></table></figure></div><p> 为新建的两文件添加可执行权限：<code>sudo chmod 777 iterm2-*</code>。之后添加 rz sz 命令的软连接：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo ln -s /opt/homebrew/bin/rz /usr/local/bin/rz</span><br><span class="line">sudo ln -s /opt/homebrew/bin/sz /usr/local/bin/sz</span><br></pre></td></tr></table></figure></div><p> 最后配置 iTerm2，选择 Preference... -> Profiles -> Default -> Advanced -> Edit （in Triggers），添加下载触发器：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 1. Regular expression 中填写</span></span><br><span class="line">rz waiting to receive.\*\*B0100</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. Action 选择</span></span><br><span class="line">Run Silent Coprocess...</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. Parameters 中填写</span></span><br><span class="line">/usr/<span class="built_in">local</span>/bin/iterm2-send-zmodem.sh</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4. Instant 不勾选</span></span><br><span class="line"><span class="comment"># 5. Enabled 勾选</span></span><br></pre></td></tr></table></figure></div><p>再添加上传触发器：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 1. Regular expression 中填写</span></span><br><span class="line">\*\*B00000000000000</span><br><span class="line"></span><br><span class="line"><span class="comment"># 2. Action 选择</span></span><br><span class="line">Run Silent Coprocess...</span><br><span class="line"></span><br><span class="line"><span class="comment"># 3. Parameters 中填写</span></span><br><span class="line">/usr/<span class="built_in">local</span>/bin/iterm2-recv-zmodem.sh</span><br><span class="line"></span><br><span class="line"><span class="comment"># 4. Instant 不勾选</span></span><br><span class="line"><span class="comment"># 5. Enabled 勾选</span></span><br></pre></td></tr></table></figure></div><p> 至此 M1 中 iTerm2 rz sz 命令配置完成。</p><h2 id="参考资料">参考资料</h2><p><a href="https://suixinblog.cn/2019/09/beautify-terminal.html">iTerm2 + zsh + Oh My Zsh + Powerlevel10k 打造 Mac 下最强终端</a></p><p><a href="https://www.jianshu.com/p/6f344a1fd2e8">Mac M1 iTerm2 配置rz sz 上传下载文件</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 记录一下 Shaun 个人的 Mac 装机配置。</p></summary>
<category term="Share" scheme="http://cniter.github.io/categories/Share/"/>
<category term="record" scheme="http://cniter.github.io/tags/record/"/>
</entry>
<entry>
<title>Scala 多线程编程小结</title>
<link href="http://cniter.github.io/posts/9c9b4035.html"/>
<id>http://cniter.github.io/posts/9c9b4035.html</id>
<published>2021-10-10T01:56:42.000Z</published>
<updated>2021-12-18T11:54:13.936Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 多线程的执行方式有两种：并发（Concurrent）和并行（Parallel），简单来说，并发就是两个线程轮流在一个 CPU 核上执行，而并行则是两个线程分别在两个 CPU 核上运行。一般而言，程序员无法直接控制线程是并发执行还是并行执行，线程的执行一般由操作系统直接控制，当然程序运行时也可以做简单调度。所以对于一般程序员来说，只需要熟练使用相关语言的多线程编程库即可，至于是并发执行还是并行执行，可能并不是那么重要，只要能达到预期效果就行。</p><span id="more"></span><p> Shaun 目前接触的 Scala 原生多线程编程语法就两个：Future 和 Parallel Collections。其中 Future 用的的最多，并且 Parallel Collections 语法非常简单，所以主要介绍 Future，附带提一下 Parallel Collections。</p><h2 id="executioncontext-篇">ExecutionContext 篇</h2><p> ExecutionContext 是 Future 的执行上下文，相当于是 Java 的线程池，Java 的线程池主要有以下两类：</p><ul><li>ThreadPool：所有线程共用一个任务队列，当线程空闲时，从队列中取一个任务执行。</li><li>ForkJoinPool：每个线程各有一个任务队列，当线程空闲时，从其他线程的任务队列中取一批任务放进自己的队列中执行。</li></ul><p> 对于少量任务，这两个池子没啥区别，只是 ThreadPool 在某些情况下会死锁，比如在一个并行度为 2 （最多两个线程）的 ThreadPool 中执行两个线程，两个线程又分别提交一个子任务，并等到子任务执行完才退出，这时会触发相互等待的死锁条件，因为没有多余的空闲线程来执行子任务，而 ForkJoinPool 中每个线程产生的子任务会放在自己的任务队列中，ForkJoinPool 可以在线程耗尽时额外创建线程，也可以挂起当前任务，执行子任务，从而防止死锁。对于大量任务，ForkJoinPool 中的空闲线程会从其他线程的任务队列中一批一批的取任务执行，所以一般会更快，当然若各个任务执行时间比较均衡，则 ThreadPool 会更快。</p><p> 根据线程池创建的参数不同，Executors 中提供了 5 种线程池：newSingleThreadExecutor（单线程线程池，可保证任务执行顺序），newFixedThreadPool（固定大小线程池，限制并行度），newCachedThreadPool（无限大小线程池，任务执行时间小采用），newScheduledThreadPool（同样无限大小，用来处理延时或定时任务），newWorkStealingPool（ForkJoinPool 线程池）。前四种都属于 ThreadPool，根据阿里的 Java 的编程规范，不推荐直接使用 Executors 创建线程池，不过对于计算密集型任务，一般使用 newFixedThreadPool 或 newWorkStealingPool 即可，线程数设置当前 CPU 数即可（Runtime.getRuntime.availableProcessors()），多了反而增加线程上下文切换次数，对CPU 的利用率不增反减。</p><p> Scala 提供了一个默认的 ExecutionContext：<code>scala.concurrent.ExecutionContext.Implicits.global</code>，其本质也是一个 ForkJoinPool，并行度默认设置为当前可用 CPU 数，当然也会根据需要（比如当前全部线程被阻塞）额外创建更多线程。一般做计算密集型任务就用默认线程池即可，特殊情况也可以自己创建 <code>ExecutionContext.fromExecutor(Executors.newFixedThreadPool(8))</code>，下面的代码就可以创建一个同步阻塞的 ExecutionContext：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> currentThreadExecutionContext = <span class="type">ExecutionContext</span>.fromExecutor(</span><br><span class="line"> <span class="keyword">new</span> <span class="type">Executor</span> {</span><br><span class="line"> <span class="comment">// Do not do this!</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">execute</span></span>(runnable: <span class="type">Runnable</span>) { runnable.run() }</span><br><span class="line">})</span><br></pre></td></tr></table></figure></div><p>原因是 <code>runnable.run()</code> 并不会新开一个线程，而是直接在主线程上执行，和调用普通函数一样。</p><h2 id="future-篇">Future 篇</h2><p> 先上一个简单的 Future 并发编程 Demo：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">////import scala.concurrent.ExecutionContext.Implicits.global</span></span><br><span class="line"><span class="comment">//val pool = Executors.newFixedThreadPool(Runtime.getRuntime.availableProcessors())</span></span><br><span class="line"><span class="keyword">val</span> pool = <span class="type">Executors</span>.newWorkStealingPool()</span><br><span class="line"><span class="keyword">implicit</span> <span class="keyword">val</span> ec = <span class="type">ExecutionContext</span>.fromExecutorService(pool)</span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> futures = <span class="type">Array</span>.range(<span class="number">0</span>, <span class="number">10000</span>).map(i => <span class="type">Future</span> {</span><br><span class="line"> println(i)</span><br><span class="line"> <span class="type">Thread</span>.sleep(<span class="number">100</span>)</span><br><span class="line"> i</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> futureSequence = <span class="type">Future</span>.sequence(futures)</span><br><span class="line">futureSequence.onComplete({</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Success</span>(results) => {</span><br><span class="line"> println(results.mkString(<span class="string">"Array("</span>, <span class="string">", "</span>, <span class="string">")"</span>))</span><br><span class="line"> println(<span class="string">s"Success"</span>)</span><br><span class="line"></span><br><span class="line"> ec.shutdown()</span><br><span class="line"> pool.shutdownNow()</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Failure</span>(e) => println(<span class="string">s"Error processing future operations, error = <span class="subst">${e.getMessage}</span>"</span>)</span><br><span class="line">})</span><br><span class="line"><span class="type">Await</span>.result(futureSequence, <span class="type">Duration</span>.<span class="type">Inf</span>)</span><br></pre></td></tr></table></figure></div><p> 如果计算机 CPU 核数为 8 核，则程序运行成功后将会从 VisualVM 中看到有 8 个线程数在运行，控制台中会每次打印 8 条记录，最后打印出完整数组。</p><p> onComplete 是 Future 的回调函数，可对 Success 和 Failure 分别处理，Await 是为了阻塞主线程，当 futureSequence 执行完成后，才继续执行下面的任务。当然，主线程的阻塞也可以使用 Java 中的 CountDownLatch 来实现，只需要在每个 Future 执行完成后调用一次 countDown() 即可，或者直接在 onComplete 的回调函数中调用一次也行。（<em>题外话：CountDownLatch 和 Golang 中的 sync.WaitGroup 感觉区别不大</em>）。</p><p> 如果不想让程序并发执行，则将 <code>Future.sequence(futures)</code> 改为 <code>Future.traverse(futures)(x => x)</code> 即可，此时就会一条条打印，但不保证打印顺序与数组一致。</p><p> 如果使用 <code>ExecutionContext.Implicits.global</code>，并将上面创建 futures 的代码改为：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> futures = <span class="type">Array</span>.range(<span class="number">0</span>, <span class="number">10000</span>).map(i => <span class="type">Future</span> {</span><br><span class="line"> blocking {</span><br><span class="line"> println(i)</span><br><span class="line"> <span class="type">Thread</span>.sleep(<span class="number">100</span>)</span><br><span class="line"> i</span><br><span class="line"> }</span><br><span class="line">})</span><br></pre></td></tr></table></figure></div><p> 则控制台会马上将数组全部打印出来，从 VisualVM 中看会有非常多的线程在运行，远远超过 8 个，这是因为 ForkJoinPool 检测到当前线程以全部阻塞，所以需要另开线程继续执行，如果将线程池改为 <code>Executors.newFixedThreadPool(8)</code>，则不会马上将数组全部打印，而是恢复原样，每次打印 8 条。<code>blocking</code> 需要慎用，如果 ForkJoinPool 中线程数太多，同样会 OOM，一般在大量运行时间短内存小的并发任务中使用。</p><hr /><p> Parallel Collections 并发编程就很简单了，demo 如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">Array</span>.range(<span class="number">0</span>, <span class="number">10000</span>).par.foreach(i => {</span><br><span class="line"> println(i)</span><br><span class="line"> <span class="type">Thread</span>.sleep(<span class="number">100</span>)</span><br><span class="line">})</span><br></pre></td></tr></table></figure></div><p> 关键字为 <code>par</code>，调用该方法即可轻松进行并发计算，不过需要注意的是并发操作的副作用（side-effects）和“乱序”（out of order）语义，副作用就是去写函数外的变量，不仅仅只读写并发操作函数内部声明的变量，乱序语义是指并发操作不会严格按照数组顺序执行，所以如果并发操作会同时操作两个数组元素（eg：reduce），则需要慎重使用，有的操作结果不变，而有的操作会导致结果不唯一。</p><h2 id="经验篇">经验篇</h2><p> Shaun 目前使用 Scala 进行多线程编程主要碰到过以下几个问题：</p><ul><li>数据竞争问题</li><li>任务拆分问题</li><li>内存占用问题</li></ul><p> 数据竞争问题算是多线程编程中最常见的问题，简单来说就是两个线程同时写同一个变量，导致变量值不确定，引发后续问题，解决该问题有很多方法，性能由高到底有：Atomic，volatile，线程安全数据结构（eg：ConcurrentHashMap），Lock，synchronized，前两个方法性能最高，但局限性也很大，如果有现成的线程安全对象使用是最好的，没有的只能用 Lock 和 synchronized，这两种各有优缺点，synchronized 用法简单，能应付绝大部分问题，但对读也会加锁并且无法中断等待线程，Lock 是个接口，有比较多的派生对象（ReentrantLock，ReadWriteLock，ReentrantReadWriteLock 等），能更灵活的控制锁，不过使用起来相对复杂，需要显式地加锁解锁。</p><p> 任务拆分问题，这个问题发生在任务量非常多（千万级以上）的时候，当需要对千万级数据进行并发处理时，单纯的生成相应的千万级 Future 在默认的 ExecutionContext 中执行会比较慢，甚至出现程序运行一段时间卡一段时间的现象（可能是内存不足，GC 卡了），此时需要人为对千万级任务进行合并。Shaun 这里有两种方案：一种是使用 grouped 将千万级任务划分为 16 组，从而降级为 16 个任务，生成 16 个Future，这时执行速度会快很多，且不会有卡的现象出现；另一种方案就是，每次只生成 10 万个 Future 放进 ExecutionContext 中执行，如此将千万级任务拆分成每次 10 万并发执行，同样能解决问题。</p><p> 内存占用问题，这个问题发生在单个任务需要占用大量内存（1G 以上）的时候，当单个任务需要 1G 以上内存，8 个任务并行则需要 8G 以上内存，内存占用过高，提高 JVM 的内存，但也只是治标不治本。Shaun 的解决方案是对单个任务进行进一步拆分，将单个任务继续拆分为 16 个子任务，再将 16 个子任务的结果进行合并，作为单个大任务的结果，8 个大任务串行执行，如此内存占用极大减少，只需要单个任务的内存即可完成全部任务，且 CPU 利用率不变，执行速度甚至会更快（Full GC 次数变少）。</p><hr /><p> Shaun 在写大文件的时候会用到 newSingleThreadExecutor 和 Future.traverse，将写文件的操作放在 Future 里面，每次只写一个大文件（不用多线程写是因为机械硬盘的顺序读写肯定比随机读写快），而生产大文件内容的操作由默认的 ExecutionContext 执行，从而使生产与消费互不干扰，写大文件操作不会阻塞生产操作。</p><p> 一个用 Future 实现的生产者消费者 demo：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> poolProducer = <span class="type">Executors</span>.newWorkStealingPool()</span><br><span class="line"><span class="keyword">implicit</span> <span class="keyword">val</span> ecProducer = <span class="type">ExecutionContext</span>.fromExecutorService(poolProducer)</span><br><span class="line"><span class="keyword">val</span> poolConsumer = <span class="type">Executors</span>.newSingleThreadExecutor()</span><br><span class="line"><span class="keyword">val</span> ecConsumer = <span class="type">ExecutionContext</span>.fromExecutorService(poolConsumer)</span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> futures = <span class="type">Array</span>.range(<span class="number">0</span>, <span class="number">1000</span>).map(i => <span class="type">Future</span> {</span><br><span class="line"> <span class="keyword">val</span> x = produce(i) <span class="comment">// produce something...</span></span><br><span class="line"> x</span><br><span class="line">}(ecProducer).andThen { <span class="keyword">case</span> <span class="type">Success</span>(x) =></span><br><span class="line"> consume(x) <span class="comment">// consume something...</span></span><br><span class="line">}(ecConsumer))</span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> futureSequence = <span class="type">Future</span>.sequence(futures)</span><br><span class="line">futureSequence.onComplete({</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Success</span>(results) => {</span><br><span class="line"> println(<span class="string">"Success."</span>)</span><br><span class="line"></span><br><span class="line"> ecProducer.shutdown()</span><br><span class="line"> poolProducer.shutdownNow()</span><br><span class="line"> ecConsumer.shutdown()</span><br><span class="line"> poolConsumer.shutdownNow()</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Failure</span>(e) => println(<span class="string">s"Error processing future operations, error = <span class="subst">${e.getMessage}</span>"</span>)</span><br><span class="line">})</span><br><span class="line"><span class="type">Await</span>.result(futureSequence, <span class="type">Duration</span>.<span class="type">Inf</span>)</span><br></pre></td></tr></table></figure></div><h2 id="后记">后记</h2><p> Shaun 这里写的 Scala 多线程编程主要是针对计算密集型任务，而 IO 密集型任务一般会用专门的一些框架，计算密集型考虑的是如何最大化利用 CPU，加快任务执行速度，线程数一般比较固定。Scala 的 Future 多线程编程相比 Java 的多线程编程要简洁了很多，唯一需要控制的就是并行度和任务拆分，Shaun 自己在用时也对 Future 做了简单封装，进一步简化了 Scala 的多线程编程，对 Iterable 的并发计算会更方便。</p><h2 id="参考资料">参考资料</h2><p>[1] <a href="https://docs.scala-lang.org/overviews/core/futures.html">Futures and Promises</a></p><p>[2] <a href="https://stackoverflow.com/questions/29068064/scala-concurrent-blocking-what-does-it-actually-do">scala.concurrent.blocking - what does it actually do?</a></p><p>[3] <a href="https://docs.scala-lang.org/overviews/parallel-collections/overview.html">Parallel Collections</a></p><p>[4] <a href="https://www.cnblogs.com/dolphin0520/p/3923167.html">Java并发编程：Lock</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 多线程的执行方式有两种：并发（Concurrent）和并行（Parallel），简单来说，并发就是两个线程轮流在一个 CPU 核上执行，而并行则是两个线程分别在两个 CPU 核上运行。一般而言，程序员无法直接控制线程是并发执行还是并行执行，线程的执行一般由操作系统直接控制，当然程序运行时也可以做简单调度。所以对于一般程序员来说，只需要熟练使用相关语言的多线程编程库即可，至于是并发执行还是并行执行，可能并不是那么重要，只要能达到预期效果就行。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="language" scheme="http://cniter.github.io/tags/language/"/>
</entry>
<entry>
<title>Google S2 Geometry 浅解</title>
<link href="http://cniter.github.io/posts/720275bd.html"/>
<id>http://cniter.github.io/posts/720275bd.html</id>
<published>2021-09-19T08:21:58.000Z</published>
<updated>2021-12-18T11:54:13.932Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> Google S2 Geometry（以下简称 S2） 是 Google 发明的基于单位球的一种地图投影和空间索引算法，该算法可快速进行覆盖以及邻域计算。更多详见 <a href="https://s2geometry.io/">S2Geometry</a>，<a href="https://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/">Google’s S2, geometry on the sphere, cells and Hilbert curve</a>，<a href="https://halfrost.com/go_s2_regioncoverer/">halfrost 的空间索引系列文章</a>。虽然使用 S2 已有一年的时间，但确实没有比较系统的看过其源码，这次借着这段空闲时间，将 Shaun 常用的功能系统的看看其具体实现，下文将结合 S2 的 C++，Java，Go 的版本一起看，由于 Java 和 Go 的都算是 C++ 的衍生版，所以以 C++ 为主，捎带写写这三种语言实现上的一些区别，Java 版本时隔 10 年更新了 2.0 版本，喜大普奔。</p><span id="more"></span><h2 id="坐标篇">坐标篇</h2><figure><img src="https://s2geometry.io/devguide/img/s2cell_global.jpg" alt="s2 projection" /><figcaption aria-hidden="true">s2 projection</figcaption></figure><p> S2 的投影方式可简单想象为一个单位球外接一个立方体，从球心发出一条射线得到球面上的点到立方体上 6 个面的投影，即将球面投影为立方体，当然中间为了使面积分布更为均匀，还做了些其他坐标变换。</p><h3 id="s2latlng-坐标">S2LatLng 坐标</h3><p> 首先是经纬度坐标，默认用弧度（Radians）构造，取值范围为经度 [-π，+π]，纬度 [-π/2，+π/2]，当然也可使用 S1Angle 将角度（Degrees）转成弧度来构造。</p><h3 id="s2point-坐标">S2Point 坐标</h3><p> 然后球面笛卡尔坐标，这是个三维坐标，由 S2LatLng 到 S2Point 相当于将单位球的极坐标表示法转换为笛卡尔坐标表示法，具体公式为 <span class="math inline">\(x=\cos(lat)cos(lng); y=cos(lat)sin(lng); z=sin(lat)\)</span>。</p><h3 id="faceuv-坐标">FaceUV 坐标</h3><p> 这个坐标并没实际的类与其对应，face 指的是立方体的面，值域为 [0,5]，而 uv 坐标是指面上的点，值域为 [-1,1]。首先需要知道 S2Point 会投影到哪个面上，可以知道 S2 的笛卡尔坐标 X 轴正向指向 0 面，Y 轴正向指向 1 面，Z 轴正向指向 2 面，X 轴负向指向 3 面，Y 轴负向指向 4 面，Z 轴负向指向 5 面，所以 S2Point xyz 哪个分量的绝对值最大，就会投影到哪个轴指向的面，若该分量为正值，则取正向指的面，若该分量为负值，则取负向指的面。至于 uv 的计算方式就是直线与平面的交点了，之前的一篇「计算几何基础」中写过，但这里的平面和直线都比较特殊，所以有快速算法，就直接贴 Go 的代码吧：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="GO"><div class="code-copy"></div><figure class="highlight hljs go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// validFaceXYZToUV given a valid face for the given point r (meaning that</span></span><br><span class="line"><span class="comment">// dot product of r with the face normal is positive), returns</span></span><br><span class="line"><span class="comment">// the corresponding u and v values, which may lie outside the range [-1,1].</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">validFaceXYZToUV</span><span class="params">(face <span class="keyword">int</span>, r r3.Vector)</span> <span class="params">(<span class="keyword">float64</span>, <span class="keyword">float64</span>)</span></span> {</span><br><span class="line"><span class="keyword">switch</span> face {</span><br><span class="line"><span class="keyword">case</span> <span class="number">0</span>:</span><br><span class="line"><span class="keyword">return</span> r.Y / r.X, r.Z / r.X</span><br><span class="line"><span class="keyword">case</span> <span class="number">1</span>:</span><br><span class="line"><span class="keyword">return</span> -r.X / r.Y, r.Z / r.Y</span><br><span class="line"><span class="keyword">case</span> <span class="number">2</span>:</span><br><span class="line"><span class="keyword">return</span> -r.X / r.Z, -r.Y / r.Z</span><br><span class="line"><span class="keyword">case</span> <span class="number">3</span>:</span><br><span class="line"><span class="keyword">return</span> r.Z / r.X, r.Y / r.X</span><br><span class="line"><span class="keyword">case</span> <span class="number">4</span>:</span><br><span class="line"><span class="keyword">return</span> r.Z / r.Y, -r.X / r.Y</span><br><span class="line">}</span><br><span class="line"><span class="keyword">return</span> -r.Y / r.Z, -r.X / r.Z</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 这里需要注意的是 S2Point xyz 三分量构成的向量与平面法向量的点积必须是正数时 uv 才算正确有效，Go 在计算时没做校验，C++ 和 Java 都有校验，使用时需要注意。</p><h3 id="facest-坐标">FaceST 坐标</h3><p> 之所以引入 ST 坐标是因为同样的球面面积映射到 UV 坐标面积大小不一，大小差距比较大（离坐标轴越近越小，越远越大），所以再做一次 ST 变换，将面积大的变小，小的变大，使面积更均匀，利于后面在立方体面上取均匀格网（cell）时，每个 cell 对应球面面积差距不大。S2 的 ST 变换有三种：1、线性变换，基本没做任何变形，只是简单将 ST 坐标的值域变换为 [0, 1]，cell 对应面积最大与最小比大约为 5.2；2、二次变换，一种非线性变换，能起到使 ST 空间面积更均匀的作用，cell 对应面积最大与最小比大约为 2.1；3、正切变换，同样能使 ST 空间面积更均匀，且 cell 对应面积最大与最小比大约为 1.4，不过其计算速度相较于二次变换要慢 3 倍，所以 S2 权衡考虑，最终采用了二次变换作为默认的 UV 到 ST 之间的变换。二次变换公式为：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="JAVA"><div class="code-copy"></div><figure class="highlight hljs java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">stToUV</span><span class="params">(<span class="keyword">double</span> s)</span> </span>{</span><br><span class="line"> <span class="keyword">if</span> (s >= <span class="number">0.5</span>) {</span><br><span class="line"> <span class="keyword">return</span> (<span class="number">1</span> / <span class="number">3.</span>) * (<span class="number">4</span> * s * s - <span class="number">1</span>);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> (<span class="number">1</span> / <span class="number">3.</span>) * (<span class="number">1</span> - <span class="number">4</span> * (<span class="number">1</span> - s) * (<span class="number">1</span> - s));</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">double</span> <span class="title">uvToST</span><span class="params">(<span class="keyword">double</span> u)</span> </span>{</span><br><span class="line"> <span class="keyword">if</span> (u >= <span class="number">0</span>) {</span><br><span class="line"> <span class="keyword">return</span> <span class="number">0.5</span> * Math.sqrt(<span class="number">1</span> + <span class="number">3</span> * u);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span> - <span class="number">0.5</span> * Math.sqrt(<span class="number">1</span> - <span class="number">3</span> * u);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><h3 id="faceij-坐标">FaceIJ 坐标</h3><p> IJ 坐标是离散化后的 ST 坐标，将 ST 空间的平面划分为 <span class="math inline">\(2^{30}×2^{30}\)</span> 个网格，取网格所在的横纵坐标得到 IJ 坐标，所以由 ST 到 IJ 坐标的变换就比较简单了：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="JAVA"><div class="code-copy"></div><figure class="highlight hljs java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">int</span> <span class="title">stToIj</span><span class="params">(<span class="keyword">double</span> s)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> Math.max(</span><br><span class="line"> <span class="number">0</span>, Math.min(<span class="number">1073741824</span> - <span class="number">1</span>, (<span class="keyword">int</span>) Math.round(<span class="number">1073741824</span> * s - <span class="number">0.5</span>))</span><br><span class="line"> );</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><h3 id="s2cellid">S2CellId</h3><p> 这个 id 其实是个一维坐标，而是利用希尔伯特空间填充曲线将 IJ 坐标从二维变换为一维，该 id 用一个 64 位整型表示，高 3 位用来表示 face（0~5），后面 61 位来保存不同的 level（0~30） 对应的希尔伯特曲线位置，每增加一个 level 增加两位，后面紧跟一个 1，最后的位数都补 0。<em>注：Java 版本的 id 是有符号 64 位整型，而 C++ 和 Go 的是无符号 64 位整型，所以在跨语言传递 id 的时候，在南极洲所属的最后一个面（即 face = 5）需要小心处理。</em></p><h4 id="hilbertcurve">HilbertCurve</h4><figure><img src="https://upload.wikimedia.org/wikipedia/commons/a/a7/Hilbert_curve_production_rules%21.svg" alt="hilbert_curve_subdivision_rules" /><figcaption aria-hidden="true">hilbert_curve_subdivision_rules</figcaption></figure><figure><img src="https://upload.wikimedia.org/wikipedia/commons/3/35/Hilbert_curve_3_Orient%21.svg" alt="hilbert_curve" /><figcaption aria-hidden="true">hilbert_curve</figcaption></figure><p> 上面两张图很明了的展示了希尔伯特曲线的构造过程，该曲线的构造基本元素由 ABCD 4 种“U”形构成，而 BCD 又可由 A 依次逆时针旋转 90 度得到，所以也可以认为只有一种“U”形，每个 U 占 4 个格子，以特定方式进行 1 分 4 得到下一阶曲线形状。</p><p>每个 U 坐标与希尔伯特位置（用二进制表示）对应关系如下：</p><ul><li>A：<code>00 -> (0,0); 01 -> (0,1); 10 -> (1,1); 11 -> (1,0);</code></li><li>B：<code>00 -> (1,1); 01 -> (0,1); 10 -> (0,0); 11 -> (1,0);</code></li><li>C：<code>00 -> (1,1); 01 -> (1,0); 10 -> (0,0); 11 -> (0,1);</code></li><li>D：<code>00 -> (0,0); 01 -> (1,0); 10 -> (1,1); 11 -> (0,1);</code></li></ul><p>每个 U 一分四对应关系如下：</p><ul><li>A：<code>D -> A -> A -> B</code></li><li>B：<code>C -> B -> B -> A</code></li><li>C：<code>B -> C -> C -> D</code></li><li>D：<code>A -> D -> D -> C</code></li></ul><p> 根据以上两个对应关系就能找到右手坐标系任意阶数的希尔伯特位置及坐标对应关系。以初始 1 阶曲线 A 为例，占据四个格子，然后进行一分四操作，四个格子分成 16 个格子，A 分为 DAAB 四个“U”形，连接起来即为 2 阶曲线，位置与坐标对应关系为（都用二进制表示）：</p><p><code>0000 -> (00, 00); 0001 -> (01, 00); 0010 -> (01, 01); 0011 -> (00, 01)</code>；</p><p><code>0100 -> (00, 10); 0101 -> (00, 11); 0110 -> (01, 11); 0111 -> (01, 10)</code>；</p><p><code>1000 -> (10, 10); 1001 -> (10, 11); 1010 -> (11, 11); 1011 -> (11, 10)</code>；</p><p><code>1100 -> (11, 01); 1101 -> (10, 01); 1110 -> (10, 00); 1111 -> (11, 00)</code>；</p><p> 从二进制中很容易看出随着阶数的增加，位置与坐标的对应关系：每增加一阶，位置往后增加两位，坐标分量各增加一位，位置增加的两位根据一分四对应关系拼接，坐标各分量增加的一位需先找到一分四对应关系，再找对应位置与坐标对应关系，将得到的坐标分量对应拼接。以一阶的 <code>01 -> (0,1)</code> 到二阶的 <code>0110 -> (01, 11)</code> 为例，首先根据 01 得到当前所属一阶第二块，查找一分四对应关系知道，下一阶这块还是 A，根据 0110 后两位 10 可知这块属于 A 的第三个位置，查找坐标得到是 <code>(1,1)</code>，结合一阶的 <code>(0,1)</code>，对应分量拼接得到坐标 <code>(01,11)</code>，即 <code>(1, 3)</code>，同理可根据第二阶的坐标反查第二阶的位置。有了这些关系，就能生成希尔伯特曲线了，下面就看看 S2 是怎么生成 id 的。</p><h4 id="s2id">S2Id</h4><p> 首先 S2 中用了两个二维数组分别保存位置到坐标以及坐标到位置的对应的关系：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// kIJtoPos[orientation][ij] -> pos</span></span><br><span class="line"><span class="keyword">const</span> <span class="keyword">int</span> kIJtoPos[<span class="number">4</span>][<span class="number">4</span>] = {</span><br><span class="line"> <span class="comment">// (0,0) (0,1) (1,0) (1,1)</span></span><br><span class="line"> { <span class="number">0</span>, <span class="number">1</span>, <span class="number">3</span>, <span class="number">2</span> }, <span class="comment">// canonical order</span></span><br><span class="line"> { <span class="number">0</span>, <span class="number">3</span>, <span class="number">1</span>, <span class="number">2</span> }, <span class="comment">// axes swapped</span></span><br><span class="line"> { <span class="number">2</span>, <span class="number">3</span>, <span class="number">1</span>, <span class="number">0</span> }, <span class="comment">// bits inverted</span></span><br><span class="line"> { <span class="number">2</span>, <span class="number">1</span>, <span class="number">3</span>, <span class="number">0</span> }, <span class="comment">// swapped & inverted</span></span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="comment">// kPosToIJ[orientation][pos] -> ij</span></span><br><span class="line"><span class="keyword">const</span> <span class="keyword">int</span> kPosToIJ[<span class="number">4</span>][<span class="number">4</span>] = {</span><br><span class="line"> <span class="comment">// 0 1 2 3</span></span><br><span class="line"> { <span class="number">0</span>, <span class="number">1</span>, <span class="number">3</span>, <span class="number">2</span> }, <span class="comment">// canonical order: (0,0), (0,1), (1,1), (1,0)</span></span><br><span class="line"> { <span class="number">0</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">1</span> }, <span class="comment">// axes swapped: (0,0), (1,0), (1,1), (0,1)</span></span><br><span class="line"> { <span class="number">3</span>, <span class="number">2</span>, <span class="number">0</span>, <span class="number">1</span> }, <span class="comment">// bits inverted: (1,1), (1,0), (0,0), (0,1)</span></span><br><span class="line"> { <span class="number">3</span>, <span class="number">1</span>, <span class="number">0</span>, <span class="number">2</span> }, <span class="comment">// swapped & inverted: (1,1), (0,1), (0,0), (1,0)</span></span><br><span class="line">};</span><br><span class="line"></span><br><span class="line"><span class="comment">// kPosToOrientation[pos] -> orientation_modifier</span></span><br><span class="line"><span class="keyword">const</span> <span class="keyword">int</span> kPosToOrientation[<span class="number">4</span>] = {<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">3</span>};</span><br></pre></td></tr></table></figure></div><p> 方向 0（canonical order）相当于上文中 A，方向 1（axes swapped）相当于上文中 D，方向 2（bits inverted）相当于上文中 C，方向 3（swapped & inverted）相当于上文中 B，kPosToOrientation 代表 S2 中方向 0 一分四的对应关系，而 方向 1，2，3 的对应关系可由该值推出，计算公式为 <code>orientation ^ kPosToOrientation</code>，eg：<code>1 -> 1^kPosToOrientation=[0, 1, 1, 2]; 3 -> 3^kPosToOrientation=[2, 3, 3, 0]</code>，与上文中一分四对应关系一致。</p><p> 随后 S2 初始化了一个 4 阶希尔伯特曲线位置与坐标的对应关系查找表，见 C++ 版的 <code>MaybeInit()</code> 方法，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">int</span> ij = (i << <span class="number">4</span>) + j;</span><br><span class="line">lookup_pos[(ij << <span class="number">2</span>) + orig_orientation] = (pos << <span class="number">2</span>) + orientation;</span><br><span class="line">lookup_ij[(pos << <span class="number">2</span>) + orig_orientation] = (ij << <span class="number">2</span>) + orientation;</span><br></pre></td></tr></table></figure></div><p> orig_orientation 代表 4 个初始方向，orientation 代表该位置或坐标下一阶一分四的方向，数组中每个元素是 16 位数，2 个字节，一个四阶希尔伯特曲线是 <span class="math inline">\(2^4×2^4=256\)</span> 个位置，一个初始方向对应一个四阶希尔伯特曲线，所以一个查找表共占内存 <span class="math inline">\(2×256×4=2048=2KB\)</span>，正好一级缓存能放下，再大的话，一级缓存可能放不下，反而会降低查找速度。这两个查找表就相当于 4 个超“U”形的位置与坐标对应关系，同时一分四对应关系保持不变，以超“U”作为基本元素做下一阶希尔伯特曲线，每增加一阶位置往后增加 8 位，IJ 坐标各往后增加 4 位，如此，以更快的速度迭代到 S2 想要的 30 阶希尔伯特曲线。C++ 的这份代码就很精妙了：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">S2CellId <span class="title">S2CellId::FromFaceIJ</span><span class="params">(<span class="keyword">int</span> face, <span class="keyword">int</span> i, <span class="keyword">int</span> j)</span> </span>{</span><br><span class="line"> <span class="comment">// 初始化超“U”形查找表</span></span><br><span class="line"> <span class="built_in">MaybeInit</span>();</span><br><span class="line"></span><br><span class="line"> <span class="comment">// face 向左移 60 位</span></span><br><span class="line"> uint64 n = absl::implicit_cast<uint64>(face) << (kPosBits - <span class="number">1</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 确定每个面的初始“U”形方向，使每个面都保持相同的右手坐标系，6 个面生成的希尔伯特曲线可以依次相连</span></span><br><span class="line"> uint64 bits = (face & kSwapMask);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 基于超“U”形得到 30 阶希尔伯特曲线 IJ 坐标对应位置</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> GET_BITS(k) do { \</span></span><br><span class="line"><span class="meta"> const int mask = (1 << kLookupBits) - 1; \</span></span><br><span class="line"><span class="meta"> bits += ((i >> (k * kLookupBits)) & mask) << (kLookupBits + 2); \</span></span><br><span class="line"><span class="meta"> bits += ((j >> (k * kLookupBits)) & mask) << 2; \</span></span><br><span class="line"><span class="meta"> bits = lookup_pos[bits]; \</span></span><br><span class="line"><span class="meta"> n |= (bits >> 2) << (k * 2 * kLookupBits); \</span></span><br><span class="line"><span class="meta"> bits &= (kSwapMask | kInvertMask); \</span></span><br><span class="line"><span class="meta"> } while (0)</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">// IJ 只有 30 位，7 这个调用只会导致位置移 4 位，后续调用都移 8 位，得到 4 + 8 * 7 = 60 位</span></span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">7</span>); </span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">6</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">5</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">4</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">3</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">2</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">1</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">0</span>);</span><br><span class="line"><span class="meta">#<span class="meta-keyword">undef</span> GET_BITS</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">// 整个 n 向右移一位，再以 1 结尾</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">S2CellId</span>(n * <span class="number">2</span> + <span class="number">1</span>);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p>再来看看根据 id 反算 IJ 坐标：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">S2CellId::ToFaceIJOrientation</span><span class="params">(<span class="keyword">int</span>* pi, <span class="keyword">int</span>* pj, <span class="keyword">int</span>* orientation)</span> <span class="keyword">const</span> </span>{</span><br><span class="line"> <span class="comment">// 与上面一样</span></span><br><span class="line"> <span class="built_in">MaybeInit</span>();</span><br><span class="line"></span><br><span class="line"> <span class="keyword">int</span> i = <span class="number">0</span>, j = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">int</span> face = <span class="keyword">this</span>-><span class="built_in">face</span>();</span><br><span class="line"> <span class="keyword">int</span> bits = (face & kSwapMask);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 反算 IJ 坐标，k == 7 时，取希尔伯特曲线位置高 4 位，IJ 各前 2 位，其余依次取位置 8 位， IJ 各 4 位</span></span><br><span class="line"><span class="meta">#<span class="meta-keyword">define</span> GET_BITS(k) do { \</span></span><br><span class="line"><span class="meta"> const int nbits = (k == 7) ? (kMaxLevel - 7 * kLookupBits) : kLookupBits; \</span></span><br><span class="line"><span class="meta"> bits += (static_cast<span class="meta-string"><int></span>(id_ >> (k * 2 * kLookupBits + 1)) \</span></span><br><span class="line"><span class="meta"> & ((1 << (2 * nbits)) - 1)) << 2; \</span></span><br><span class="line"><span class="meta"> bits = lookup_ij[bits]; \</span></span><br><span class="line"><span class="meta"> i += (bits >> (kLookupBits + 2)) << (k * kLookupBits); \</span></span><br><span class="line"><span class="meta"> j += ((bits >> 2) & ((1 << kLookupBits) - 1)) << (k * kLookupBits); \</span></span><br><span class="line"><span class="meta"> bits &= (kSwapMask | kInvertMask); \</span></span><br><span class="line"><span class="meta"> } while (0)</span></span><br><span class="line"></span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">7</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">6</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">5</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">4</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">3</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">2</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">1</span>);</span><br><span class="line"> <span class="built_in">GET_BITS</span>(<span class="number">0</span>);</span><br><span class="line"><span class="meta">#<span class="meta-keyword">undef</span> GET_BITS</span></span><br><span class="line"></span><br><span class="line"> *pi = i;</span><br><span class="line"> *pj = j;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (orientation != <span class="literal">nullptr</span>) {</span><br><span class="line"> <span class="built_in">S2_DCHECK_EQ</span>(<span class="number">0</span>, kPosToOrientation[<span class="number">2</span>]);</span><br><span class="line"> <span class="built_in">S2_DCHECK_EQ</span>(kSwapMask, kPosToOrientation[<span class="number">0</span>]);</span><br><span class="line"> <span class="comment">// 0x1111111111111111ULL may be better?</span></span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">lsb</span>() & <span class="number">0x1111111111111110</span>ULL) {</span><br><span class="line"> bits ^= kSwapMask;</span><br><span class="line"> }</span><br><span class="line"> *orientation = bits;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> face;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 这里的 orientation 实际是指当前位置的方向，即其周围必有 3 个位置与其方向相同，最后一行注释 Shaun 之所以认为应该是 0x1111111111111111ULL，是因为第 30 阶希尔伯特曲线位置（leaf cell）按理说同样需要做异或操作得到方向，不过整个 S2 库都没有需要用到 leaf cell 的方向，所以这就倒无关紧要了。之所以需要做异或操作，是因为 bits 是该位置下一阶一分四的方向，而对于同一个希尔伯特曲线位置，奇数阶与奇数阶下一阶一分四方向相同，偶数阶与偶数阶下一阶一分四方向相同，lsb() 表示二进制 id 从右往左数第一个 1 所代表的数， 所以有 0x1111111111111110ULL 这一魔术数，而异或操作正好能将下一阶一分四方向调整为当前阶方向。</p><p> 如此 S2 的坐标以及 id 的生成以及反算就很明了了，下面就是 S2 如何使用 id 做计算了。</p><hr /><h3 id="facesiti-坐标">FaceSiTi 坐标</h3><p> 这个是 S2 内部计算使用的坐标，一般用来计算 cell 的中心坐标，以及根据当前 s 和 t 坐标的精度（小数点后几位）判断对应的级别（level）。由于 S2 本身并不显式存储 ST 坐标（有存 UV 坐标），所以 ST 坐标只能计算出来，每个 cell 的中心点同样如此。计算公式为 <span class="math inline">\(Si=s*2^{31};Ti=t*2^{31}\)</span>。至于为啥是 <span class="math inline">\(2^{31}\)</span>，是因为该坐标是用来描述从 0~ 31 阶希尔伯特曲线网格的中心坐标，0 阶中心以 <span class="math inline">\(1/2^1\)</span> 递增，1 阶中心以 <span class="math inline">\(1/2^2\)</span> 递增，2 阶中心以 <span class="math inline">\(1/2^3\)</span> 递增，……，30 阶中心以 <span class="math inline">\(1/2^{31}\)</span> 递增。S2 计算 id 对应的格子中心坐标，首先就会计算 SiTi 坐标，再将 SiTi 转成 ST 坐标。</p><h2 id="算法篇">算法篇</h2><h3 id="邻域算法">邻域算法</h3><p> S2 计算邻域，最关键的是计算不同面相邻的 leaf cell id，即：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function">S2CellId <span class="title">S2CellId::FromFaceIJWrap</span><span class="params">(<span class="keyword">int</span> face, <span class="keyword">int</span> i, <span class="keyword">int</span> j)</span> </span>{</span><br><span class="line"> <span class="comment">// 限制 IJ 最大最小取值为 -1~2^30, 刚好能超出 IJ 正常表示范围 0~2^30-1</span></span><br><span class="line"> i = <span class="built_in">max</span>(<span class="number">-1</span>, <span class="built_in">min</span>(kMaxSize, i));</span><br><span class="line"> j = <span class="built_in">max</span>(<span class="number">-1</span>, <span class="built_in">min</span>(kMaxSize, j));</span><br><span class="line"></span><br><span class="line"> <span class="keyword">static</span> <span class="keyword">const</span> <span class="keyword">double</span> kScale = <span class="number">1.0</span> / kMaxSize;</span><br><span class="line"> <span class="keyword">static</span> <span class="keyword">const</span> <span class="keyword">double</span> kLimit = <span class="number">1.0</span> + DBL_EPSILON;</span><br><span class="line"> <span class="built_in">S2_DCHECK_EQ</span>(<span class="number">0</span>, kMaxSize % <span class="number">2</span>);</span><br><span class="line"> <span class="comment">// IJ -> SiTi -> ST -> UV</span></span><br><span class="line"> <span class="keyword">double</span> u = <span class="built_in">max</span>(-kLimit, <span class="built_in">min</span>(kLimit, kScale * (<span class="number">2</span> * (i - kMaxSize / <span class="number">2</span>) + <span class="number">1</span>)));</span><br><span class="line"> <span class="keyword">double</span> v = <span class="built_in">max</span>(-kLimit, <span class="built_in">min</span>(kLimit, kScale * (<span class="number">2</span> * (j - kMaxSize / <span class="number">2</span>) + <span class="number">1</span>)));</span><br><span class="line"></span><br><span class="line"> face = S2::<span class="built_in">XYZtoFaceUV</span>(S2::<span class="built_in">FaceUVtoXYZ</span>(face, u, v), &u, &v);</span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">FromFaceIJ</span>(face, S2::<span class="built_in">STtoIJ</span>(<span class="number">0.5</span>*(u+<span class="number">1</span>)), S2::<span class="built_in">STtoIJ</span>(<span class="number">0.5</span>*(v+<span class="number">1</span>)));</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 这个算法主要用来计算超出范围（0~2^30-1）的 IJ 对应的 id，核心思想是先将 FaceIJ 转为 XYZ，再使用 XYZ 反算得到正常的 FaceIJ，进而得到正常的 id。中间 IJ -> UV 中坐标实际经过了 3 步，对于 leaf cell，IJ -> SiTi 的公式为 <span class="math inline">\(Si=2×I+1\)</span>，而对于 ST -> UV，这里没有采用二次变换，就是线性变换 <span class="math inline">\(u=2*s-1\)</span>，官方注释上说明用哪个变换效果都一样，所以采用最简单的就行。</p><h4 id="边邻域">边邻域</h4><p> 边邻域代码很简单，也很好理解：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">S2CellId::GetEdgeNeighbors</span><span class="params">(S2CellId neighbors[<span class="number">4</span>])</span> <span class="keyword">const</span> </span>{</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> <span class="keyword">int</span> level = <span class="keyword">this</span>-><span class="built_in">level</span>();</span><br><span class="line"> <span class="comment">// 计算当前 level 一行或一列对应多少个 30 级的 cell（leaf cell） 2^(30-level)</span></span><br><span class="line"> <span class="keyword">int</span> size = <span class="built_in">GetSizeIJ</span>(level);</span><br><span class="line"> <span class="keyword">int</span> face = <span class="built_in">ToFaceIJOrientation</span>(&i, &j, <span class="literal">nullptr</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Edges 0, 1, 2, 3 are in the down, right, up, left directions.</span></span><br><span class="line"> neighbors[<span class="number">0</span>] = <span class="built_in">FromFaceIJSame</span>(face, i, j - size, j - size >= <span class="number">0</span>)</span><br><span class="line"> .<span class="built_in">parent</span>(level);</span><br><span class="line"> neighbors[<span class="number">1</span>] = <span class="built_in">FromFaceIJSame</span>(face, i + size, j, i + size < kMaxSize)</span><br><span class="line"> .<span class="built_in">parent</span>(level);</span><br><span class="line"> neighbors[<span class="number">2</span>] = <span class="built_in">FromFaceIJSame</span>(face, i, j + size, j + size < kMaxSize)</span><br><span class="line"> .<span class="built_in">parent</span>(level);</span><br><span class="line"> neighbors[<span class="number">3</span>] = <span class="built_in">FromFaceIJSame</span>(face, i - size, j, i - size >= <span class="number">0</span>)</span><br><span class="line"> .<span class="built_in">parent</span>(level);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 分别计算当前 IJ 坐标下右上左坐标对应 id，FromFaceIJSame 表示若邻域在相同面，则走 FromFaceIJ，否则走 FromFaceIJWrap，由于这两个函数得到都是 leaf cell，要上升到指定 level，需要用到 parent 方法，即将希尔伯特曲线位置去掉右 <span class="math inline">\(2*(30-level)\)</span> 位，再组合成新的 id，位运算也很有意思：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> uint64 <span class="title">lsb_for_level</span><span class="params">(<span class="keyword">int</span> level)</span> </span>{</span><br><span class="line"> <span class="keyword">return</span> uint64{<span class="number">1</span>} << (<span class="number">2</span> * (kMaxLevel - level));</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">inline</span> S2CellId <span class="title">S2CellId::parent</span><span class="params">(<span class="keyword">int</span> level)</span> <span class="keyword">const</span> </span>{</span><br><span class="line"> uint64 new_lsb = <span class="built_in">lsb_for_level</span>(level);</span><br><span class="line"> <span class="comment">// 取反加一实际是取负数</span></span><br><span class="line"> <span class="keyword">return</span> <span class="built_in">S2CellId</span>((id_ & (~new_lsb + <span class="number">1</span>)) | new_lsb);</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><h4 id="点邻域">点邻域</h4><p> S2 的点邻域并不是指常规意义上 4 个顶点相邻左上右上右下左下的 id，而是一种比较特殊的相邻关系，以直角坐标系 (0,0),(0,1),(1,1),(1,0) 为例，(0,0) 的点邻域为 (0,0),(0,-1),(-1,-1),(-1,0)，(0,1) 的点邻域为 (0,1),(0,2),(-1,2),(-1,1)，(1,1) 的点邻域为 (1,1),(1,2),(2,2),(2,1)，(1,0) 的点邻域为 (1,0),(1,-1),(2,-1),(2,0)。具体代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">S2CellId::AppendVertexNeighbors</span><span class="params">(<span class="keyword">int</span> level,</span></span></span><br><span class="line"><span class="params"><span class="function"> vector<S2CellId>* output)</span> <span class="keyword">const</span> </span>{</span><br><span class="line"> <span class="comment">// level < this->level()</span></span><br><span class="line"> <span class="built_in">S2_DCHECK_LT</span>(level, <span class="keyword">this</span>-><span class="built_in">level</span>());</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> <span class="keyword">int</span> face = <span class="built_in">ToFaceIJOrientation</span>(&i, &j, <span class="literal">nullptr</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 判断 IJ 落在 level 对应 cell 的哪个方位？（左下左上右上右下，对应上文的(0,0),(0,1),(1,1),(1,0)坐标）</span></span><br><span class="line"> <span class="keyword">int</span> halfsize = <span class="built_in">GetSizeIJ</span>(level + <span class="number">1</span>);</span><br><span class="line"> <span class="keyword">int</span> size = halfsize << <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">bool</span> isame, jsame;</span><br><span class="line"> <span class="keyword">int</span> ioffset, joffset;</span><br><span class="line"> <span class="keyword">if</span> (i & halfsize) {</span><br><span class="line"> ioffset = size;</span><br><span class="line"> isame = (i + size) < kMaxSize;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> ioffset = -size;</span><br><span class="line"> isame = (i - size) >= <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> (j & halfsize) {</span><br><span class="line"> joffset = size;</span><br><span class="line"> jsame = (j + size) < kMaxSize;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> joffset = -size;</span><br><span class="line"> jsame = (j - size) >= <span class="number">0</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">parent</span>(level));</span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i + ioffset, j, isame).<span class="built_in">parent</span>(level));</span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i, j + joffset, jsame).<span class="built_in">parent</span>(level));</span><br><span class="line"> <span class="comment">// 则邻域的 IJ 与当前 cell 都不在同一个面，则说明只有三个点邻域</span></span><br><span class="line"> <span class="keyword">if</span> (isame || jsame) {</span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i + ioffset, j + joffset,</span><br><span class="line"> isame && jsame).<span class="built_in">parent</span>(level));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 上面的代码算是比较清晰了，3 个点邻域的情况一般出现在当前 id 位于立方体 6 个面的角落，<em>该方法的参数 level 必须比当前 id 的 level 要小</em>。</p><h4 id="全邻域">全邻域</h4><p> 所谓全邻域，即为当前 id 对应 cell 周围一圈 cell 对应的 id，若周围一圈 cell 的 level 与 当前 id 的 level 一样，则所求即为正常的 9 邻域。具体代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="C++"><div class="code-copy"></div><figure class="highlight hljs c++"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">void</span> <span class="title">S2CellId::AppendAllNeighbors</span><span class="params">(<span class="keyword">int</span> nbr_level,</span></span></span><br><span class="line"><span class="params"><span class="function"> vector<S2CellId>* output)</span> <span class="keyword">const</span> </span>{</span><br><span class="line"> <span class="comment">// nbr_level >= level</span></span><br><span class="line"> <span class="built_in">S2_DCHECK_GE</span>(nbr_level, <span class="built_in">level</span>());</span><br><span class="line"> <span class="keyword">int</span> i, j;</span><br><span class="line"> <span class="keyword">int</span> face = <span class="built_in">ToFaceIJOrientation</span>(&i, &j, <span class="literal">nullptr</span>);</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 先归一 IJ 坐标，将 IJ 坐标调整为当前 cell 左下角 leaf cell 的坐标</span></span><br><span class="line"> <span class="keyword">int</span> size = <span class="built_in">GetSizeIJ</span>();</span><br><span class="line"> i &= -size;</span><br><span class="line"> j &= -size;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">int</span> nbr_size = <span class="built_in">GetSizeIJ</span>(nbr_level);</span><br><span class="line"> <span class="built_in">S2_DCHECK_LE</span>(nbr_size, size);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">int</span> k = -nbr_size; ; k += nbr_size) {</span><br><span class="line"> <span class="keyword">bool</span> same_face;</span><br><span class="line"> <span class="keyword">if</span> (k < <span class="number">0</span>) {</span><br><span class="line"> same_face = (j + k >= <span class="number">0</span>);</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (k >= size) {</span><br><span class="line"> same_face = (j + k < kMaxSize);</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> same_face = <span class="literal">true</span>;</span><br><span class="line"> <span class="comment">// 生成外包围圈下上两边的 id, 顺序为从左往右</span></span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i + k, j - nbr_size,</span><br><span class="line"> j - size >= <span class="number">0</span>).<span class="built_in">parent</span>(nbr_level));</span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i + k, j + size,</span><br><span class="line"> j + size < kMaxSize).<span class="built_in">parent</span>(nbr_level));</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// 生成外包围圈左右两边以及四个边角的 id, 顺序为从下往上</span></span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i - nbr_size, j + k,</span><br><span class="line"> same_face && i - size >= <span class="number">0</span>)</span><br><span class="line"> .<span class="built_in">parent</span>(nbr_level));</span><br><span class="line"> output-><span class="built_in">push_back</span>(<span class="built_in">FromFaceIJSame</span>(face, i + size, j + k,</span><br><span class="line"> same_face && i + size < kMaxSize)</span><br><span class="line"> .<span class="built_in">parent</span>(nbr_level));</span><br><span class="line"> <span class="keyword">if</span> (k >= size) <span class="keyword">break</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 知道这个函数的作用，再看代码就很明了了，<em>这个方法的参数 nbr_level 必须大于或等于当前 id 的 level</em>，因为一旦外包围圈的 cell 面积比当前 cell 还大，就无法得到正确的外包围圈。</p><h3 id="覆盖算法">覆盖算法</h3><p> S2 的覆盖，是指给定一块区域，能用多少 id 对应的 cell 完全覆盖该区域（GetCovering），当然也有尽量覆盖的算法（GetInteriorCovering），下面主要解析 GetCovering，因为 GetInteriorCovering 也差不多，就是覆盖策略略有不同。</p><p>GetCovering 的区域入参是 S2Region，比较典型的 S2Region 有以下几种：</p><ul><li>S2Cell：S2 id 对应的网格，会保存左下右上两个 UV 坐标，也是覆盖算法使用的基本元素；</li><li>S2CellUnion：多个 S2Cell 集合体，GetCovering 的返回值；</li><li>S2LatLngRect：经纬度矩形区域；</li><li>S2Cap：球帽区域，类比于二维圆的圆弧，球帽的构造比较奇怪，球帽的中心 S2Point 是需要，但另一个变量不是球帽的圆弧角，而是半个圆弧角（S2 代码库对应的 S1Angle 弧度，90 度代表半球，180 度代表全球）所对应弦长的平方，最大值为 4，之所以采用弦长的平方作为默认构造，是因为这就是 3 维中距离，在进行距离比较的场景时会更方便，比如测试是否包含一个 S2Point，计算覆盖多边形时，就不用再比较角度，毕竟角度计算代价比较大；</li><li>S2Loop：多边形的基本组成元素，第一个点与最后一个点隐式连接，逆时针代表封闭，顺时针代表开孔取外围区域，不允许自相交；</li><li>S2Polygon：非常正常的复杂多边形，由多个 S2Loop 构成，S2Loop 之间不能相交；</li><li>S2Polyline：一条折线，同样不能自相交；</li><li>还有些其它不常用的：S2R2Rect（S2Point 矩形区域），S2RegionIntersection（集合相交区域），S2RegionUnion（集合合并区域），……等。</li></ul><p> S2 覆盖算法的本质是一种启发式算法，先取满足当前条件最基本的元素，再依照条件进行迭代优化，所以该算法得到的只是一个近似最优解。GetCovering 需要依次满足以下条件：</p><ol type="1"><li>生成的 S2Cell level 不能比指定的 minLevel 小；（必须满足）</li><li>生成的 S2Cell 的个数不能比指定的 maxCells 多；（可以满足，当满足 1 时，数目已经 maxCells 多，迭代停止）</li><li>生成的 S2Cell level 不能比指定的 maxLevel 大；（必须满足）</li></ol><p> 以上 3 个条件对应 GetCovering 的其他三个参数，当然还有一个参数是 levelModel，表示从 minLevel 向下分到 maxLevel 时，是 1 分 4，还是 1 分 16，还是 1 分 64，对应一次升 1 阶曲线，还是一次升 2 阶，或是一次升 3 阶。下面就来具体看看 GetCovering 的算法流程（代码就不贴了，太多了）：</p><ol type="1"><li>首先获取候选种子 S2Cell。先构造一个临时覆盖器，设置 maxCells 为 4，minLevel 为 0，以快速得到初始覆盖结果，做法为：先得到覆盖输入区域的 S2Cap，再用 S2CellUnion 覆盖该 S2Cap，根据 S2Cap 圆弧度计算 S2Cell 的 level，若最终 level < 0，则说明 S2Cap 非常大，需要取 6 个面对应的 S2Cell，否则只需要取 S2Cap 中心点对应 S2Cell 的 level 级的点邻域 4 个 S2Cell 作为初始候选 S2Cell。</li><li>然后标准化候选种子。第一步，如果候选 S2Cell level 比 maxLevel 大或者候选 S2Cell 的 level 不符合 levelModel，则调整候选 S2Cell 的 level，用指定父级 S2Cell 来代替；第二步，归一化候选 S2Cell，先对 S2Cell 按 id 排序，去除被包含的 id，以及对 id 剪枝（若连续 4 个 S2Cell 共有同一个 parent，则用 parent 代替这 4 个 S2Cell）；第三步，反归一化候选 S2Cell，若候选 S2Cell level 比 minLevel 小或不满足 levelModel，则需要将 S2Cell 分裂，用指定级别的孩子来取代该 S2Cell；第四步，检查是否满足全部条件，若满足，则标准化完成，若不满足，则看候选 S2Cell 的数目是否足够多，若足够多，则需要迭代进行 GetCovering，这样会极大降低算法性能，若不是很多，则迭代合并相同祖先的两个 S2Cell（当然祖先的 level 不能比 minLevel 小），最后再次检查所有候选 S2Cell 是否达到标准化要求，并调整 S2Cell level。</li><li>构造优先级队列。将符合条件（与入参区域相交）的候选 S2Cell 放进一个优先级队列中，优先级会依次根据三个参数进行判断，1、S2Cell 的大小（level 越大，S2Cell 越小），越大的优先级越高；2、入参区域与候选 S2Cell 孩子相交（这里的相交是指相交但不完全包含）的个数，越少优先级越高；3、入参区域完全包含候选 S2Cell 孩子和与无法再细分的孩子的个数，同样是越少优先级越高。在构造这个优先级队列的同时，会输出一些候选 S2Cell 作为覆盖算法的正式结果，这些 S2Cell 满足任意以下条件：1、被入参区域完全覆盖；2、与入参区域相交但不可再细分；3、入参区域包含或相交全部孩子。如此留在优先级队列中的，就都是些与入参区域边界相交的 S2Cell，这些就是真正的候选 S2Cell。</li><li>最后，处理优先级队列中的 S2Cell。处理方式也比较简单粗暴，继续细分并入队，满足上面3个出队条件的任意一个，即可出队作为正式结果，当然，若分到后面可能正式的 S2Cell 太多，甚至超过 maxCells，这时不再细分强行出队作为正式结果。最后，再对正式结果做一次标准化处理，即进行第 2 步，得到最终的覆盖结果。</li></ol><p> 以上就是 S2 覆盖算法的大致流程，更加细节的东西，还是得看代码，文字有些不是很好描述，代码里面计算候选 S2Cell 的优先级就很有意思。</p><hr /><p> 当然 S2 中还有很多其他算法（凸包，相交，距离），这里就不做太多介绍了，Shaun 平常用的最多的就是覆盖算法，之前一直没有细看，就简单用用 api，同时为了对一块大的 S2Cell 做多线程处理，需要了解 S2Cell 一分四的方向，经过这次对 S2 的了解，发现之前的用法存在一些问题，可见调包侠同样需要对包有一定的了解才能调好包 ╮(╯▽╰)╭。</p><h2 id="后记">后记</h2><p> 正如许多经典的算法一样，看完之后总有种我上我也行的感觉，但实际完全不行，S2 全程看下来有些地方确实比较晦涩，而且这一连串的想法也很精妙（单位球立方体投影，ST 空间面积优化，64 位 id 生成等），Shaun 或许能有部分想法，但这么多奇思妙想组合起来，就完全不行。</p><h2 id="附录">附录</h2><h3 id="hilbertcurve-绘制">HilbertCurve 绘制</h3><p> 在网上随便找了三种实现方式，并用 threejs 简单绘制了一下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> * <span class="keyword">as</span> THREE <span class="keyword">from</span> <span class="string">"three"</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">export</span> <span class="keyword">default</span> <span class="class"><span class="keyword">class</span> <span class="title">HilbertCurve</span> </span>{</span><br><span class="line"> order = <span class="number">3</span>; <span class="comment">// 阶数</span></span><br><span class="line"> size = <span class="number">1</span> << <span class="built_in">this</span>.order; <span class="comment">// 行列数</span></span><br><span class="line"> totalSize = <span class="built_in">this</span>.size * <span class="built_in">this</span>.size; <span class="comment">// 总网格数，希尔伯特长度</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">// https://www.youtube.com/watch?v=dSK-MW-zuAc</span></span><br><span class="line"> <span class="function"><span class="title">getPath_V1</span>(<span class="params"></span>)</span> {</span><br><span class="line"> <span class="keyword">let</span> path = [];</span><br><span class="line"> <span class="keyword">let</span> origOrientation = [</span><br><span class="line"> [<span class="number">0</span>, <span class="number">0</span>],</span><br><span class="line"> [<span class="number">0</span>, <span class="number">1</span>],</span><br><span class="line"> [<span class="number">1</span>, <span class="number">1</span>],</span><br><span class="line"> [<span class="number">1</span>, <span class="number">0</span>],</span><br><span class="line"> ]; <span class="comment">// 倒 U 形</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">0</span>; i < <span class="built_in">this</span>.totalSize; i++) {</span><br><span class="line"> path.push(hilbertToXY(i, <span class="built_in">this</span>.order));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> path;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">hilbertToXY</span>(<span class="params">i: <span class="built_in">number</span>, order: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> index = i & <span class="number">3</span>;</span><br><span class="line"> <span class="keyword">let</span> curCoord = origOrientation[index].slice();</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> ord = <span class="number">1</span>; ord < order; ord++) {</span><br><span class="line"> i = i >>> <span class="number">2</span>;</span><br><span class="line"> index = i & <span class="number">3</span>;</span><br><span class="line"> <span class="keyword">let</span> delta = <span class="number">1</span> << ord;</span><br><span class="line"> <span class="keyword">if</span> (index === <span class="number">0</span>) {</span><br><span class="line"> <span class="comment">// 顺时针旋转 90°</span></span><br><span class="line"> <span class="keyword">let</span> tmp = curCoord[<span class="number">0</span>];</span><br><span class="line"> curCoord[<span class="number">0</span>] = curCoord[<span class="number">1</span>];</span><br><span class="line"> curCoord[<span class="number">1</span>] = tmp;</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (index === <span class="number">1</span>) {</span><br><span class="line"> curCoord[<span class="number">1</span>] += delta;</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (index === <span class="number">2</span>) {</span><br><span class="line"> curCoord[<span class="number">0</span>] += delta;</span><br><span class="line"> curCoord[<span class="number">1</span>] += delta;</span><br><span class="line"> } <span class="keyword">else</span> <span class="keyword">if</span> (index === <span class="number">3</span>) {</span><br><span class="line"> <span class="comment">// 逆时针旋转 90°</span></span><br><span class="line"> <span class="keyword">let</span> tmp = delta - <span class="number">1</span> - curCoord[<span class="number">0</span>];</span><br><span class="line"> curCoord[<span class="number">0</span>] = delta - <span class="number">1</span> - curCoord[<span class="number">1</span>];</span><br><span class="line"> curCoord[<span class="number">1</span>] = tmp;</span><br><span class="line"> curCoord[<span class="number">0</span>] += delta;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> curCoord;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Hacker's Delight</span></span><br><span class="line"> <span class="function"><span class="title">getPath_V2</span>(<span class="params"></span>)</span> {</span><br><span class="line"> <span class="keyword">let</span> path: <span class="built_in">number</span>[][] = [];</span><br><span class="line"> <span class="keyword">let</span> x = -<span class="number">1</span>,</span><br><span class="line"> y = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">let</span> s = <span class="number">0</span>; <span class="comment">// along the curve</span></span><br><span class="line"></span><br><span class="line"> step(<span class="number">0</span>);</span><br><span class="line"> hilbert(<span class="number">0</span>, <span class="number">1</span>, <span class="built_in">this</span>.order);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> path;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">step</span>(<span class="params">dir: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">switch</span> (dir & <span class="number">3</span>) {</span><br><span class="line"> <span class="keyword">case</span> <span class="number">0</span>:</span><br><span class="line"> x += <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">1</span>:</span><br><span class="line"> y += <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">2</span>:</span><br><span class="line"> x -= <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> <span class="keyword">case</span> <span class="number">3</span>:</span><br><span class="line"> y -= <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> path[s] = [x, y];</span><br><span class="line"></span><br><span class="line"> s += <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">hilbert</span>(<span class="params">dir: <span class="built_in">number</span>, rot: <span class="built_in">number</span>, order: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (order === <span class="number">0</span>) <span class="keyword">return</span>;</span><br><span class="line"></span><br><span class="line"> dir += rot;</span><br><span class="line"> hilbert(dir, -rot, order - <span class="number">1</span>);</span><br><span class="line"> step(dir);</span><br><span class="line"></span><br><span class="line"> dir -= rot;</span><br><span class="line"> hilbert(dir, rot, order - <span class="number">1</span>);</span><br><span class="line"> step(dir);</span><br><span class="line"></span><br><span class="line"> hilbert(dir, rot, order - <span class="number">1</span>);</span><br><span class="line"></span><br><span class="line"> dir -= rot;</span><br><span class="line"> step(dir);</span><br><span class="line"> hilbert(dir, -rot, order - <span class="number">1</span>);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// https://en.wikipedia.org/wiki/Hilbert_curve</span></span><br><span class="line"> <span class="function"><span class="title">getPath_V3</span>(<span class="params"></span>)</span> {</span><br><span class="line"> <span class="keyword">let</span> path: <span class="built_in">number</span>[][] = [];</span><br><span class="line"></span><br><span class="line"> <span class="comment">// for (let i = 0; i < this.totalSize; i++) {</span></span><br><span class="line"> <span class="comment">// path.push(hilbertToXY(this.size, i));</span></span><br><span class="line"> <span class="comment">// }</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> y = <span class="number">0</span>; y < <span class="built_in">this</span>.size; y++) {</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> x = <span class="number">0</span>; x < <span class="built_in">this</span>.size; x++) {</span><br><span class="line"> path[xyToHilbert(<span class="built_in">this</span>.size, x, y)] = [x, y];</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> path;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">rot</span>(<span class="params">N: <span class="built_in">number</span>, rx: <span class="built_in">number</span>, ry: <span class="built_in">number</span>, xy: <span class="built_in">number</span>[]</span>) </span>{</span><br><span class="line"> <span class="keyword">if</span> (ry === <span class="number">0</span>) {</span><br><span class="line"> <span class="keyword">if</span> (rx === <span class="number">1</span>) {</span><br><span class="line"> xy[<span class="number">0</span>] = N - <span class="number">1</span> - xy[<span class="number">0</span>];</span><br><span class="line"> xy[<span class="number">1</span>] = N - <span class="number">1</span> - xy[<span class="number">1</span>];</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> t = xy[<span class="number">0</span>];</span><br><span class="line"> xy[<span class="number">0</span>] = xy[<span class="number">1</span>];</span><br><span class="line"> xy[<span class="number">1</span>] = t;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">hilbertToXY</span>(<span class="params">N: <span class="built_in">number</span>, h: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> t = h;</span><br><span class="line"> <span class="keyword">let</span> xy = [<span class="number">0</span>, <span class="number">0</span>];</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> s = <span class="number">1</span>; s < N; s *= <span class="number">2</span>) {</span><br><span class="line"> <span class="keyword">let</span> rx = <span class="number">1</span> & (t / <span class="number">2</span>);</span><br><span class="line"> <span class="keyword">let</span> ry = <span class="number">1</span> & (t ^ rx);</span><br><span class="line"> rot(s, rx, ry, xy);</span><br><span class="line"> xy[<span class="number">0</span>] += s * rx;</span><br><span class="line"> xy[<span class="number">1</span>] += s * ry;</span><br><span class="line"> t /= <span class="number">4</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> xy;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">xyToHilbert</span>(<span class="params">N: <span class="built_in">number</span>, x: <span class="built_in">number</span>, y: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> h = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">let</span> xy = [x, y];</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> s = N / <span class="number">2</span>; s > <span class="number">0</span>; s /= <span class="number">2</span>) {</span><br><span class="line"> <span class="keyword">let</span> rx = (xy[<span class="number">0</span>] & s) > <span class="number">0</span> ? <span class="number">1</span> : <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">let</span> ry = (xy[<span class="number">1</span>] & s) > <span class="number">0</span> ? <span class="number">1</span> : <span class="number">0</span>;</span><br><span class="line"> h += s * s * ((<span class="number">3</span> * rx) ^ ry);</span><br><span class="line"> rot(N, rx, ry, xy);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> h;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="title">draw</span>(<span class="params"></span>)</span> {</span><br><span class="line"> <span class="keyword">let</span> lineGeometry = <span class="keyword">new</span> THREE.Geometry();</span><br><span class="line"> <span class="built_in">this</span>.getPath_V3().forEach(<span class="function">(<span class="params">vertice</span>) =></span> {</span><br><span class="line"> <span class="keyword">let</span> vecot = <span class="keyword">new</span> THREE.Vector3().fromArray(vertice);</span><br><span class="line"> vecot.setZ(<span class="number">0</span>);</span><br><span class="line"> lineGeometry.vertices.push(vecot);</span><br><span class="line"> });</span><br><span class="line"> <span class="keyword">let</span> lineMaterial = <span class="keyword">new</span> THREE.LineBasicMaterial({ <span class="attr">color</span>: <span class="number">0x00ffff</span>, <span class="attr">linewidth</span>: <span class="number">1</span> });</span><br><span class="line"> <span class="keyword">let</span> line = <span class="keyword">new</span> THREE.Line(lineGeometry, lineMaterial);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> line;</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br></pre></td></tr></table></figure></div>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> Google S2 Geometry（以下简称 S2） 是 Google 发明的基于单位球的一种地图投影和空间索引算法，该算法可快速进行覆盖以及邻域计算。更多详见 <a href="https://s2geometry.io/">S2Geometry</a>，<a href="https://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/">Google’s S2, geometry on the sphere, cells and Hilbert curve</a>，<a href="https://halfrost.com/go_s2_regioncoverer/">halfrost 的空间索引系列文章</a>。虽然使用 S2 已有一年的时间，但确实没有比较系统的看过其源码，这次借着这段空闲时间，将 Shaun 常用的功能系统的看看其具体实现，下文将结合 S2 的 C++，Java，Go 的版本一起看，由于 Java 和 Go 的都算是 C++ 的衍生版，所以以 C++ 为主，捎带写写这三种语言实现上的一些区别，Java 版本时隔 10 年更新了 2.0 版本，喜大普奔。</p></summary>
<category term="Mathematics" scheme="http://cniter.github.io/categories/Mathematics/"/>
<category term="geometry" scheme="http://cniter.github.io/tags/geometry/"/>
</entry>
<entry>
<title>K8S 应用开发指北</title>
<link href="http://cniter.github.io/posts/55e674ff.html"/>
<id>http://cniter.github.io/posts/55e674ff.html</id>
<published>2021-08-28T08:21:58.000Z</published>
<updated>2021-12-18T11:54:13.934Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 在周志明的『凤凰架构』中需要思考这样一个问题，如何用不可靠的部件来构造一个可靠的系统？对于程序员来说，写的代码从某种程度上来说都是不可靠的，但这些代码组成的一些系统却可以是可靠的。程序员对于错误的处理可以分为两派，一派是必须对错误进行处理，以保证系统的稳定行；另一派不对错误进行处理，任由程序 crash，只要有兜底方案，后面再不断完善。这两派并无孰优孰劣，只是两种不同的思维方式，甚至在同一个程序中，有些错误会处理，有些错误不会处理，这都是可能的。K8S 作为事实上的云原生操作系统，其目的就是为了将程序员写的各个程序组装成一个稳定的系统，并减少运维成本。</p><span id="more"></span><h2 id="基础篇">基础篇</h2><p> K8S 调度的基本单元是 Pod，Pod 也是 K8S 自带的一个资源对象，其可以简单理解为是一个容器集合体，程序员可控的容器有两类（Pause 容器除外），一类是 InitContainer，另一类是普通业务容器，InitContainer 按数组顺序创建，顺序执行，若一个失败，则整个 Pod 创建失败，普通业务容器同样按数组顺序创建，但异步执行，所以执行顺序不可控（可以通过 postStart Hook 简单控制一下）。由于 InitContainer 先于 Pod 其他容器执行，所以一般用来做普通业务容器执行前置条件的一些事情，比如：下载文件，初始化配置，状态消息通知等。</p><p> 同一 Pod 中存储卷和网络可以共享。存储卷共享是指 Pod 内各容器可以挂载相同存储卷，从而数据共享。K8S 目前支持的存储卷共有三种：第一种是 emptyDir，这种存储是临时的，只能在 Pod 内使用，当 Pod 被销毁时，该存储的内容也会消失，只能在同一 Pod 内共享数据；第二种是 hostPath，这种存储会直接和集群中物理机存储相关联，是一种跨 Pod 持久化存储，但仅限该物理机，当 pod 被调度到其他物理机时就无法实现跨 Pod 共享数据；最后一种是外部存储（NFS，Ceph，GlusterFS，AWS EBS 等），这种方式可以真正实现数据持久化并共享，而且可以支持存储与计算分离，对系统会更友好一些，当然运维的成本也会更大。当然除了 K8S 自身提供的存储卷挂载可以实现数据共享，从程序的角度上，使用传统的方式一样也能数据共享，如数据库，DFS，OSS 等。</p><p> 而网络共享是指 Pod 内各容器直接可以使用 localhost 以及容器暴露的端口进行相互通信，K8S 的端口有三种，分别为：容器端口（containerPort，容器中对外暴露的端口），集群内端口（port，集群内 pod 相互通信的端口），集群外端口（nodePort，集群外请求集群内的端口），其中容器端口和集群内是正常的动态端口，取值范围为 [1024, 65535]，集群外端口只能设置为 [30000, 32767]，若集群中服务不与集群外通信，则只需要设置集群内端口就行。K8S 中 IP 也同样有三种，分别为：Pod IP（两不同 Pod 资源对象相互通信的地址，集群外不可访问），Cluster IP（Service 资源对象的通信地址，集群外不可访问），Node IP（K8S 物理节点的 IP 地址，是真实的物理网络，集群外配合 nodePort 即可访问）。集群内端口和集群外端口由 K8S 的 Service 资源提供设置。<em>在创建 Service 时需要注意，一个 Pod 资源对应一个 Service 资源，不要想着一个 Service 管理两个 Pod 暴露的端口，这样做会使 Service 提供服务的能力异常，经常会接口超时</em>。</p><p> K8S 编程可以简单称之为面向 config 编程，一切需要动态变化的程序初始化变量，都应该以 config 的形式提供，然后交给运维就行，这样可以避免程序员频繁的修改程序，减少运维负担，K8S 的 config 有三种形式，第一种是程序启动参数，通过创建容器时的 args 参数配置；第二种是系统环境变量，通过创建容器时的 env 参数配置；最后一种是 K8S 提供的 ConfigMap 资源，该资源可以从文件，目录或 key-value 字符串创建，创建后的 ConfinMap 被全集群同命名空间所共享，可以通过 volumes 参数挂载到 pod 中，进而 mount 进容器中，被程序读取。前两种 config 方式对于配置变量少的可以使用，当配置变量很多或配置参数很长时，还是使用 ConfigMap 比较合适。</p><h2 id="调度篇">调度篇</h2><p> 调度，广义上的调度可指一切管理安排，CPU 的指令执行就涉及到三级缓存的调度，程序运行时的 GC 可认为是运行时对内存资源的调度，操作系统的进程轮转可认为是系统对进程的调度，而 K8S 中的调度可简单理解为是对操作系统的调度。</p><p> K8S 的调度可简单分为两个层面上的调度，最底层的调度自然是 K8S 自身的调度策略，根据不同的资源用度和调度策略将 Pod 分配到不同的物理节点之上执行，根据指定的重启或恢复策略启动相应的 Pod，这个层面上的调度，K8S 有一套默认的调度器，对于特殊的调度需求，K8S 也支持自定义调度器，使用外部调度器代替默认调度器，这个层面的调度器 Shaun 没做太多研究，所以在这篇里对这层面的调度器不做过多描述。Shaun 接触过的是更上层的调度器，业务层面的调度服务，业务调度服务一般与业务紧密相关，但最核心的一点就是能够从业务入手，负责 Pod 的创建和销毁，并能掌握其运行状态，就算是完成了一个基础的业务调度服务器。</p><p> 在设计业务调度服务时，有一种通用的模式，可以称之为 master-worker 模式，与同名的并发模式细节上有所不同，这里的 master 是指调度服务本体，只负责对外服务，资源监控，以及任务分发，任务状态感知等，不负责做具体的任务，一般也不关心任务的输入输出。在部署 master 时，一般会创建一个 Service 资源对象，毕竟其主要功能就是对外服务，master 一般由运维进行部署创建销毁。而 worker 是指真正做任务的 Pod，该 Pod 中可能会有多个容器，主容器负责真正执行任务，其他一些容器可能会负责保障任务的前置条件（输入，配置等），以及向 master 汇报任务执行状态信息（执行任务的主容器可能并不知道 master 的存在）等。worker 对应的 Pod 一般由 master 进行创建销毁，worker 的一些配置信息则可能会由运维管理。</p><p> 由于 K8S 并没有在整个集群物理资源之上抽象出一层集群资源，所以 K8S 分配的节点实际还是在物理机上，若所有物理机剩余资源（是单个剩余资源，而不是所有剩余资源之和）都不满足 Pod 所需资源，则该 Pod 无法调度，类比内存碎片化，可以称之为资源碎片化。所以在创建 Pod 时，所需资源最好不要太多，以免调度失败。</p><h2 id="实践篇">实践篇</h2><p> Shaun 目前在 K8S 上开发的主要就是重计算（单机计算时间以小时计）调度服务。这类调度服务其实也分两种，一种是并发调度，一种是流水线（pipeline）式的串行调度，当然也可以将这两种混合起来，串行中有并行。在设计这类调度服务时，需要考虑集群上的资源（内存，CPU）是否足够，若不足，则可以考虑加入一个简单的等待机制，将任务放进一个队列中，当然加入这样一个等待机制，又会增加系统复杂性，需要考虑队列容量，队列优先级等。所以可执行的最小任务消耗的资源越少约好，否则集群中可能完全无法执行相关任务。</p><p> 由于 Shaun 是独立开发，能完全控制 master 和 worker 的编写，所以 worker 设计的比较简单，一个主容器即完成了前置数据处理，主任务执行，执行状态汇报等全部事情，这是从时间和性能上以及系统复杂度上等多方面权衡的结果，当然在时间足够人手够的情况，是应该把现有的 worker 进一步分离的，而 master 就是比较通用的设计，资源监控，任务队列，任务 Pod 创建与销毁，任务状态信息保存，服务接口等，其中常规的服务接口应该有添加任务，开始任务，停止任务，恢复任务，删除任务，任务状态查询，任务日志查询，任务状态汇报等接口，如果任务是并行且无依赖的，还应该支持开始指定子任务等接口。</p><p> 在工作中，Shaun 也接触到一个 pipeline 式的任务调度服务，pipeline 式的工作流有个特点就是下一个子任务的输入必定依赖上一个子任务的输出，在这个任务调度服务中，其子任务的输入输出都是文件态，并且 master 不关心子任务的输入输出，子任务的执行程序也不知道 master 的存在，尽量低耦合。在云上，文件态的存储载体比较好的自然是 OSS，但原本的子任务执行程序只支持本地读取文件，而且在原来的程序中引入 OSS 的读写逻辑并不十分合适，所以在 K8S 中引入了 NFS，由 master 负责将 NFS 挂载到各子任务的 Pod 中，并在挂载到主容器时使用 SubPath 完成 pipeline 之间的资源隔离，使用 emptyDir 完成各子任务之间的资源隔离，每条 pipeline 开始的子任务是从 OSS 中拉取文件到 NFS 中对应的 SubPath 目录中，结束的子任务是将 NFS 中对应的 SubPath 目录中约定好的生成物上传到 OSS 中，并清空该 SubPath 目录，从而使原来的程序在 IO 这块完全不用改动。在监听任务运行状态方面，有两种方案：一种是利用 K8S 的 InitContainer，另一种是借助 K8S 的 shareProcessNamespace。InitContainer 的方案比较简单，InitContainer 第一个容器只做汇报子任务开始这一件事， 第二个容器则是真正执行子任务的容器，而业务容器只做汇报子任务结束这一件事，该方案利用 InitContainer 顺序且先于业务容器执行这两特点，并且若执行子任务的容器失败，则 Pod 也会创建失败，查询 Pod 状态即可知道子任务是否正常运行。而 shareProcessNamespace 的方案稍微复杂一些，同样使用一个 InitContainer 做汇报子任务开始这件事，而业务容器中放两个容器：一个主容器和一个 sidecar 容器（希望 K8S 原生支持的 SideCar 早日做好 ╯△╰），sidecar 容器中以轮询的方式监听主容器的运行状态（查询是否存在主进程）以及是否正常退出（获取容器退出码），并向 master 推送状态信息，该方案借助进程空间共享，使 sidecar 容器能直接查询主容器中的进程，从而达到监听主容器运行状态的目的，该方案的执行还需要一个小 trick，就是要让主容器先执行，由两种方案：一种是借助 postStart Hook，另一种是直接让 sidecar 容器先休眠个 10s 钟。关于 sidecar 容器的另外一种应用方案可参考 <a href="https://zhuanlan.zhihu.com/p/143845408">Nginx容器配置文件如何热更新？</a> 。</p><p> 虽然分布式任务调度框架有很多，eg：<a href="https://airflow.apache.org/">Airflow</a>、<a href="https://github.com/spotify/luigi">Luigi</a> 以及 <a href="https://dolphinscheduler.apache.org">DolphinScheduler</a> 等，但目前与 K8S 联系最紧密的应该就是 <a href="https://argoproj.github.io/">Argo</a> 了，其利用 K8S 的自定义资源对 K8S 已有功能进行扩展，仅使用 YAML 即可完成整个 pipeline 的任务调度和部署，虽然在并发任务调度时有一定的缺陷，但仅使用 YAML 表示其对 K8S 运维的足够友好性，对于常规 pipeline 式任务，Argo 已足以应付，除特殊需求外，程序员可少写很多代码。</p><h3 id="附录">附录</h3><p> 对于 Spring 编写的程序，在 K8S 中运行，在导出日志时可参考 <a href="https://www.cnblogs.com/varyuan/p/14243472.html">k8s:获取pod的ip</a>，通过 valueFrom 使用 Pod 的 metadata 作为环境变量，以区分日志的来源，不过挂载存储时最好还是用外部存储，用 hostPath 的话就需要保证每个物理节点都有相同的日志存储目录。</p><h2 id="后记">后记</h2><p> K8S 作为云原生时代的操作系统，不要求人人都完全掌握，但至少需要了解，知道什么该开发干，什么该运维干，这样才能充分发挥各个角色（包括 K8S）的价值。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 在周志明的『凤凰架构』中需要思考这样一个问题，如何用不可靠的部件来构造一个可靠的系统？对于程序员来说，写的代码从某种程度上来说都是不可靠的，但这些代码组成的一些系统却可以是可靠的。程序员对于错误的处理可以分为两派，一派是必须对错误进行处理，以保证系统的稳定行；另一派不对错误进行处理，任由程序 crash，只要有兜底方案，后面再不断完善。这两派并无孰优孰劣，只是两种不同的思维方式，甚至在同一个程序中，有些错误会处理，有些错误不会处理，这都是可能的。K8S 作为事实上的云原生操作系统，其目的就是为了将程序员写的各个程序组装成一个稳定的系统，并减少运维成本。</p></summary>
<category term="CloudNative" scheme="http://cniter.github.io/categories/CloudNative/"/>
<category term="k8s" scheme="http://cniter.github.io/tags/k8s/"/>
</entry>
<entry>
<title>OpenGL坐标系统与渲染管线</title>
<link href="http://cniter.github.io/posts/81321445.html"/>
<id>http://cniter.github.io/posts/81321445.html</id>
<published>2021-05-28T10:22:41.000Z</published>
<updated>2025-01-08T13:53:02.940Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 图形学中最基础的东西就是坐标系统，三维的东西如何在二维中显示，这中间经历了数次坐标变换，同时坐标变换也贯穿了整个计算机图形渲染管线。</p><span id="more"></span><h2 id="坐标篇">坐标篇</h2><figure><img src="https://learnopengl-cn.github.io/img/01/08/coordinate_systems.png" alt="coordinate_systems" /><figcaption aria-hidden="true">coordinate_systems</figcaption></figure><p> 在计算机图形世界中，为更灵活的控制三维物体显示在二维中，将变换的过程大致分为 5 个空间：1、局部空间（Local Space，或者称为物体空间（Object Space））；2、世界空间（World Space）；3、观察空间（View Space，或者称为视觉空间（Eye Space））；4、裁剪空间（Clip Space）；5、屏幕空间（Screen Space）。局部空间中是物体相对于坐标原点的坐标，也是物体的固有坐标，在依次经历过缩放旋转平移，也即模型矩阵（Model Matrix）变换后，物体局部坐标变换为世界坐标，世界坐标中即定义了物体所在的位置，以及产生的旋转和缩放。在世界空间中加入相机，以相机的视角看世界中的物体，即通过观察矩阵（View Matrix，也称视图矩阵）变换后，将世界坐标转换为观察坐标，由于一张屏幕能显示的东西是有限的，而三维世界中的物体是无限，所以需要通过投影矩阵（Projection Matrix）对三维空间进行裁剪，以决定哪些物体能显示在屏幕上，为方便的计算机判断，处于裁剪空间内的坐标会被转换为 [-1, 1]，为顺利在屏幕上显示，又需要通过视窗变换（Viewport Transform）将 [-1, 1] 映射为 viewport 中的图元坐标，再通过渲染管线的其他流程输出为屏幕上的像素点。</p><h2 id="变换篇">变换篇</h2><p> 矩阵相乘一般有左乘和右乘之分，左乘和右乘的区别在于坐标是按列还是按行排列（OpenGL 中是按列，所以是左乘，DX 中按行，所以是右乘，同一种变换，传入 DX 中的矩阵与传入 OpenGL 中的矩阵互为转置），坐标与矩阵相乘越靠近坐标的矩阵表示该坐标越先做相应矩阵变换。</p><p> 模型矩阵，视图矩阵，投影矩阵，在简单的顶点着色器编程中，这三个矩阵一般会合并成一个 MVP 矩阵传入 GPU 中。</p><h3 id="模型矩阵">模型矩阵</h3><p> 模型矩阵一般定义了物体的缩放旋转平移状态，缩放矩阵的构造很简单，若物体在 <span class="math inline">\((x,y,z)\)</span> 方向上缩放尺度分别为 <span class="math inline">\((S_x, S_y, S_z)\)</span>，则缩放矩阵为： <span class="math display">\[M_{scaling} = \begin{bmatrix} S_x & 0 & 0 & 0 \\ 0 & S_y & 0 & 0 \\ 0 & 0 & S_z & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}\]</span> 旋转矩阵就非常麻烦了，这里暂且不讨论其如何计算，只给出矩阵，物体绕任意轴 <span class="math inline">\((R_X, R_y, R_z)\)</span> 旋转 θ 角的矩阵为： <span class="math display">\[M_{rotation} = \begin{bmatrix} cos\theta+R_x^2(1-cos\theta) & R_xR_y(1-cos\theta)-R_zsin\theta & R_xR_z(1-cos\theta)+R_ysin\theta & 0 \\ R_yR_x(1-cos\theta)+R_zsin\theta & cos\theta+R_y^2(1-cos\theta) & R_yR_z(1-cos\theta)-R_xsin\theta & 0 \\ R_zR_x(1-cos\theta)-R_ysin\theta & R_zR_y(1-cos\theta)+R_xsin\theta & cos\theta+R_z^2(1-cos\theta) & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}\]</span> 当然，由于万向节锁的存在，一般不会直接使用欧拉角和旋转轴计算旋转矩阵，而是会通过四元数得到旋转矩阵，这样既高效又能避免万向节锁，详情可看「LearnOpenGL」译者的<a href="https://krasjet.github.io/quaternion/">教程</a>。</p><p> 至于平移矩阵也非常简单，若物体在 <span class="math inline">\((x,y,z)\)</span> 方向上平移量分别为 <span class="math inline">\((T_x, T_y, T_z)\)</span>，则平移矩阵为： <span class="math display">\[M_{translation} = \begin{bmatrix} 1 & 0 & 0 & T_x \\ 0 & 1 & 0 & T_y \\ 0 & 0 & 1 & T_z \\ 0 & 0 & 0 & 1 \end{bmatrix}\]</span> 前面的缩放和旋转矩阵其实只需要用到 3×3 的矩阵，而之所以用 4×4 的表示也是因为平移矩阵，普通的 3 维坐标必须增加一维 <span class="math inline">\(w\)</span> 构成齐次坐标才能进行平移操作，<span class="math inline">\(w\)</span> 一般都是 1.0，而从齐次坐标<span class="math inline">\((x,y,z,w)\)</span> 变为普通的 3 维坐标需要每个分量除以 <span class="math inline">\(w\)</span>，即 <span class="math inline">\((x/w, y/w, z/w)\)</span> 。</p><p>则模型矩阵 <span class="math inline">\(M_{model} = M_{translation} \cdot M_{rotation} \cdot M_{scaling}\)</span>。</p><h3 id="视图矩阵">视图矩阵</h3><p> 视图矩阵描述的是三维场景中模拟相机的状态，根据模拟相机的状态确定一套以相机为原点的相机坐标系，从而使用视图矩阵进行坐标变换，至于为啥是模拟相机，是因为 OpenGL 本身并没有相机的概念，通过模拟相机来实现在三维场景中的漫游。</p><figure><img src="https://learnopengl-cn.github.io/img/01/09/camera_axes.png" alt="camera_axes" /><figcaption aria-hidden="true">camera_axes</figcaption></figure><p> 模拟相机有三个关键点，分别为相机位置（cameraPos），相机朝向点（cameraTarget），相机上向量（top），根据相机位置和相机朝向点可确定相机坐标系的 z 轴正向向量 <span class="math inline">\(cameraDirection = (cameraPos - cameraTarget).normalize\)</span>，叉乘相机上向量和相机 z 轴正向向量可得到相机坐标系 x 轴正向向量 <span class="math inline">\(cameraRight = top.cross(cameraDirection).normalize\)</span>，最后将相机 z 轴正向向量与 x 轴正向向量叉乘得到 y 轴正向向量 <span class="math inline">\(cameraUp = cameraDirection.cross(cameraRight)\)</span>，如此即可建立完整的相机坐标系，从而得到变换矩阵，即视图矩阵： <span class="math display">\[M_{view} = \begin{bmatrix} R_x & R_y & R_z & 0 \\ U_x & U_y & U_z & 0 \\ D_x & D_y & D_z & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 & -P_x \\ 0 & 1 & 0 & -P_y \\ 0 & 0 & 1 & -P_z \\ 0 & 0 & 0 & 1 \end{bmatrix}\]</span> 其中 <span class="math inline">\(R\)</span> 是相机 x 轴正向向量，<span class="math inline">\(U\)</span> 是相机 y 轴正向向量，<span class="math inline">\(D\)</span> 是相机 z 轴正向向量， <span class="math inline">\(P\)</span> 是相机位置向量。</p><h3 id="投影矩阵">投影矩阵</h3><p> 投影矩阵描述的是摄像机前的可视区域（Frustum），根据可视区域的形状可分为正射投影（Orthographic Projection）和透视投影（Perspective Projection）。</p><p><img src="https://learnopengl-cn.github.io/img/01/08/orthographic_frustum.png" alt="orthographic projection frustum" /> <img src="https://learnopengl-cn.github.io/img/01/08/perspective_frustum.png" alt="perspective_frustum" /></p><p> 对于这两种投影，都有远（far）近（near）参数，不同的是，正射投影是个立方体，所以有左（left）右（right）上（top）下（bottom）四个参数，而透视投影是个类梯形台，所以还有垂直方向视野（Field of View，fov），以及一个宽高比（aspect）两个参数。远近两个参数决定摄像机能看到多近和多远的物体，太近和太远都会看不见，一般可设 near = 0.1，far = 1000；若渲染视窗（viewport）宽为 W，高为 H，则一般 <span class="math inline">\(left=-W/2, right=W/2, top=H/2, bottom=-H/2\)</span> ；透视投影的 fov 是角度，一般设为 45.0，而 <span class="math inline">\(aspect = W/H\)</span> 。这两种投影的矩阵分别为： <span class="math display">\[M_{orth} = \begin{bmatrix} \frac{2}{right-left} & 0 & 0 & -\frac{right+left}{right-left} \\ 0 & \frac{2}{top-bottom} & 0 & -\frac{top+bottom}{top-bottom} \\ 0 & 0 & \frac{-2}{far-near} & -\frac{far+near}{far-near} \\ 0 & 0 & 0 & 1 \end{bmatrix} \\M_{pers} = \begin{bmatrix} \frac{2near}{right-left} & 0 & \frac{right+left}{right-left} & 0 \\ 0 & \frac{2near}{top-bottom} & \frac{top+bottom}{top-bottom} & 0 \\ 0 & 0 & \frac{-(far+near)}{far-near} & \frac{-2far*near}{far-near} \\ 0 & 0 & -1 & 0 \end{bmatrix}\]</span></p><p> 在 three.js 中，对于透视投影矩阵中 left, right, top, bottom 计算方式为：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="JAVASCRIPT"><div class="code-copy"></div><figure class="highlight hljs javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> top = near * <span class="built_in">Math</span>.tan( _Math.DEG2RAD * <span class="number">0.5</span> * <span class="built_in">this</span>.fov ) / <span class="built_in">this</span>.zoom;</span><br><span class="line"><span class="keyword">let</span> height = <span class="number">2</span> * top;</span><br><span class="line"><span class="keyword">let</span> width = <span class="built_in">this</span>.aspect * height;</span><br><span class="line"><span class="keyword">let</span> left = - <span class="number">0.5</span> * width;</span><br><span class="line"><span class="keyword">let</span> right = left + width;</span><br><span class="line"><span class="keyword">let</span> bottom = top - height;</span><br></pre></td></tr></table></figure></div><p> 对于透视投影，由于计算出的齐次坐标 w 分量显然不为 1.0，所以必须进行透视除法（x,y,z 各分量分别除以 w），得到真正的 3 维坐标。</p><p> 正射投影一般用来模拟 2D 空间，透视投影用来模拟 3D 空间，当透视投影 near 和 far 设置的相差太大时，很容易引发 z-fighting 现象，原因是离近平面越远时，计算出的深度精度越低，three.js 中为解决这一问题，引入了一个 logarithmicDepthBuffer 参数来决定是否开启使用对数函数优化深度计算，具体可看源码中的 logdepthbuf_vertex.glsl.js 和 logdepthbuf_fragment.glsl.js 文件，开启该参数会造成渲染性能下降。</p><h3 id="小结">小结</h3><p> <span class="math inline">\(M_{mvp} = M_{projection}M_{view}M_{model}\)</span>，一个局部坐标 <span class="math inline">\(V_{local}\)</span> 在经过 MVP 矩阵变换之后可得到裁剪坐标 <span class="math inline">\(V_{clip} = M_{mvp}V_{local}\)</span> ，在 OpenGL 中，<span class="math inline">\(V_{clip}\)</span> 会被赋值到顶点着色器中的 <code>gl_Position</code>，并且 OpenGL 会自动进行透视除法和裁剪。</p><p> 3 维中的相机一般可分为两种，第一人称相机（常规 FPS 游戏）和第三人称相机（常规 ARPG 游戏），第一人称相机的特点是灵活，相机往往可以任意改变位置和朝向，所以会对某些人造成一种 “晕 3D” 的现象，而第三人称相机虽然可以改变相机朝向点和位置，但当朝向点和到朝向点的距离一旦固定，则相机只能沿着以朝向点为球心，以到朝向点的距离为半径的球面上运动，这两种相机一般看具体业务需求进行选择。</p><p> 缩放操作是很常规的一种操作，镜头拉近代表放大，拉远代表缩小。在使用透视投影的 3 维场景中，只需要改变相机到朝向点的距离即可简单实现缩放操作，而在使用正射投影的场景中，改变距离并不能实现缩放，而是需要改变 左右上下 四个参数，所以在相机中往往会在引入一个 zoom 的参数，用 左右上下 四个参数分别除以 zoom 得到真正的 左右上下，从而改变 zoom，就可以改变相机参数，进而实现正射投影的缩放。</p><h2 id="管线篇">管线篇</h2><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" id="processonSvg1000" viewBox="-14.0 -19.5 759.0 557.625" width="600" height="500"><defs id="ProcessOnDefs1001"><marker id="ProcessOnMarker1009" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1010" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1017" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1018" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1025" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1026" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1038" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1039" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1047" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1048" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1055" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1056" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1063" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1064" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1071" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1072" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1083" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1084" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1089" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1090" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1105" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1106" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1116" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1117" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1122" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1123" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker><marker id="ProcessOnMarker1138" markerUnits="userSpaceOnUse" orient="auto" markerWidth="16.23606797749979" markerHeight="10.550836550532098" viewBox="-1.0 -1.3763819204711736 16.23606797749979 10.550836550532098" refX="-1.0" refY="3.8990363547948754"><path id="ProcessOnPath1139" d="M12.0 3.8990363547948754L0.0 7.798072709589751V0.0Z" stroke="#999" stroke-width="2.0" fill="#999" transform="matrix(1.0,0.0,0.0,1.0,0.0,0.0)"></path></marker></defs><g id="ProcessOnG1002"><path id="ProcessOnPath1003" d="M-14.0 -19.5H745.0V538.125H-14.0V-19.5Z" fill="none"></path><g id="ProcessOnG1004"><g id="ProcessOnG1005" transform="matrix(1.0,0.0,0.0,1.0,6.0,122.0)" opacity="1.0"><path id="ProcessOnPath1006" d="M0.0 39.5C0.0 -13.166666666666666 84.0 -13.166666666666666 84.0 39.5C84.0 92.16666666666667 0.0 92.16666666666667 0.0 39.5Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1007"><path id="ProcessOnPath1008" d="M90.0 161.5L137.3829796415478 161.5" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1009)"></path></g><g id="ProcessOnG1011" transform="matrix(1.0,0.0,0.0,1.0,152.6190476190476,126.5)" opacity="1.0"><path id="ProcessOnPath1012" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1013" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1014" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="16" x="39.0" y="16.4">顶点着色器</text></g></g><g id="ProcessOnG1015"><path id="ProcessOnPath1016" d="M252.6190476190476 161.5L263.7639320225002 161.5" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1017)"></path></g><g id="ProcessOnG1019" transform="matrix(1.0,0.0,0.0,1.0,279.0,126.5)" opacity="1.0"><path id="ProcessOnPath1020" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1021" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1022" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="16" x="39.0" y="16.4">图元装配</text></g></g><g id="ProcessOnG1023"><path id="ProcessOnPath1024" d="M379.0 161.5L413.7639320225002 161.5" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1025)"></path></g><g id="ProcessOnG1027" transform="matrix(1.0,0.0,0.0,1.0,429.0,126.5)" opacity="1.0"><path id="ProcessOnPath1028" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1029" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1030" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="16" x="39.0" y="16.4">光栅器</text></g></g><g id="ProcessOnG1031" transform="matrix(1.0,0.0,0.0,1.0,24.5,150.0)" opacity="1.0"><path id="ProcessOnPath1032" d="M0.0 0.0L47.0 0.0L47.0 23.0L0.0 23.0Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1033" transform="matrix(1.0,0.0,0.0,1.0,0.0,-7.25)"><text id="ProcessOnText1034" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="22.5" y="15.375">顶点</text><text id="ProcessOnText1035" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="22.5" y="34.125">缓冲区</text></g></g><g id="ProcessOnG1036"><path id="ProcessOnPath1037" d="M529.0 161.5L548.5 161.5L548.5 161.5L552.7639320225002 161.5" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1038)"></path></g><g id="ProcessOnG1040" transform="matrix(1.0,0.0,0.0,1.0,568.0,126.5)" opacity="1.0"><path id="ProcessOnPath1041" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1042" transform="matrix(1.0,0.0,0.0,1.0,10.0,15.0)"><text id="ProcessOnText1043" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">片元着色</text><text id="ProcessOnText1044" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="36.4">器</text></g></g><g id="ProcessOnG1045"><path id="ProcessOnPath1046" d="M668.0 161.5L725.0 161.5L725.0 271.8354430379747L710.2360679774998 271.8354430379747" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1047)"></path></g><g id="ProcessOnG1049" transform="matrix(1.0,0.0,0.0,1.0,595.0,236.8354430379747)" opacity="1.0"><path id="ProcessOnPath1050" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1051" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1052" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">归属测试</text></g></g><g id="ProcessOnG1053"><path id="ProcessOnPath1054" d="M595.0 271.8354430379747L556.0 271.8354430379747L556.0 271.8354430379747L532.2360679774998 271.8354430379747" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1055)"></path></g><g id="ProcessOnG1057" transform="matrix(1.0,0.0,0.0,1.0,417.0,236.8354430379747)" opacity="1.0"><path id="ProcessOnPath1058" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1059" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1060" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">模板测试</text></g></g><g id="ProcessOnG1061"><path id="ProcessOnPath1062" d="M417.0 271.8354430379747L378.5 271.8354430379747L378.5 271.8354430379747L355.2360679774998 271.8354430379747" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1063)"></path></g><g id="ProcessOnG1065" transform="matrix(1.0,0.0,0.0,1.0,240.0,236.8354430379747)" opacity="1.0"><path id="ProcessOnPath1066" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1067" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1068" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">深度测试</text></g></g><g id="ProcessOnG1069"><path id="ProcessOnPath1070" d="M240.0 271.8354430379747L207.0 271.8354430379747L207.0 271.8354430379747L189.2360679774998 271.8354430379747" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1071)"></path></g><g id="ProcessOnG1073" transform="matrix(1.0,0.0,0.0,1.0,74.0,236.8354430379747)" opacity="1.0"><path id="ProcessOnPath1074" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1075" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1076" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">融合</text></g></g><g id="ProcessOnG1077" transform="matrix(1.0,0.0,0.0,1.0,74.0,444.25)" opacity="1.0"><path id="ProcessOnPath1078" d="M0.0 4.0Q0.0 0.0 4.0 0.0L96.0 0.0Q100.0 0.0 100.0 4.0L100.0 66.0Q100.0 70.0 96.0 70.0L4.0 70.0Q0.0 70.0 0.0 66.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1079" transform="matrix(1.0,0.0,0.0,1.0,10.0,25.0)"><text id="ProcessOnText1080" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="微软雅黑" text-anchor="middle" font-size="16" x="39.0" y="16.4">抖动</text></g></g><g id="ProcessOnG1081"><path id="ProcessOnPath1082" d="M74.0 271.8354430379747L20.0 271.8354430379747L20.0 479.25L58.763932022500214 479.25" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1083)"></path></g><g id="ProcessOnG1085" transform="matrix(1.0,0.0,0.0,1.0,250.05952380952385,440.375)" opacity="1.0"><path id="ProcessOnPath1086" d="M0.0 38.875C0.0 -12.958333333333334 79.88095238095235 -12.958333333333334 79.88095238095235 38.875C79.88095238095235 90.70833333333333 0.0 90.70833333333333 0.0 38.875Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1087"><path id="ProcessOnPath1088" d="M174.0 479.25L212.02976190476193 479.25L212.02976190476193 479.25L234.82345583202405 479.25" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1089)"></path></g><g id="ProcessOnG1091" transform="matrix(1.0,0.0,0.0,1.0,268.73809523809524,467.75)" opacity="1.0"><path id="ProcessOnPath1092" d="M0.0 0.0L47.0 0.0L47.0 23.0L0.0 23.0Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1093" transform="matrix(1.0,0.0,0.0,1.0,0.0,-7.25)"><text id="ProcessOnText1094" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="22.5" y="15.375">颜色</text><text id="ProcessOnText1095" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="22.5" y="34.125">缓冲区</text></g></g><g id="ProcessOnG1096" transform="matrix(1.0,0.0,0.0,1.0,624.0,3.500000000000014)" opacity="1.0"><path id="ProcessOnPath1097" d="M0.0 38.49999999999999C0.0 -12.83333333333333 82.0 -12.83333333333333 82.0 38.49999999999999C82.0 89.83333333333331 0.0 89.83333333333331 0.0 38.49999999999999Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1098" transform="matrix(1.0,0.0,0.0,1.0,639.2619047619048,30.16455696202533)" opacity="1.0"><path id="ProcessOnPath1099" d="M0.0 0.0L51.476190476190474 0.0L51.476190476190474 25.32911392405063L0.0 25.32911392405063Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1100" transform="matrix(1.0,0.0,0.0,1.0,0.0,-6.085443037974684)"><text id="ProcessOnText1101" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="24.738095238095237" y="15.375">纹理</text><text id="ProcessOnText1102" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="24.738095238095237" y="34.125">缓冲区</text></g></g><g id="ProcessOnG1103"><path id="ProcessOnPath1104" d="M665.0 80.5L665.0 103.5L618.0 103.5L618.0 111.26393202250021" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1105)"></path></g><g id="ProcessOnG1107" transform="matrix(1.0,0.0,0.0,1.0,252.6190476190476,346.0)" opacity="1.0"><path id="ProcessOnPath1108" d="M0.0 37.0C0.0 -12.333333333333334 74.76190476190476 -12.333333333333334 74.76190476190476 37.0C74.76190476190476 86.33333333333333 0.0 86.33333333333333 0.0 37.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1109" transform="matrix(1.0,0.0,0.0,1.0,264.26190476190476,370.33544303797464)" opacity="1.0"><path id="ProcessOnPath1110" d="M0.0 0.0L51.476190476190474 0.0L51.476190476190474 25.32911392405063L0.0 25.32911392405063Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1111" transform="matrix(1.0,0.0,0.0,1.0,0.0,-6.085443037974684)"><text id="ProcessOnText1112" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="24.738095238095237" y="15.375">深度</text><text id="ProcessOnText1113" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="24.738095238095237" y="34.125">缓冲区</text></g></g><g id="ProcessOnG1114"><path id="ProcessOnPath1115" d="M290.0 306.8354430379747L290.0 326.4177215189874L289.99999999999994 326.4177215189874L289.99999999999994 330.7639320225002" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1116)"></path></g><g id="ProcessOnG1118" transform="matrix(1.0,0.0,0.0,1.0,78.0,25.0)" opacity="1.0"><path id="ProcessOnPath1119" d="M0.0 39.5C0.0 -13.166666666666666 84.0 -13.166666666666666 84.0 39.5C84.0 92.16666666666667 0.0 92.16666666666667 0.0 39.5Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1120"><path id="ProcessOnPath1121" d="M120.0 104.0L120.0 161.5L137.3829796415478 161.5" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1122)"></path></g><g id="ProcessOnG1124" transform="matrix(1.0,0.0,0.0,1.0,78.75,48.5)" opacity="1.0"><path id="ProcessOnPath1125" d="M0.0 0.0L82.5 0.0L82.5 32.0L0.0 32.0Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1126" transform="matrix(1.0,0.0,0.0,1.0,0.0,-2.75)"><text id="ProcessOnText1127" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="40.25" y="15.375">uniform</text><text id="ProcessOnText1128" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="40.25" y="34.125">数据</text></g></g><g id="ProcessOnG1129" transform="matrix(1.0,0.0,0.0,1.0,529.0,0.5)" opacity="1.0"><path id="ProcessOnPath1130" d="M0.0 40.0C0.0 -13.333333333333334 82.0 -13.333333333333334 82.0 40.0C82.0 93.33333333333333 0.0 93.33333333333333 0.0 40.0Z" stroke="#999" stroke-width="2.0" stroke-dasharray="none" opacity="1.0" fill="none"></path></g><g id="ProcessOnG1131" transform="matrix(1.0,0.0,0.0,1.0,529.75,26.829113924050645)" opacity="1.0"><path id="ProcessOnPath1132" d="M0.0 0.0L82.5 0.0L82.5 32.0L0.0 32.0Z" stroke="#999" stroke-width="0.0" stroke-dasharray="none" opacity="1.0" fill="none"></path><g id="ProcessOnG1133" transform="matrix(1.0,0.0,0.0,1.0,0.0,-2.75)"><text id="ProcessOnText1134" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="40.25" y="15.375">uniform</text><text id="ProcessOnText1135" fill="#999" font-weight="normal" font-style="normal" text-decoration="none" font-family="Arial,宋体" text-anchor="middle" font-size="15" x="40.25" y="34.125">数据</text></g></g><g id="ProcessOnG1136"><path id="ProcessOnPath1137" d="M570.0 80.5L570.0 103.5L618.0 103.5L618.0 111.26393202250021" stroke="#999" stroke-width="2.0" stroke-dasharray="none" fill="none" marker-end="url(#ProcessOnMarker1138)"></path></g></g></g></svg><p> 渲染管线，图形学中最重要的概念之一，既然称之为管线，自然有像流水线一样的步骤，各个步骤具体做的事情如下：</p><ol type="1"><li>顶点着色器：负责将顶点数据进行坐标变换，该着色器中一般存在 MVP 矩阵，负责将三维坐标变换为二维坐标，该阶段也可以优化每个点的深度值，以便管线后续进行深度测试，也可以利用光照简单优化每个顶点的颜色；</li><li>图元装配：将输入的顶点数据进行组装，形成图元，常见的图元包括：点(GL_POINTS)、线(GL_LINES)、线条(GL_LINE_STRIP)、三角面(GL_TRIANGLES)，在该过程中，一般 GPU 会做一些裁剪和背面剔除等操作，以减少图元的数量，同时完成透视除法以进行屏幕映射；</li><li>光栅化：负责计算每个图元到屏幕像素点的映射。光栅化会计算每个图元所覆盖的片元，同时利用顶点属性插值计算每个片元的属性，片元可认为是候选像素，经过后续管线阶段即可变为真正的像素。</li><li>片元着色器：将光栅化得到的片元进行颜色计算。图形学中几乎所有的高级特效都会在这一步完成，光照计算，阴影处理，纹理，材质，统统在这一步进行处理；</li><li>归属测试：即测试片元所在位置是否位于当前上下文视窗内，若一个显示帧缓冲区视窗被另一个视窗所遮蔽，则剔除该部分片元。</li><li>模板测试：即测试片元是否满足一定条件（可大于或小于某个值等），若测试不满足，则剔除该该片元， OpenGL 可自行选择开启或关闭模板测试。</li><li>深度测试：用来测试片元的远近，远的片元被遮挡。在深度测试，若两片元深度值接近，则可能会引起 Z-fighting 现象，即像素闪烁，这是因为此时 GPU 无法确定该剔除哪个片元，导致这一帧可能绘制这个片元，下一帧绘制另一个片元。若开启 Alpha 测试，即启用透明度，则会在下一阶段进行 Alpha 混合，从而达到透明效果。</li><li>混合：将新生成的片元颜色和帧缓冲区中对应位置的颜色进行混合，得到像素颜色。</li><li>抖动：一种以牺牲分辨率为代价来增加颜色表示范围技术，从视觉效果上来看就是颜色过度更平滑。</li></ol><p> 以上这些阶段中，能完全被编程控制的也就顶点着色器和片元着色器两个阶段，其余阶段要么完全无法控制，要么只能通过已有的参数进行设置，当然也可以通过顶点着色器和片元着色器影响余下阶段，顶点着色器和片元着色器也统称 Shader 编程。</p><p> 有时候为了做更好看的特效，需要进行多次渲染，将上一次渲染的结果作为下一次渲染的输入，此时可以将颜色缓冲区作为一张纹理，并构造新的帧缓冲区，将该纹理作为输入，重新放进渲染管线中，这种操作方式也叫后期处理（Post Processing），虽然好看，但对 GPU 的负载很大，需要合理使用。</p><p> 对于渲染管线，Shaun 的理解也就到此为止了，非常粗浅，Shader 也只是刚入门的水平，Shaun 在图形学方面做的更多是降低 Draw-Call 和 CPU 层面的 Tessellation，以及 Geometry 上的事，对纹理材质颜色光照阴影等方面涉及的较少。</p><h2 id="后记">后记</h2><p> 虽然目前 OpenGL 已停止更新，但学习图形学编程，OpenGL 总是绕不过去（至少暂时以及未来很长一段时间都会是这样），而且图形学基础知识本质都是相同的，不管是 DirectX 还是 Vulkan，变的只是写法形式而已，数学知识总是在那里，两种 shader 也同样需要，所以了解这些东西还是有必要的。</p><h2 id="附录">附录</h2><h3 id="二维图像的图像透视投影变换">二维图像的图像透视投影变换</h3><p> 图像的透视投影变换常用于图像的矫正，OpenCV 中就有现成的 api（getPerspectiveTransform 和 warpPerspective），用于将不规整的四边形区域变换为规整的矩形区域。其基本的数学原理为，先构造一个投影变换等式： <span class="math display">\[\begin{bmatrix} XW \\ YW \\ W \end{bmatrix} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}\]</span> 设四边形中四个点分别为 <span class="math inline">\((X_1, Y_1),(X_2, Y_2),(X_3, Y_3),(X_4, Y_4)\)</span> ，对应矩形中四个点为 <span class="math inline">\((x_1, y_1),(x_2, y_2),(x_3, y_3),(x_4, y_4)\)</span>。则可构造齐次线性方程组： <span class="math display">\[\begin{bmatrix} x_1 & y_1 & 1 & 0 & 0 & 0 & -X_1x_1 & -X_1y_1 \\ 0 & 0 & 0 & x_1 & y_1 & 1 & -Y_1x_1 & -Y_1y_1 \\ x_2 & y_2 & 1 & 0 & 0 & 0 & -X_2x_2 & -X_2y_2 \\ 0 & 0 & 0 & x_2 & y_2 & 1 & -Y_2x_2 & -Y_2y_2 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ x_n & y_n & 1 & 0 & 0 & 0 & -X_nx_n & -X_ny_n \\ 0 & 0 & 0 & x_n & y_n & 1 & -Y_nx_n & -Y_ny_n \end{bmatrix} \begin{bmatrix} a \\ b \\ c \\ d \\ e \\ f \\ g \\ h \end{bmatrix} = \begin{bmatrix} X_1 \\ Y_1 \\ X_2 \\ Y_2 \\ \vdots \\ X_n \\ Y_n \end{bmatrix}\]</span> 解这个方程组得到 abcdefg ，使用上面的投影变换等式可计算 <span class="math inline">\(X = XW / W, Y = YW / W\)</span> ，从而使用插值得到规整矩形图形的各个像素值。</p><h3 id="shader-学习资料">Shader 学习资料</h3><p>shader 入门书：https://thebookofshaders.com，在线编写 shader ：https://thebookofshaders.com/edit.php</p><p>glslsandbox 网站：http://glslsandbox.com/</p><p>shadertoy 网站：https://www.shadertoy.com/</p><p>threejs shader 系列教程：https://www.cnblogs.com/heymar/category/2432299.html</p><h2 id="参考资料">参考资料</h2><p>[1] <a href="https://learnopengl-cn.github.io/01%20Getting%20started/08%20Coordinate%20Systems/">坐标系统</a>（https://learnopengl-cn.github.io）</p><p>[2] <a href="http://www.yanhuangxueyuan.com/webgl_course/hardware.html">WebGL图形系统、渲染管线_郭隆邦技术博客</a></p><p>[3] <a href="http://www.songho.ca/opengl/gl_projectionmatrix.html">OpenGL Projection Matrix</a></p><p>[4] <a href="https://www.cnblogs.com/dojo-lzz/p/11250327.html">WebGL着色器32位浮点数精度损失问题</a></p><p>[5] <a href="https://stackoverflow.com/questions/3190483/transform-quadrilateral-into-a-rectangle">Transform quadrilateral into a rectangle?</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 图形学中最基础的东西就是坐标系统，三维的东西如何在二维中显示，这中间经历了数次坐标变换，同时坐标变换也贯穿了整个计算机图形渲染管线。</p></summary>
<category term="Image&Graphic" scheme="http://cniter.github.io/categories/Image-Graphic/"/>
<category term="algorithm" scheme="http://cniter.github.io/tags/algorithm/"/>
</entry>
<entry>
<title>Scala 学习小结</title>
<link href="http://cniter.github.io/posts/8d3d87a2.html"/>
<id>http://cniter.github.io/posts/8d3d87a2.html</id>
<published>2021-02-16T16:12:28.000Z</published>
<updated>2021-12-18T11:54:13.936Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 最近要改行做大数据相关的东西了，经调研大数据开发的语言还是用 Scala 好，当然 Java 也可以，毕竟都运行在 JVM 上，不过 Java 也有很长时间没用过了，所以对于 Shaun 来说用 Scala 和 Java 的代价是一样的，都需要学习一下，所以决定用对大数据更友好的 Scala。</p><span id="more"></span><p> 以 Martin Odersky 14 年写的「Scala By Example」为参考，虽然是 14 年的，但 Scala 的基本语法还是没变的，就学习本身而言没问题，毕竟不兼容的只是更上层的 API，Shaun 学习用的 Scala 版本为 2.12.12。Alvin Alexander 的「Scala Cookbook, 2nd Edition」预计今年 8 月会出版，到时可能这本书用来入门更好，但 Shaun 不需要系统的学，就简单的能上手写出比较理想的 Scala 代码就行了。</p><h2 id="学习篇">学习篇</h2><h3 id="第一章入门基础">第一章：入门基础</h3><h4 id="helloworld">HelloWorld</h4><p> 由于「Scala By Example」第一章没啥内容，也为了在正式写 Scala 之前简单熟悉一下，这里先用「A Scala Tutorial for Java Programmers」简单上手一下，首先写个 HelloWorld，具体代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">HelloWorld</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> println(<span class="string">"Hello, world!"</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 和 C 语言类似，程序唯一入口函数都是 main 函数，但 Scala 的变量在前，声明的类型在后，相比常规的语言是有点奇怪了，但这种语法规则和 Typescript 一样，所以很容易接受，但其模板的表示就有点奇怪了，Array[String] 表示一个 String 类型的数组，即表示方法为 Array[T]，常规的模板方式为 <code>Array<T></code> 或 <code>T[]</code>，def 关键字用来定义一个函数，object 用来表示一个单例类，即在定义类的同时，又创建了一个类的实例。Scala 中没有 static 关键字，需要用 static 修饰的都放在 object 中即可。</p><h4 id="调用-java">调用 Java</h4><p>Scala 中默认已导入 java.lang 中的全部类，但其它类需要显式导入，以格式化输出本地日期为例：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> java.util.{<span class="type">Date</span>, <span class="type">Locale</span>}</span><br><span class="line"><span class="keyword">import</span> java.text.<span class="type">DateFormat</span>._</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">LocalDate</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> now = <span class="keyword">new</span> <span class="type">Date</span></span><br><span class="line"> <span class="keyword">val</span> df = getDateInstance(<span class="type">LONG</span>, <span class="type">Locale</span>.<span class="type">CHINA</span>)</span><br><span class="line"> println(df format now) <span class="comment">// df format(now)</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> Scala 中的导入和 java 中 import 基本一样，但功能更强大，可以使用 <code>{}</code> 导入部分，也使用 <code>_</code> 导入全部（java 导入全部为 <code>*</code>，这不一样），当一个函数只有一个参数，可以通过 <em>空格+参数</em> 的形式调用，而不需要使用 <em>括号包裹</em> 的形式。这里采用 <code>val</code> 关键字声明的是常量，而要声明变量需要用 <code>var</code>。</p><h4 id="对象">对象</h4><p>Scala 中万物皆对象，一个数字也是一个对象，一个函数也是一个对象，具体如下图：</p><figure><img src="http://coredumper.cn/wordpress/wp-content/uploads/2017/06/MacHi-2017-06-03-17-18-18.png" alt="enter image description here" /><figcaption aria-hidden="true">enter image description here</figcaption></figure><p>以简单计时器函数为例：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Timer</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">oncePerSecond</span></span>(callback: () => <span class="type">Unit</span>) {</span><br><span class="line"> <span class="keyword">while</span> (<span class="literal">true</span>) {</span><br><span class="line"> callback();</span><br><span class="line"> <span class="type">Thread</span> sleep <span class="number">1000</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">timeFiles</span></span>() {</span><br><span class="line"> println(<span class="string">"time files like an arrow..."</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="comment">// oncePerSecond(timeFiles);</span></span><br><span class="line"> oncePerSecond(() => {</span><br><span class="line"> println(<span class="string">"time files like an arrow..."</span>);</span><br><span class="line"> });</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 这个和 Typescript 函数式编程的用法基本差不多，唯一不同这里声明的函数返回的是 <code>Unit</code> ，这个 Unit 可认为是无返回的函数，大部分情况等同于 void，在 Scala 中真正的没有值指的是 Nothing。</p><h4 id="类">类</h4><p>Scala 中同样有类，具体代码示例如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Complex</span>(<span class="params">real: <span class="type">Double</span>, imaginary: <span class="type">Double</span></span>) </span>{</span><br><span class="line"> <span class="comment">// def re() = real;</span></span><br><span class="line"> <span class="comment">// def im() = imaginary;</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">re</span> </span>= real;</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">im</span> </span>= imaginary;</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">toString</span></span>(): <span class="type">String</span> = <span class="string">""</span> + re + (<span class="keyword">if</span> (im < <span class="number">0</span>) <span class="string">""</span> <span class="keyword">else</span> <span class="string">"+"</span>) + im + <span class="string">"i"</span>;</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">ComplexNumbers</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> c = <span class="keyword">new</span> <span class="type">Complex</span>(<span class="number">1.2</span>, <span class="number">-3.4</span>);</span><br><span class="line"> <span class="comment">// println("real part: " + c.re() + " imaginary part: " + c.im());</span></span><br><span class="line"> println(c.toString());</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 在 Scala 中所有类都会继承某个父类，若没有显式声明父类，则默认继承 scala.AnyRef 类，如上面的 Complex 类，若需要覆盖父类的函数，则需要在函数声明前加上 override 关键字。当函数没有参数时，可以不用加括号，在调用时也不用加括号，如上面示例的注释和非注释的代码。</p><h4 id="模式匹配与条件类">模式匹配与条件类</h4><p> 接下来用 Scala 来写一个树结构表示表达式的示例代码，树的非叶节点表示操作符，叶子节点表示数值（这里为常量或变量），具体代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">Tree</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Sum</span>(<span class="params">l: <span class="type">Tree</span>, r: <span class="type">Tree</span></span>) <span class="keyword">extends</span> <span class="title">Tree</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Var</span>(<span class="params">n: <span class="type">String</span></span>) <span class="keyword">extends</span> <span class="title">Tree</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Const</span>(<span class="params">v: <span class="type">Int</span></span>) <span class="keyword">extends</span> <span class="title">Tree</span></span></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Expression</span> </span>{</span><br><span class="line"> <span class="class"><span class="keyword">type</span> <span class="title">Environment</span> </span>= <span class="type">String</span> => <span class="type">Int</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">eval</span></span>(t: <span class="type">Tree</span>, env: <span class="type">Environment</span>): <span class="type">Int</span> = t <span class="keyword">match</span> {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Sum</span>(l, r) => eval(l, env) + eval(r, env)</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Var</span>(n) => env(n)</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Const</span>(v) => v</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">derive</span></span>(t: <span class="type">Tree</span>, v: <span class="type">String</span>): <span class="type">Tree</span> = t <span class="keyword">match</span> {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Sum</span>(l, r) => <span class="type">Sum</span>(derive(l, v), derive(r, v))</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Var</span>(n) <span class="keyword">if</span> (v == n) => <span class="type">Const</span>(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">case</span> _ => <span class="type">Const</span>(<span class="number">0</span>)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> exp: <span class="type">Tree</span> = <span class="type">Sum</span>(<span class="type">Sum</span>(<span class="type">Var</span>(<span class="string">"x"</span>), <span class="type">Var</span>(<span class="string">"x"</span>)), <span class="type">Sum</span>(<span class="type">Const</span>(<span class="number">7</span>), <span class="type">Var</span>(<span class="string">"y"</span>))) </span><br><span class="line"> <span class="keyword">val</span> env: <span class="type">Environment</span> = {<span class="keyword">case</span> <span class="string">"x"</span> => <span class="number">5</span> <span class="keyword">case</span> <span class="string">"y"</span> => <span class="number">7</span>}</span><br><span class="line"> println(<span class="string">"Expression: "</span> + exp)</span><br><span class="line"> println(<span class="string">"Evalution with x=5, y=7: "</span> + eval(exp, env))</span><br><span class="line"> println(<span class="string">"Derivative relative to x:\n"</span> + derive(exp, <span class="string">"x"</span>))</span><br><span class="line"> println(<span class="string">"Derivative relative to y:\n"</span> + derive(exp, <span class="string">"y"</span>))</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 该示例主要用来说明两种 case 关键字，分别为：case class 和 ... match case ...，前者可认为是一个结构体，实例化时可以省略 new 关键字，参数有默认的 getter 函数，整个 case class 有默认的 equals 和 hashCode 方法实现，通过这两个方式可实现根据值判断类的两个实例是否相等，而不是通过引用，条件类同样有默认的 toString 方法实现；后者可认为是一种特殊的 switch case ，只不过 case 的判定和执行是函数式的，case class 可直接参与 match case 的判定（判定是不是属于该类）。第 7 行中有个 type 关键字，可认为是定义了一种新的类型（不是数据类型），示例中是函数类型，通过这个 type ，可直接将字符串映射为整型，23 行中将这个 type 与 case 结合使用，定义多个字符串映射多个整型的变量。第 18 行中有个 <code>_</code> ，这是 scala 中的通配符，不同的语义下表示的含义不同，这里的含义是指，当上面的模式都不匹配时，将执行这个，相当于 switch case 中的 default。</p><h4 id="scala-中的-trait">Scala 中的 trait</h4><p> 简单理解就是 Java 中的 Interface（接口），Scala 中没有 interface 关键字，但是 trait 比 Interface 的功能更多，其中可直接定义属性和方法的实现，Scala 中可通过 trait 来实现多重继承。下面的示例用 trait 简单实现了一个比较接口：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">trait</span> <span class="title">Ord</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title"><</span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title"><=</span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span> = (<span class="keyword">this</span> < that) || (<span class="keyword">this</span> == that)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">></span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span> = !(<span class="keyword">this</span> <= that)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">>=</span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span> = !(<span class="keyword">this</span> < that)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Date</span>(<span class="params">y: <span class="type">Int</span>, m: <span class="type">Int</span>, d: <span class="type">Int</span></span>) <span class="keyword">extends</span> <span class="title">Ord</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">year</span> </span>= y</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">month</span> </span>= m</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">day</span> </span>= d</span><br><span class="line"></span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">toString</span></span>(): <span class="type">String</span> = year + <span class="string">"-"</span> + month + <span class="string">"-"</span> + day</span><br><span class="line"></span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">equals</span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span> = {</span><br><span class="line"> that.isInstanceOf[<span class="type">Date</span>] && {</span><br><span class="line"> <span class="keyword">val</span> o = that.asInstanceOf[<span class="type">Date</span>]</span><br><span class="line"> o.day == day && o.month == month && o.year == year</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title"><</span></span>(that: <span class="type">Any</span>): <span class="type">Boolean</span> = {</span><br><span class="line"> <span class="keyword">if</span> (!that.isInstanceOf[<span class="type">Date</span>]) {</span><br><span class="line"> sys.error(<span class="string">"cannot compare "</span> + that + <span class="string">" and a Date"</span>)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> o = that.asInstanceOf[<span class="type">Date</span>]</span><br><span class="line"> (year < o.year) || (year == o.year && (month < o.month || (month == o.month && day < o.day)))</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Comparable</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> d1 = <span class="keyword">new</span> <span class="type">Date</span>(<span class="number">2021</span>, <span class="number">1</span>, <span class="number">3</span>);</span><br><span class="line"> <span class="keyword">val</span> d2 = <span class="keyword">new</span> <span class="type">Date</span>(<span class="number">2021</span>, <span class="number">1</span>, <span class="number">3</span>);</span><br><span class="line"></span><br><span class="line"> println(d1 < d2)</span><br><span class="line"> println(d1 <= d2)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 比较关系一般只需要确定 小于 和 等于 关系即可，其它关系都可由这两关系推出来，由于等于方法默认存在于所有对象中，所以只需要重写小于即可， 其它的比较方法都可以在 trait 中定义好。在上面的示例中有两个函数 isInstanceOf 和 asInstanceOf，前者用来判断对象是否是指定类型，后者用来将对象转换为指定类型，一般用在将父类转为子类时，在使用 asInstanceOf 之前一般需要先使用 isInstanceOf。</p><h4 id="泛型">泛型</h4><p> 这东西没啥好说的，基本有编程经验的或见过或用过，只是 Scala 的泛型语法确实有点奇怪就是了，可能也是为了函数式那些乱七八糟的操作符，具体示例代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Reference</span>[<span class="type">T</span>] </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">var</span> contents: <span class="type">T</span> = _</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">set</span></span>(value: <span class="type">T</span>) {</span><br><span class="line"> contents = value</span><br><span class="line"> }</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">get</span></span>: <span class="type">T</span> = contents</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">IntegerReference</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> cell = <span class="keyword">new</span> <span class="type">Reference</span>[<span class="type">Int</span>]</span><br><span class="line"> cell.set(<span class="number">13</span>)</span><br><span class="line"> println(<span class="string">"Reference contains the half of "</span> + (cell.get * <span class="number">2</span>))</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 这里同样有个 <code>_</code>，这里表示的是默认值，对于数字类型来说是 0，对于 boolean 来说是 false，对于 Unit（函数签名）来说是()（无参数无返回），对于其他来说是 null。</p><p>简单的了解 Scala 就到这里了。</p><hr /><h3 id="第二章快排">第二章：快排</h3><p>开场就是一个快排，示例代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">QuickSort</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">qSort</span></span>(xs: <span class="type">Array</span>[<span class="type">Int</span>]) {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">swap</span></span>(i: <span class="type">Int</span>, j: <span class="type">Int</span>) {</span><br><span class="line"> <span class="keyword">val</span> t = xs(i); xs(i) = xs(j); xs(j) = t;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">sort</span></span>(l: <span class="type">Int</span>, r: <span class="type">Int</span>) {</span><br><span class="line"> <span class="keyword">val</span> pivot = xs(l);</span><br><span class="line"> <span class="keyword">var</span> i = l+<span class="number">1</span>; <span class="keyword">var</span> j = r;</span><br><span class="line"> <span class="keyword">while</span> (i < j) {</span><br><span class="line"> <span class="keyword">while</span> (i <= r && xs(i) < pivot) i += <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">while</span> (j > l && xs(j) > pivot) j -= <span class="number">1</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (i < j) {</span><br><span class="line"> swap(i, j);</span><br><span class="line"> i += <span class="number">1</span>;</span><br><span class="line"> j -= <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (i > j) {</span><br><span class="line"> i = j;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">while</span> (i > l && xs(i) > pivot) {</span><br><span class="line"> i -= <span class="number">1</span>; j -= <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"> swap(i, l);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (l < j<span class="number">-1</span>) sort(l, j<span class="number">-1</span>);</span><br><span class="line"> <span class="keyword">if</span> (j+<span class="number">1</span> < r) sort(j+<span class="number">1</span>, r);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> sort(<span class="number">0</span>, xs.length<span class="number">-1</span>);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="comment">// val xs = Array(4, 1, 2, 5, 6);</span></span><br><span class="line"> <span class="comment">// val xs = Array(1, 2, 4, 4, 55, 5, 6);</span></span><br><span class="line"> <span class="comment">// val xs = Array(55, 6, 6);</span></span><br><span class="line"> <span class="keyword">val</span> xs = <span class="type">Array</span>(<span class="number">4</span>, <span class="number">1</span>, <span class="number">5</span>, <span class="number">7</span>,<span class="number">7</span>,<span class="number">7</span>,<span class="number">7</span>, <span class="number">2</span>, <span class="number">6</span>);</span><br><span class="line"> qSort(xs);</span><br><span class="line"> println(xs.mkString(<span class="string">" "</span>))</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 从这段快排代码可看出，Scala 支持函数嵌套和闭包，即在函数内部定义子函数，子函数可直接使用父函数的变量，同时，这里也简单说明一下 Scala 中数组的一些使用方法，用下标取数组元素时使用的是小括号 <code>()</code>，而不是其它语言常见的中括号 <code>[]</code>。当然 Scala 作为一种函数式语言，提供了非常多的函数式操作符，这篇也只会简单介绍。</p><h3 id="第三章actor">第三章：Actor</h3><p> Actor，Scala 中的多线程编程模型，下方的示例代码在 Scala 2.11 及之后的版本无法运行，因为 Actor 已从 Scala 库独立出来，见 <a href="https://stackoverflow.com/questions/29343770/object-actors-is-not-a-member-of-package-scala">object-actors-is-not-a-member-of-package-scala</a>。</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> scala.actors.<span class="type">Actor</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">AuctionMessage</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Offer</span>(<span class="params">bin: <span class="type">Int</span>, client: <span class="type">Actor</span></span>) <span class="keyword">extends</span> <span class="title">AuctionMessage</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Inquire</span>(<span class="params">client: <span class="type">Actor</span></span>) <span class="keyword">extends</span> <span class="title">AuctionMessage</span></span></span><br><span class="line"></span><br><span class="line"><span class="keyword">abstract</span> <span class="class"><span class="keyword">class</span> <span class="title">AuctionReply</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">Status</span>(<span class="params">asked: <span class="type">Int</span>, expire: <span class="type">Date</span></span>) <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">object</span> <span class="title">BestOffer</span> <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">BeatenOffer</span>(<span class="params">maxBid: <span class="type">Int</span></span>) <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">class</span> <span class="title">AuctionConCluded</span>(<span class="params">seller: <span class="type">Actor</span>, client: <span class="type">Actor</span></span>) <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">object</span> <span class="title">AuctionFailed</span> <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"><span class="keyword">case</span> <span class="class"><span class="keyword">object</span> <span class="title">AuctionOver</span> <span class="keyword">extends</span> <span class="title">AuctionReply</span></span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Auction</span>(<span class="params">seller: <span class="type">Actor</span>, minBid: <span class="type">Int</span>, closing: <span class="type">Date</span></span>) <span class="keyword">extends</span> <span class="title">Actor</span> </span>{</span><br><span class="line"> <span class="keyword">val</span> timeToShutdown = <span class="number">36000000</span> <span class="comment">// msec</span></span><br><span class="line"> <span class="keyword">val</span> bidIncrement = <span class="number">10</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">act</span></span>() {</span><br><span class="line"> <span class="keyword">var</span> maxBid = minBid - bidIncrement</span><br><span class="line"> <span class="keyword">var</span> maxBidder: <span class="type">Actor</span> = <span class="literal">null</span></span><br><span class="line"> <span class="keyword">var</span> running = <span class="literal">true</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">while</span> (running) {</span><br><span class="line"> receiveWithin ((closing.getTime() - <span class="keyword">new</span> <span class="type">Date</span>().getTime())) {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Offer</span>(bid, client) => {</span><br><span class="line"> <span class="keyword">if</span> (bid >= maxBid + bidIncrement) {</span><br><span class="line"> <span class="keyword">if</span> (maxBid >= minBid) maxBidder ! <span class="type">BeatenOffer</span>(bid)</span><br><span class="line"> maxBid = bid; maxBidder = client; client ! <span class="type">BestOffer</span></span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> client ! <span class="type">BeatenOffer</span>(maxBid)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Inquire</span>(client) => {</span><br><span class="line"> client ! <span class="type">BeatenOffer</span>(maxBid)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">case</span> <span class="type">TIMEOUT</span> => {</span><br><span class="line"> <span class="keyword">if</span> (maxBid >= minBid) {</span><br><span class="line"> <span class="keyword">val</span> reply = <span class="type">AuctionConCluded</span>(seller, maxBidder)</span><br><span class="line"> maxBidder ! reply; seller ! reply</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> seller ! <span class="type">AuctionFailed</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> receiveWithin(timeToShutdown) {</span><br><span class="line"> <span class="keyword">case</span> <span class="type">Offer</span>(_, client) => client ! <span class="type">AuctionOver</span></span><br><span class="line"> <span class="keyword">case</span> <span class="type">TIMEOUT</span> => running = <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">HelloActor</span> <span class="keyword">extends</span> <span class="title">Actor</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">act</span></span>() {</span><br><span class="line"> <span class="keyword">while</span> (<span class="literal">true</span>) {</span><br><span class="line"> receive {</span><br><span class="line"> <span class="keyword">case</span> name: <span class="type">String</span> => println(<span class="string">"Hello, "</span> + name)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">AuctionService</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> seller: <span class="type">Actor</span> = <span class="keyword">new</span> <span class="type">HelloActor</span></span><br><span class="line"> <span class="keyword">val</span> client: <span class="type">Actor</span> = <span class="keyword">new</span> <span class="type">HelloActor</span></span><br><span class="line"> <span class="keyword">val</span> minBid = <span class="number">10</span></span><br><span class="line"> <span class="keyword">val</span> closing = <span class="keyword">new</span> <span class="type">Date</span>()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> helloActor = <span class="keyword">new</span> <span class="type">HelloActor</span></span><br><span class="line"> helloActor.start()</span><br><span class="line"> helloActor ! <span class="string">"leo"</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 通过重写 Actor 中的 <code>act</code> 方法即可简单的实现多线程编程，Actor 中有个特殊的标识符 <code>!</code>，该符号其实是是一种缩写，即可将 <code>helloActor.!("leo")</code> 缩写为 <code>helloActor ! "leo"</code>，代表将数据传递给 Actor，由 Actor 内部的 <code>receive case</code> 接受数据并处理，当然也可通过 <code>receiveWithin</code> 控制数据传递时间，若超时，则默认触发 <code>TIMEOUT</code> 处理模式。</p><h3 id="第四章表达式与简单函数">第四章：表达式与简单函数</h3><p>该章主要有两个例子：1、牛顿法求平方根；2、尾递归，具体如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Sqrt</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">sqrt</span></span>(x: <span class="type">Double</span>): <span class="type">Double</span> = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">sqrtIter</span></span>(guess: <span class="type">Double</span>, x: <span class="type">Double</span>): <span class="type">Double</span> = {</span><br><span class="line"> <span class="keyword">if</span> (isGoodEnough(guess, x)) guess</span><br><span class="line"> <span class="keyword">else</span> sqrtIter(improve(guess, x), x)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">improve</span></span>(guess: <span class="type">Double</span>, x: <span class="type">Double</span>) = {</span><br><span class="line"> (guess + x / guess) / <span class="number">2</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">isGoodEnough</span></span>(guess: <span class="type">Double</span>, x: <span class="type">Double</span>) = (guess * guess - x).abs < <span class="number">0.001</span> <span class="comment">// guess * guess == x</span></span><br><span class="line"></span><br><span class="line"> sqrtIter(<span class="number">1.0</span>, x)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">TailRecursion</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">gcd</span></span>(a: <span class="type">Int</span>, b: <span class="type">Int</span>): <span class="type">Int</span> = <span class="keyword">if</span> (b == <span class="number">0</span>) a <span class="keyword">else</span> gcd(b, a % b)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">facorial</span></span>(n: <span class="type">Int</span>): <span class="type">Int</span> = <span class="keyword">if</span> (n == <span class="number">0</span>) <span class="number">1</span> <span class="keyword">else</span> n * facorial(n<span class="number">-1</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">facorialTail</span></span>(n: <span class="type">Int</span>): <span class="type">Int</span> = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">facorialIter</span></span>(n: <span class="type">Int</span>, res: <span class="type">Int</span>): <span class="type">Int</span> = {</span><br><span class="line"> <span class="keyword">if</span> (n == <span class="number">0</span>) res</span><br><span class="line"> <span class="keyword">else</span> facorialIter(n<span class="number">-1</span>, res * n)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> facorialIter(n, <span class="number">1</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">SimpleFunc</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> sqrtValue = <span class="type">Sqrt</span>.sqrt(<span class="number">0.01</span>)</span><br><span class="line"> println(sqrtValue)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> gcdValue = <span class="type">TailRecursion</span>.gcd(<span class="number">14</span>,<span class="number">21</span>)</span><br><span class="line"> println(gcdValue)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> facorialValue = <span class="type">TailRecursion</span>.facorial(<span class="number">5</span>)</span><br><span class="line"> println(facorialValue)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> facorialTailValue = <span class="type">TailRecursion</span>.facorialTail(<span class="number">5</span>)</span><br><span class="line"> println(facorialTailValue)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 由于并没有引入新的语法，就简单聊聊这两个例子吧。牛顿法求平方根主要在于构造一个特殊的二分函数 <span class="math inline">\(y_{i+1} = (y_i + x / y_i)/2, i=0,1,2,3,..., y_0=1\)</span> ，如此迭代，直到 <span class="math inline">\(|y_i^2-x| < \epsilon\)</span> ，得到 <span class="math inline">\(y_i\)</span> 即为 x 的平方根，更朴素一点的求多次方根就是利用二分法，分 [0, 1] 和 [1, +∞] 两个区间即可，对应从 [x, 1] 和 [1, x] 开始二分取值。至于尾递归，以前简单的写过一点，即最后递归调用原函数时，原函数不会再参与任何计算表达式。尾递归的好处在于当编译器或解释器支持尾递归时，将不会产生普通递归时的压栈操作，即不用担心递归层次太深，尾递归将类似循环迭代处理。</p><h3 id="第五章高阶函数">第五章：高阶函数</h3><p> 高阶函数（First-Class Functions），支持以函数作为参数或返回值，也可将函数赋值给其它变量，由此也可引出闭包和柯里化，闭包是指将内嵌函数作为返回值，而柯里化是指将多个参数分解为独立参数传递给函数，如：<span class="math inline">\(f(args_1,args_2,...,args_n)=f(args_1)(args_2)(...)(args_n)\)</span>。下面以求函数的不动点为例：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">FirstClassFunctions</span> </span>{</span><br><span class="line"> <span class="keyword">val</span> tolerance = <span class="number">0.0001</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">isCloseEnough</span></span>(x: <span class="type">Double</span>, y: <span class="type">Double</span>) = ((x-y) / x).abs < tolerance</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">fixedPoint</span></span>(f: <span class="type">Double</span> => <span class="type">Double</span>)(firstGuess: <span class="type">Double</span>) = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">iterate</span></span>(guess: <span class="type">Double</span>): <span class="type">Double</span> = {</span><br><span class="line"> <span class="keyword">val</span> next = f(guess)</span><br><span class="line"> <span class="keyword">if</span> (isCloseEnough(guess, next)) next</span><br><span class="line"> <span class="keyword">else</span> iterate(next)</span><br><span class="line"> }</span><br><span class="line"> iterate(firstGuess)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">averageDamp</span></span>(f: <span class="type">Double</span> => <span class="type">Double</span>)(x: <span class="type">Double</span>) = (x + f(x)) / <span class="number">2</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">sqrt</span></span>(x: <span class="type">Double</span>) = fixedPoint(averageDamp(y => x/y))(<span class="number">1.0</span>)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> println(sqrt(<span class="number">0.01</span>));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 该示例简单明了的展示了 Scala 中匿名函数，函数柯里化以及闭包。</p><h3 id="第六章类和对象">第六章：类和对象</h3><p>直接看下面的有理数示例吧，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 主构造函数</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">Rational</span>(<span class="params">n: <span class="type">Int</span>, d: <span class="type">Int</span></span>) <span class="keyword">extends</span> <span class="title">AnyRef</span> </span>{</span><br><span class="line"> <span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">gcd</span></span>(x: <span class="type">Int</span>, y: <span class="type">Int</span>): <span class="type">Int</span> = {</span><br><span class="line"> <span class="keyword">if</span> (x == <span class="number">0</span>) y</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span> (x < <span class="number">0</span>) gcd(-x, y)</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span> (y < <span class="number">0</span>) -gcd(x, -y)</span><br><span class="line"> <span class="keyword">else</span> gcd(y % x, x)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">val</span> g = gcd(n, d)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 构造函数重载（辅助构造函数）</span></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">this</span></span>() {</span><br><span class="line"> <span class="keyword">this</span>(<span class="number">0</span>, <span class="number">0</span>) <span class="comment">// 调用主构造函数</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> number: <span class="type">Int</span> = <span class="keyword">if</span> (g != <span class="number">0</span>) n / g <span class="keyword">else</span> <span class="number">0</span></span><br><span class="line"> <span class="keyword">val</span> denom: <span class="type">Int</span> = <span class="keyword">if</span> (g != <span class="number">0</span>) d / g <span class="keyword">else</span> <span class="number">0</span></span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">+</span></span>(that: <span class="type">Rational</span>) = <span class="keyword">new</span> <span class="type">Rational</span>(number * that.denom + that.number * denom, denom * that.denom)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">-</span></span>(that: <span class="type">Rational</span>) = <span class="keyword">new</span> <span class="type">Rational</span>(number * that.denom - that.number * denom, denom * that.denom)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">*</span></span>(that: <span class="type">Rational</span>) = <span class="keyword">new</span> <span class="type">Rational</span>(number * that.number, denom * that.denom)</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">/</span></span>(that: <span class="type">Rational</span>) = <span class="keyword">new</span> <span class="type">Rational</span>(number * that.denom, denom * that.number)</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">toNumber</span></span>: <span class="type">Double</span> = <span class="keyword">if</span> (denom != <span class="number">0</span>) number.toDouble / denom <span class="keyword">else</span> <span class="number">0.0</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">toString</span> </span>= <span class="string">""</span> + number + <span class="string">"/"</span> + denom</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Rational</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> rational = <span class="keyword">new</span> <span class="type">Rational</span>(<span class="number">2</span>,<span class="number">1</span>) / <span class="keyword">new</span> <span class="type">Rational</span>()</span><br><span class="line"> println(rational.toNumber);</span><br><span class="line"> println(rational.toString);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 从有理数这个示例可以看出，Scala 的类支持操作符重载，也支持构造函数重载，同样支持继承，多继承也是支持的，每个父类用 <code>with</code> 关键字分隔就行。</p><h3 id="第七章条件类和模式匹配">第七章：条件类和模式匹配</h3><p>大致和第一章内容差不多，就不重复写了。</p><h3 id="第八章泛型">第八章：泛型</h3><p> 大致也和第一章内容差不多，<em>值得一提的书中实现的泛型栈本质是一个链表，实现方法挺有意思的</em>。通过 <code><:</code> 标识符可约束泛型的类型，如 <code>[T <: P[T]]</code> 表明泛型 T 必须类型 P 的子类型。而标识符 <code><%</code> 比 <code><:</code> 约束性弱一点，只要 T 能够通过隐式类型变换为 P 即可。若想约束为父类型，则需使用 <code>>:</code> 标识符。</p><p> Scala 中有一种特殊的泛型，就是变化型注解，<code>trait List[+T]</code> 代表协变，表示当 B 类型是 A 类型子类时，<code>List[B]</code> 也可认为是 <code>List[A]</code> 的子类；<code>trait List[-T]</code> 代表逆变，当 B 类型是 A 类型子类时，<code>List[B]</code> 可认为是 <code>List[A]</code> 的父类。</p><p> Scala 中同样有元组，使用时也很方便，简单使用直接用括号声明即可，如 <code>def divmod(x: Int, y: Int): (Int, Int) = (x / y, x % y)</code>，该函数即返回一个元组，也可声明一个元组 <code>case class Tuple2[A, B](_1: A, _2: B)</code>，若需要取元组的元素可通过 <code>_i</code> 的方式，如 <code>val xy = divmod(3, 4); xy._1; xy._2;</code>，也可通过 match-case 语句取，如 <code>xy match { case (n, d) => println("quotient: " + n + ", rest: " + d) }</code>。</p><h3 id="第九章list">第九章：List</h3><p> Scala 中的 List 其实是数组结构，并且是不可变的，可认为是 C++ 里的静态数组，不能往其中添加或删除元素，下面用数组排序示例下 List 的用法：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">Sort</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">insertSort</span></span>(xsl: <span class="type">List</span>[<span class="type">Int</span>]): <span class="type">List</span>[<span class="type">Int</span>] = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">insert</span></span>(x: <span class="type">Int</span>, xs: <span class="type">List</span>[<span class="type">Int</span>]): <span class="type">List</span>[<span class="type">Int</span>] = {</span><br><span class="line"> xs <span class="keyword">match</span> {</span><br><span class="line"> <span class="comment">// case Nil => List(x)</span></span><br><span class="line"> <span class="keyword">case</span> <span class="type">List</span>() => <span class="type">List</span>(x)</span><br><span class="line"> <span class="keyword">case</span> y :: ys => <span class="keyword">if</span> (x <= y) x :: xs <span class="keyword">else</span> y :: insert(x, ys)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (xsl.isEmpty) <span class="type">Nil</span></span><br><span class="line"> <span class="keyword">else</span> insert(xsl.head, insertSort(xsl.tail))</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">mergeSort</span></span>[<span class="type">A</span>](less: (<span class="type">A</span>, <span class="type">A</span>) => <span class="type">Boolean</span>)(xs: <span class="type">List</span>[<span class="type">A</span>]): <span class="type">List</span>[<span class="type">A</span>] = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">merge</span></span>(xs1: <span class="type">List</span>[<span class="type">A</span>], xs2: <span class="type">List</span>[<span class="type">A</span>]): <span class="type">List</span>[<span class="type">A</span>] = {</span><br><span class="line"> <span class="keyword">if</span> (xs1.isEmpty) xs2</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span> (xs2.isEmpty) xs1</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">if</span> (less(xs1.head, xs2.head)) xs1.head :: merge(xs1.tail, xs2)</span><br><span class="line"> <span class="keyword">else</span> xs2.head :: merge(xs1, xs2.tail)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> n = xs.length / <span class="number">2</span></span><br><span class="line"> <span class="keyword">if</span> (n == <span class="number">0</span>) xs</span><br><span class="line"> <span class="keyword">else</span> merge(mergeSort(less)(xs take n), mergeSort(less)(xs drop n))</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> xs = <span class="type">List</span>(<span class="number">4</span>, <span class="number">1</span>, <span class="number">5</span>, <span class="number">7</span>,<span class="number">7</span>,<span class="number">7</span>,<span class="number">7</span>, <span class="number">2</span>, <span class="number">6</span>);</span><br><span class="line"> <span class="comment">// val xs = 3::2::1::1::Nil;</span></span><br><span class="line"> println(xs(<span class="number">0</span>), xs(<span class="number">1</span>), xs(xs.length<span class="number">-1</span>)) <span class="comment">// (4,1,6)</span></span><br><span class="line"> <span class="comment">// val ys = insertSort(xs);</span></span><br><span class="line"> <span class="keyword">val</span> ys = mergeSort((x: <span class="type">Int</span>, y: <span class="type">Int</span>) => x > y)(xs);</span><br><span class="line"> println(ys.mkString(<span class="string">" "</span>))</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> List 中有两个操作符非常类似，即 <code>::</code> 和 <code>:::</code>， 前者用于 List 中的元素和 List 连接，即创建一个新 List，新 List 为原 List 头插入元素后的 List，后者用于连接两个 List，即创建一个新 List ，新 List 为将第二个 List 的元素全部放入第一个 List 尾部的 List。字符 <code>Nil</code> 代表空 List 和 <code>List()</code> 等效，<code>head</code> 方法返回 List 的第一个元素，<code>tail</code> 方法返回除第一个元素之外的其它所有元素，还是一个 List，<code>isEmpty</code> 方法当 List 为空时返回 <code>true</code>。List 的 case-match 方法中，<code>case y :: ys</code> 其中 y 代表 xs.head，ys 代表 xs.tail。<code>(xs take n)</code> 表示取 List 前 n 个元素，<code>(xs drop n)</code> 表示取 List 前 n 个元素之外的元素，即与 (xs take n) 取得元素正好互补，而 <code>(xs split n)</code> 返回一个元组，元组中第一个元素为 (xs take n)，第二个元素为 (xs drop n)。关于 List 还有些更高阶得方法：filter，map, flatMap, reduceRight, foldRight 等方法就不继续写了。至于动态 List 可用 <code>ListBuffer</code> 结构，当然 Scala 中直接用 <code>Seq</code> 作为返回值和参数一般会更好些。</p><h3 id="第十章序列理解">第十章：序列理解</h3><p> Scala 中用来做序列理解的表达式是 <code>For-Comprehensions</code>，具体示例如下：<code>for (p <persons if p.age > 20) yield p.name</code> 相当于 <code>persons filter (p => p.age > 20) map (p => p.name)</code>，可以简单认为 for-yield 方法是 filter 和 map 的集合体。下面具体用个 N-皇后（特例是 8 皇后）的示例来具体说明：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">NQueen</span> </span>{</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">queens</span></span>(n: <span class="type">Int</span>): <span class="type">List</span>[<span class="type">List</span>[<span class="type">Int</span>]] = {</span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">isSafe</span></span>(col: <span class="type">Int</span>, queenList: <span class="type">List</span>[<span class="type">Int</span>], delta: <span class="type">Int</span>): <span class="type">Boolean</span> = {</span><br><span class="line"> <span class="keyword">val</span> curRow = queenList.length<span class="number">-1</span> + delta</span><br><span class="line"> <span class="keyword">for</span> (row <- <span class="type">List</span>.range(<span class="number">0</span>, queenList.length)) {</span><br><span class="line"> <span class="keyword">val</span> queenCol = queenList(row)</span><br><span class="line"> <span class="keyword">val</span> queenRow = queenList.length<span class="number">-1</span> - row</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (queenCol == col) <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> <span class="keyword">if</span> (queenRow == curRow) <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> <span class="keyword">if</span> ((queenCol - col).abs == (queenRow - curRow).abs) <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">placeQueens</span></span>(k: <span class="type">Int</span>): <span class="type">List</span>[<span class="type">List</span>[<span class="type">Int</span>]] = {</span><br><span class="line"> <span class="keyword">if</span> (k == <span class="number">0</span>) <span class="type">List</span>(<span class="type">List</span>())</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">for</span> { </span><br><span class="line"> queens <- placeQueens(k<span class="number">-1</span>);</span><br><span class="line"> column <- <span class="type">List</span>.range(<span class="number">0</span>, n);</span><br><span class="line"> <span class="keyword">if</span> isSafe(column, queens, <span class="number">1</span>) </span><br><span class="line"> } <span class="keyword">yield</span> column :: queens</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> placeQueens(n)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]) {</span><br><span class="line"> <span class="keyword">val</span> queenList = queens(<span class="number">8</span>);</span><br><span class="line"> println(<span class="string">"queenCount: "</span> + queenList.length) <span class="comment">// 92</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p>for-yield 表达式中 for 中可以写多条语句，代表多重循环，第 5 行的 for 代表 for 循环，<code><-</code> 表示取 List 中的元素。</p><hr /><p> 剩下的几章就没啥特别要写的，重点就两个特性，一个是 Stream ，一个 Lazy，Stream 和 List 有点类似，主要区别在于 Stream 是即时返回的，算一个返回一个，而 List 一般是全部计算完再返回一个 List；Lazy 一般用作常量的修饰符，主要作用是只用该常量被用到时才赋值，否则一直为空，有点类似常见的先判空再取值的封装。</p><h2 id="后记">后记</h2><p> 曾看到过通过刷题去学习新语言的方式，一直都以为很粗暴，但这次照着「Scala By Example」敲下来，感觉还挺有效的，同时也巩固了一下基本的算法知识，后续再把 twitter 的 「Effective Scala」再看一下应该就差不多了。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 最近要改行做大数据相关的东西了，经调研大数据开发的语言还是用 Scala 好，当然 Java 也可以，毕竟都运行在 JVM 上，不过 Java 也有很长时间没用过了，所以对于 Shaun 来说用 Scala 和 Java 的代价是一样的，都需要学习一下，所以决定用对大数据更友好的 Scala。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="language" scheme="http://cniter.github.io/tags/language/"/>
</entry>
<entry>
<title>2020年小结</title>
<link href="http://cniter.github.io/posts/28416d7c.html"/>
<id>http://cniter.github.io/posts/28416d7c.html</id>
<published>2021-02-12T12:15:16.000Z</published>
<updated>2021-12-18T11:54:13.930Z</updated>
<content type="html"><![CDATA[<p> 社畜不易，行者多难，披荆斩棘，前路莫测，步履不停。</p><span id="more"></span><h2 id="前言">前言</h2><p> 20 年，也算是正式步入社畜生活的第一年，新鲜感自然少不了，但也没持续很长时间。这一年中基本都在学习，工作中生活中都在熟悉新事物新模式。</p><h2 id="技术篇">技术篇</h2><p> 开始独立负责项目，从无到有写完了一个产品，做了半个地图可视化项目，图形学相关知识从完全不会到熟练使用 Shader 做简单特效，学习新语言使用新工具，这就是 Shaun 过去一年在工作中的写照。</p><p> 在产品中，Shaun 基本独立完成了调研设计编码的全过程，这款网页版的 OpenDrive 路网编辑器，让 Shaun 基本熟悉的前端开发的主流框架和打包流程，甚至基于这款编辑器继续引申出两个 SDK，虽然开发模式和真正的前端有所区别（Shaun 是把 Typescript 当 C# 用的，将网页程序当客户端程序开发），但感觉现在的浏览器完全能撑的住，完全可以将更多的计算和存储任务直接在前端全部做完，但同时也感到了纯前端的无力，没有后端，前端网页能呈现的数据和效果确实有限，网页的内存有限制，webgl 渲染的三角形也有限制，只能做些小东西，大场景就很难施展。路网编辑器中涉及的前端技术栈也有很多，主要是现在无论开发一个什么应用，都不可能从语言最底层的 api 写起，总会用到别人写好的库，熟悉，吸收，再修改，用着用着就需要自己写了，从用轮子到造轮子，从而产生更多的轮子，也算是一种良性循环。</p><p> 半个地图可视化项目，主要用的 mapbox-gl + geoserver 显示地图，做完这个项目，同时也基本了解了国内的百度和高德两家的地图突然变好看了的原因，其背后的技术也同样源自于 mapbox，一家真正小而美的公司，定义了一套前端渲染地图的数据标准（Vector Tile），在非 3D 地图上，这套标准就是业内通用的标准了，如今的导航地图用的都是这套前端渲染技术，美观又高效。</p><h2 id="生活篇">生活篇</h2><p> 整个 20 年出去玩的时间也不多，工作地所在能玩的地方基本也玩的差不多了，大部分时间都是宅在屋里看电影，学技术，感觉就非常平淡，也没啥特别好说的。20年，开始学习理财，锻炼买入卖出的感觉，由于整个 20 年股市一片良好，以至于 Shaun 这个新手也赚了些钱，但由于本钱不多，赚的也非常有限，赚大钱的机会，要么拿不住，要么下不去手，最终都失之交臂，这样一来，赚的就更少了，不过，股市中赚到的钱终究只是个数字，到手的才是赚到的，没到手是赚是亏还不好说，作为新手而言，Shaun 也就当玩玩而已，亏也不多，主要是锻炼自己的感觉或承受能力，反正理财是一辈子的事，不急于这一时。</p><h2 id="总结">总结</h2><p> 生活一年如一日的平淡如水，依旧独自前行，由于疫情的原因，出去看看都嫌太麻烦，只能周边走走，着实无聊，好在工作上的东西对 Shaun 来说是新的知识，稍微有点挑战，每解决一个问题，总会带来一些成就感，冲淡些许无聊，可这成就感越来越少了，或许哪天成就感完全消失，就是 Shaun 换个新环境的时候。</p><div style="text-align:center; font-family: Allura, Consolas, Helvetica, Tahoma, Arial, Microsoft YaHei, 微软雅黑, SimSun, 宋体, Heiti, 黑体, sans-serif; font-size:1.3em; color:#4094c3; font-weight:700; margin:.5em auto;">20 年获得技能：<strong><em>触类旁通</em></strong><br />20 年获得成就：<strong><em>独挡一面</em></strong></div>]]></content>
<summary type="html"><p> 社畜不易，行者多难，披荆斩棘，前路莫测，步履不停。</p></summary>
<category term="Life" scheme="http://cniter.github.io/categories/Life/"/>
<category term="record" scheme="http://cniter.github.io/tags/record/"/>
</entry>
<entry>
<title>Linux服务器运维文档</title>
<link href="http://cniter.github.io/posts/b59e3e7b.html"/>
<id>http://cniter.github.io/posts/b59e3e7b.html</id>
<published>2021-02-09T12:06:27.000Z</published>
<updated>2024-12-30T16:24:32.281Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 记录一下服务器问题排查常用的一些命令。</p><span id="more"></span><h2 id="常用篇">常用篇</h2><h3 id="linux">Linux</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 只列出含 XXX 的文件</span></span><br><span class="line">ll | grep "XXX"</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 按列显示文件名</span></span><br><span class="line">ls -1</span><br><span class="line">ls -l | grep ^[^d] | awk '{print $9}'</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 返回进入当前目录之前的目录</span></span><br><span class="line">cd -</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 在文件中查找带 XXX 的行，并输出到 /tmp/99</span></span><br><span class="line">fgrep "XXX" a.txt > /tmp/99</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 在当前文件夹中查找带 XXX 的行，并输出到 /tmp/99</span></span><br><span class="line">fgrep "XXX" -r ./* > /tmp/99</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 显示前5行</span></span><br><span class="line">head -n 5 a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 显示倒数第5行</span></span><br><span class="line">tail -n 5 a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 显示第5行至末尾</span></span><br><span class="line">tail -n +5 a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 提取第二行 [linux系统中sed命令输出指定的行](https://www.cnblogs.com/superbaby11/p/16556602.html)</span></span><br><span class="line">sed -n '2p' a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 以;分隔每一行，并提取第一列和第三列</span></span><br><span class="line">awk -F ';' '{print $1,$3}' a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 以:分隔每一行，并提取第一列和第三列</span></span><br><span class="line">awk -F '[:]' '{print $1,$3}' a.txt</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看 8080 端口占用</span></span><br><span class="line">lsof -i:8080</span><br><span class="line">netstat -tnlp | grep :8080</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看系统运行状态</span></span><br><span class="line">top</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看一定时间内进程cpu占用情况</span></span><br><span class="line">pidstat</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看运行进程</span></span><br><span class="line">ps -ef</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看postgres数据库连接状态，并按cpu使用率排序</span></span><br><span class="line">ps -aux | grep postgres | sort -nrk 3,3</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看磁盘占用大小</span></span><br><span class="line">du -sh *</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看磁盘剩余空间</span></span><br><span class="line">df -h</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看程序被 killed 的原因</span></span><br><span class="line">dmesg | egrep -i -B100 'killed process'</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看 url 请求时间</span></span><br><span class="line">curl -o /dev/null -s -w %{time_namelookup}:%{time_connect}:%{time_starttransfer}:%{time_total} [url]</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看硬盘序列号</span></span><br><span class="line">sudo lshw -class disk | grep serial</span><br><span class="line"></span><br></pre></td></tr></table></figure></div><h4 id="正则表达式">正则表达式</h4><p>常用正则：<a href="https://ihateregex.io">i Hate Regex</a></p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">// 匹配 hello 之前的字符</span><br><span class="line">(.+(?=hello))</span><br><span class="line"></span><br><span class="line">// 匹配其他数字和英文字母但不匹配结尾的 2</span><br><span class="line">([a-zA-Z_0-9]+[^2])</span><br><span class="line"></span><br><span class="line">// 提取包含test以及time后的数字</span><br><span class="line">test[a-zA-Z0-9\-\_\=\|\ ]*time=([\d+])</span><br><span class="line"></span><br><span class="line">// 提取中括号里的内容</span><br><span class="line">[\[](.*?)[\]]</span><br></pre></td></tr></table></figure></div><h4 id="工具">工具</h4><ul><li><strong>crontab</strong>：设置定时任务工具；</li><li><strong>Socat</strong>：网络工具（透明代理，端口转发，文件传输等），<a href="https://zhuanlan.zhihu.com/p/347722248">新版瑞士军刀：socat</a></li></ul><h4 id="服务器之间文件传输">服务器之间文件传输</h4><p>参考资料：<a href="https://blog.csdn.net/AnChenliang_1002/article/details/131466784">Linux下的SCP指令详解</a></p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash"> 本地主机传输文件到远程主机</span></span><br><span class="line">scp [本地文件路径] [用户名]@[远程主机IP地址]:[目标路径]</span><br><span class="line"><span class="meta">#</span><span class="bash"> eg:</span></span><br><span class="line">scp file.txt [email protected]:/home/user/</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 远程主机传输文件到本地主机</span></span><br><span class="line">scp [用户名]@[远程主机IP地址]:[远程文件路径] [本地目标路径]</span><br><span class="line"><span class="meta">#</span><span class="bash"> eg:</span></span><br><span class="line">scp [email protected]:/home/user/file.txt /path/to/local/</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 传输本地主机整个目录到远程主机</span></span><br><span class="line">scp -r [本地目录路径] [用户名]@[远程主机IP地址]:[目标路径]</span><br><span class="line"><span class="meta">#</span><span class="bash"> eg:</span></span><br><span class="line">scp -r directory/ [email protected]:/home/user/</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 若远程主机的SSH服务器端口不是默认的22端口，则需要指定端口号</span></span><br><span class="line">scp -P [端口号] [本地文件路径] [用户名]@[远程主机IP地址]:[目标路径]</span><br></pre></td></tr></table></figure></div><h3 id="postgresql">PostgreSQL</h3><h4 id="编译安装">编译安装</h4><p><em>参考自：<a href="https://blog.csdn.net/GJ454221763/article/details/113700717">【CentOS7】PostgreSQL-10.3的安装</a></em></p><ol type="1"><li><p>安装编译工具：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">yum install -y vim lrzsz tree wget gcc gcc-c++ readline-devel zlib-devel</span><br></pre></td></tr></table></figure></div></li><li><p>进入/usr/local/目录下：<code>cd /usr/local</code></p></li><li><p>下载 tar 包：<code>curl -O https://ftp.postgresql.org/pub/source/v16.2/postgresql-16.2.tar.gz</code></p></li><li><p>解压：<code>tar -xzvf postgresql-16.2.tar.gz</code></p></li><li><p>编译安装：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">cd /usr/local/postgresql-16.2</span><br><span class="line">./configure --prefix=/usr/local/pgsql-16.2 # /usr/local/pgsql-16.2 为安装目录</span><br><span class="line">make && make install</span><br><span class="line"><span class="meta"></span></span><br><span class="line"><span class="meta">#</span><span class="bash"> Two thousand years later，出现「PostgreSQL installation complete.」代表安装成功</span></span><br></pre></td></tr></table></figure></div></li><li><p>配置系统环境变量：<code>vi /etc/profile</code></p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line"><span class="comment"># /etc/profile 文件末尾添加</span></span><br><span class="line"><span class="built_in">export</span> PGHOME=/usr/<span class="built_in">local</span>/pgsql-16.2</span><br><span class="line"><span class="built_in">export</span> PGDATA=<span class="variable">$PGHOME</span>/data</span><br><span class="line"><span class="built_in">export</span> LD_LIBRARY_PATH=<span class="variable">$PGHOME</span>/lib:<span class="variable">$LD_LIBRARY_PATH</span></span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PGHOME</span>/bin:<span class="variable">$PATH</span></span><br></pre></td></tr></table></figure></div></li><li><p>使配置文件立即生效：<code>source /etc/profile</code></p></li><li><p>创建数据库用户：<code>useradd -m -d /home/postgres postgres</code></p></li><li><p>切换到数据库用户：<code>su postgres</code></p></li><li><p>初始化数据库：<code>pg_ctl init -D /home/postgres/db_data</code></p></li><li><p>启动数据库：<code>pg_ctl start -D /home/postgres/db_data</code></p></li></ol><h4 id="自启动设置">自启动设置</h4><p>复制 PostgreSQL 自启动文件：<code>cp /usr/local/postgresql-16.2/contrib/start-scripts/linux /etc/init.d/postgresql</code></p><p>修改自启动文件：<code>vi /etc/init.d/postgresql</code>，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#! /bin/sh</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># chkconfig: 2345 98 02</span></span><br><span class="line"><span class="comment"># description: PostgreSQL RDBMS</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># This is an example of a start/stop script for SysV-style init, such</span></span><br><span class="line"><span class="comment"># as is used on Linux systems. You should edit some of the variables</span></span><br><span class="line"><span class="comment"># and maybe the 'echo' commands.</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Place this file at /etc/init.d/postgresql (or</span></span><br><span class="line"><span class="comment"># /etc/rc.d/init.d/postgresql) and make symlinks to</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc0.d/K02postgresql</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc1.d/K02postgresql</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc2.d/K02postgresql</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc3.d/S98postgresql</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc4.d/S98postgresql</span></span><br><span class="line"><span class="comment"># /etc/rc.d/rc5.d/S98postgresql</span></span><br><span class="line"><span class="comment"># Or, if you have chkconfig, simply:</span></span><br><span class="line"><span class="comment"># chkconfig --add postgresql</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Proper init scripts on Linux systems normally require setting lock</span></span><br><span class="line"><span class="comment"># and pid files under /var/run as well as reacting to network</span></span><br><span class="line"><span class="comment"># settings, so you should treat this with care.</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Original author: Ryan Kirkpatrick <[email protected]></span></span><br><span class="line"></span><br><span class="line"><span class="comment"># contrib/start-scripts/linux</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## EDIT FROM HERE</span></span><br><span class="line"></span><br><span class="line"><span class="comment">###### 上面不改 #####################</span></span><br><span class="line"><span class="comment"># Installation prefix</span></span><br><span class="line">prefix=/usr/<span class="built_in">local</span>/pgsql-16.2</span><br><span class="line"></span><br><span class="line"><span class="comment"># Data directory</span></span><br><span class="line">PGDATA=<span class="string">"/home/postgres/db_data"</span></span><br><span class="line"><span class="comment">###### 下面不改 #####################</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Who to run postgres as, usually "postgres". (NOT "root")</span></span><br><span class="line">PGUSER=postgres</span><br><span class="line"></span><br><span class="line"><span class="comment"># Where to keep a log file</span></span><br><span class="line">PGLOG=<span class="string">"<span class="variable">$PGDATA</span>/serverlog"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># It's often a good idea to protect the postmaster from being killed by the</span></span><br><span class="line"><span class="comment"># OOM killer (which will tend to preferentially kill the postmaster because</span></span><br><span class="line"><span class="comment"># of the way it accounts for shared memory). To do that, uncomment these</span></span><br><span class="line"><span class="comment"># three lines:</span></span><br><span class="line"><span class="comment">#PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj</span></span><br><span class="line"><span class="comment">#PG_MASTER_OOM_SCORE_ADJ=-1000</span></span><br><span class="line"><span class="comment">#PG_CHILD_OOM_SCORE_ADJ=0</span></span><br><span class="line"><span class="comment"># Older Linux kernels may not have /proc/self/oom_score_adj, but instead</span></span><br><span class="line"><span class="comment"># /proc/self/oom_adj, which works similarly except for having a different</span></span><br><span class="line"><span class="comment"># range of scores. For such a system, uncomment these three lines instead:</span></span><br><span class="line"><span class="comment">#PG_OOM_ADJUST_FILE=/proc/self/oom_adj</span></span><br><span class="line"><span class="comment">#PG_MASTER_OOM_SCORE_ADJ=-17</span></span><br><span class="line"><span class="comment">#PG_CHILD_OOM_SCORE_ADJ=0</span></span><br><span class="line"></span><br><span class="line"><span class="comment">## STOP EDITING HERE</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># The path that is to be used for the script</span></span><br><span class="line">PATH=/usr/<span class="built_in">local</span>/sbin:/usr/<span class="built_in">local</span>/bin:/sbin:/bin:/usr/sbin:/usr/bin</span><br><span class="line"></span><br><span class="line"><span class="comment"># What to use to start up postgres. (If you want the script to wait</span></span><br><span class="line"><span class="comment"># until the server has started, you could use "pg_ctl start" here.)</span></span><br><span class="line">DAEMON=<span class="string">"<span class="variable">$prefix</span>/bin/postgres"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># What to use to shut down postgres</span></span><br><span class="line">PGCTL=<span class="string">"<span class="variable">$prefix</span>/bin/pg_ctl"</span></span><br><span class="line"></span><br><span class="line"><span class="built_in">set</span> -e</span><br><span class="line"></span><br><span class="line"><span class="comment"># Only start if we can find postgres.</span></span><br><span class="line"><span class="built_in">test</span> -x <span class="variable">$DAEMON</span> ||</span><br><span class="line">{</span><br><span class="line"><span class="built_in">echo</span> <span class="string">"<span class="variable">$DAEMON</span> not found"</span></span><br><span class="line"><span class="keyword">if</span> [ <span class="string">"<span class="variable">$1</span>"</span> = <span class="string">"stop"</span> ]</span><br><span class="line"><span class="keyword">then</span> <span class="built_in">exit</span> 0</span><br><span class="line"><span class="keyword">else</span> <span class="built_in">exit</span> 5</span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment"># If we want to tell child processes to adjust their OOM scores, set up the</span></span><br><span class="line"><span class="comment"># necessary environment variables. Can't just export them through the "su".</span></span><br><span class="line"><span class="keyword">if</span> [ -e <span class="string">"<span class="variable">$PG_OOM_ADJUST_FILE</span>"</span> -a -n <span class="string">"<span class="variable">$PG_CHILD_OOM_SCORE_ADJ</span>"</span> ]</span><br><span class="line"><span class="keyword">then</span></span><br><span class="line">DAEMON_ENV=<span class="string">"PG_OOM_ADJUST_FILE=<span class="variable">$PG_OOM_ADJUST_FILE</span> PG_OOM_ADJUST_VALUE=<span class="variable">$PG_CHILD_OOM_SCORE_ADJ</span>"</span></span><br><span class="line"><span class="keyword">fi</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment"># Parse command line parameters.</span></span><br><span class="line"><span class="keyword">case</span> <span class="variable">$1</span> <span class="keyword">in</span></span><br><span class="line"> start)</span><br><span class="line"><span class="built_in">echo</span> -n <span class="string">"Starting PostgreSQL: "</span></span><br><span class="line"><span class="built_in">test</span> -e <span class="string">"<span class="variable">$PG_OOM_ADJUST_FILE</span>"</span> && <span class="built_in">echo</span> <span class="string">"<span class="variable">$PG_MASTER_OOM_SCORE_ADJ</span>"</span> > <span class="string">"<span class="variable">$PG_OOM_ADJUST_FILE</span>"</span></span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$DAEMON_ENV</span> <span class="variable">$DAEMON</span> -D '<span class="variable">$PGDATA</span>' >><span class="variable">$PGLOG</span> 2>&1 &"</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"ok"</span></span><br><span class="line">;;</span><br><span class="line"> stop)</span><br><span class="line"><span class="built_in">echo</span> -n <span class="string">"Stopping PostgreSQL: "</span></span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$PGCTL</span> stop -D '<span class="variable">$PGDATA</span>' -s"</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"ok"</span></span><br><span class="line">;;</span><br><span class="line"> restart)</span><br><span class="line"><span class="built_in">echo</span> -n <span class="string">"Restarting PostgreSQL: "</span></span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$PGCTL</span> stop -D '<span class="variable">$PGDATA</span>' -s"</span></span><br><span class="line"><span class="built_in">test</span> -e <span class="string">"<span class="variable">$PG_OOM_ADJUST_FILE</span>"</span> && <span class="built_in">echo</span> <span class="string">"<span class="variable">$PG_MASTER_OOM_SCORE_ADJ</span>"</span> > <span class="string">"<span class="variable">$PG_OOM_ADJUST_FILE</span>"</span></span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$DAEMON_ENV</span> <span class="variable">$DAEMON</span> -D '<span class="variable">$PGDATA</span>' >><span class="variable">$PGLOG</span> 2>&1 &"</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"ok"</span></span><br><span class="line">;;</span><br><span class="line"> reload)</span><br><span class="line"><span class="built_in">echo</span> -n <span class="string">"Reload PostgreSQL: "</span></span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$PGCTL</span> reload -D '<span class="variable">$PGDATA</span>' -s"</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"ok"</span></span><br><span class="line">;;</span><br><span class="line"> status)</span><br><span class="line">su - <span class="variable">$PGUSER</span> -c <span class="string">"<span class="variable">$PGCTL</span> status -D '<span class="variable">$PGDATA</span>'"</span></span><br><span class="line">;;</span><br><span class="line"> *)</span><br><span class="line"><span class="comment"># Print help</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"Usage: <span class="variable">$0</span> {start|stop|restart|reload|status}"</span> 1>&2</span><br><span class="line"><span class="built_in">exit</span> 1</span><br><span class="line">;;</span><br><span class="line"><span class="keyword">esac</span></span><br><span class="line"></span><br><span class="line"><span class="built_in">exit</span> 0</span><br></pre></td></tr></table></figure></div><hr /><p>接下来有两种方式：</p><p>一种是直接执行：<code>cd /etc/rc.d/init.d/ && chkconfig --add postgresql</code>；</p><p>一种是修改 <code>/etc/rc.d/rc.local</code> 文件：<code>vi /etc/rc.d/rc.local</code>，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/bin/bash</span></span><br><span class="line"><span class="comment"># THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># It is highly advisable to create own systemd services or udev rules</span></span><br><span class="line"><span class="comment"># to run scripts during boot instead of using this file.</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># In contrast to previous versions due to parallel execution during boot</span></span><br><span class="line"><span class="comment"># this script will NOT be run after all other services.</span></span><br><span class="line"><span class="comment">#</span></span><br><span class="line"><span class="comment"># Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure</span></span><br><span class="line"><span class="comment"># that this script will be executed during boot.</span></span><br><span class="line"> </span><br><span class="line"><span class="built_in">exec</span> 2> /tmp/rc.local.log <span class="comment"># send stderr from rc.local to a log file</span></span><br><span class="line"><span class="built_in">exec</span> 1>&2 <span class="comment"># send stdout to the same log file</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"rc.local starting..."</span> <span class="comment"># show start of execution</span></span><br><span class="line"><span class="built_in">set</span> -x</span><br><span class="line"></span><br><span class="line">touch /var/lock/subsys/<span class="built_in">local</span></span><br><span class="line"></span><br><span class="line"><span class="built_in">cd</span> /etc/rc.d/init.d/</span><br><span class="line">sudo sh postgresql start & <span class="comment"># 以root执行，不然可能会出现权限错误，&表示后台执行</span></span><br><span class="line"> </span><br><span class="line"><span class="comment"># 脚本执行完后也给个日志</span></span><br><span class="line"><span class="built_in">echo</span> <span class="string">"rc.local completed"</span></span><br></pre></td></tr></table></figure></div><p>添加可执行权限：<code>chmod a+x /etc/rc.d/rc.local</code>，最后查看一下 rc.local 服务是否启动：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">systemctl status rc-local.serives</span><br><span class="line"><span class="meta"> </span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 启动命令</span></span><br><span class="line">systemctl enable rc-local.service</span><br><span class="line">systemctl start rc-local.service</span><br><span class="line"><span class="meta"> </span></span><br><span class="line"><span class="meta">#</span><span class="bash"> 查看数据库服务</span></span><br><span class="line">ps -ef | grep postgres</span><br></pre></td></tr></table></figure></div><hr /><p>若要在容器中设置自启动，在没给容器提权的情况下，则需要第三种方式：将 <code>/etc/rc.d/init.d/postgresql</code> 放进 <code>/root/.bashrc</code> 中启动，<code>vi /root/.bashrc</code>，</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line"><span class="comment"># /root/.bashrc 文件末尾添加</span></span><br><span class="line"><span class="keyword">if</span> [ -f /etc/rc.d/init.d/postgresql ]; <span class="keyword">then</span></span><br><span class="line"> sh /etc/rc.d/init.d/postgresql start > /tmp/postgresql.start.log 2>&1</span><br><span class="line"><span class="keyword">fi</span></span><br></pre></td></tr></table></figure></div><p>原理是：docker 容器在启动时，会自动执行 <code>~/.bashrc</code> 文件，加载环境变量，当有其他命令在该文件时，也会一起执行。</p><p>当然，容器中自启动更普遍的方式应该是在镜像/容器中通过 CMD 或者 ENTRYPOINT 直接指定 shell 脚本启动执行。</p><h4 id="配置文件设置">配置文件设置</h4><p>PG 电子书：<a href="https://postgres-internals.cn/docs/">PostgreSQL 14 Internals</a></p><p>配置参数解析文档：<a href="https://postgresqlco.nf/doc/zh/param/">PostgresqlCO.NF: 人类的PostgreSQL配置</a></p><p>自动化参数调优：<a href="https://pgtune.leopard.in.ua/">PGTune</a></p><p>PG13 一个推荐的配置解析（SSD，48 核，128GB 内存，机器资源独占，混布相当于降低内存和 cpu）</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br></pre></td><td class="code"><pre><span class="line"># 允许任何机器连接。默认只允许本地连接</span><br><span class="line">listen_addresses = '*'</span><br><span class="line"></span><br><span class="line"># 数据库连接端口。默认为5432</span><br><span class="line">port = 5432</span><br><span class="line"></span><br><span class="line"># 最大允许512个连接。默认为100</span><br><span class="line">max_connections = 512</span><br><span class="line"></span><br><span class="line"># 锁超时20s。默认为0，不超时</span><br><span class="line">lock_timeout = 20000</span><br><span class="line"># sql超时60s。默认为0，不超时</span><br><span class="line">statement_timeout = 60000</span><br><span class="line"></span><br><span class="line"># 数据库用于缓存数据的使用内存大小，一般设置为系统内存的 25%～30%，不宜过大，最多不超过40%。默认为128MB</span><br><span class="line">shared_buffers = 64GB</span><br><span class="line"></span><br><span class="line"># 查询优化器可用的内存大小，只是预估，不实际使用，值越大，越倾向于索引扫描，一般设置为系统内存的30%～50%，最大不超过90%。默认为4GB</span><br><span class="line">effective_cache_size = 96GB</span><br><span class="line"></span><br><span class="line"># 数据库维护性操作使用的内存（eg：vacuum, create index等），若需加快维护速度，可临时增大该参数 set maintenance_work_mem = 2GB;。默认为64MB</span><br><span class="line">maintenance_work_mem = 64MB</span><br><span class="line"></span><br><span class="line"># 尚未写入磁盘的WAL数据的共享内存量，增大该值有利于提高写入性能，不建议太大，最多不超过 128MB。默认与shared_buffers一致</span><br><span class="line">wal_buffers = 16MB</span><br><span class="line"></span><br><span class="line"># 查询优化器中统计信息的详细程度，越大越详细，查询优化器的决策越好，但会增加 ANALYZE 耗时。默认为100</span><br><span class="line">default_statistics_target = 100</span><br><span class="line"></span><br><span class="line"># 查询优化器获取一个随机页的cost（相比于一个顺序扫描页（seq_page_cost=1）的cost为1），该值相对seq_page_cost越小，越倾向于索引扫描，但不可低于 seq_page_cost。默认为4</span><br><span class="line"># 默认值可以被想成把随机访问建模为比顺序访问慢 40 倍，而期望 90% 的随机读取会被内存缓存。</span><br><span class="line">random_page_cost = 1.1</span><br><span class="line"></span><br><span class="line"># 顺序扫描时并行 I/O 操作的最大数量。默认为1</span><br><span class="line">effective_io_concurrency = 8</span><br><span class="line"></span><br><span class="line"># 每个排序操作、哈希表等操作所能使用的内存大小，增大该值可以提高某些查询性能，但设置过高可能会导致内存耗尽。默认为4MB</span><br><span class="line">work_mem = 4MB</span><br><span class="line"></span><br><span class="line"># 是否为主共享内存区域请求巨型页，巨型页面的使用会导致更小的页面表以及花费在内存管理上的 CPU 时间更少，从而提高性能。默认为try</span><br><span class="line">huge_pages = try</span><br><span class="line"></span><br><span class="line"># 最大工作进程数，增加该值可以增加数据库并行处理能力，过大可能导致资源消耗过多，一般可以设置为CPU核数。默认为8</span><br><span class="line">max_worker_processes = 16</span><br><span class="line"></span><br><span class="line"># 并行查询的最大并行数。默认为2</span><br><span class="line">max_parallel_workers_per_gather = 4</span><br><span class="line"></span><br><span class="line"># 与 max_worker_processes 相同。默认为8</span><br><span class="line">max_parallel_workers = 16</span><br><span class="line"></span><br><span class="line"># 数据库维护性操作的最大并行数。默认为2</span><br><span class="line">max_parallel_maintenance_workers = 4</span><br><span class="line"></span><br><span class="line"># WAL级别，minimal<replica<logical，级别越高记录的WAL越详细，replica用于物理复制，logical用于逻辑复制。默认为replica</span><br><span class="line">wal_level = replica</span><br><span class="line"></span><br><span class="line"># 启用文件系统同步，确保即使系统发生崩溃或断电等异常情况，数据也不会丢失，在高写入负载下，会导致性能下降。默认为on</span><br><span class="line">fsync = on</span><br><span class="line"></span><br><span class="line"># 最小的 WAL 文件大小，WAL 文件用于确保数据的持久性和恢复能力。默认为80MB</span><br><span class="line">min_wal_size = 128MB</span><br><span class="line"></span><br><span class="line"># 最大的 WAL 文件大小，过小会导致频繁的 checkpoint，从而影响性能，过大则可能会占用过多存储空间。默认为1GB</span><br><span class="line">max_wal_size = 4GB</span><br><span class="line"></span><br><span class="line"># 控制checkpoint（用来保证内存数据和磁盘数据一致性和完整性）分散写入，值越大，越分散，写入耗时越长，系统负载越小，一般设置为0.7～0.9，对于写入较大的数据库，该值越大越好。默认为0.5</span><br><span class="line">checkpoint_completion_target = 0.9</span><br><span class="line"></span><br><span class="line">### --- 主从同步相关参数 ---</span><br><span class="line">## 主库设置</span><br><span class="line">## 确保 wal_level 为 replica或logical</span><br><span class="line"># 最大的从库连接数，需大于当前从库数。默认为10</span><br><span class="line">max_wal_senders = 10</span><br><span class="line"></span><br><span class="line"># WAL文件保留的最小磁盘空间。默认为0，不保留</span><br><span class="line">wal_keep_size = 1GB</span><br><span class="line"></span><br><span class="line"># 主库等待从库接收WAL文件后响应的超时时间。默认为60s</span><br><span class="line">wal_sender_timeout = 300s</span><br><span class="line"></span><br><span class="line"># 最大复制槽数量，和 max_wal_senders 相同。默认为10</span><br><span class="line">max_replication_slots = 10</span><br><span class="line"></span><br><span class="line">## 从库设置</span><br><span class="line"># 连接主库的信息</span><br><span class="line">primary_conninfo = "host=master-db-host port=5432 user=replicator password=pwd"</span><br><span class="line"></span><br><span class="line"># 指定主库的复制槽名称</span><br><span class="line">primary_slot_name = 'xxx'</span><br><span class="line"></span><br><span class="line"># 允许从库进行只读查询。默认为on</span><br><span class="line">hot_standby = on</span><br><span class="line"></span><br><span class="line"># 从库向主库发送状态信息的时间间隔（状态信息包括 WAL 接收器的状态、当前接收进度等数据，主数据库可以使用这些信息监控复制的健康状况和同步延迟）。默认为10s</span><br><span class="line">wal_receiver_status_interval = 10s</span><br><span class="line"></span><br><span class="line"># 允许从库向主库发送反馈信息，以减少查询延迟和 WAL 日志的删除（启用该配置需要确保有足够的磁盘空间，并定期监控主库的 WAL 文件状态）。默认为off</span><br><span class="line">hot_standby_feedback = on</span><br><span class="line"></span><br><span class="line"># 从库等待 WAL 发送的超时时间。默认为60s</span><br><span class="line">wal_receiver_timeout = 300s</span><br><span class="line"></span><br><span class="line">### --- log 相关参数 ---</span><br><span class="line"># 将日志输出到标准错误输出</span><br><span class="line">log_destination = 'stderr'</span><br><span class="line"># 启用日志收集器（按照 log_directory 和 log_filename 指定的路径保存）。默认为off</span><br><span class="line">logging_collector = on</span><br><span class="line"># 日志文件的存储目录</span><br><span class="line">log_directory = 'log'</span><br><span class="line"># 日志文件的命名格式</span><br><span class="line">log_filename = 'pgsql-%Y%m%d_%H%M%S.log'</span><br><span class="line"># 日志文件切分周期</span><br><span class="line">log_rotation_age = 1d</span><br><span class="line"># 不根据文件大小切分</span><br><span class="line">log_rotation_size = 0</span><br><span class="line"># 日志记录的最低级别</span><br><span class="line">log_min_messages = warning</span><br><span class="line"># 记录 SQL 语句的最低错误级别</span><br><span class="line">log_min_error_statement = error</span><br><span class="line"># 记录慢查询时间，单位毫秒，超过该值会记录到日志中。默认不记录</span><br><span class="line">log_min_duration_statement = 5000</span><br><span class="line"># 日志格式。默认只记录时间和进程id</span><br><span class="line">log_line_prefix = '<%m [%p] %r %u@%d> '</span><br><span class="line"># 记录等待锁时间超过deadlock_timeout的日志。默认为off，不记录</span><br><span class="line">log_lock_waits = on</span><br><span class="line"></span><br><span class="line">### --- autovacuum 相关参数（需根据表大小，表数据更新频率调整，系统资源） ---</span><br><span class="line"># 启用自动清理，需同时开启track_counts。默认为on</span><br><span class="line">autovacuum = on</span><br><span class="line"># 执行自动清理的最大并发数。默认为3</span><br><span class="line">autovacuum_max_workers = 3</span><br><span class="line"># 每分钟启动一次自动清理进程。默认为1min</span><br><span class="line">autovacuum_naptime = 1min</span><br><span class="line"># 当表中的死行数超过该阈值时，触发 VACUUM 操作。默认为50</span><br><span class="line">autovacuum_vacuum_threshold = 10000</span><br><span class="line"># 在表中插入的行数超过此阈值时，触发 VACUUM 操作。默认为1000</span><br><span class="line">autovacuum_vacuum_insert_threshold = 10000</span><br><span class="line"># 当表中有足够的变化（如插入、更新、删除）且行数超过该阈值时，触发 ANALYZE 操作以更新统计信息。默认为50</span><br><span class="line">autovacuum_analyze_threshold = 5000</span><br><span class="line"># 当表中死行数达到表行数的5%时触发 VACUUM。默认为0.2</span><br><span class="line">autovacuum_vacuum_scale_factor = 0.05</span><br><span class="line"># 在表中插入的行数超过5%时，触发 VACUUM 操作。默认为0.2</span><br><span class="line">autovacuum_vacuum_insert_scale_factor = 0.05</span><br><span class="line"># 当数据变化超过表大小的 5% 时，触发 ANALYZE 操作，更新表的统计信息。默认为0.1</span><br><span class="line">autovacuum_analyze_scale_factor = 0.05</span><br><span class="line"># 每次vacuum操作执行一定量的 I/O 操作后休眠的时间（毫秒），目的是限制自动清理操作对磁盘 I/O 的影响，避免过多的 I/O 操作导致系统性能下降，可增加该值以减少对系统性能的影响。默认是2ms，需与autovacuum_vacuum_cost_limit配合使用</span><br><span class="line">autovacuum_vacuum_cost_delay = 20ms</span><br><span class="line"># 每次vacuum操作的最大 I/O 成本。默认是 -1（即使用 vacuum_cost_limit），可降低该值以减少对系统性能的影响</span><br><span class="line">autovacuum_vacuum_cost_limit = 200</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">可单独针对表设置vacuum参数：</span><br><span class="line">ALTER TABLE large_table</span><br><span class="line">SET (</span><br><span class="line"> autovacuum_vacuum_threshold = 10000,</span><br><span class="line"> autovacuum_vacuum_scale_factor = 0.05,</span><br><span class="line"> autovacuum_analyze_threshold = 5000,</span><br><span class="line"> autovacuum_analyze_scale_factor = 0.05</span><br><span class="line">);</span><br></pre></td></tr></table></figure></div><h4 id="psql">psql</h4><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">nohup psql postgresql://user:password@host:port/dbname -f update.sql > update.sql 2>&1 & # 刷库命令，update.sql 文件以 begin; 开始，commit; 结束</span><br><span class="line">\q # 退出数据库</span><br><span class="line">\c exampledb # 切换数据库</span><br><span class="line">\l+ # 查看全部数据库</span><br><span class="line">\du+ # 查看全部用户</span><br><span class="line">\d+ # 查看全部表</span><br><span class="line">\dt+ [table_name] # 查看表大小</span><br><span class="line">\di+ [index_name] # 查看索引大小</span><br><span class="line">\dn+ # 查看全部schema</span><br><span class="line">\dp [table_name] # 查看表的权限详情</span><br><span class="line">\x # 竖式显示记录</span><br></pre></td></tr></table></figure></div><h4 id="sql">sql</h4><h5 id="查看锁等待状态">查看锁等待状态</h5><p><a href="https://developer.aliyun.com/ask/54578?spm=a2c6h.13159736">pg中关于AccessShareLock和ExclusiveLock的问题</a>：</p><blockquote><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 1. 先用一个函数来将锁转换为数字，</span></span><br><span class="line"><span class="keyword">create</span> <span class="keyword">function</span> f_lock_level(i_mode text) <span class="keyword">returns</span> <span class="type">int</span> <span class="keyword">as</span> </span><br><span class="line">$$</span><br><span class="line"></span><br><span class="line"><span class="keyword">declare</span></span><br><span class="line"><span class="keyword">begin</span></span><br><span class="line"><span class="keyword">case</span> i_mode</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'INVALID'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'AccessShareLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'RowShareLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'RowExclusiveLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">3</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'ShareUpdateExclusiveLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">4</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'ShareLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">5</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'ShareRowExclusiveLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">6</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'ExclusiveLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">7</span>;</span><br><span class="line"> <span class="keyword">when</span> <span class="string">'AccessExclusiveLock'</span> <span class="keyword">then</span> <span class="keyword">return</span> <span class="number">8</span>;</span><br><span class="line"> <span class="keyword">else</span> <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line"><span class="keyword">end</span> <span class="keyword">case</span>;</span><br><span class="line"><span class="keyword">end</span>; </span><br><span class="line"></span><br><span class="line">$$</span><br><span class="line"><span class="keyword">language</span> plpgsql strict;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 2. 修改查询语句，按锁级别排序：</span></span><br><span class="line"><span class="keyword">with</span> t_wait <span class="keyword">as</span> </span><br><span class="line">(<span class="keyword">select</span> a.mode,a.locktype,a.database,a.relation,a.page,a.tuple,a.classid,a.objid,a.objsubid,</span><br><span class="line">a.pid,a.virtualtransaction,a.virtualxid,a,transactionid,b.query,b.xact_start,b.query_start,</span><br><span class="line">b.usename,b.datname <span class="keyword">from</span> pg_locks a,pg_stat_activity b <span class="keyword">where</span> a.pid<span class="operator">=</span>b.pid <span class="keyword">and</span> <span class="keyword">not</span> a.granted),</span><br><span class="line">t_run <span class="keyword">as</span> </span><br><span class="line">(<span class="keyword">select</span> a.mode,a.locktype,a.database,a.relation,a.page,a.tuple,a.classid,a.objid,a.objsubid,</span><br><span class="line">a.pid,a.virtualtransaction,a.virtualxid,a,transactionid,b.query,b.xact_start,b.query_start,</span><br><span class="line">b.usename,b.datname <span class="keyword">from</span> pg_locks a,pg_stat_activity b <span class="keyword">where</span> a.pid<span class="operator">=</span>b.pid <span class="keyword">and</span> a.granted) </span><br><span class="line"><span class="keyword">select</span> r.locktype,r.mode r_mode,r.usename r_user,r.datname r_db,r.relation::regclass,r.pid r_pid,</span><br><span class="line">r.page r_page,r.tuple r_tuple,r.xact_start r_xact_start,r.query_start r_query_start,</span><br><span class="line">now()<span class="operator">-</span>r.query_start r_locktime,r.query r_query,w.mode w_mode,w.pid w_pid,w.page w_page,</span><br><span class="line">w.tuple w_tuple,w.xact_start w_xact_start,w.query_start w_query_start,</span><br><span class="line">now()<span class="operator">-</span>w.query_start w_locktime,w.query w_query </span><br><span class="line"><span class="keyword">from</span> t_wait w,t_run r <span class="keyword">where</span></span><br><span class="line">r.locktype <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.locktype <span class="keyword">and</span></span><br><span class="line">r.database <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.database <span class="keyword">and</span></span><br><span class="line">r.relation <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.relation <span class="keyword">and</span></span><br><span class="line">r.page <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.page <span class="keyword">and</span></span><br><span class="line">r.tuple <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.tuple <span class="keyword">and</span></span><br><span class="line">r.classid <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.classid <span class="keyword">and</span></span><br><span class="line">r.objid <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.objid <span class="keyword">and</span></span><br><span class="line">r.objsubid <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.objsubid <span class="keyword">and</span></span><br><span class="line">r.transactionid <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">distinct</span> <span class="keyword">from</span> w.transactionid <span class="keyword">and</span></span><br><span class="line">r.pid <span class="operator"><></span> w.pid</span><br><span class="line"><span class="keyword">order</span> <span class="keyword">by</span> f_lock_level(w.mode)<span class="operator">+</span>f_lock_level(r.mode) <span class="keyword">desc</span>,r.xact_start;</span><br></pre></td></tr></table></figure></div><p>现在可以排在前面的就是锁级别高的等待，优先干掉这个。</p><p>-[ RECORD 1 ]-+----------------------------------------------------------</p><p>locktype | relation -- 冲突类型</p><p>r_mode | ShareUpdateExclusiveLock -- 持锁模式</p><p>r_user | postgres -- 持锁用户</p><p>r_db | postgres -- 持锁数据库</p><p>relation | tbl -- 持锁对象</p><p>r_pid | 25656 -- 持锁进程</p><p>r_xact_start | 2015-05-10 14:11:16.08318+08 -- 持锁事务开始时间</p><p>r_query_start | 2015-05-10 14:11:16.08318+08 -- 持锁SQL开始时间</p><p>r_locktime | 00:01:49.460779 -- 持锁时长</p><p>r_query | vacuum freeze tbl; -- 持锁SQL,注意不一定是这个SQL带来的锁,也有可能是这个事务在之前执行的SQL加的锁</p><p>w_mode | AccessExclusiveLock -- 等待锁模式</p><p>w_pid | 26731 -- 等待锁进程</p><p>w_xact_start | 2015-05-10 14:11:17.987362+08 -- 等待锁事务开始时间</p><p>w_query_start | 2015-05-10 14:11:17.987362+08 -- 等待锁SQL开始时间</p><p>w_locktime | 00:01:47.556597 -- 等待锁时长</p><p>w_query | truncate tbl; -- 等待锁SQL</p><p>-[ RECORD 2 ]-+----------------------------------------------------------</p><p>locktype | relation</p><p>r_mode | ShareUpdateExclusiveLock</p><p>r_user | postgres</p><p>r_db | postgres</p><p>relation | tbl</p><p>r_pid | 25656</p><p>r_xact_start | 2015-05-10 14:11:16.08318+08</p><p>r_query_start | 2015-05-10 14:11:16.08318+08</p><p>r_locktime | 00:01:49.460779</p><p>r_query | vacuum freeze tbl;</p><p>w_mode | RowExclusiveLock</p><p>w_pid | 25582</p><p>w_xact_start | 2015-05-10 14:11:22.845+08</p><p>w_query_start | 2015-05-10 14:11:22.845+08</p><p>w_locktime | 00:01:42.698959</p><p>w_query | insert into tbl(crt_time) select now() from generate_series(1,1000); -- 这个SQL其实等待的是truncate tbl的锁;</p><p>......</p></blockquote><h5 id="统计数据库表以及索引存储空间">统计数据库表以及索引存储空间</h5><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 按从大到小排序输出数据库每个索引大小</span></span><br><span class="line"><span class="keyword">select</span> indexrelname, pg_size_pretty(pg_relation_size(indexrelid)) <span class="keyword">as</span> size <span class="keyword">from</span> pg_stat_user_indexes <span class="keyword">where</span> schemaname<span class="operator">=</span><span class="string">'public'</span> <span class="keyword">order</span> <span class="keyword">by</span> pg_relation_size(<span class="string">'public'</span><span class="operator">||</span><span class="string">'.'</span><span class="operator">||</span>indexrelname) <span class="keyword">desc</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- [PostgreSQL中查询 每个表的总大小、索引大小和数据大小，并按总大小降序排序](https://blog.csdn.net/sunny_day_day/article/details/131455635)</span></span><br><span class="line"><span class="keyword">SELECT</span></span><br><span class="line"> pg_size_pretty(pg_total_relation_size(c.oid)) <span class="keyword">AS</span> total_size,</span><br><span class="line"> pg_size_pretty(pg_indexes_size(c.oid)) <span class="keyword">AS</span> index_size,</span><br><span class="line"> pg_size_pretty(pg_total_relation_size(c.oid) <span class="operator">-</span> pg_indexes_size(c.oid)) <span class="keyword">AS</span> data_size,</span><br><span class="line"> nspname <span class="keyword">AS</span> schema_name,</span><br><span class="line"> relname <span class="keyword">AS</span> table_name</span><br><span class="line"><span class="keyword">FROM</span></span><br><span class="line"> pg_class c</span><br><span class="line"><span class="keyword">LEFT</span> <span class="keyword">JOIN</span></span><br><span class="line"> pg_namespace n <span class="keyword">ON</span> n.oid <span class="operator">=</span> c.relnamespace</span><br><span class="line"><span class="keyword">WHERE</span></span><br><span class="line"> relkind <span class="operator">=</span> <span class="string">'r'</span></span><br><span class="line"> <span class="keyword">AND</span> nspname <span class="keyword">NOT</span> <span class="keyword">LIKE</span> <span class="string">'pg_%'</span></span><br><span class="line"> <span class="keyword">AND</span> nspname <span class="operator">!=</span> <span class="string">'information_schema'</span></span><br><span class="line"><span class="keyword">ORDER</span> <span class="keyword">BY</span></span><br><span class="line"> pg_total_relation_size(c.oid) <span class="keyword">DESC</span>;</span><br><span class="line"></span><br></pre></td></tr></table></figure></div><h5 id="常用sql语句">常用sql语句</h5><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 查找超过1小时的长事务</span></span><br><span class="line"><span class="keyword">select</span> <span class="built_in">count</span>(<span class="operator">*</span>) <span class="keyword">from</span> pg_stat_activity <span class="keyword">where</span> state <span class="operator"><></span> <span class="string">'idle'</span> <span class="keyword">and</span> (backend_xid <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">null</span> <span class="keyword">or</span> backend_xmin <span class="keyword">is</span> <span class="keyword">not</span> <span class="keyword">null</span>) <span class="keyword">and</span> now()<span class="operator">-</span>xact_start <span class="operator">></span> <span class="type">interval</span> <span class="string">'3600 sec'</span>::<span class="type">interval</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 查看处于等待锁状态</span></span><br><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> pg_locks <span class="keyword">where</span> <span class="keyword">not</span> granted;</span><br><span class="line"><span class="comment">-- 查看等待锁的关系（表，索引，序列等）</span></span><br><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> pg_class <span class="keyword">where</span> oid<span class="operator">=</span>[上面查出来的relation];</span><br><span class="line"><span class="comment">-- 查看等待锁的数据库</span></span><br><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> pg_database <span class="keyword">where</span> oid<span class="operator">=</span>[上面查出来的database];</span><br><span class="line"><span class="comment">-- 锁表状态</span></span><br><span class="line"><span class="keyword">select</span> oid <span class="keyword">from</span> pg_class <span class="keyword">where</span> relname<span class="operator">=</span><span class="string">'可能锁表了的表'</span>;</span><br><span class="line"><span class="comment">-- 查询出结果则被锁</span></span><br><span class="line"><span class="keyword">select</span> pid <span class="keyword">from</span> pg_locks <span class="keyword">where</span> relation<span class="operator">=</span><span class="string">'上面查出的oid'</span>; </span><br><span class="line"></span><br><span class="line"><span class="comment">-- 关闭事务并回滚</span></span><br><span class="line"><span class="keyword">select</span> pg_cancel_backend(pid);</span><br><span class="line"><span class="comment">-- 若无法关闭，则强制杀死进程连接</span></span><br><span class="line"><span class="keyword">select</span> pg_terminate_backend(pid);</span><br><span class="line"> </span><br><span class="line"><span class="comment">-- 查看连接信息，重点关注state处于idle in transaction</span></span><br><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> pg_stat_activity;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 替换数据库名称</span></span><br><span class="line">update pg_database <span class="keyword">set</span> datname <span class="operator">=</span> <span class="string">'destniationDb'</span> <span class="keyword">where</span> datname <span class="operator">=</span> <span class="string">'sourceDb'</span>;</span><br><span class="line"><span class="comment">-- 清除数据库所有连接</span></span><br><span class="line"><span class="keyword">SELECT</span> pg_terminate_backend(pg_stat_activity.pid) <span class="keyword">FROM</span> pg_stat_activity <span class="keyword">WHERE</span> datname<span class="operator">=</span><span class="string">'test_db'</span> <span class="keyword">AND</span> pid<span class="operator"><></span>pg_backend_pid();</span><br><span class="line"><span class="comment">-- 复制数据库，需断开sourceDb的全部连接</span></span><br><span class="line"><span class="keyword">CREATE</span> DATABASE destniationDb TEMPLATE sourceDb OWNER test_user; </span><br><span class="line"></span><br><span class="line"><span class="comment">-- 清空表并重置自增序列</span></span><br><span class="line"><span class="keyword">truncate</span> <span class="keyword">table</span> table1,table2 RESTART <span class="keyword">IDENTITY</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 导出数据库中数据，HEADER 可不带</span></span><br><span class="line">\<span class="keyword">COPY</span> (<span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> table1) <span class="keyword">TO</span> <span class="string">'/tmp/sql_output.csv'</span> <span class="keyword">WITH</span> CSV HEADER;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 输出删除全部表的sql</span></span><br><span class="line">\<span class="keyword">COPY</span> (<span class="keyword">SELECT</span> <span class="string">'DROP TABLE IF EXISTS "'</span> <span class="operator">||</span> tablename <span class="operator">||</span> <span class="string">'" CASCADE;'</span> <span class="keyword">from</span> pg_tables <span class="keyword">WHERE</span> schemaname <span class="operator">=</span> <span class="string">'public'</span>) <span class="keyword">TO</span> <span class="string">'/tmp/sql_output.sql'</span>;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 添加部分索引（满足条件才建立索引）, where 和 select 语句的一致</span></span><br><span class="line"><span class="keyword">create</span> index [XXX] <span class="keyword">where</span> [XXX]</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 查看当前连接事务执行超时时间</span></span><br><span class="line"><span class="keyword">show</span> statement_timeout;</span><br><span class="line"><span class="comment">-- 设置数据库事务执行超时时间为 60 秒</span></span><br><span class="line"><span class="keyword">AlTER</span> DATABASE mydatabse <span class="keyword">SET</span> statement_timeout<span class="operator">=</span><span class="string">'60s'</span>;</span><br><span class="line"><span class="comment">-- 设置用户事务执行超时时间为 5 分钟</span></span><br><span class="line"><span class="keyword">ALTER</span> ROLE guest <span class="keyword">SET</span> statement_timeout<span class="operator">=</span><span class="string">'5min'</span>;</span><br></pre></td></tr></table></figure></div><h5 id="子查询优化">子查询优化</h5><p>PG 的子查询实际有两种，分为子连接（Sublink）和子查询（SubQuery），按子句的位置不同，出现在 from 关键字后的是子查询，出现在 where/on 等约束条件中或投影中的子句是子连接。</p><p>子查询：<code>select a.* from table_a a, （select a_id from table_b where id=1) b where b.a_id = a.id;</code></p><p>子连接：<code>select * from table_a where id in（select a_id from table_b where id=1);</code></p><p>在简单的子连接查询下，PG 数据库查询优化器一般会将其转化为内连接的方式：<code>select a.* from table_a a, table_b b where a.id=b.a_id and b.id=1;</code>，正常索引没问题情况下这两种方式都能得一样的结果，最终执行的都是索引内连接结果。但在某些情况下，PG 查询优化器在子连接的 SQL 下，子连接的查询会走索引，而主查询会顺序扫描（Seq Scan），原因是当 table_a 的数据量很大时，索引值又有很多重复的，同时查询优化器也不知道子连接返回的具体数据，这时查询优化器可能会认为顺序扫描更快，从而不走索引，导致耗时增加，所以为减少查询优化器的不确定性，最好是直接使用内连接的方式代替 in 语句。 <em>当然，对于特别复杂的查询业务，还是开启事务，分多次查询，在代码层做一些业务逻辑处理更合适，别让数据库把事情全做了，这也能减轻数据库的压力</em>。 PG 查询计划执行路径可以看看： <a href="http://www.postgres.cn/news/viewone/1/156">PostgreSQL 查询语句优化</a>，<a href="https://www.jb51.net/article/203010.htm">postgresql通过索引优化查询速度操作</a></p><h5 id="tricks">tricks</h5><ul><li>由于 <a href="https://stackoverflow.com/questions/309786/how-do-i-force-postgres-to-use-a-particular-index">pg 无法强制使用索引</a>，所以只能通过一些其他方法来引导查询优化器使用索引，比如调整查询条件；</li><li><a href="https://stackoverflow.com/questions/3800551/select-first-row-in-each-group-by-group">获取分组中最大值对应的一行数据</a>；</li></ul><hr /><p>权限配置，<a href="https://blog.csdn.net/zou8944/article/details/121528128">PostgreSQL权限管理详解</a>：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 创建只读组</span></span><br><span class="line"><span class="keyword">create</span> role readonly_group;</span><br><span class="line"><span class="comment">-- 创建只读用户继承只读组</span></span><br><span class="line"><span class="keyword">create</span> <span class="keyword">user</span> reader <span class="keyword">with</span> password <span class="string">'reader'</span> <span class="keyword">in</span> role readonly_group;</span><br><span class="line"><span class="comment">-- 删除用户</span></span><br><span class="line"><span class="keyword">drop</span> <span class="keyword">user</span> reader;</span><br><span class="line"><span class="comment">-- 将只读组权限赋给只读用户</span></span><br><span class="line"><span class="keyword">grant</span> readonly_group <span class="keyword">to</span> reader;</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 读权限</span></span><br><span class="line"><span class="keyword">GRANT</span> <span class="keyword">SELECT</span> <span class="keyword">ON</span> <span class="keyword">ALL</span> TABLES <span class="keyword">IN</span> SCHEMA public <span class="keyword">TO</span> readonly_group;</span><br><span class="line"><span class="keyword">GRANT</span> <span class="keyword">SELECT</span> <span class="keyword">ON</span> <span class="keyword">ALL</span> SEQUENCES <span class="keyword">IN</span> SCHEMA public <span class="keyword">TO</span> readonly_group;</span><br><span class="line"><span class="keyword">GRANT</span> <span class="keyword">EXECUTE</span> <span class="keyword">ON</span> <span class="keyword">ALL</span> FUNCTIONS <span class="keyword">IN</span> SCHEMA public <span class="keyword">TO</span> readonly_group;</span><br><span class="line"><span class="comment">-- 写权限</span></span><br><span class="line"><span class="keyword">GRANT</span> <span class="keyword">INSERT</span>, UPDATE, <span class="keyword">DELETE</span> <span class="keyword">ON</span> <span class="keyword">ALL</span> TABLES <span class="keyword">IN</span> SCHEMA public <span class="keyword">TO</span> write_group;</span><br><span class="line"><span class="keyword">GRANT</span> USAGE <span class="keyword">ON</span> <span class="keyword">ALL</span> SEQUENCES <span class="keyword">IN</span> SCHEMA public <span class="keyword">TO</span> write_group;</span><br></pre></td></tr></table></figure></div><hr /><h4 id="主从备份">主从&备份</h4><p>参考资料：<a href="https://www.jianshu.com/p/478d07af204d">postgresql流式复制(Streaming Replication)</a>、<a href="https://www.cnblogs.com/abclife/p/16391659.html">【PostgreSQL】PostgreSQL复制的监控</a>、<a href="https://blog.csdn.net/weixin_41093846/article/details/132429564">【PostgreSQL】导出数据库表(或序列)的结构和数据</a>、<a href="http://www.postgres.cn/docs/13/app-pg-ctl.html">pg_ctl</a>、<a href="http://www.postgres.cn/docs/13/app-pgbasebackup.html">pg_basebackup</a></p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 创建流复制备份用户（主从）</span></span><br><span class="line"><span class="keyword">create</span> <span class="keyword">user</span> replicator replication login encrypted password <span class="string">'replicator'</span></span><br><span class="line"></span><br><span class="line"><span class="comment">-- 在主库创建一个物理复制槽（PG9.4引入，一个从库一个复制槽）</span></span><br><span class="line"><span class="keyword">select</span> pg_create_physical_replication_slot(<span class="string">'phy_repl_slot_1'</span>);</span><br><span class="line"></span><br><span class="line"><span class="comment">-- 查看复制槽状态</span></span><br><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> pg_replication_slots;</span><br></pre></td></tr></table></figure></div><p>相关命令：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="BASH"><div class="code-copy"></div><figure class="highlight hljs bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 冷备数据库(结合刷库命令可恢复数据库)，加 -s 参数只导出数据库表结构</span></span><br><span class="line">nohup pg_dump postgresql://user:password@host:port/dbname -f db_dump.20240107.sql > dump.log 2>&1 &</span><br><span class="line"></span><br><span class="line"><span class="comment"># 新建一个数据库</span></span><br><span class="line">pg_ctl init -D /home/postgres/db_data_dir</span><br><span class="line"></span><br><span class="line"><span class="comment"># 修改配置后重新加载配置</span></span><br><span class="line">pg_ctl reload -D /home/postgres/db_data_dir</span><br><span class="line"><span class="comment"># 或者重启数据库</span></span><br><span class="line">pg_ctl restart -D /home/postgres/db_data_dir</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置数据库默认连接密码</span></span><br><span class="line"><span class="built_in">export</span> PGPASSWORD=test_pswd</span><br><span class="line"></span><br><span class="line"><span class="comment"># 完整复制数据库（作为从库）</span></span><br><span class="line">nohup pg_basebackup -h localhost -p port -U replicator -D /home/postgres/db_data1_dir -v -P -R -Xs > ./backup.log 2>&1 &</span><br><span class="line"></span><br><span class="line"><span class="comment"># 从库提升为主库</span></span><br><span class="line">pg_ctl promote -D /home/postgres/db_data_dir</span><br></pre></td></tr></table></figure></div><p>设置主库：postgresql.conf</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">wal_level = hot_standby </span><br><span class="line"># PG12 之后，wal_level = replica</span><br><span class="line"></span><br><span class="line"># 主备机不同步时，re_wind恢复结点</span><br><span class="line">wal_log_hints = on</span><br><span class="line"># 设置最大流复制数（从库数）</span><br><span class="line">max_wal_senders = 3</span><br><span class="line">wal_keep_segments = 64</span><br><span class="line"># 支持从库读，以及从库再拉从库</span><br><span class="line">hot_standby = on</span><br></pre></td></tr></table></figure></div><p>设置主库：pg_hba.conf</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"># Allow replication connections from localhost, by a user with the</span><br><span class="line"># replication privilege.</span><br><span class="line">local replication all trust</span><br><span class="line">host replication all 127.0.0.1/32 trust</span><br><span class="line">host replication all ::1/128 trust</span><br><span class="line">host replication all 0.0.0.0/0 md5</span><br></pre></td></tr></table></figure></div><p>设置从库 recovery.conf（自 Postgresql 12 起，recovery.conf 并入 postgresql.conf）：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">standby_mode = 'on' # PG12之后，删除该配置项</span><br><span class="line">primary_conninfo = 'host=db_addr port=db_port user=replicator password=<password>'</span><br><span class="line">primary_slot_name = 'phy_repl_slot_1'</span><br></pre></td></tr></table></figure></div><h5 id="区分主库从库">区分主库从库</h5><p>主要方式：从库的根目录下存在 recovery.conf 文件（PG12 之后无该文件，而是存在一个 0KB 的 standby.signal 文件）。</p><p><code>SELECT * FROM pg_stat_replication;</code> 如果有结果（显示所有连接到该节点的从库），则表示当前节点为主库。</p><p>主库一般配置参数：</p><ul><li>PG12 之后，wal_level = replica<code>或</code>logical；</li><li>max_wal_senders 一般设置较大，允许多个从库；</li><li>hot_standby，主库一般为 off；</li></ul><p>从库一般配置参数：</p><ul><li>hot_standby，从库为 on；</li><li>primary_conninfo，有连接到主库的相关配置信息；</li></ul><h4 id="并发-dumprestore-数据库">并发 dump&restore 数据库</h4><ol type="1"><li><p>导出数据库全部表结构</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pg_dump -d postgresql://user:pswd@host:port/db_name --schema-only -f db_name_schema.sql</span><br></pre></td></tr></table></figure></div></li><li><p>导出外键约束</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line">psql -d postgresql://owner_user:pswd@host:port/db_name -t -A -F"," -c "</span><br><span class="line">SELECT DISTINCT</span><br><span class="line"> 'ALTER TABLE ' || quote_ident(nsp.nspname) || '.' || quote_ident(cls.relname) || </span><br><span class="line"> ' ADD CONSTRAINT ' || quote_ident(con.conname) || </span><br><span class="line"> ' FOREIGN KEY (' || array_to_string(ARRAY(</span><br><span class="line"> SELECT quote_ident(att.attname)</span><br><span class="line"> FROM pg_attribute att</span><br><span class="line"> WHERE att.attnum = ANY(con.conkey)</span><br><span class="line"> AND att.attrelid = cls.oid), ', ') || </span><br><span class="line"> ') REFERENCES ' || quote_ident(f_nsp.nspname) || '.' || quote_ident(f_cls.relname) || </span><br><span class="line"> ' (' || array_to_string(ARRAY(</span><br><span class="line"> SELECT quote_ident(att.attname)</span><br><span class="line"> FROM pg_attribute att</span><br><span class="line"> WHERE att.attnum = ANY(con.confkey)</span><br><span class="line"> AND att.attrelid = f_cls.oid), ', ') || </span><br><span class="line"> ') ON DELETE ' || CASE con.confdeltype</span><br><span class="line"> WHEN 'a' THEN 'NO ACTION'</span><br><span class="line"> WHEN 'r' THEN 'RESTRICT'</span><br><span class="line"> WHEN 'c' THEN 'CASCADE'</span><br><span class="line"> WHEN 'n' THEN 'SET NULL'</span><br><span class="line"> WHEN 'd' THEN 'SET DEFAULT'</span><br><span class="line"> END ||</span><br><span class="line"> ' ON UPDATE ' || CASE con.confupdtype</span><br><span class="line"> WHEN 'a' THEN 'NO ACTION'</span><br><span class="line"> WHEN 'r' THEN 'RESTRICT'</span><br><span class="line"> WHEN 'c' THEN 'CASCADE'</span><br><span class="line"> WHEN 'n' THEN 'SET NULL'</span><br><span class="line"> WHEN 'd' THEN 'SET DEFAULT'</span><br><span class="line"> END || ';'</span><br><span class="line">FROM pg_constraint con</span><br><span class="line">JOIN pg_class cls ON con.conrelid = cls.oid</span><br><span class="line">JOIN pg_namespace nsp ON cls.relnamespace = nsp.oid</span><br><span class="line">JOIN pg_class f_cls ON con.confrelid = f_cls.oid</span><br><span class="line">JOIN pg_namespace f_nsp ON f_cls.relnamespace = f_nsp.oid</span><br><span class="line">WHERE con.contype = 'f';" > db_name_fkeys.sql</span><br></pre></td></tr></table></figure></div></li><li><p>导出数据库全局用户/权限</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pg_dumpall -d postgresql://superuser:pswd@host:port --globals-only -f db_name_user.sql</span><br></pre></td></tr></table></figure></div></li><li><p>4个并行任务导出全部数据</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pg_dump -d postgresql://user:pswd@host:port/db_name --data-only -F d -j 4 -f ./db_name_data_dir</span><br></pre></td></tr></table></figure></div></li><li><p>新建数据库实例</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pg_ctl init -D ~/new_db_data</span><br></pre></td></tr></table></figure></div></li><li><p>导入数据库全局用户/权限</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">psql -U superuser -p port -f db_name_user.sql</span><br></pre></td></tr></table></figure></div></li><li><p>新建数据库</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">create</span> database new_db_name owner owner_user</span><br></pre></td></tr></table></figure></div></li><li><p>导入数据库全部表结构</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">psql -U superuser -p port -f db_name_schema.sql</span><br></pre></td></tr></table></figure></div></li><li><p>移除新库外键约束</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">psql -d postgresql://owner_user:pswd@host:port/db_name <<EOF</span><br><span class="line">DO \$\$ </span><br><span class="line">DECLARE</span><br><span class="line"> r RECORD;</span><br><span class="line">BEGIN</span><br><span class="line"> FOR r IN (SELECT conname, conrelid::regclass</span><br><span class="line"> FROM pg_constraint</span><br><span class="line"> WHERE contype = 'f') LOOP</span><br><span class="line"> EXECUTE 'ALTER TABLE ' || r.conrelid || ' DROP CONSTRAINT ' || r.conname;</span><br><span class="line"> END LOOP;</span><br><span class="line">END \$\$;</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure></div></li><li><p>4个并行任务导入数据</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pg_restore -d postgresql://owner_user:pswd@host:port/db_name -j 4 ./db_name_data_dir</span><br></pre></td></tr></table></figure></div></li><li><p>恢复新库外键约束</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">psql -d postgresql://owner_user:pswd@host:port/db_name -f db_name_fkeys.sql</span><br></pre></td></tr></table></figure></div></li></ol><hr /><h4 id="mvcc数据碎片索引膨胀freeze">MVCC/数据碎片/索引膨胀/FREEZE</h4><p>参考自：<a href="https://www.modb.pro/db/48183">PostgreSQL | 空间又告警了，先从整理索引碎片开始</a>，<a href="https://blog.csdn.net/wonder191/article/details/131787424">正确的评估postgres index膨胀</a>，<a href="https://www.cnblogs.com/dbadaily/p/vacuum1.html">PostgreSQL VACUUM 之深入浅出 (一)</a>，<a href="https://blog.csdn.net/qq_44866828/article/details/132031913">深入理解 PostgreSQL 中的 MVCC（多版本并发控制）机制</a>，<a href="https://cloud.tencent.com/developer/article/1555360">硬核-深度剖析PostgreSQL数据库“冻结炸弹”原理机制</a></p><p>简单总结一下：</p><ul><li>mvcc 主要通过锁或乐观并发控制机制来解决冲突，通过事务号实现多版本及查询可见性（当前事务只能看到当前事务启动前已提交的数据，即只可能大事务号看到小事务号的数据），当事务号达到设定值时，事务号会发生回卷，此时需要以单用户模式执行 vacuum freeze 操作，将所有事务号置为2，代表冻结事务，对所有事务可见，当然可通过设置参数实现自动 freeze，减少人工介入维护时间；</li><li>由于 postgres 的 mvcc 机制，更新和删除以及新增的回滚都会造成数据碎片，虽然有 vacuum，但仍然存在部分数据碎片无法再被重复利用（连续空间释放中间一部分，再重新分配后，可能导致少许剩余空间太小无法再利用，实时清理或合并这些小空间的代价又太大），且索引的膨胀不可避免（当数据被删除标记为死元组时，被删除数据的索引仍然存在，而 vacuum 不会清理无效索引），所以当发现索引碎片率超过 30% 时，需要进行重建索引 REINDEX，但常规的 REINDEX 会锁表，在 pg12 之后才有 REINDEX CONCURRENTLY，可在线重建，不会锁表，重建完之后需要执行 ANALYZE 更新一下统计信息使索引立即生效。</li></ul><h4 id="空间清理">空间清理</h4><p> <a href="http://www.postgres.cn/docs/14/routine-vacuuming.html">由于标准的 vacuum 无法释放空间归还给操作系统</a>，只是在数据库内部清理/释放/使用（<em>所以 vacuum 只对于未造成空间膨胀的数据库有效，而且当存在大量更新/删除操作时，vacuum 也不一定能及时控制数据库大小，导致数据库空间一步步变大</em>）。而 VACUUM FULL 或者 CLUSTER 在清理磁盘时会进行锁表（SELECT、INSERT、UPDATE 和 DELETE 等操作都无法正常进行，基本可认为是需要停机维护），对于已经占用大量存储空间的数据库，可以使用 <a href="https://github.com/reorg/pg_repack">pg_repack</a> 进行在线清理/释放表空间，相比 CLUSTER 或 VACUUM FULL，<a href="https://help.aliyun.com/zh/rds/apsaradb-rds-for-postgresql/use-the-pg-repack-extension-to-clear-tablespaces">pg_repack 无需获取排它锁</a>，更轻量。</p><p> 针对 vacuum 不及时导致一直新申请磁盘空间膨胀的问题，PG 支持设置 autovacuum，根据系统资源调整相关参数后，可以使用 pg_stat_user_tables 视图监控表的膨胀情况，关注 n_dead_tup（死元组数量）和 last_autovacuum（上次vacuum时间）：<code>SELECT relname, n_live_tup, n_dead_tup, last_vacuum, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC;</code>，以及使用 pg_stat_activity 视图检查 vacuum 进程的执行情况和影响：<code>SELECT datname, pid, usename, query_start, state, query FROM pg_stat_activity WHERE query LIKE '%vacuum%';</code>。</p><p> 对于既成事实占用存储空间超大的数据库，缩减空间一个可能的方案是先 dump 数据，同时开始记录原数据库增量的 dml sql（log_statement=mod），新建一个数据库，用 dump sql 文件写入，记录 dump 最新的节点（时间或者啥 id，再将原数据库节点之外的数据迁移到新数据库中（用之前记录的增量 dml sql，需过滤回滚的事务），再用新数据库替换原数据库，如此达到释放空间的目的（该方案同样适用于数据库版本升级）。（当然也可以用时间字段过滤出增量数据）</p><hr /><p>常见问题：</p><ol type="1"><li><p>当自增主键报 <code>duplicate key value violates unique constraint</code> 主键冲突时，一般是因为存在手动分配 id 的数据（复制表或着手动插入分配了 id），自增主键 seqence TABLE_COLUMN_seq 没有更新，新插入一个值自增 id 和数据库已插入的分配 id 冲突，此时需要执行 <code>SELECT setval('TABLE_COLUMN_seq', (SELECT max(COLUMN) FROM "TABLE"))</code> 更新自增主键;</p></li><li><p>分析 sql 性能时，可在 sql 语句前增加 <code>EXPLAIN</code> 关键字，查看执行计划，EXPLAIN 一般不会实际执行 sql，但 sql 中带有子语句时，子语句可能会执行，所以为保险起见，最好是在事务中使用 EXPLAIN；eg:</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SQL"><div class="code-copy"></div><figure class="highlight hljs sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">begin</span>;</span><br><span class="line">EXPLAIN <span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> table1 <span class="keyword">where</span> id<span class="operator">=</span><span class="number">1</span>;</span><br><span class="line"><span class="keyword">rollback</span>;</span><br></pre></td></tr></table></figure></div><p>若要分析实际执行时间，可以使用 EXPLAIN ANALYZE，<em>该选项会实际执行 SQL</em>，也可以组合参数一起分析执行命令 <code>explain (analyze,verbose,costs,buffers,timing) select * from table1 where id=1;</code>。</p></li><li><p>如果业务数据无法直接使用批量写入数据库，就最好在一个事务中写入（当然也得看数据量），在同一个事务中写入，不仅能利用事务本身的 ACID 特性，而且比单独分次执行 sql 效率更高；</p></li><li><p>PG 数据库中，如果要使用 order 排序查询时，一般带主键的复合索引比单个字段索引更有效，因为 PG 数据在数据更新后，一般会乱序存储，导致单字段索引在查询时需要访问的页面会更多；</p></li><li><p>PG 刚创建/删除索引后，不一定会及时生效，需要数据库运行一段时间后才会开始生效，如需要立即生效，可执行 <code>ANALYZE VERBOSE table_name;</code>命令，离线或者低负载的时候可以执行 <code>VACUUM VERBOSE ANALYZE table_name</code>，清理表的同时更新统计信息，得到更好的 SQL 执行计划。</p></li></ol><h2 id="后记">后记</h2><p> 后面持续更新。。。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 记录一下服务器问题排查常用的一些命令。</p></summary>
<category term="Wiki" scheme="http://cniter.github.io/categories/Wiki/"/>
<category term="note" scheme="http://cniter.github.io/tags/note/"/>
<category term="unix-like" scheme="http://cniter.github.io/tags/unix-like/"/>
</entry>
<entry>
<title>时空查询之ECQL</title>
<link href="http://cniter.github.io/posts/489fa7b3.html"/>
<id>http://cniter.github.io/posts/489fa7b3.html</id>
<published>2021-01-23T12:59:21.000Z</published>
<updated>2021-12-18T11:54:13.940Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> ECQL 是 CQL 的扩展，CQL 是 OGC 标准查询语言，而 ECQL 是 GeoTools 为更好的方便查询，在编程实现时扩展了 CQL，主要扩展在于其移除了 CQL 的一些限制（属性必须在比较运算符的左边，不能创建 Id Filter 进行查询等限制），也和 SQL 更相似。所以可简单认为 CQL 是书面上的标准，而 ECQL 是事实上的标准。</p><span id="more"></span><h2 id="谓词篇">谓词篇</h2><p>时间查询主要有以下几个查询谓词：</p><table><colgroup><col style="width: 36%" /><col style="width: 63%" /></colgroup><thead><tr class="header"><th>谓词</th><th>作用</th></tr></thead><tbody><tr class="odd"><td>T <strong>TEQUALS</strong> Time</td><td>测试 T 和给定时间相等，相当于 T == Time。</td></tr><tr class="even"><td>T <strong>BEFORE</strong> Time</td><td>测试 T 在给定时间之前，相当于 T < Time。</td></tr><tr class="odd"><td>T <strong>BEFORE OR DURING</strong> Time Period</td><td>测试 T 在给定时间段之前或其中，相当于 T <= TimePeriod[1]。</td></tr><tr class="even"><td>T <strong>DURING</strong> Time Period</td><td>测试 T 在给定时间段其中，相当于 TimePeriod[0] <= T <= TimePeriod[1]。</td></tr><tr class="odd"><td>T <strong>DURING OR AFTER</strong> Time Period</td><td>测试 T 在给定时间段其中或之后，相当于 TimePeriod[0] <= T。</td></tr><tr class="even"><td>T <strong>AFTER</strong> Time</td><td>测试 T 在给定时间之后，相当于 T > Time。</td></tr></tbody></table><p>时间段以 <code>/</code> 分隔符区分前后两个时间，时间格式一般为 yyyy-MM-dd'T'HH:mm:ss.SSS'Z'。</p><p>空间查询主要有以下几个查询谓词：</p><table><colgroup><col style="width: 50%" /><col style="width: 50%" /></colgroup><thead><tr class="header"><th>谓词</th><th>作用</th></tr></thead><tbody><tr class="odd"><td><strong>INTERSECTS</strong>(A: Geometry, B: Geometry)</td><td>测试 A 与 B 相交，与 DISJOINT 相反。</td></tr><tr class="even"><td><strong>DISJOINT</strong>(A: Geometry, B: Geometry)</td><td>测试 A 与 B 不相交，与 INTERSECTS 相反。</td></tr><tr class="odd"><td><strong>CONTAINS</strong>(A: Geometry, B: Geometry)</td><td>测试 A 包含 B，与 WITHIN 相反。</td></tr><tr class="even"><td><strong>WITHIN</strong>(A: Geometry, B: Geometry)</td><td>测试 B 包含 A，即 A 在 B 中，与 CONTAINS 相反。</td></tr><tr class="odd"><td><strong>TOUCHES</strong>(A: Geometry, B: Geometry)</td><td>测试 A 的边界是否与 B 的边界接触，但内部不相交。</td></tr><tr class="even"><td><strong>CROSSES</strong>(A: Geometry, B: Geometry)</td><td>测试 A 与 B 是否相交，但不存在包含关系。</td></tr><tr class="odd"><td><strong>OVERLAPS</strong>(A: Geometry, B: Geometry)</td><td>测试 A 与 B 是否重叠，需满足 A 与 B 是同一类型（如都是 POLYGON），并且相交区域同样是 A 和 B 的类型（只能是 POLYGON，不能是 POINT）。</td></tr><tr class="even"><td><strong>EQUALS</strong>(A: Geometry, B: Geometry)</td><td>测试 A 与 B 完全相等。</td></tr><tr class="odd"><td><strong>RELATE</strong>(A: Geometry, B: Geometry, nineIntersectionModel: String)</td><td>测试 A 与 B 是否满足 <strong>DE-9IM</strong> 模型，该模型可模拟上述所有情况。</td></tr><tr class="even"><td><strong>DWITHIN</strong>(A: Geometry, B: Geometry, distance: double, units: String)</td><td>测试 A 与 B 的最短距离是否不超过多少距离，单位有（<code>feet</code>, <code>meters</code>, <code>statute miles</code>, <code>nautical miles</code>, <code>kilometers</code>）。</td></tr><tr class="odd"><td><strong>BEYOND</strong>(A: Geometry, B: Geometry, distance: Double, units: String)</td><td>测试 A 与 B 的最短距离是否超过多少距离。</td></tr><tr class="even"><td><strong>BBOX</strong>(A: Geometry, leftBottomLng: Double, leftBottomLat: Double, rightTopLng: Double, rightTopLat: Double, crs="EPSG:4326")</td><td>测试 A 是否与给定 box 相交。</td></tr></tbody></table><p>Geometry 是指 WKT 格式的数据，主要有以下几种：</p><table><colgroup><col style="width: 26%" /><col style="width: 73%" /></colgroup><thead><tr class="header"><th>类型</th><th>示例</th></tr></thead><tbody><tr class="odd"><td><strong>POINT</strong></td><td>POINT(6 10)</td></tr><tr class="even"><td><strong>LINESTRING</strong></td><td>LINESTRING(3 4,10 50,20 25)</td></tr><tr class="odd"><td><strong>POLYGON</strong></td><td>POLYGON((1 1,5 1,5 5,1 5,1 1),(2 2,2 3,3 3,3 2,2 2))</td></tr><tr class="even"><td><strong>MULTIPOINT</strong></td><td>MULTIPOINT(3.5 5.6, 4.8 10.5)</td></tr><tr class="odd"><td><strong>MULTILINESTRING</strong></td><td>MULTILINESTRING((3 4,10 50,20 25),(-5 -8,-10 -8,-15 -4))</td></tr><tr class="even"><td><strong>MULTIPOLYGON</strong></td><td>MULTIPOLYGON(((1 1,5 1,5 5,1 5,1 1),(2 2,2 3,3 3,3 2,2 2)),((6 3,9 2,9 4,6 3)))</td></tr><tr class="odd"><td><strong>GEOMETRYCOLLECTION</strong></td><td>GEOMETRYCOLLECTION(POINT(4 6),LINESTRING(4 6,7 10))</td></tr></tbody></table><p><strong><em>※注：</em></strong> POLYGON 中的边界点必须闭合，即首尾点相同，若存在多个边界，则需要遵循 逆时针,顺时针,逆时针,顺时针... 的点排列顺序，逆时针封闭，顺时针开孔，以形成具有岛和洞的复杂多边形。</p><p> 由于 WKT 标准只支持二维的坐标，为支持三维坐标以及齐次线性计算，所以在 PostGIS 中又有 EWKT 标准实现，EWKT 扩展了 WKT，带 <code>Z</code> 结尾用来支持三维坐标，带 <code>M</code> 结尾用来支持齐次线性计算，如 <code>POINTZ(6 10 3)</code>，<code>POINTM(6 10 1)</code>，<code>POINTZM(6 10 3 1)</code>，同时还支持坐标内嵌空间参考系，如 <code>SRID=4326;LINESTRING(-134.921387 58.687767, -135.303391 59.092838)</code>。GeoTools 19.0 之后也默认以 EWKT 进行解析和编码。</p><h2 id="查询篇">查询篇</h2><h3 id="属性字段查询">属性字段查询</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line">// 查询属性 ATTR1 小于 7 的数据</span><br><span class="line">Filter filter = ECQL.toFilter("ATTR1 < (1 + ((3 / 2) * 4))" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 小于属性 ATTR2 绝对值的数据</span><br><span class="line">Filter filter = ECQL.toFilter("ATTR1 < abs(ATTR2)" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 为 test 字符串的数据</span><br><span class="line">Filter filter = ECQL.toFilter("ATTR1 == 'test'" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 在 10 和 20 之间的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 BETWEEN 10 AND 20" );</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 >= 10 AND ATTR1 <= 20" );</span><br><span class="line"></span><br><span class="line">// 多条件查询</span><br><span class="line">Filter filter = ECQL.toFilter("ATTR1 < 10 AND ATTR2 < 2 OR ATTR3 > 10" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 为 silver 或 oil 或 gold 的数据</span><br><span class="line">Filter filter = ECQL.toFilter("ATTR1 IN ('silver','oil', 'gold' )");</span><br><span class="line"></span><br><span class="line">// 以 ID 主键进行查询</span><br><span class="line">Filter filter = ECQL.toFilter("IN ('river.1', 'river.2')");</span><br><span class="line">Filter filter = ECQL.toFilter("IN (300, 301)");</span><br></pre></td></tr></table></figure></div><h3 id="模糊查询">模糊查询</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">// 查询属性 ATTR1 包含 abc 字符串的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 LIKE '%abc%'" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 开头不为 abc 字符串的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 NOT LIKE 'abc%'" );</span><br><span class="line"></span><br><span class="line">// 查询属性 cityName 开头为 new 的数据，忽略 new 的大小写</span><br><span class="line">Filter filter = ECQL.toFilter("cityName ILIKE 'new%'");</span><br><span class="line"></span><br><span class="line">// 测试字符串是否包含</span><br><span class="line">Filter filter = ECQL.toFilter("'aabbcc' LIKE '%bb%'");</span><br></pre></td></tr></table></figure></div><h3 id="空属性查询">空属性查询</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">// 查询有属性 ATTR1 存在的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 EXISTS" );</span><br><span class="line"></span><br><span class="line">// 查询属性 ATTR1 不存在的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "ATTR1 DOES-NOT-EXIST" );</span><br><span class="line"></span><br><span class="line">// 查询 Name 为 NULL 的数据</span><br><span class="line">Filter filter = ECQL.toFilter("Name IS NULL");</span><br></pre></td></tr></table></figure></div><h3 id="时间查询">时间查询</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line">// 查询时间属性 dtg 等于的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "dtg TEQUALS 2006-11-30T01:30:00Z" );</span><br><span class="line"></span><br><span class="line">// 查询时间属性 dtg 在之后的数据</span><br><span class="line">Filter filter = ECQL.toFilter("dtg AFTER 2006-11-30T01:30:00Z");</span><br><span class="line"></span><br><span class="line">// 查询时间属性 dtg 在之前的数据</span><br><span class="line">Filter filter = ECQL.toFilter("dtg BEFORE 2006-11-30T01:30:00Z");</span><br><span class="line"></span><br><span class="line">// 查询时间属性 dtg 在之间的数据，+3:00 代表 GMT 时间 +3 小时，以 Z 结尾的时间就是 GMT 时间</span><br><span class="line">Filter filter = ECQL.toFilter( "dtg DURING 2006-11-30T00:30:00+03:00/2006-11-30T01:30:00+03:00 ");</span><br><span class="line"></span><br><span class="line">// 查询时间属性 dtg 等于的数据</span><br><span class="line">Filter filter = ECQL.toFilter("dtg = 1981-06-20");</span><br><span class="line"></span><br><span class="line">// 查询时间属性 dtg 小于等于的数据</span><br><span class="line">Filter filter = ECQL.toFilter("dtg <= 1981-06-20T12:30:01Z");</span><br></pre></td></tr></table></figure></div><h3 id="空间查询">空间查询</h3><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line">// 查询空间属性 geom 包含点的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "CONTAINS(geom, POINT(1 2))" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与 box 相交的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "BBOX(geom, 10,20,30,40)" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与点最短距离不超过 10 千米的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "DWITHIN(geom, POINT(1 2), 10, kilometers)" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与线相交的数据（geom 也必须是线）</span><br><span class="line">Filter filter = ECQL.toFilter( "CROSS(geom, LINESTRING(1 2, 10 15))" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与 GEOMETRYCOLLECTION 相交的数据（geom 也必须是 GEOMETRYCOLLECTION）</span><br><span class="line">Filter filter = ECQL.toFilter( "INTERSECT(geom, GEOMETRYCOLLECTION (POINT (10 10),POINT (30 30),LINESTRING (15 15, 20 20)) )" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与线相交的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "CROSSES(geom, LINESTRING(1 2, 10 15))" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与 GEOMETRYCOLLECTION 相交的数据</span><br><span class="line">Filter filter = ECQL.toFilter( "INTERSECTS(geom, GEOMETRYCOLLECTION (POINT (10 10),POINT (30 30),LINESTRING (15 15, 20 20)) )" );</span><br><span class="line"></span><br><span class="line">// 查询空间属性 geom 与包含线的数据</span><br><span class="line">Filter filter = ECQL.toFilter("RELATE(geom, LINESTRING (-134.921387 58.687767, -135.303391 59.092838), T*****FF*)");</span><br></pre></td></tr></table></figure></div><hr /><p> 在 GeoTools 中，可通过 FilterFactory 来构造 Filter，而不是直接写字符串，具体示例如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="JAVA"><div class="code-copy"></div><figure class="highlight hljs java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">FilterFactory2 ff = CommonFactoryFinder.getFilterFactory2();</span><br><span class="line"></span><br><span class="line"><span class="comment">// 相当于 Filter filter1 = ECQL.toFilter("ATTR1 = 1 AND ATTR2 < 4" );</span></span><br><span class="line">List<Filter> filterList = ECQL.toFilterList(<span class="string">"ATTR1=1; ATTR2<4"</span>);</span><br><span class="line">Filter filter1 = ff.and(filterList);</span><br><span class="line"></span><br><span class="line"><span class="comment">// 相当于 Filter filter2 = ECQL.toFilter( "BBOX(geom, 10,20,30,40)" );</span></span><br><span class="line">Filter filter2 = ff.bbox(<span class="string">"geom"</span>, <span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>, <span class="string">"EPSG:4326"</span>);</span><br><span class="line"></span><br><span class="line"><span class="comment">// 相当于 Filter filter3 = ECQL.toFilter( "dtg DURING 2006-11-29T00:30:00Z/2006-11-30T00:30:00Z");</span></span><br><span class="line">Date startTime = ZonedDateTime.of(<span class="number">2006</span>, <span class="number">11</span>, <span class="number">29</span>, <span class="number">0</span>, <span class="number">30</span>, <span class="number">0</span>, <span class="number">0</span>, ZoneOffset.UTC);</span><br><span class="line">Date endTime = Date.from(startTime.plusDays(<span class="number">1</span>).toInstant());</span><br><span class="line">Filter filter3 = ff.between(ff.property(<span class="string">"dtg"</span>), ff.literal(startTime), ff.literal(endTime));</span><br></pre></td></tr></table></figure></div><h2 id="后记">后记</h2><p> 基本可认为 CQL 和 SQL 中查询条件差不多，虽然不支持分组查询等复杂 SQL 特性，但对于一般的时空查询基本够用，CQL 中还有些空间操作函数就不继续写了，如取面积，取缓冲区，取交集，取长度等等，有需要的可自行查询 <a href="http://udig.github.io/docs/user/concepts/Constraint%20Query%20Language.html">uDig Common Query Language</a>。</p><h2 id="参考资料">参考资料</h2><p><a href="http://docs.geotools.org/latest/userguide/library/cql/cql.html">GeoTools CQL</a></p><p><a href="http://docs.geotools.org/stable/userguide/library/cql/ecql.html">GeoTools ECQL</a></p><p><a href="https://docs.geoserver.org/latest/en/user/filter/ecql_reference.html#ecql-reference">GeoServer ECQL Reference</a> / <a href="https://blog.csdn.net/neimeng0/article/details/79914880">GeoServer 属性查询和空间查询支持 CQL / ECQL过滤器语言</a></p><p><a href="https://blog.csdn.net/ucs426/article/details/99780891">WKT解读</a></p><p><a href="https://www.cnblogs.com/denny402/p/4968201.html">GEOS库学习之三：空间关系、DE-9IM和谓词</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> ECQL 是 CQL 的扩展，CQL 是 OGC 标准查询语言，而 ECQL 是 GeoTools 为更好的方便查询，在编程实现时扩展了 CQL，主要扩展在于其移除了 CQL 的一些限制（属性必须在比较运算符的左边，不能创建 Id Filter 进行查询等限制），也和 SQL 更相似。所以可简单认为 CQL 是书面上的标准，而 ECQL 是事实上的标准。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="language" scheme="http://cniter.github.io/tags/language/"/>
</entry>
<entry>
<title>GeoMesa踩坑指北</title>
<link href="http://cniter.github.io/posts/9978824c.html"/>
<id>http://cniter.github.io/posts/9978824c.html</id>
<published>2021-01-16T09:52:09.000Z</published>
<updated>2021-12-18T11:54:13.932Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 需要做个 GeoMesa 的微服务，简单熟悉一下 GeoMesa。</p><span id="more"></span><h2 id="基础篇">基础篇</h2><p> GeoMesa 可以说是大数据中的 PostGIS，主要用来在存储和处理 GIS 数据时提供相应的索引，从而加快处理速度。GeoMesa 基于 GeoTools，其中最重要的两个概念就是 SimpleFeatureType 和 SimpleFeature，SimpleFeatureType 对应的是关系型数据库中表的描述（表明，表的列字段属性信息等），而 SimpleFeature 对应的是表中每行数据。下面重点谈谈 GeoMesa 中的 SimpleFeatureType 以及其创建索引方式。</p><p> 在 GeoMesa 中通常使用 SimpleFeatureTypes.createType 方法进行创建，该方法有两个重载，以没有 namespace 参数的方法为例：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">createType</span></span>(typeName: <span class="type">String</span>, spec: <span class="type">String</span>): <span class="type">SimpleFeatureType</span> = {</span><br><span class="line"> <span class="keyword">val</span> (namespace, name) = parseTypeName(typeName)</span><br><span class="line"> createType(namespace, name, spec)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p>先通过 parseTypeName 解析 typeName，以 <code>:</code> 作为分隔符，取最后一个有效（不为空）字符串作为表名（name），其余部分如有效则作为 namespace，否则 namespace 则为 null。spec 参数的通用形式有以下几种：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> spec = <span class="string">"name:String,dtg:Date,*geom:Point:srid=4326"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> spec = <span class="string">"name:String,dtg:Date,*geom:Point:srid=4326;geomesa.indices.enabled='z2,id,z3'"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> spec = <span class="string">"name:String:index=true,tags:String:json=true,dtg:Date:default=true,*geom:Point:srid=4326;geomesa.indices.enabled='z2,id,z3'"</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> spec = <span class="string">"userId:String,trackId:String,altitude:Double,dtg:Date,*geom:Point:srid=4326;geomesa.index.dtg='dtg',geomesa.table.sharing='true',geomesa.indices='z3:4:3,z2:3:3,id:2:3',geomesa.table.sharing.prefix='\\u0001'"</span></span><br></pre></td></tr></table></figure></div><p>先使用 <code>;</code> 分隔符，再使用 <code>,</code> 分隔符，最后使用 <code>:</code> 分隔符。<code>;</code> 分隔符将 spec 分割为两个字符串：前者表示表中的全部列属性信息，列属性经过 <code>,</code> 分隔符分割为多列，列又经过 <code>:</code> 分隔符分割为 列名，列数据类型，列的一些属性（是否是索引，json 数据，默认索引等），而列名首字母 <code>*</code> 代表该字段是用于索引的 geometry 类型，一般采用 WKT 格式进行描述，当然存在数据库时会以字节码进行压缩；后者表示创建表时的 userData，同样经过 <code>,</code> 分隔符分割为多个 userData，userData 的一些默认属性可在 SimpleFeatureTypes.Configs 中看到，其它的可以用户自定义，这里重点说一下 <code>geomesa.indices.enabled</code> 属性，目前 GeoMesa 支持 8 种索引，分别为：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">"attr", // 属性索引</span><br><span class="line">"id", // 主键索引</span><br><span class="line">"s2", // Hilbert 曲线点空间索引</span><br><span class="line">"s3", // Hilbert 曲线点时空索引</span><br><span class="line">"z2", // Z 型曲线点空间索引</span><br><span class="line">"xz2", // Z 型曲线线面空间索引</span><br><span class="line">"z3", // Z 型曲线点时空索引</span><br><span class="line">"xz3" // Z 型曲线线面时空索引</span><br></pre></td></tr></table></figure></div><p> 由于 GeoMesa 中的索引一般存在多个版本，而 <code>geomesa.indices.enabled</code> 默认使用最新的版本，若需要指定版本，需要使用 <code>geomesa.indices</code>，<strong><em>该属性是 geomesa 内部属性，不对外开放</em></strong>，通用格式为：</p><p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">s"<span class="subst">$name</span>:<span class="subst">$version</span>:<span class="subst">${mode.flag}</span>:<span class="subst">${attributes.mkString(":")}</span>"</span></span><br></pre></td></tr></table></figure></div></p><p>name 代表索引类别，version 代表索引版本，mode.flag 代表索引模式（是否支持读写，一般为3，支持读也支持写），attributes 代表是哪些字段需要建立该索引。spec 参数可以只有描述列属性的字段，即不带任何 useData 信息，GeoMesa 会默认添加索引信息，若存在空间和时间字段，则会默认建立 z3（空间字段为点 Point 类型） 或 xz3（空间字段为线面 非Point 类型） 索引，若有多个空间和时间字段，建立索引的字段为第一个空间和第一个时间字段；若只存在空间字段，则会建立 z2 或 xz2 索引；若只有时间字段，则默认建立时间属性索引。当然如没有在 spec 指明索引信息，可以在后续继续添加信息，如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="JAVA"><div class="code-copy"></div><figure class="highlight hljs java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> org.locationtech.geomesa.utils.interop.SimpleFeatureTypes;</span><br><span class="line"></span><br><span class="line">String spec = <span class="string">"name:String,dtg:Date,*geom:Point:srid=4326"</span>;</span><br><span class="line">SimpleFeatureType sft = SimpleFeatureTypes.createType(<span class="string">"mySft"</span>, spec);</span><br><span class="line"><span class="comment">// enable a default z3 and a default attribute index</span></span><br><span class="line">sft.getUserData().put(<span class="string">"geomesa.indices.enabled"</span>, <span class="string">"z3,attr:name"</span>);</span><br><span class="line"><span class="comment">// or, enable a default z3 and an attribute index with a Z2 secondary index</span></span><br><span class="line">sft.getUserData().put(<span class="string">"geomesa.indices.enabled"</span>, <span class="string">"z3,attr:name:geom"</span>);</span><br><span class="line"><span class="comment">// or, enable a default z3 and an attribute index with a temporal secondary index</span></span><br><span class="line">sft.getUserData().put(<span class="string">"geomesa.indices.enabled"</span>, <span class="string">"z3,attr:name:dtg"</span>);</span><br></pre></td></tr></table></figure></div><h2 id="坑篇">坑篇</h2><h3 id="导入-osm-数据问题">导入 OSM 数据问题</h3><p> 在<a href="https://www.geomesa.org/documentation/stable/user/convert/premade/osm.html">导入 osm 数据</a>时，若使用 osm-ways 作为 SimpleFeatureType，则 geomesa 会使用数据库存储 node 临时使用，这时其默认使用 H2 Database，若想使用其它数据库，则需要在 lib 导入相应 jdbc 包，若使用 postgresql 数据库，则 geomesa 会触发一个 bug，因为 postgresql 没有 double 类型，只有 double precision 类型，这将导致建表出错。详情见 geomesa/geomesa-convert/geomesa-convert-osm/src/main/scala/org/locationtech/geomesa/convert/osm/OsmWaysConverter.scala 中</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SCALA"><div class="code-copy"></div><figure class="highlight hljs scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="function"><span class="keyword">def</span> <span class="title">createNodesTable</span></span>(): <span class="type">Unit</span> = {</span><br><span class="line"> <span class="keyword">val</span> sql = <span class="string">"create table nodes(id BIGINT NOT NULL PRIMARY KEY, lon DOUBLE, lat DOUBLE);"</span></span><br><span class="line"> <span class="type">WithClose</span>(connection.prepareStatement(sql))(_.execute())</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p>所以若需要使用 geomesa-convert-osm 导入 osm 数据时，需要进入 geomesa/geomesa-convert/geomesa-convert-osm 文件夹中输入命令</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mvn dependency:copy-dependencies -DoutputDirectory=./depLib</span><br></pre></td></tr></table></figure></div><p>导出 geomesa-convert-osm 依赖包，将其中的 h2，osm4j，dynsax，trove4j 等一系列库放入 $GEOMESA_HBASE_HOME/lib 中。</p><h3 id="s2-索引问题">s2 索引问题</h3><p> s2 索引即 Google S2 Geometry 算法基于 Hilbert 曲线生成一种索引，GeoMesa 的 s2 索引是一个国人提交的，目前 3.2 版本只支持点的时空索引，不支持线面的时空索引，当然官方也在实现自己的 Hilbert 曲线，希望后续 GeoMesa 中会有 h2 索引。Shaun 在导入 osm 数据并启用 s2 索引时，报错，被提示不支持，对比 geomesa-index-api2Index.scala 和 geomesa-index-api2Index.scala 两文件的 defaults 函数可发现 S2Index 直接返回空，而在 geomesa-index-api.scala 中 fromName 函数需要调用 defaults 函数，从而导致 s2 索引不支持，修改 S2Index 的 defaults 函数即可（别忘了在 S2Index 类中首行加上 <code>import org.locationtech.geomesa.utils.geotools.RichSimpleFeatureType.RichSimpleFeatureType</code>）。</p><h2 id="后记">后记</h2><p> 暂时就了解了这么多，等后续熟悉的更多再继续更吧 (ง •_•)ง。</p><h2 id="附录">附录</h2><h3 id="geomesa-命令行工具部分参数">GeoMesa 命令行工具部分参数</h3><p>Geomesa 命令行参数：</p><table><colgroup><col style="width: 25%" /><col style="width: 74%" /></colgroup><thead><tr class="header"><th>参数</th><th>描述</th></tr></thead><tbody><tr class="odd"><td>-c, --catalog *</td><td>存放 schema 元数据的catalog 表（相当于数据库）</td></tr><tr class="even"><td>-f, --feature-name</td><td>schema 名（相当于数据库中的表）</td></tr><tr class="odd"><td>-s, --spec</td><td>要创建 SimpleFeatureType 的说明（即表中列的描述信息，表的 schema，如 "name:String,age:Int,dtg:Date,*geom:Point:srid=4326"）</td></tr><tr class="even"><td>-C, --converter</td><td>指定转换器，必须为一下之一：1、已经在classpath中的converter 名；2、converter 的配置（一个字符串）；3、包括converter的配置的名</td></tr><tr class="odd"><td>–converter-error-mode</td><td>自定义的转换器的error mode</td></tr><tr class="even"><td>-t, --threads</td><td>指定并行度</td></tr><tr class="odd"><td>–input-format</td><td>指定输入源格式（如csv, tsv, avro, shp, json,）</td></tr><tr class="even"><td>–no-tracking</td><td>指定提交的 ingest job何时终止（在脚本中常用）</td></tr><tr class="odd"><td>–run-mode</td><td>指定运行模式，必须为：local（本地）、distributed （分布式）、distributedcombine（分布式组合）之一</td></tr><tr class="even"><td>–split-max-size</td><td>在分布式中，指定切片最大大小（字节）</td></tr><tr class="odd"><td>–src-list</td><td>输入文件为文本文件，按行输入</td></tr><tr class="even"><td>–force</td><td>禁用任何的提示</td></tr><tr class="odd"><td>[files]…</td><td>指定输入的文件</td></tr></tbody></table><p>参考资料：<a href="https://blog.csdn.net/qq_21705851/article/details/93392202">GeoMesa命令行工具---摄取命令</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 需要做个 GeoMesa 的微服务，简单熟悉一下 GeoMesa。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="bigdata" scheme="http://cniter.github.io/tags/bigdata/"/>
<category term="geomesa" scheme="http://cniter.github.io/tags/geomesa/"/>
</entry>
<entry>
<title>IDEA使用Docker环境开发调试</title>
<link href="http://cniter.github.io/posts/3692cd6.html"/>
<id>http://cniter.github.io/posts/3692cd6.html</id>
<published>2021-01-10T08:50:12.000Z</published>
<updated>2021-12-18T11:54:13.933Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> IDEA 以前基本没用过，只是简单用过 Android Studio，还基本都忘记了 ( ╯□╰ )，以后应该会用 Scala 做一些大数据方面的东西，而大数据的环境都是 Linux 下的，而 Shaun 日常都是在 Windows 下开发，所以需要用日前做的容器环境来测试调试运行程序，简单记录一下 IDEA 在这方面的使用方法。</p><span id="more"></span><h2 id="运行篇">运行篇</h2><p> 右键项目名（HelloWorld），新建文件（New =》File），指定文件名为 <code>Dockerfile</code> 。写入内容示例如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="DOCKERFILE"><div class="code-copy"></div><figure class="highlight hljs dockerfile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">FROM</span> stc:<span class="number">2.0</span></span><br><span class="line"><span class="keyword">COPY</span><span class="bash"> ./target/classes/ /tmp</span></span><br><span class="line"><span class="keyword">WORKDIR</span><span class="bash"> /tmp</span></span><br><span class="line"><span class="keyword">ENTRYPOINT</span><span class="bash"> [<span class="string">"scala"</span>,<span class="string">"HelloWorld"</span>]</span></span><br></pre></td></tr></table></figure></div><p>点击左上角绿色双箭头，可编辑 Dockerfile（Edit 'Dockerfile'） ，指定当前上下文目录（Context folder），Contaier name 等容器启动选项。直接运行 Dockerfile（Run 'Dockerfile'），IDEA 即可自动创建容器，并在容器中运行程序，程序运行完则容器自动停止，若需要运行存在外部依赖的程序，则只能以 jar 包的方式运行。</p><p> 设置 IDEA 生成 jar 包如下：在最上面的菜单栏中 File =》Project Structure =》Artifacts =》+ =》JAR =》From modules with dependencies，选择 Main Class，点击右边的文件夹图标即可选择相应类，由于存在外部依赖，所以不能直接用默认的 extract to the target JAR，而是应该选择下面的 <strong>copy to the output directory and link via manifest</strong>，点击 OK 后，自动或手动选择导出的依赖 jar 包，点击 OK。在最上面的菜单栏中 Build =》Build Artifacts...，可在 out/artifacts/HelloWorld_jar 文件夹中生成所有 jar 包。之后编辑 Dockerfile， <strong>更改 Dockerfile 上下文目录</strong>为 out/artifacts/HelloWorld_jar ，指定容器名，在 Command 中输入 <code>java -jar HelloWorld.jar</code> 修改 Dockerfile 中第 2 行命令为 <code>COPY . /tmp</code>，修改第 4 行命令为 <code>CMD ["java", "-jar", "HelloWorld.jar"]</code>。之后运行 Dockerfile 即可在下面 Services 栏对应 Docker 容器 Attached Console 中看到程序运行结果。</p><h2 id="调试篇">调试篇</h2><p> 除了使用 IDEA 生成 jar 包外，还需要使用 IDEA 的远程调试功能，设置 IDEA 远程调试功能如下：在最上面的菜单栏中 Run =》Edit Configurations... =》+ =》Remote JVM Debug，上方的 Debugger mode 中使用默认的 Attach to remote JVM， 在下面的 Before launch 添加 Launch Docker before debug。在弹窗中选择相应 Dockerfile，在下方的 Custom command 中输入 <code>java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 -jar HelloWorld.jar</code>， 完成后即可使用该配置在 IDEA 调试容器中运行的程序。</p><h2 id="后记">后记</h2><p> 用这种方式使用 IDEA 确实达到了 Shaun 理想的结果，Windows 下开发，Docker 中调试和运行，应付简单的代码调试和运行确实是没问题，但是在复杂的分布式环境下总会碰到一些莫名奇妙的问题，这些问题就是纯粹的经验了。</p><h2 id="参考资料">参考资料</h2><p><a href="https://www.jetbrains.com/help/idea/supported-languages.html/running-a-java-app-in-a-container.html">Run a Java application in a Docker container</a></p><p><a href="https://www.jetbrains.com/help/idea/supported-languages.html/debug-a-java-application-using-a-dockerfile.html#create-dockerfile-run-config">Debug a Java application using a Dockerfile</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> IDEA 以前基本没用过，只是简单用过 Android Studio，还基本都忘记了 ( ╯□╰ )，以后应该会用 Scala 做一些大数据方面的东西，而大数据的环境都是 Linux 下的，而 Shaun 日常都是在 Windows 下开发，所以需要用日前做的容器环境来测试调试运行程序，简单记录一下 IDEA 在这方面的使用方法。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="devtool" scheme="http://cniter.github.io/tags/devtool/"/>
</entry>
<entry>
<title>大数据环境搭建笔记</title>
<link href="http://cniter.github.io/posts/af5e9ace.html"/>
<id>http://cniter.github.io/posts/af5e9ace.html</id>
<published>2021-01-03T01:50:26.000Z</published>
<updated>2021-12-18T11:54:13.940Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 准备开始搞时空数据了，先简单搭一下环境。</p><span id="more"></span><p>准备搭的环境为：jdk-1.8.0，hadoop-3.2.1，hbase-2.2.6，geomesa-hbase_2.11-3.1.0，spark-3.0.1-bin-hadoop3.2，geoserver-2.16.5-bin，geomesa-hbase_2.11-3.2.0-SNAPSHOT，所用的包都已下好并解压到 /home 目录下。</p><p><strong><em>※注</em></strong>： <em>hbase-2.2.6 暂不支持最新的 hadoop-3.3.0</em>，Hadoop 也最好使用 jdk-1.8.0，java-11 会有问题。</p><h2 id="hadoop-环境">Hadoop 环境</h2><p> 首先修改 /etc/hosts 文件中本机 ip 对应的名称为 master，若在容器中安装则需要在 run 开启容器就指定 <code>--hostname master</code>，否则改了也没用，下次启动容器时 hostname 又会回到初始状态，下面开启正式的配置。</p><p>修改 /home/hadoop-3.2.1/etc/hadoop/hadoop-env.sh 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> JAVA_HOME=<span class="variable">$JAVA_HOME</span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hadoop-3.2.1/etc/hadoop/core-site.xml 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="XML"><div class="code-copy"></div><figure class="highlight hljs xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- master 前面配置的主机名称 --></span></span><br><span class="line"> <span class="comment"><!-- <property></span></span><br><span class="line"><span class="comment"> <name>fs.default.name</name></span></span><br><span class="line"><span class="comment"> <value>hdfs://master:9000</value></span></span><br><span class="line"><span class="comment"> </property> --></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>fs.defaultFS<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://master:9000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hadoop.tmp.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/home/hadoop/data/tmp<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hadoop-3.2.1/etc/hadoop/hdfs-site.xml 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="XML"><div class="code-copy"></div><figure class="highlight hljs xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="comment"><!--指定SecondaryNameNode位置--></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.namenode.secondary.http-address<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>master:9001<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hadoop-3.2.1/etc/hadoop/yarn-site.xml 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="XML"><div class="code-copy"></div><figure class="highlight hljs xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"></span><br><span class="line"><span class="comment"><!-- Site specific YARN configuration properties --></span></span><br><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>yarn.nodemanager.aux-services<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>mapreduce_shuffle<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"><span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hadoop-3.2.1/etc/hadoop/mapred-site.xml 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="XML"><div class="code-copy"></div><figure class="highlight hljs xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>mapreduce.framework.name<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>yarn<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure></div><p>在 /home/hadoop-3.2.1/sbin/start-dfs.sh 和 /home/hadoop-3.2.1/sbin/stop-dfs.sh 文件头添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/usr/bin/env bash</span></span><br><span class="line">HDFS_DATANODE_USER=root</span><br><span class="line">HDFS_DATANODE_SECURE_USER=hdfs</span><br><span class="line">HDFS_NAMENODE_USER=root</span><br><span class="line">HDFS_SECONDARYNAMENODE_USER=root</span><br></pre></td></tr></table></figure></div><p>在 /home/hadoop-3.2.1/sbin/start-yarn.sh 和 /home/hadoop-3.2.1/sbin/stop-yarn.sh 文件头添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#!/usr/bin/env bash</span></span><br><span class="line">YARN_RESOURCEMANAGER_USER=root</span><br><span class="line">HADOOP_SECURE_DN_USER=yarn</span><br><span class="line">YARN_NODEMANAGER_USER=root</span><br></pre></td></tr></table></figure></div><p>设置环境变量，在 /etc/profile 中添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">Hadoop Environment Setting</span></span><br><span class="line">export HADOOP_HOME=/home/hadoop-3.2.1</span><br><span class="line">export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native</span><br><span class="line">export LD_LIBRARY_PATH=$JAVA_LIBRARY_PATH</span><br><span class="line">export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin</span><br></pre></td></tr></table></figure></div><p>由于容器中默认为 root 用户，所以在 /root/.bashrc 文件末尾添加 <code>source /etc/profile</code>，以开机启用设置的环境变量。</p><p>在启动 Hadoop 之前需要执行 <code>hdfs namenode -format</code> 进行格式化，启动命令为 <code>/home/hadoop-3.2.1/sbin/start-all.sh</code>。<em>后续若需要清空并重新设置 Hadoop 时，必须先删除 /home/hadoop/ 目录，再重新进行格式化。</em></p><h2 id="hbase-环境">HBase 环境</h2><p>修改 /home/hbase-2.2.6/conf/hbase-env.sh 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> JAVA_HOME=<span class="variable">$JAVA_HOME</span></span><br><span class="line"><span class="comment"># 使用自带的ZooKeeper管理</span></span><br><span class="line"><span class="built_in">export</span> HBASE_MANAGES_ZK=<span class="literal">true</span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hbase-2.2.6/conf/hbase-site.xml 文件，添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="XML"><div class="code-copy"></div><figure class="highlight hljs xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> </span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.rootdir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://master:9000/hbase<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.dynamic.jars.dir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>hdfs://master:9000/hbase/lib<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.cluster.distributed<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>true<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>dfs.replication<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>1<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.master.maxclockskew<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>180000<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">description</span>></span>Time difference of regionserver from master<span class="tag"></<span class="name">description</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.zookeeper.quorum<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>localhost<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="comment"><!-- 修改默认8080 端口--></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.rest.port<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>8088<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- 2181 默认端口，尽量不要修改，geomesa-hbase 导入数据时默认连接端口为 2181--></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.zookeeper.property.clientPort<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>2181<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.zookeeper.property.dataDir<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>/home/hbase/data<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.unsafe.stream.capability.enforce<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>false<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- geomesa-hbase --></span></span><br><span class="line"> <span class="tag"><<span class="name">property</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>hbase.coprocessor.user.region.classes<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">value</span>></span>org.locationtech.geomesa.hbase.server.coprocessor.GeoMesaCoprocessor<span class="tag"></<span class="name">value</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">property</span>></span></span><br><span class="line"></span><br><span class="line"><span class="tag"></<span class="name">configuration</span>></span></span><br></pre></td></tr></table></figure></div><p>修改 /home/hbase-2.2.6/conf/regionservers 文件，修改为（原来为 localhost）</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="PLAINTEXT"><div class="code-copy"></div><figure class="highlight hljs plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">master</span><br></pre></td></tr></table></figure></div><p>设置环境变量，在 /etc/profile 中添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#</span><span class="bash">HBase Environment Setting</span></span><br><span class="line">export HBASE_HOME=/home/hbase-2.2.6</span><br><span class="line">export PATH=$PATH:$HBASE_HOME/bin</span><br></pre></td></tr></table></figure></div><p>配置好之后，执行 <code>start-hbase.sh</code> 启动 HBase。</p><h2 id="spark-环境">Spark 环境</h2><p>修改 /home/spark-3.0.1-bin-hadoop3.2/conf/spark-env.sh 文件，在文件末尾添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 配置JAVA_HOME，一般来说，不配置也可以，但是可能会出现问题，还是配上吧</span></span><br><span class="line"><span class="built_in">export</span> JAVA_HOME=<span class="variable">$JAVA_HOME</span></span><br><span class="line"><span class="comment"># 一般来说，spark任务有很大可能性需要去HDFS上读取文件，所以配置上</span></span><br><span class="line"><span class="comment"># 如果说你的spark就读取本地文件，也不需要yarn管理，不用配</span></span><br><span class="line"><span class="built_in">export</span> HADOOP_CONF_DIR=<span class="variable">$HADOOP_HOME</span>/etc/hadoop</span><br><span class="line"></span><br><span class="line"><span class="comment"># 设置Master的主机名</span></span><br><span class="line"><span class="built_in">export</span> SPARK_MASTER_HOST=master</span><br><span class="line"><span class="comment"># 提交Application的端口，默认就是这个，万一要改呢，改这里</span></span><br><span class="line"><span class="built_in">export</span> SPARK_MASTER_PORT=7077</span><br><span class="line"><span class="comment"># 每一个Worker最多可以使用的cpu core的个数，我虚拟机就一个...</span></span><br><span class="line"><span class="comment"># 真实服务器如果有32个，你可以设置为32个</span></span><br><span class="line"><span class="built_in">export</span> SPARK_WORKER_CORES=1</span><br><span class="line"><span class="comment"># 每一个Worker最多可以使用的内存，我的虚拟机就2g</span></span><br><span class="line"><span class="comment"># 真实服务器如果有128G，你可以设置为100G</span></span><br><span class="line"><span class="built_in">export</span> SPARK_WORKER_MEMORY=2g</span><br><span class="line"><span class="comment"># master web UI端口默认8080</span></span><br><span class="line"><span class="built_in">export</span> SPARK_MASTER_WEBUI_PORT=8090</span><br><span class="line"><span class="comment"># worker web UI端口默认8081</span></span><br><span class="line"><span class="built_in">export</span> SPARK_WORKER_WEBUI_PORT=8089</span><br></pre></td></tr></table></figure></div><p>复制 /home/spark-3.0.1-bin-hadoop3.2/conf/slaves.template 文件，并重命名为 slaves，将该文件尾修改为</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 里面的内容原来为localhost，改为master </span></span><br><span class="line">master</span><br></pre></td></tr></table></figure></div><p>设置环境变量，在 /etc/profile 中添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SHELL"><div class="code-copy"></div><figure class="highlight hljs shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">export SPARK_HOME=/home/spark-3.0.1-bin-hadoop3.2</span><br><span class="line">export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin</span><br></pre></td></tr></table></figure></div><p>将 /home/spark-3.0.1-bin-hadoop3.2/sbin/start-all.sh 重命名为 start-spark-all.sh，将 /home/spark-3.0.1-bin-hadoop3.2/sbin/stop-all.sh 重命名为 stop-spark-all.sh，执行 <code>start-spark-all.sh</code> 启动 Spark。</p><h2 id="geomesa-hbase-环境">geomesa-hbase 环境</h2><h3 id="编译-geomesa">编译 geomesa</h3><p>克隆 <a href="https://gitee.com/mirrors/geomesa">LocationTech GeoMesa</a> ，<strong>修改 pom.xml</strong>，即修改对应依赖的 hadoop 和 hbase 以及 spark 版本（spark 最新的3.0.1版本由 Scala-2.12 编译，而 Geomesa 编译目前采用 Scala-2.11， 所以 Spark 不能使用最新的版本，只能用 2.4.7）。进入 geomesa 根目录，使用命令</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">mvn clean install -DskipTests</span><br><span class="line"></span><br><span class="line"><span class="comment"># 或仅编译 geomesa-hbase</span></span><br><span class="line">mvn clean install -pl geomesa-hbase -am -DskipTests</span><br></pre></td></tr></table></figure></div><p>编译 geomesa，中间可能会失败很多次，包下不来，可能需要挂代理或换源，重复使用命令多次即可。</p><h3 id="配置-geomesa-hbase">配置 geomesa-hbase</h3><p>将 /home/geomesa/geomesa-hbase/geomesa-hbase-dist/target/geomesa-hbase_2.11-3.2.0-SNAPSHOT-bin.tar.gz 解压为 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT，<strong>将 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT/dist/hbase/geomesa-hbase-distributed-runtime-hbase2_2.11-3.2.0-SNAPSHOT.jar 复制到 /home/hbase-2.2.6/lib/ 文件夹中</strong>，修改 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT/conf/dependencies.sh 文件，设置正确的Hadoop 和 hbase 版本，依次执行 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT/bin/install-dependencies.sh 和 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT/bin/install-shapefile-support.sh。设置环境变量，在 /etc/profile 中添加</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> GEOMESA_HBASE_HOME=/home/geomesa-hbase_2.11-3.2.0-SNAPSHOT</span><br><span class="line"><span class="built_in">export</span> GEOMESA_LIB=<span class="variable">$GEOMESA_HBASE_HOME</span>/lib</span><br><span class="line"><span class="built_in">export</span> GEOMESA_CONF_DIR=<span class="variable">${GEOMESA_HBASE_HOME}</span>/conf</span><br><span class="line"><span class="built_in">export</span> CLASSPATH=<span class="variable">$CLASSPATH</span>:<span class="variable">$GEOMESA_LIB</span>:<span class="variable">$GEOMESA_CONF_DIR</span></span><br><span class="line"><span class="built_in">export</span> PATH=<span class="variable">$PATH</span>:<span class="variable">$GEOMESA_HBASE_HOME</span>/bin</span><br></pre></td></tr></table></figure></div><h3 id="测试-geomesa-hbase">测试 geomesa-hbase</h3><p>启动 Hadoop 和 HBase 之后，可直接使用命令</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">geomesa-hbase ingest --catalog TestGeomesa --feature-name road --input-format shp <span class="string">"/home/shpdata/road.shp"</span></span><br></pre></td></tr></table></figure></div><p>导入 shp 数据，<strong>shp 不能有 id 字段</strong>，因为 Geomesa 在创建表时会默认生成一个 id 字段。</p><p>也可克隆 <a href="https://gitee.com/xfilove/geomesa-tutorials">geomesa-tutorials</a> ，同样修改其中的 pom.xml 文件，进入 geomesa-tutorials 根目录，使用命令</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">mvn clean install -pl geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart -am</span><br></pre></td></tr></table></figure></div><p>编译 geomesa-tutorials，编译完成后，使用命令</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">java -cp geomesa-tutorials-hbase/geomesa-tutorials-hbase-quickstart/target/geomesa-tutorials-hbase-quickstart-3.2.0-SNAPSHOT.jar org.geomesa.example.hbase.HBaseQuickStart --hbase.zookeepers localhost --hbase.catalog geomesaTest</span><br></pre></td></tr></table></figure></div><p>导入数据进 Hbase，导入成功后可通过 <code>hbase shell</code> 进入 hbase，在 hbase shell 中通过 <code>list</code> 查看 hbase 现有的表。</p><h3 id="整合-geoserver">整合 geoserver</h3><p>导入依赖插件</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">manage-geoserver-plugins.sh -l <span class="variable">${GEOSERVER_HOME}</span>/webapps/geoserver/WEB-INF/lib/ -i</span><br></pre></td></tr></table></figure></div><p>修改 /home/geomesa-hbase_2.11-3.2.0-SNAPSHOT/bin/install-dependencies.sh 中第33行：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="SH"><div class="code-copy"></div><figure class="highlight hljs sh"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># install_dir="${GEOMESA_HBASE_HOME}/lib"</span></span><br><span class="line">install_dir=<span class="string">"<span class="variable">${GEOSERVER_HOME}</span>/webapps/geoserver/WEB-INF/classes"</span></span><br></pre></td></tr></table></figure></div><p>执行 install-dependencies.sh 安装插件，安装完后将 classes 中的 lib 都移到 ${GEOSERVER_HOME}/webapps/geoserver/WEB-INF/lib中。</p><h2 id="后记">后记</h2><p> 环境搞起来真麻烦，在编译和运行 Geomesa 时总能遇到一些莫名奇妙的问题，Java 系的这一套确实很麻烦，尤其是各种依赖关系，不过最后总算是搞好了，能直接在 geoserver 中看到 geomesa 存在 hbase 里的地图。</p><h2 id="参考资料">参考资料</h2><p><a href="https://www.cnblogs.com/Mr-lin66/p/13378248.html">Centos7系统 Hadoop+HBase+Spark环境搭建</a></p><p><a href="https://blog.csdn.net/weixin_41834634/article/details/89176473">GeoMesa-HBase操作篇——安装</a></p><p><a href="https://blog.csdn.net/hsg77/article/details/82221531">centos7安装geomesa2.0.2_hbase_geoserver2.13.2的方法</a></p><p><a href="https://www.cnblogs.com/heidsoft/p/8558419.html">hadoop fs 命令使用</a></p><p><a href="https://www.geomesa.org/documentation/stable/tutorials/geomesa-quickstart-hbase.html">GeoMesa HBase Quick Start</a></p><p><a href="https://www.geomesa.org/documentation/stable/user/hbase/install.html">Installing GeoMesa HBase</a></p><p><a href="https://www.cnblogs.com/yszd/p/10039249.html">Spark完全分布式集群搭建【Spark2.4.4+Hadoop3.2.1】</a></p><h2 id="附录">附录</h2><p>最后附上一些常用的端口及说明：</p><h3 id="hbase">Hbase</h3><table><thead><tr class="header"><th>配置</th><th>端口</th><th>说明</th><th></th></tr></thead><tbody><tr class="odd"><td>hbase.master.port</td><td>16000</td><td>HMaster绑定端口</td><td></td></tr><tr class="even"><td><strong>hbase.master.info.port</strong></td><td><strong>16010</strong></td><td><strong>HBase Master的Web UI端口</strong></td><td></td></tr><tr class="odd"><td>hbase.regionserver.port</td><td>16020</td><td>HBase RegionServer绑定的端口</td><td></td></tr><tr class="even"><td><strong>hbase.regionserver.info.port</strong></td><td><strong>16030</strong></td><td><strong>HBase RegionServer的Web UI端口</strong></td><td></td></tr><tr class="odd"><td>hbase.zookeeper.property.clientPort</td><td>2181</td><td>Zookeeper客户端连接端口</td><td></td></tr><tr class="even"><td>hbase.zookeeper.peerport</td><td>2888</td><td>Zookeeper节点内部之间通信的端口</td><td></td></tr><tr class="odd"><td>hbase.zookeeper.leaderport</td><td>3888</td><td>Zookeeper用来选举主节点的端口</td><td></td></tr><tr class="even"><td>hbase.rest.port</td><td>8080</td><td>HBase REST server的端口</td><td></td></tr><tr class="odd"><td>hbase.master.port</td><td>60000</td><td>HMaster的RPC端口</td><td></td></tr><tr class="even"><td>hbase.master.info.port</td><td>60010</td><td>HMaster的http端口</td><td></td></tr><tr class="odd"><td>hbase.regionserver.port</td><td>60020</td><td>HRegionServer的RPC端口</td><td></td></tr><tr class="even"><td>hbase.regionserver.info.port</td><td>60030</td><td>HRegionServer的http端口</td><td></td></tr><tr class="odd"><td></td><td></td><td></td><td></td></tr></tbody></table><h3 id="hadoop">Hadoop</h3><table><thead><tr class="header"><th>配置</th><th>端口</th><th>说明</th><th></th></tr></thead><tbody><tr class="odd"><td>fs.defaultFS</td><td>9000</td><td>hdfs访问端口</td><td></td></tr><tr class="even"><td>dfs.namenode.rpc-address</td><td>9001</td><td>DataNode会连接这个端口</td><td></td></tr><tr class="odd"><td>dfs.datanode.address</td><td>9866</td><td>DataNode的数据传输端口</td><td></td></tr><tr class="even"><td><strong>dfs.namenode.http-address</strong></td><td><strong>9870</strong></td><td><strong>namenode的web UI 端口</strong></td><td></td></tr><tr class="odd"><td><strong>yarn.resourcemanager.webapp.address</strong></td><td><strong>8088</strong></td><td><strong>YARN的http端口</strong></td><td></td></tr></tbody></table><h3 id="spark">Spark</h3><table><thead><tr class="header"><th></th><th>端口</th><th>说明</th><th></th></tr></thead><tbody><tr class="odd"><td></td><td>8080</td><td>master的webUI，Tomcat的端口号（已修改为8090）</td><td></td></tr><tr class="even"><td></td><td>8081</td><td>worker的webUI的端口号（已修改为8089）</td><td></td></tr><tr class="odd"><td></td><td>18080</td><td>historyServer的webUI的端口号</td><td></td></tr></tbody></table><p>需开放端口 22，2181，5432，8080，8088，8089，8090，9870，16010，16030。</p><p><code>docker run -dit --privileged=true --name STC2 --hostname master -v E:/Docker/ShareFile:/mnt/sharefile -p 22:22 -p 80:80 -p 2181:2181 -p 5432:5432 -p 8080-8090:8080-8090 -p 9870:9870 -p 16010:16010 -p 16030:16030 stc:2.0 init</code></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 准备开始搞时空数据了，先简单搭一下环境。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="note" scheme="http://cniter.github.io/tags/note/"/>
<category term="bigdata" scheme="http://cniter.github.io/tags/bigdata/"/>
</entry>
<entry>
<title>设计模式浅谈</title>
<link href="http://cniter.github.io/posts/ae780057.html"/>
<id>http://cniter.github.io/posts/ae780057.html</id>
<published>2020-12-27T10:28:56.000Z</published>
<updated>2021-12-18T11:54:13.943Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 进入职场一年半以来，Shaun 完全独立从 0 到 1 开发了 1.5 个项目（当然也有参与其它项目，但不是 Shaun 独立从 0 到 1 开发的，没多少控制权，就不谈了），一个网页版的高精地图编辑器，半个地图可视化系统，这里面 Shaun 用了不少设计模式，这篇就谈谈 Shaun 用过的和没用过的一些设计模式。</p><span id="more"></span><p> 以「Head First 设计模式」为参考，Shaun 用 C++ 实现了一遍书中的例子（代理模式及其后面的模式除外），下面进入正文。</p><h2 id="模式篇">模式篇</h2><h3 id="策略模式">策略模式</h3><p> Shaun 个人认为最能体现面向对象编程思想（抽象封装继承多态）的一种模式，换句话说，只要真正理解和运用面向对象编程，一定会自然而然的用到策略模式。Shaun 在做高精地图编辑器时，需要设计一个渲染模块，渲染模块会包含高亮行为，高亮有两种，一种是直接改变颜色，一种是使用后期处理（OutlinePass 或 UnrealBloomPass 等）进行高亮，这时就需要在渲染类中组合高亮行为。</p><p> 策略模式中涉及到的原则有：1、封装变化；2、多用组合，少用继承；3、针对接口编程，不针对实现编程。封装变化这点很考验程序员的经验水平，在写代码之初，往往预料不到变化，所以这一点一般是在编码过程中逐渐完善的，不断进行抽象，从而生成比较合理的基类；第二点一般也是对的，但有时在编码过程中难免会碰到到底是用继承还是组合的问题，这时候可以多想想，组合并不是万能的，有时继承更合适，这时可以请教身边更有经验的程序员，组合的优势在于当子类不想要这个对象时，可以随时丢弃，而继承的优势在于，当子类不想实现这个行为时，可以有默认的行为，而且有些时候只能用继承；针对接口编程没啥好说的，就是抽象。</p><h3 id="观察者模式">观察者模式</h3><p> 这个模式如果在分布式系统中又叫发布订阅模式，该模式常用于消息通知。前端有个 RxJS 的库将这一模式玩出花来了，Shaun 在高精地图编辑器的事件流管理中就使用了该库。在 threejs 中所有渲染对象的都有一个统一的基类 EventDispatcher，该类中就实现了一个简单的观察者模式，用来处理事件监听触发和通知，和真正的观察者相比，区别在于观察者只是一个函数，而不是一个类，其实浏览器的事件监听机制实现方式也和这个类差不多。</p><p> 观察者模式中涉及到原则有：松耦合。这里的松耦合是指主题和观察者之间是隔离的，观察者可自行实现自己的更新行为，而主题同样可实现自己的通知机制，两者虽有关联但互不影响。松耦合原则说起来人人会说，但真正能实现松耦合的却不多，实现松耦合的关键在于怎样分离两个系统，两个系统的连接点在哪，这有时很难理清，从而造成逻辑混乱，bug 丛生。</p><h3 id="装饰者模式">装饰者模式</h3><p> 利用该模式可以很方便的扩展一些方法和属性，尤其是遇到主体和配件这样的关系时，可以很轻松的添加配件到主体上。Shaun 没用过这个模式，本来在扩展 threejs 一个类时想用，但确实没找到非常明确的主体和配件这样的关系，最后还是简单的使用继承了。</p><p> 装饰者模式涉及到的原则有：开放——封闭原则。设计一个类需要考虑对扩展开放，对修改关闭。修改和扩展看似矛盾，但实则可以独立存在，装饰者的配件可以无限加，这是扩展，是开放，而在加配件时无需修改现有代码，这是封闭。当然这一原则并不独属于装饰者模式，应该说是只要用到面向对象的思想开发程序，就一定会用到该原则，否则也就失去了面向对象的意义。但有时这个原则又没必要贯彻彻底，因为对于有些需求可能很难弄清修改和扩展的界限，这时就达到能尽量重用父类的方法就好。</p><h3 id="工厂模式">工厂模式</h3><p> 该模式在稍微大一点的系统中应该都会用到，根据不同的输入，生成不同的对象，这就是工厂模式的本质。至于工厂模式的实现方式一般会根据需求的复杂度来决定：1、只有一个工厂，一类产品，只是为了集中一层 if-else，可用简单工厂模式，甚至一个 builder 函数即可；2、有多个工厂，还是只有一类产品，用工厂模式，多个工厂继承一个工厂父类即可，相当于多个简单工厂组合；3、有多个工厂，多类产品，哪个工厂生产什么产品可能有变化，这时需要用到抽象工厂模式，除正常的继承之外，还需使用组合，组合组成产品的父类，相当于再组合一个工厂。Shaun 在高精地图编辑器中当然是大量使用的工厂模式和简单工厂模式，主要是为了集中 if-else 的处理，比如根据不同的数据类型创建不同的属性栏界面（枚举用下拉框，字符串用文本框，数字用数字栏等），根据不同的路网元素创建对应的渲染器对象以及对应的属性界面等。</p><p> 工厂模式涉及到的原则有：依赖倒置原则。尽量依赖抽象，而不是具体类。这其实也是抽象一种作用或好处，即在使用过程中尽量使用最上层的父类，具体类只在创建实例时使用。</p><h3 id="单例模式">单例模式</h3><p> 写程序的基本都会用到该模式，主要用来创建全局唯一对象，可用来存储和处理系统中常用的各个模块都会用到的一些数据。Shaun 在编辑器中单例模式用了好几个，比如全局唯一的 viewport，用力绘制 3d 图形；全局唯一的路网数据；当然系统中存在太多的单例模式也不好，最好是只有一个，如 Shaun 的编辑器中最好的模式就是创建一个单例的 Editor 类，需要做单例的对象都可以放在该类中，如此保证系统中只有一个单例类，以进行统一管理。</p><p> 该模式与面向对象倒是没多大关系了，可以认为是全局变量的优化版，毕竟大的系统中全局变量基本不可避免，这时就可以使用单例模式。</p><h3 id="命令模式">命令模式</h3><p> 该模式主要用来将函数方法封装成类，这样做的好处就是可以更灵活的执行该方法（将方法放进队列中依次执行，将方法持久化以便系统启动执行），同时也可以保存该方法的一些中间状态，以便撤销操作，回到系统执行该方法前的状态。Shaun 在编辑器中主要用命令模式做撤销重做功能，这基本也是编辑器的必备功能了，可以说没有撤销重做功能的编辑器是不完整的，要实现撤销重做功能除了基本的命令模式之外，还要提供撤销和重做两个栈以保存操作命令。</p><p> 该模式与面向对象也没很大关系，只是提供了一个实现一些特殊功能的标准或通用方案。</p><h3 id="适配器模式">适配器模式</h3><p> 该模式正如其名，主要用来转换接口，将一个类的方法用其它类封装一下，以达到兼容其它类接口的目的，同时对外可接口保持不变，该模式通过对象组合实现。Shaun 没使用过该模式，就 Shaun 感觉这个模式应该可以用在维护了好几年的系统上，当新作的东西需要兼容老接口时，可以用适配器模式将新街口封装一下。</p><p> 该模式同样只是提供了一种新接口兼容老接口的一种优良方案，当然实际使用过程中可能很难这么完美，但算是一种思路。</p><h3 id="外观模式">外观模式</h3><p> 该模式算是封装的一种体现。当一个功能需要经过多次函数调用才能完成时，这时可以用另一个方法将这些函数都封装起来，从而简化调用方式。Shaun 用该模式处理整个渲染模块的初始化和资源释放，因为初始化时需要分配好很多东西（光照，viewport，固定图层，地面，天空盒等），而释放时同样需要释放这些东西。该模式同样只能算是提供了一种好的编程实践，实际使用过程可能每个函数都有很多参数，调用顺序可能有变，这时简化调用反而没有必要，让用户自己决定怎样调用更好。</p><p> 外观模式涉及到的原则有：最少知识原则。该原则主要用来减少对象依赖，即尽量不将类中组合的对象直接暴露出去，而应该将组合对象的方法再简单封装一下，再将封装后的方法暴露出去，以减少另外的类又依赖类中组合对象的现象。该原则可以适当遵守，因为有时直接使用更方便一点，多次封装之后反而显得逻辑混乱，增加系统的复杂度。</p><h3 id="模板方法模式">模板方法模式</h3><p> 该模式是抽象的一种体现。首先抽象出一套固定化的流程，流程中每个步骤的具体行为并一致，有些默认，有些可以重写，父类固定流程，子类负责重写流程中每个步骤，这就时模板方法模式。Shaun 没写过完全符合该模式的代码，只是写了个类似该模式的模块，该模块有三个功能（编辑道路节点，编辑车道节点，编辑车道线），做完前两个功能后，发现这里有一套逻辑是通用的，那就是滑过节点高亮，选择节点，出现 gizmo，拖动 gizmo，完成编辑（当然还有选择节点后不拖动 gizmo 等一套 if-else 中间处理状态），于是 Shaun 把这一套流程抽象出来，固化方法，这三个功能都继承该类，方法该重写的重写，不仅减少了代码量，同时整个流程也更清晰了，很快完成了第三个功能。</p><p> 模板方法涉及到的原则有：好莱坞原则。即由父类负责函数调用，而子类负责重写被调用的函数，不用管父类的调用逻辑，也最好不要调用父类的函数。该原则用来理清流程很方便，只需要看父类即可，但实际编程过程中可能也会遇到子类不可避免的会调用父类的一些公共函数的情况，Shaun 觉得只要流程没问题的话，调用父类函数也能接受，并不需要严格遵守模式。</p><h3 id="迭代器模式">迭代器模式</h3><p> 迭代器，即对遍历进行封装，一般只能顺序执行，提供 next() 方法获取下一个元素，集合容器的遍历方式一般都会用迭代器进行封装。Shaun 在这一个半项目里没写过迭代器，毕竟这是非常底层的一个模式，语言库本身有的数据结构大多自己实现了迭代器，除非需要设计一个新的集合或容器数据结构，才需要提供相应的迭代器。因为 js 没有 SortedMap 数据结构，为了高效分配路网元素 id，Shaun 利用 object 简单实现了一个，提供了相应的 forEach 方法。</p><p> 迭代器模式涉及到的原则有：单一责任原则。即一类只做一件事，这个原则对于涉及最最底层的接口很实用，而大多具体类很难只做一件事。迭代器模式对于顺序访问来说还是非常有用的，毕竟使用迭代器的人不需要管底层到底用的什么数据结构，反正可以顺序遍历即可。</p><h3 id="组合模式">组合模式</h3><p> 组合模式与其说是一种模式，更不如说就是一颗树，只是树的节点都是可供继承的类。在标准的组合模式中，父类中一定会有全部子类的全部函数，即所有子类的函数要么是继承自父类，要么是重写父类函数的，这其实是违背上面单一责任原则的，因为这必然会造成有些子类不需要这么多函数。而从组合模式会存储孩子节点这点来看，和装饰者模式有点类似，只不过装饰者只会存一个孩子，而组合模式可能会存多个，当然两者做的事是不一样，只是实现手法类似而已。Shaun 没写过标准的组合模式，如果只要符合树形模式都可认为是组合模式，那在高精地图编辑器中，所有路网元素都会继承一个父类，而道路中又包含车道簇，车道簇中包含车道，这也算组合模式。在 threejs 中有个 Object3D 的基类，所有渲染对象都会继承该类，该类中又包含若干孩子，threejs 计算 Model 矩阵时就是一层层孩子的 Model 矩阵乘下去，直到最后的孩子，结果就是最后 Shader 中的 Model 矩阵。</p><h3 id="状态模式">状态模式</h3><p> 状态机的状态转移可以说是程序设计中最麻烦的部分之一了，这部分如果写不好的话，根本没法修改维护，同时会造成 bug 频发。在高精地图编辑器中鼠标操作有两类模式，一种是选择模式，另一种是编辑模式，选择模式又分为点选和框选，而编辑模式就非常多了，针对路网的不同元素，编辑模式的具体实现都不会一样，Shaun 首先使用 RxJS 封装了一个鼠标操作类（左键右键中键移动等），后续的鼠标操作都继承自该类，可以算是状态模式的父类，后续的鼠标操作就针对相应的需求实现相应的方法即可，当然其中鼠标操作自身也存在状态转移（左键到右键，左键到鼠标移动等），这些一般都是针对特定需求来的，所以这些小的状态转移一般在鼠标操作内部实现，但需要支持随时从编辑模式到选择模式，这意味着编辑模式编辑到一半的东西都需要支持随时释放，恢复编辑前的样子，这算是一个麻烦的地方，有时忘了释放就会出现问题。</p><p> 状态模式算是为解决状态转移问题提供一种理想的方案，但其具体实现并不一定要和书上一样，Shaun 在用 C++ 实现时就采用另一套方案，状态类是独立的，控制状态转移的代码都在状态机内，而不是书中这种直接在状态类中控制状态机。好处坏处都有，看具体需求，Shaun 的方式就是状态类和状态机是分离的，状态类不需要管状态机怎么实现的，只需要管当前状态的情况，但需要在状态机中管理状态转移，而书中实现方式状态机的状态转移放到状态类中了，也因此状态类需要依赖状态机。</p><hr /><p><em>剩下的模式，Shaun 就没直接写代码实践了，因为大多都需要跨模块实现，有的甚至就是个小项目了，所以就简要谈谈 Shaun 的个人理解</em>。</p><h3 id="代理模式">代理模式</h3><p> 主要可以用来做权限控制，在模块与模块之间的调用过程中，有时不想要一个模块可以访问另一个模块的全部内容，这时可以使用代理模式，创建一个中间模块，避免两个模块直接调用，同时进行访问控制。代理模式在如今的互联网时代不可避免的会用到，或直接或间接，往最小的说，对象组合也可用来实现代理模式。</p><h3 id="复合模式">复合模式</h3><p> 将多种模式组合在一起使用，比如 MVC 模式，这种模式与其说是模式，更不如说就是一种架构，一种开发包含客户端系统的通用架构，当然每一层都会有很多模式进行组合，从而造成具体实现差异非常大。</p><h3 id="反模式">反模式</h3><p> 反模式指的是用“不好的解决方案”去解决一个问题，Shaun 主要想谈谈开发反模式，因为这非常常见。有时候一个解决方案好不好要从多个角度进行衡量，比如现有技术，长期短期，上手难度，开发效率，维护难度等角度，当出现一个新问题时，往往意味着就有解决方案有缺陷，这种缺陷可能很容易弥补，更可能很难，当很难解决时，往往要采用全新的解决方案，这时团队对新解决方案可能都不熟，也没有魄力去采用新解决方案，只能去老解决方案继续强行打补丁，直到最后没法维护，白白浪费了大量的人力和时间，这是非常典型的一种反模式。</p><h3 id="桥接模式">桥接模式</h3><p> 将抽象接口和实现分开，并独立派生，以支持抽象和实现的同时改变，并相互独立，可适用在需要跨平台跨不同设备的系统上。</p><h3 id="生成器模式">生成器模式</h3><p> 有点像是模板方法模式和工厂模式的结合版，使用多个步骤创建一个对象，但步骤没有固定顺序，可适用于流程复杂的规划系统中。</p><h3 id="责任链模式">责任链模式</h3><p> 可以认为是模板方法模式的进阶版，只是模板的步骤方法变成了一个个对象，并且支持步骤的增加和删除，以及更换顺序，一旦某个步骤成功执行，则整个链条终止，可适用于消除显式的 if-else，处理键盘或鼠标事件时，需要针对不同按键触发不同操作，这时可以采用该模式，缺点是链条很长时，要写很多类，导致执行啥很不直观。</p><h3 id="蝇量模式">蝇量模式</h3><p> 这个模式算是一种优化内存占用的方案，通过牺牲类的独立性来减少内存，更彻底一点就是不创建类，直接用函数调用来处理就行。</p><h3 id="解释器模式">解释器模式</h3><p> 可用来实现简单语法规则语言的解释器或编译器，每个语法规则都由一个类进行解析，然后组合。</p><h3 id="中介者模式">中介者模式</h3><p> 可认为是状态模式和代理模式的结合版，不过各个状态可以是不同类，由中介者控制系统流转，集中控制逻辑，使被控制对象解耦，但可能会造成中介者本身非常复杂。</p><h3 id="备忘录模式">备忘录模式</h3><p> 可用于系统（游戏）存档，存储系统中关键对象的运行状态，通常实现的方案一般是序列化/持久化，为了效率考虑，难的是有时需要增量存档。</p><h3 id="原型模式">原型模式</h3><p> js 的原型链应该是原型模式的典型，不仅实现了动态扩展实例，更实现了动态扩展对象，即继承。在高精地图编辑器中，由于需要做自动保存，所以在做序列化和反序列化的同时也简单实现了对象的 clone()，即从当前实例中创建一个完全一样的实例，可认为是 C++ 中的深拷贝。</p><h3 id="访问者模式">访问者模式</h3><p> 相当于加个中间层，从而以最小的代价修改现有系统（一般是增加一个方法），达到外部可以取得系统内部信息的目的。</p><h2 id="后记">后记</h2><p> 曾看过这样一句话：抽象能力决定编程能力，Shaun 个人认为，所谓抽象即提炼事物的共同点，这也是设计模式中反复使用接口的原因，接口即一组具体类的共同点，接口中的函数和变量即为这些具体类共有的，虽然具体行为可以不一样，但行为本身总是存在的。而又有这样一句话：程序等于数据结构加算法，Shaun 的理解是，狭义上的程序确实是这样，一段代码解决一个问题，这是程序，多段代码解决一个问题，这也是程序，多段代码解决多个问题，这亦是程序，一个软件，一个系统，一个平台，都是程序，但显然这些程序不是简单的数据结构和算法就能概括的，其内部必然有一套合理的逻辑进行组织，这套逻辑可以称之为“设计模式”，但这个“设计模式”不仅仅是上面谈的这些模式概念。Shaun 认为好的数据结构和算法确实能使程序达到当前最优，但对于一个大型系统或平台来说，这种最优只能是一种局部最优，这对整个系统的全局最优来说可能是微不足道的，而“设计模式”要解决的是怎样使一个系统达到全局最优，怎么合理组织局部最优。面对现代的超大型系统或平台，传统意义上的设计模式也只能达到局部最优，全局最优基本很少有人能驾驭，都是针对特定的业务需要，不断的试错改进优化，逐渐趋于稳定，但这种稳定可能很难抽象，放进其它的业务中，又得花费大量的人力物力去修改。</p><p> Shaun 个人对现代大型系统架构的理解就是分层分模块，功能太多分模块，模块太多就分层，一层不够分两层，两层不够分三层，三层不够继续分，根据数据流的处理分层，根据功能的不同分模块，模块内部依靠设计模式进行组织，设计模式调度的就是数据结构与算法。Shaun 目前的设计原则就是：每层一个独立的服务控制模块，每个模块一个对外服务功能（或事件或 socket ），同层的各模块之间尽量保持独立，不相互依赖，若各模块存在共同点，则将共同点抽出来，将其作为公共模块或独立为小层，层与层之间通过服务控制模块进行数据流的传输，除服务控制模块之外，模块之间尽量避免相互通信，即每个模块的对外服务功能一般只对本层服务控制模块提供服务，最好是只负责接收数据。如果系统实在太大，就只能保持纵向分层，横向保证各模块间数据流依次传输，并在特定模块节点上进行上下层的数据传输。</p><p> 数据结构与算法重不重要？当然重要，数据结构与算法不过关，面试都过不去 ( ╯□╰ )，工作都没有，还何谈什么设计模式，什么架构。设计模式重不重要？当然也重要，不会合理使用设计模式，写出一堆别人没法维护的垃圾代码（当然，这或许是好事 :p ），改个需求要半天，加个功能要半个月，效率太低，这样即使有好的数据结构与算法作用也不大。但是设计模式也不是万能的，针对不同的需求，同一种设计模式有不同的实现方式，所以书中的设计模式也仅供参考而已，与其说设计模式重要，还不如说书中那几个设计原则更重要些。同时一味的追求设计模式也不见得是件好事，设计模式可以参考，但不能生搬硬套，毕竟人是活的，需求也是活的，固定的模式也需要有所改变，总而言之，能以最小的代价解决问题完成需求的模式就是好模式。</p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 进入职场一年半以来，Shaun 完全独立从 0 到 1 开发了 1.5 个项目（当然也有参与其它项目，但不是 Shaun 独立从 0 到 1 开发的，没多少控制权，就不谈了），一个网页版的高精地图编辑器，半个地图可视化系统，这里面 Shaun 用了不少设计模式，这篇就谈谈 Shaun 用过的和没用过的一些设计模式。</p></summary>
<category term="Study" scheme="http://cniter.github.io/categories/Study/"/>
<category term="thought" scheme="http://cniter.github.io/tags/thought/"/>
</entry>
<entry>
<title>积分计算</title>
<link href="http://cniter.github.io/posts/eee1c041.html"/>
<id>http://cniter.github.io/posts/eee1c041.html</id>
<published>2020-11-29T10:28:56.000Z</published>
<updated>2021-12-18T11:54:13.941Z</updated>
<content type="html"><![CDATA[<h2 id="前言">前言</h2><p> 最近需要计算一下曲线长度，无法直接得到被积函数的原函数，常规的积分解法牛顿莱布尼茨公式没法使用，所以只能使用数值积分计算积分。</p><span id="more"></span><p> 下面主要介绍两种数值积分方法：龙贝格积分（Romberg Quadrature） 和 高斯-克朗罗德积分（Gauss-kronrod Quadrature）。 <em>下面附带的代码只做简单测试，不保证正确性，语言使用 Typescript。</em></p><h2 id="romberg-篇">Romberg 篇</h2><p> 计算积分最简单的当然是使用复化梯形公式，即 <span class="math inline">\(I=\int_a^b{f(x)dx}=\frac{b-a}{2n}[f(a)+f(b)+2\sum\limits_{i=1}^{n-1}f(x_i)]= T_n, x_i=a+i*h, h=(b-a)/n\)</span> ，若将 n 段每段一分为 2，可得到 <span class="math inline">\(T_{2n}=T_n/2+\frac{b-a}{2n}\sum\limits_{i=0}^{n-1}f(x_{i+1/2})\)</span> 。考虑数列 <span class="math inline">\(T=\{T_1,T_2,T_{2^2},...,T_{2^k}\}\)</span>，显然该数列必收敛，最后收敛为对应积分，当 <span class="math inline">\(|T_{2^k}-T_{2^{k-1}}| < ε\)</span> （<span class="math inline">\(ε\)</span> 为精度）时，可取 <span class="math inline">\(T_{2^k}\)</span> 作为最后的积分结果。但是，直接利用梯形公司求解，收敛很慢，导致计算效率很差，所以需要使用理查德森（Richardson）外推法加速收敛，设 <span class="math inline">\(T_{2^k}^{(m)}\)</span> 为 m 次加速值，当 m 为 0 时，表示没加速，为原梯形公司，则 <span class="math inline">\(T_{2^k}^{(m)} = \frac{4^m}{4^m-1}T_{2^{k+1}}^{(m-1)}-\frac{1}{4^m-1}T_{2^k}^{(m-1)}\)</span>，当 <span class="math inline">\(|T_{2^{k+1}}^{(m)}-T_{2^k}^{(m)}| < ε\)</span> 时，则收敛，并取其中一值作为最终的积分值。未经修饰的代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">rombergIntegrator</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, tolerance = <span class="number">1e-8</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> h = b - a;</span><br><span class="line"> <span class="keyword">let</span> n = <span class="number">1</span>;</span><br><span class="line"> <span class="keyword">let</span> preTK = ((f(a) + f(b)) * h) / <span class="number">2</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> tkmArray = [];</span><br><span class="line"> <span class="keyword">let</span> m = <span class="number">0</span>;</span><br><span class="line"> tkmArray[m] = preTK;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">while</span> (<span class="literal">true</span>) {</span><br><span class="line"> m += <span class="number">1</span>;</span><br><span class="line"> <span class="built_in">console</span>.log(m, tkmArray[m - <span class="number">1</span>]);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> tk = getTK(f, preTK, n, a, h);</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Math</span>.abs(tk - preTK) < tolerance) <span class="keyword">return</span> tk;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> preTKM = tkmArray[<span class="number">0</span>];</span><br><span class="line"> <span class="keyword">let</span> preTK1M = tk;</span><br><span class="line"> tkmArray[<span class="number">0</span>] = tk;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">1</span>; i <= m; i++) {</span><br><span class="line"> <span class="keyword">let</span> newTKM = getTKM(preTK1M, preTKM, i);</span><br><span class="line"> preTKM = tkmArray[i];</span><br><span class="line"> preTK1M = newTKM;</span><br><span class="line"> tkmArray[i] = newTKM;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (preTKM !== <span class="literal">undefined</span> && <span class="built_in">Math</span>.abs(preTK1M - preTKM) < tolerance) <span class="keyword">return</span> preTK1M;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> preTK = tk;</span><br><span class="line"> n *= <span class="number">2</span>;</span><br><span class="line"> h /= <span class="number">2</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">getTK</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, preTK: <span class="built_in">number</span>, n: <span class="built_in">number</span>, a: <span class="built_in">number</span>, h: <span class="built_in">number</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> sum = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">0</span>; i < n; i++) {</span><br><span class="line"> <span class="keyword">let</span> x = a + (i + <span class="number">0.5</span>) * h;</span><br><span class="line"> sum += f(x);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> (preTK + h * sum) / <span class="number">2</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">getTKM</span>(<span class="params">preTK1M: <span class="built_in">number</span>, preTKM: <span class="built_in">number</span>, m = <span class="number">0</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> m4 = <span class="number">1</span> << (<span class="number">2</span> * m); <span class="comment">// 4 ** m;</span></span><br><span class="line"> <span class="keyword">return</span> (m4 * preTK1M - preTKM) / (m4 - <span class="number">1</span>);</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 由于采用闭型积分规则（积分上下限值参与积分计算），所以以上代码不适合计算两端点值被积函数值无限大的情况（如 1/4 圆弧积分等）。而且该方法不合适求取被积函数在积分区间内导数值变化较大（如被积函数在积分下限附近剧烈波动，在积分上限附近不变化等）的积分，因为该方法是均匀分段，这种情况将导致计算量剧增，这时就需要用到下面的自适应积分。</p><h2 id="自适应篇">自适应篇</h2><p> 自适应积分主要包括两类：全局自适应积分和局部自适应积分，通常情况下全局自适应积分的会比局部自适应积分的表现要好，全局自适应积分一般通过二分递归实现，当满足一定条件时，递归终止，即通过二分分别计算两边的积分，若一边满足一定条件，则不继续划分，从而减少计算量。全局自适应积分中比较经典的有基于辛普森（Simpson）公式的自适应算法，普通辛普森积分公式为：<span class="math inline">\(I=\int_a^b{f(x)dx}=\frac{b-a}{6}[f(a)+4f(m)+f(b)]= S(a,b), m=(a+b)/2\)</span>，复化辛普森公式为 <span class="math inline">\(I=\int_a^b{f(x)dx}=\frac{h}{3}[f(a)+4\sum\limits_{i=1}^{n}f(x_{2i-1})+2\sum\limits_{i=1}^{n-1}f(x_{2i})+f(b)]= S(a,b)\)</span>，其中 <span class="math inline">\(x_i=a+i*h (i=1,2,3,...,2n),h=\frac{b-a}{2n}\)</span>，基于辛普森公式的自适应基本原理如下：令 <span class="math inline">\(S_2(a,b) = S(a,m)+S(m,b)\)</span>，m 为 a,b 中值，若 <span class="math inline">\(|S(a,b) - S_2(a,b)| < 15ε\)</span>，则取 <span class="math inline">\(S_2(a,b)\)</span> 或 <span class="math inline">\(S_2(a,b)+(S(a,b)-S_2(a,b))/15\)</span> 作为该区间的积分值，否则，将区间二分递归，同时因为误差会累积，所以每次递归都需要将精度提高两倍，即 <span class="math inline">\(ε = ε/2\)</span>，如此最后的精度才能达到最初的精度。具体 ts 代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">adaptiveSimpsonIntegrator</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, epsilon = <span class="number">1e-8</span></span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> S = complexSimpson(f, a, b);</span><br><span class="line"> <span class="keyword">return</span> adsimp(f, a, b, epsilon, S);</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">adsimp</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, epsilon = <span class="number">1e-8</span>, S = <span class="number">0</span></span>): <span class="title">number</span> </span>{</span><br><span class="line"> <span class="keyword">let</span> m = a + (b - a) / <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">let</span> LS = complexSimpson(f, a, m);</span><br><span class="line"> <span class="keyword">let</span> RS = complexSimpson(f, m, b);</span><br><span class="line"> <span class="keyword">const</span> S2 = LS + RS;</span><br><span class="line"> <span class="keyword">const</span> tolerance = <span class="number">15</span> * epsilon;</span><br><span class="line"> <span class="keyword">let</span> delta = S - S2;</span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Math</span>.abs(delta) < tolerance) <span class="keyword">return</span> S2 + delta / <span class="number">15</span>;</span><br><span class="line"> <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">let</span> doubleEPS = epsilon / <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">return</span> adsimp(f, a, m, doubleEPS, LS) + adsimp(f, m, b, doubleEPS, RS);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">complexSimpson</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, n = <span class="number">1</span></span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> n2 = n * <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">const</span> h = (b - a) / n2;</span><br><span class="line"> <span class="keyword">let</span> sum = f(a) + f(b);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">1</span>; i < n2; i += <span class="number">2</span>) {</span><br><span class="line"> sum += <span class="number">4</span> * f(a + i * h);</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">2</span>; i < n2 - <span class="number">1</span>; i += <span class="number">2</span>) {</span><br><span class="line"> sum += <span class="number">2</span> * f(a + i * h);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> (sum * h) / <span class="number">3</span>;</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 在 D.V. Fedorov 写的「Introduction to Numerical Methods with examples in Javascript」一书中介绍了一种全局自适应方法，即分别使用高阶和低阶的权值分别计算积分，两者之间的差值 <span class="math inline">\(E\)</span> 作为误差估计，设绝对精度为 <span class="math inline">\(\delta\)</span> ，相对精度为 <span class="math inline">\(\epsilon\)</span> ，若 <span class="math inline">\(|E|<\delta+\epsilon*Q\)</span>，Q 为高阶权值计算的积分，则取 Q 作为积分值，否则将积分区间二分，同时使 <span class="math inline">\(\delta/\sqrt{2}\)</span>，<span class="math inline">\(\epsilon\)</span> 保持不变。具体 ts 代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">recursiveAdaptiveIntegrator</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, accuracy = <span class="number">1e-15</span></span>) </span>{</span><br><span class="line"> <span class="keyword">return</span> recursiveAdaptiveIntegrate(f, a, b, accuracy);</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">recursiveAdaptiveIntegrate</span>(<span class="params"></span></span></span><br><span class="line"><span class="params"><span class="function"> f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>,</span></span></span><br><span class="line"><span class="params"><span class="function"> a: <span class="built_in">number</span>,</span></span></span><br><span class="line"><span class="params"><span class="function"> b: <span class="built_in">number</span>,</span></span></span><br><span class="line"><span class="params"><span class="function"> accuracy = <span class="number">1e-15</span>,</span></span></span><br><span class="line"><span class="params"><span class="function"> epsilon = <span class="built_in">Number</span>.EPSILON,</span></span></span><br><span class="line"><span class="params"><span class="function"> preFValue?: <span class="built_in">number</span>[],</span></span></span><br><span class="line"><span class="params"><span class="function"> </span>): <span class="title">number</span> </span>{</span><br><span class="line"> <span class="keyword">const</span> abscissae = [<span class="number">1</span> / <span class="number">6</span>, <span class="number">2</span> / <span class="number">6</span>, <span class="number">4</span> / <span class="number">6</span>, <span class="number">5</span> / <span class="number">6</span>];</span><br><span class="line"> <span class="keyword">const</span> highOrderWeights = [<span class="number">2</span> / <span class="number">6</span>, <span class="number">1</span> / <span class="number">6</span>, <span class="number">1</span> / <span class="number">6</span>, <span class="number">2</span> / <span class="number">6</span>];</span><br><span class="line"> <span class="keyword">const</span> lowOrderWeights = [<span class="number">1</span> / <span class="number">4</span>, <span class="number">1</span> / <span class="number">4</span>, <span class="number">1</span> / <span class="number">4</span>, <span class="number">1</span> / <span class="number">4</span>];</span><br><span class="line"> <span class="keyword">const</span> isRecompute = [<span class="number">1</span>, <span class="number">0</span>, <span class="number">0</span>, <span class="number">1</span>];</span><br><span class="line"> <span class="keyword">const</span> h = b - a;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> fValue: <span class="built_in">number</span>[] = [];</span><br><span class="line"> <span class="keyword">if</span> (preFValue === <span class="literal">undefined</span>) {</span><br><span class="line"> abscissae.forEach(<span class="function">(<span class="params">abscissa</span>) =></span> {</span><br><span class="line"> <span class="keyword">const</span> x = a + abscissa * h;</span><br><span class="line"> fValue.push(f(x));</span><br><span class="line"> });</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> k = <span class="number">0</span>, i = <span class="number">0</span>; i < abscissae.length; i++) {</span><br><span class="line"> <span class="keyword">if</span> (isRecompute[i]) fValue.push(f(a + abscissae[i] * h));</span><br><span class="line"> <span class="keyword">else</span> fValue.push(preFValue[k++]);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> highResult = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">let</span> lowResult = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">0</span>; i < highOrderWeights.length; i++) {</span><br><span class="line"> highResult += highOrderWeights[i] * fValue[i];</span><br><span class="line"> lowResult += lowOrderWeights[i] * fValue[i];</span><br><span class="line"> }</span><br><span class="line"> highResult *= h;</span><br><span class="line"> lowResult *= h;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> tolerance = accuracy + epsilon * <span class="built_in">Math</span>.abs(highResult);</span><br><span class="line"> <span class="keyword">let</span> errorEstimate = <span class="built_in">Math</span>.abs(highResult - lowResult) / <span class="number">3</span>;</span><br><span class="line"> <span class="keyword">if</span> (errorEstimate < tolerance) {</span><br><span class="line"> <span class="keyword">return</span> highResult;</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> accuracy /= <span class="built_in">Math</span>.sqrt(<span class="number">2</span>);</span><br><span class="line"> <span class="keyword">let</span> m = a + h / <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">let</span> midIndex = <span class="built_in">Math</span>.trunc(abscissae.length / <span class="number">2</span>);</span><br><span class="line"> <span class="keyword">let</span> leftFValue = fValue.slice(<span class="number">0</span>, midIndex);</span><br><span class="line"> <span class="keyword">let</span> rightFValue = fValue.slice(midIndex);</span><br><span class="line"> <span class="keyword">return</span> (</span><br><span class="line"> recursiveAdaptiveIntegrate(f, a, m, accuracy, epsilon, leftFValue) +</span><br><span class="line"> recursiveAdaptiveIntegrate(f, m, b, accuracy, epsilon, rightFValue)</span><br><span class="line"> );</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 该方法很巧妙的设计了一组插值点，使得当前计算的函数值正好可以被下次迭代所使用，从而提高性能，同时该方法可以得到 1/4 圆弧长，虽然精度只能到小数点后 8 位，至于 Shaun 写的其它测试函数，都能得到理想精度。</p><h2 id="gauss-篇">Gauss 篇</h2><p> 高斯求积法是一种多项式插值积分法，同时由于不计算被积函数在区间两个端点处的值，所以高斯积分法采用的开型积分规则，高斯积分法的衍生方法有很多种，下面主要介绍高斯-勒让德（Gauss-Legendre Quadrature）以及其迭代改良的高斯-克朗罗德法。高斯-勒让德积分法的公式为积分的原始形态，即 <span class="math inline">\(\int_a^bf(x)dx=\sum\limits_{i=1}^{∞}w_if(x_{i})\approx\sum\limits_{i=1}^{n}w_if(x_{i})\)</span> ，只不过 <span class="math inline">\(x_i \in [-1,1]\)</span>，并且 <span class="math inline">\(x_i\)</span> 和 <span class="math inline">\(w_i\)</span> 都通过勒让德多项式求出，所以其原则上只能用在积分区间为 [-1, 1] 上的积分，但是可以将积分从任意区间通过简单的变形变换到 [-1, 1] 上，即 <span class="math inline">\(\int_a^b{f(x)dx} = \frac{b-a}{2}\int_{-1}^1{f(\frac{b-a}{2}t+\frac{b+a}{2})dt}\)</span> ，从而可以将高斯-勒让德方法扩展到任意积分上。由于每个 n 对应的 <span class="math inline">\(x_i\)</span> 和 <span class="math inline">\(w_i\)</span> 都可以查表可得，所以具体代码方面就很简单了，以 n = 4，即插值点个数为 4 为例，ts 代码如下：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">gaussLegendreIntegrate</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, n: <span class="number">4</span> | <span class="number">8</span> = <span class="number">4</span></span>) </span>{</span><br><span class="line"> <span class="keyword">const</span> weightsAndAbscissae = getWeightsAndAbscissae(n);</span><br><span class="line"> <span class="keyword">const</span> weights = weightsAndAbscissae.weights;</span><br><span class="line"> <span class="keyword">const</span> abscissae = weightsAndAbscissae.abscissae;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> halfH = (b - a) / <span class="number">2</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> sum = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">0</span>; i < abscissae.length; i++) {</span><br><span class="line"> <span class="keyword">let</span> xi = halfH * abscissae[i] + a + halfH;</span><br><span class="line"> sum += weights[i] * f(xi);</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> sum * halfH;</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">getWeightsAndAbscissae</span>(<span class="params">n: <span class="number">4</span> | <span class="number">8</span> = <span class="number">4</span></span>) </span>{</span><br><span class="line"> <span class="keyword">switch</span> (n) {</span><br><span class="line"> <span class="keyword">case</span> <span class="number">8</span>:</span><br><span class="line"> <span class="keyword">return</span> {</span><br><span class="line"> <span class="attr">weights</span>: [</span><br><span class="line"> <span class="number">0.362683783378362</span>,</span><br><span class="line"> <span class="number">0.362683783378362</span>,</span><br><span class="line"> <span class="number">0.3137066458778873</span>,</span><br><span class="line"> <span class="number">0.3137066458778873</span>,</span><br><span class="line"> <span class="number">0.2223810344533745</span>,</span><br><span class="line"> <span class="number">0.2223810344533745</span>,</span><br><span class="line"> <span class="number">0.1012285362903763</span>,</span><br><span class="line"> <span class="number">0.1012285362903763</span>,</span><br><span class="line"> ],</span><br><span class="line"> <span class="attr">abscissae</span>: [</span><br><span class="line"> -<span class="number">0.1834346424956498</span>,</span><br><span class="line"> <span class="number">0.1834346424956498</span>,</span><br><span class="line"> -<span class="number">0.525532409916329</span>,</span><br><span class="line"> <span class="number">0.525532409916329</span>,</span><br><span class="line"> -<span class="number">0.7966664774136267</span>,</span><br><span class="line"> <span class="number">0.7966664774136267</span>,</span><br><span class="line"> -<span class="number">0.9602898564975363</span>,</span><br><span class="line"> <span class="number">0.9602898564975363</span>,</span><br><span class="line"> ],</span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">case</span> <span class="number">4</span>:</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> <span class="keyword">return</span> {</span><br><span class="line"> <span class="attr">weights</span>: [<span class="number">0.6521451548625461</span>, <span class="number">0.6521451548625461</span>, <span class="number">0.3478548451374538</span>, <span class="number">0.3478548451374538</span>],</span><br><span class="line"> <span class="attr">abscissae</span>: [-<span class="number">0.3399810435848563</span>, <span class="number">0.3399810435848563</span>, -<span class="number">0.8611363115940526</span>, <span class="number">0.8611363115940526</span>],</span><br><span class="line"> };</span><br><span class="line"> <span class="keyword">break</span>;</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 若要提高高斯-勒让德积分法的精度，可通过增加插值点或分多个区间进行积分来实现，但是由于没有误差估计，所以还是没法精确控制精度，对与某些被积函数积分精度高，但对于其它被积函数，积分精度却有限，当然可以简单的引入一些常用的误差估计法，但一般需要重新计算积分，导致效率很低，而高斯-克朗罗德法为其引入了一种基于 Kronrod 点的误差估计法，可充分利用现有计算值，从而达到有效控制精度的同时，性能没有太大的损失。设 <span class="math inline">\(G(f,n)=\sum\limits_{i=1}^{n}w_if(x_{i})\)</span> 为具有 n 个插值点的高斯-勒让德法计算结果，<span class="math inline">\(GK(f,n) = \sum\limits_{i=1}^{n}w'_if(x_{i})+\sum\limits_{k=n+1}^{2n+1}w'_kf(x_{k})\)</span> 为高斯-克朗罗德法的计算结果，则 <span class="math inline">\(|GK(f,n)-G(f,n)|\)</span> 可作为误差估计，有了误差估计，最后再使用全局自适应策略，即可得到精度可控的高斯积分结果。具体 ts 代码如下，以 n = 7 为例：</p><div class="highlight-wrap"autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" contenteditable="false"data-lang="TYPESCRIPT"><div class="code-copy"></div><figure class="highlight hljs typescript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">function</span> <span class="title">gaussKronrodIntegrator</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, accuracy = <span class="number">1e-15</span></span>) </span>{</span><br><span class="line"> <span class="keyword">return</span> recursiveAdaptiveIntegrate(f, a, b, accuracy);</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">function</span> <span class="title">recursiveAdaptiveIntegrate</span>(<span class="params">f: (x: <span class="built_in">number</span>) => <span class="built_in">number</span>, a: <span class="built_in">number</span>, b: <span class="built_in">number</span>, accuracy = <span class="number">1e-12</span></span>): <span class="title">number</span> </span>{</span><br><span class="line"> <span class="keyword">const</span> gaussAbscissae = [</span><br><span class="line"> <span class="number">0.0</span>,</span><br><span class="line"> -<span class="number">4.058451513773971669066064120769615e-1</span>,</span><br><span class="line"> <span class="number">4.058451513773971669066064120769615e-1</span>,</span><br><span class="line"> -<span class="number">7.415311855993944398638647732807884e-1</span>,</span><br><span class="line"> <span class="number">7.415311855993944398638647732807884e-1</span>,</span><br><span class="line"> -<span class="number">9.491079123427585245261896840478513e-1</span>,</span><br><span class="line"> <span class="number">9.491079123427585245261896840478513e-1</span>,</span><br><span class="line"> ];</span><br><span class="line"> <span class="keyword">const</span> gaussWeights = [</span><br><span class="line"> <span class="number">4.179591836734693877551020408163265e-1</span>,</span><br><span class="line"> <span class="number">3.818300505051189449503697754889751e-1</span>,</span><br><span class="line"> <span class="number">3.818300505051189449503697754889751e-1</span>,</span><br><span class="line"> <span class="number">2.797053914892766679014677714237796e-1</span>,</span><br><span class="line"> <span class="number">2.797053914892766679014677714237796e-1</span>,</span><br><span class="line"> <span class="number">1.29484966168869693270611432679082e-1</span>,</span><br><span class="line"> <span class="number">1.29484966168869693270611432679082e-1</span>,</span><br><span class="line"> ];</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> kronrodAbscissae = gaussAbscissae.concat([</span><br><span class="line"> -<span class="number">2.077849550078984676006894037732449e-1</span>,</span><br><span class="line"> <span class="number">2.077849550078984676006894037732449e-1</span>,</span><br><span class="line"> -<span class="number">5.860872354676911302941448382587296e-1</span>,</span><br><span class="line"> <span class="number">5.860872354676911302941448382587296e-1</span>,</span><br><span class="line"> -<span class="number">8.648644233597690727897127886409262e-1</span>,</span><br><span class="line"> <span class="number">8.648644233597690727897127886409262e-1</span>,</span><br><span class="line"> -<span class="number">9.914553711208126392068546975263285e-1</span>,</span><br><span class="line"> <span class="number">9.914553711208126392068546975263285e-1</span>,</span><br><span class="line"> ]);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> kronrodWeights = [</span><br><span class="line"> <span class="number">2.094821410847278280129991748917143e-1</span>,</span><br><span class="line"> <span class="number">1.903505780647854099132564024210137e-1</span>,</span><br><span class="line"> <span class="number">1.903505780647854099132564024210137e-1</span>,</span><br><span class="line"> <span class="number">1.406532597155259187451895905102379e-1</span>,</span><br><span class="line"> <span class="number">1.406532597155259187451895905102379e-1</span>,</span><br><span class="line"> <span class="number">6.309209262997855329070066318920429e-2</span>,</span><br><span class="line"> <span class="number">6.309209262997855329070066318920429e-2</span>,</span><br><span class="line"> <span class="number">2.044329400752988924141619992346491e-1</span>,</span><br><span class="line"> <span class="number">2.044329400752988924141619992346491e-1</span>,</span><br><span class="line"> <span class="number">1.690047266392679028265834265985503e-1</span>,</span><br><span class="line"> <span class="number">1.690047266392679028265834265985503e-1</span>,</span><br><span class="line"> <span class="number">1.04790010322250183839876322541518e-1</span>,</span><br><span class="line"> <span class="number">1.04790010322250183839876322541518e-1</span>,</span><br><span class="line"> <span class="number">2.293532201052922496373200805896959e-2</span>,</span><br><span class="line"> <span class="number">2.293532201052922496373200805896959e-2</span>,</span><br><span class="line"> ];</span><br><span class="line"></span><br><span class="line"> <span class="keyword">const</span> halfH = (b - a) / <span class="number">2</span>;</span><br><span class="line"></span><br><span class="line"> <span class="keyword">let</span> guassResult = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">let</span> kronrodResult = <span class="number">0</span>;</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = <span class="number">0</span>; i < gaussAbscissae.length; i++) {</span><br><span class="line"> <span class="keyword">let</span> xi = halfH * gaussAbscissae[i] + a + halfH;</span><br><span class="line"> <span class="keyword">let</span> yi = f(xi);</span><br><span class="line"> guassResult += gaussWeights[i] * yi;</span><br><span class="line"> kronrodResult += kronrodWeights[i] * yi;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">for</span> (<span class="keyword">let</span> i = gaussAbscissae.length; i < kronrodAbscissae.length; i++) {</span><br><span class="line"> <span class="keyword">let</span> xi = halfH * kronrodAbscissae[i] + a + halfH;</span><br><span class="line"> <span class="keyword">let</span> yi = f(xi);</span><br><span class="line"> kronrodResult += kronrodWeights[i] * yi;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> (<span class="built_in">Math</span>.abs(kronrodResult - guassResult) < accuracy / halfH) <span class="keyword">return</span> kronrodResult * halfH;</span><br><span class="line"> <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">let</span> m = a + (b - a) / <span class="number">2</span>;</span><br><span class="line"> accuracy /= <span class="number">2</span>;</span><br><span class="line"> <span class="keyword">return</span> recursiveAdaptiveIntegrate(f, a, m, accuracy) + recursiveAdaptiveIntegrate(f, m, b, accuracy);</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></div><p> 简单测试了一下，Shaun 这里写的 gaussKronrodIntegrator 方法最大精度只能到 1e-15，到 16 位就报错递归深度太大了，圆的 1/4 弧长也没法算出来，当然这些问题可通过设置最大递归深度以及处理异常值来解决，Shaun 这里就不继续写了。</p><h2 id="后记">后记</h2><p> 数值积分策略非常多，尤其是针对一些特殊的函数，可能只能使用一些特殊的策略才能计算，Shaun 这里只是介绍了一些比较基础常用的积分方法，能解决大部分积分问题，唯一需要注意一点的就是如何追求性能与精度之间的平衡，因为积分常常涉及到迭代求值，通常而言精度越高，迭代越多，求积时，同时也需要注意被积函数的异常值（如无穷大等），这时可能需要拆分或变换积分区间，并且使用开型积分规则的积分方法进行重新计算。</p><h2 id="附录">附录</h2><h3 id="一些常见的积分变换">一些常见的积分变换</h3><p><span class="math display">\[\int_a^bf(x) = \int_0^{b-a}f(a+t)dt \\\int_a^b{f(x)dx} = \frac{b-a}{2}\int_{-1}^1{f(\frac{b-a}{2}t+\frac{b+a}{2})dt} \\\int_{-∞}^{+∞}{f(x)dx} = \int_{-1}^1{f(\frac{t}{1-t^2})\frac{1+t^2}{(1-t^2)^2}dt} \\\int_{a}^{+∞}{f(x)dx} = \int_{0}^1{f(a + \frac{t}{1-t})\frac{1}{(1-t)^2}dt} \\\int_{-∞}^{a}{f(x)dx} = \int_{-1}^0{f(a + \frac{t}{1+t})\frac{-1}{(1+t)^2}dt}\]</span></p><h2 id="参考资料">参考资料</h2><p>[1] <a href="https://reference.wolfram.com/language/tutorial/NIntegrateIntegrationStrategies.html#155948475">积分策略</a></p><p>[2] <a href="https://pomax.github.io/bezierinfo/legendre-gauss.html">Gaussian Quadrature Weights and Abscissae</a></p><p>[3] <a href="https://www.boost.org/doc/libs/release/libs/math/doc/html/math_toolkit/gauss_kronrod.html">Gauss-Kronrod Quadrature</a></p><p>[4] <a href="https://www.advanpix.com/2011/11/07/gauss-kronrod-quadrature-nodes-weights/">Gauss-Kronrod Quadrature Nodes and Weights</a></p>]]></content>
<summary type="html"><h2 id="前言">前言</h2>
<p> 最近需要计算一下曲线长度，无法直接得到被积函数的原函数，常规的积分解法牛顿莱布尼茨公式没法使用，所以只能使用数值积分计算积分。</p></summary>
<category term="Mathematics" scheme="http://cniter.github.io/categories/Mathematics/"/>
<category term="numerical" scheme="http://cniter.github.io/tags/numerical/"/>
<category term="integration" scheme="http://cniter.github.io/tags/integration/"/>
</entry>
</feed>