forked from Exiv2/exiv2
-
Notifications
You must be signed in to change notification settings - Fork 0
/
WORK-IN-PROGRESS
856 lines (689 loc) · 38 KB
/
WORK-IN-PROGRESS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
T A B L E o f C O N T E N T S
---------------------------------
1 Building Adobe XMPsdk and Samples in Terminal with the ./Generate_XXX_mac.sh scripts
1.1 Amazing Discovery 1 DumpFile is linked to libstdc++.6.dylib
1.2 Amazing Discovery 2 Millions of "weak symbol/visibility" messages
2 Puzzle with autotools/--with-adobe MacOS-X Only
2.1 To build without the external Adobe SDK:
2.2 To add the external Adobe SDK to the mix:
2.3 Notes (mostly to remind me of painful discoveries)
2.4 Compiler options used to build Adobe XMPsdk
2.5 Compiler options used to build DumpFile sample Application
2.6 Linker Options used to link DumpFile
3 Features to Add to the CMake code
4 Build design for v0.26.1
4.1 Backwards compatibility
4.2 New Options for ./configure
4.3 New Options for msvc
4.4 New Options for CMake
4.5 New Options for contrib/cmake/msvc
4.6 Scripts xmpsdk/buildXMPsdk.sh and xmpsdk/buildXMPsdk.cmd
4.7 Support for 32/64 bit builds, Release/Debug, Shared/Static
4.8 Support for MinGW
4.9 Linking XMPsdk with libexiv2
5 Refactoring the Tiff Code
5.1 Background
5.2 How does Exiv2 decode the ExifData in a JPEG?
5.3 How is metadata organized in Exiv2
5.4 Where are the tags defined?
5.5 How do the MakerNotes get decoded?
5.6 How do the encoders work?
6 Using external XMP SDK via Conan
==========================================================================
1 Building XMPsdk and Samples in Terminal with the ./Generate_XXX_mac.sh scripts
Added : 2017-08-18
Modified: 2017-08-18
$ cd <exiv2dir>/xmpsdk
# XMP-Toolkit-SDK-CC201607 or XMP-Toolkit-SDK-CC201407
$ mkdir Adobe
$ curl -O http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC201607.zip
$ find third-party -type d -maxdepth 1 -exec cp -R '{}' Adobe/XMP-Toolkit-SDK-CC201607/third-party ';'
$ cd XMP-Toolkit-SDK-CC201607/build
$ echo '3' | ./GenerateXMPToolkitSDK_mac.sh
$ xcodebuild -project xcode/static/intel_64/XMPToolkitSDK64.xcodeproj/ -target XMPCoreStatic -configuration Release
$ xcodebuild -project xcode/static/intel_64/XMPToolkitSDK64.xcodeproj/ -target XMPFilesStatic -configuration Release
Building Samples:
$ cd Adobe
$ curl --insecure -O https://cmake.org/files/v3.5/cmake-3.5.2-Darwin-x86_64.tar.gz
$ tar xzf cmake-3.5.2-Darwin-x86_64.tar.gz
$ mkdir XMP-Toolkit-SDK-CC201607/tools/cmake/bin
$ ditto cmake-3.5.2-Darwin-x86_64/CMake.app/ XMP-Toolkit-SDK-CC201607/tools/cmake/bin/cmake.app
$ cd XMP-Toolkit-SDK-CC201607/samples/build
$ echo '2' | ./GenerateSamples_mac.sh
$ xcodebuild -project xcode/intel_64/XMP_Samples_64.xcodeproj -configuration Release [-target Sample]
$ cd XMP-Toolkit-SDK-CC201607
$ samples/target/macintosh/intel_64/Release/DumpFile ~/Stonehenge.jpg
Samples:
CustomSchema
CustomSchemaNewDOM
DumpFile
DumpMainXMP
DumpScannedXMP
ModifyingXMP
ModifyingXMPNewDOM
ReadingXMP
ReadingXMPNewDOM
XMPCommand
XMPCoreCoverage
XMPFilesCoverage
XMPIterations
1.1 Amazing Discovery 1 DumpFile is linked to libstdc++.6.dylib
------------------------------------------------------------------
XMP-Toolkit-SDK-CC201607/samples $ otool -L target/macintosh/intel_64/Release/DumpFile
target/macintosh/intel_64/Release/DumpFile:
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 104.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.4.0)
XMP-Toolkit-SDK-CC201607/samples $
Conclusion: We should link -lXMPCore with libexiv2.26.dylib
We should not link -lXMPCore with applications such as exiv2(.exe)
We should build exiv2 code with visibility disabled "CXXFLAGS=-fvisibility=hidden -fvisibility-inlines-hidden"
We must compile and link everything with "CXXFLAGS=-stdlib=libstdc++ -Wno-deprecated"
1.2 Amazing Discovery 2 Millions of "weak symbol/visibility" messages
------------------------------------------------------------------------
$ cd samples/build
$ rm -rf cmake/DumpFile/build_x64 # remove the build of DumpFile
$ env VERBOSE=1 xcodebuild -project xcode/intel_64/XMP_Samples_64.xcodeproj -configuration Release -target DumpFile
==========================================================================
2 Puzzle with autotools/--with-adobe MacOS-X Only
Added : 2017-08-17
Modified: 2017-08-18
The --with-adobe code is new (added on Monday/Tuesday 2017-08-15).
It's very much "Work in Progress" and will get lots more polishing.
This builds and links on Mac and Linux (Cygwin isn't ready yet).
Autotools isn't used with Visual Studio
Default build (which compile/links xmpsdk/src) is fine (and passes test suite)
$ sudo make distclean ; make config ; ./configure ; make ; sudo make install ; make samples ; make tests
Status: The 2016/AdobeXMPsdk requires an entry point that's not in the C++ STL library
./configure is building and passing the test suite
./configure --with-adobe=2014 is broken on __ZNSs4_Rep11_S_terminalE
./configure --with-adobe=2016 is broken on __ZNSiD0Ev
The missing entry points are in /usr/lib/libstdc++.6.dylib
When I use install_name_tool to change to libstdc++.6.dylib, I crash in this:
$ c++filt __ZNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEED1Ev
std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::~basic_string()
$
There's something mixed up concerning the libraries used by Adobe's XMPSDK and libexiv2.26.dylib
-stdlib=libstdc++ will link /usr/lib/libstdc++.6.dylib, however the compiler warns:
clang: warning: libstdc++ is deprecated; move to libc++ [-Wdeprecated]
2.1 To build without the external Adobe SDK:
--------------------------------------------
Edit config/config.mk.in and modify:
XMPSDK_LIBS= -lXMPCore
to
XMPSDK_LIBS = ''
$ sudo make distclean ; make config ; ./configure --with-adobe ; make
The following command works:
$ bin/exiv2 http://clanmills.com/Stonehenge.jpg is working
On 2017-08-15 The following command crashes when it attempts to execute XMPsdk code
$ bin/exiv2 --verbose --version
........
config_path=/Users/rmills/.exiv2
uid=501
euid=501
gid=20
dyld: lazy symbol binding failed: Symbol not found: _WXMPMeta_Initialize_1
Referenced from: /Users/rmills/gnu/github/exiv2/src/.libs/libexiv2.26.dylib
Expected in: flat namespace
dyld: Symbol not found: _WXMPMeta_Initialize_1
Referenced from: /Users/rmills/gnu/github/exiv2/src/.libs/libexiv2.26.dylib
Expected in: flat namespace
Abort trap: 6
$
This is correct. libXMPCore.a is not in the mix.
ls -alt src/.libs/libexiv2.26.dylib
-rwxr-xr-x+ 1 rmills staff 2238124 Aug 17 11:07 src/.libs/libexiv2.26.dylib
-------
2.2 To add the external Adobe SDK to the mix:
---------------------------------------------
Edit src/Makefile#242 and change:
@$(LIBTOOL) --mode=link $(LINK.cc) -o ../bin/$@ $(LIBRARY) $(EXIV2OBJ) $(EXIV2COBJ) -rpath $(libdir)
to:
@$(LIBTOOL) --mode=link $(LINK.cc) -lXMPCore -o ../bin/$@ $(LIBRARY) $(EXIV2OBJ) $(EXIV2COBJ) -rpath $(libdir)
$ touch src/version.cpp src/exiv2.cpp ; make
$ ls -alt src/.libs/libexiv2.26.dylib
-rwxr-xr-x+ 1 rmills staff 4312308 Aug 17 11:17 src/.libs/libexiv2.26.dylib
-------
$ bin/exiv2
dyld: Symbol not found: __ZNSiD0Ev
Referenced from: /Users/rmills/gnu/github/exiv2/src/.libs/libexiv2.26.dylib
Expected in: flat namespace
in /Users/rmills/gnu/github/exiv2/src/.libs/libexiv2.26.dylib
Abort trap: 6
605 rmills@rmillsmbp:~/gnu/github/exiv2 $
$ c++filt _ZNSiD0Ev
std::basic_istream<char, std::char_traits<char> >::~basic_istream()
This code is being accessed by Adobe XMPSDK 2016 libXMPCore.a and isn't in the c++ library.
$ nm -g xmpsdk/Adobe/libXMPCore.a | grep _ZNSiD0Ev
U __ZNSiD0Ev
$
Adobe XMPSDK 2014 has a similar problem with __ZNSs4_Rep11_S_terminalE
$ nm -g Adobe/XMP-Toolkit-SDK-CC201412/libXMPCore.a | grep __ZNSs4_Rep11_S_terminalE | sort --unique
U __ZNSs4_Rep11_S_terminalE
$ c++filt __ZNSs4_Rep11_S_terminalE
std::string::_Rep::_S_terminal
2.3 Notes (mostly to remind me of painful discoveries)
------------------------------------------------------
How does the build get generated?
$ make config
This generates ./configure by reading config/configure.ac
When you use --with-adobe, CXXFLAGS and LDFLAGS are modified and ENABLE_XMP is set to 2016.
Makefile has code to execute the script xmpsdk/buildXMPsdk.sh which
downloads and builds the XMPsdk to create libCore.a (which we need) and libFiles.a
config/config.mk.in is converted into config/config.mk and included in xmpsdk/src/Makefile
This is really ugly. config.mk also messes with CPPFLAGS and LDFLAGS
To get -lXMPCore into the mix, set XMPSDK_LIBS in config/config.mk.in
XMPSDK_LIBS = -lXMPCore
508 rmills@rmillsmbp:~/gnu/github/exiv2 $ finder Makefile | xargs grep -H config.mk | grep include
./contrib/organize/Makefile:include $(top_srcdir)/config/config.mk
./doc/Makefile:include $(top_srcdir)/config/config.mk
./samples/Makefile:include $(top_srcdir)/config/config.mk
./src/Makefile:include $(top_srcdir)/config/config.mk
./xmpsdk/src/Makefile:include $(top_srcdir)/config/config.mk
509 rmills@rmillsmbp:~/gnu/github/exiv2 $
2.4 Compiler options used to build Adobe XMPsdk
-----------------------------------------------
For sure, Adobe's code is being built with MacOSX10.8.sdk and -std=c++11
/usr/bin/c++
-DBUILDING_XMPCORE_AS_STATIC=1
-DBUILDING_XMPCORE_LIB=1
-DENABLE_CPP_DOM_MODEL=1
-DHAVE_EXPAT_CONFIG_H=1
-DMAC_ENV=1
-DXML_STATIC=1
-DXMP_64=1
-DXMP_COMPONENT_INT_NAMESPACE=AdobeXMPCore_Int
-DXMP_StaticBuild=1
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/build/../..
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/build/../../public/include
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/build/../../third-party/expat/public/lib
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/build/../resource/mac
-funsigned-char
-fshort-enums
-fno-common
-Wall
-Wextra
-Wno-missing-field-initializers
-Wno-shadow
-Wno-reorder
-std=c++11
-Wnon-virtual-dtor
-Woverloaded-virtual
-Wno-unused-variable
-Wno-unused-function
-Wno-unused-parameter
-fstack-protector
-O3
-DNDEBUG=1
-D_NDEBUG=1
-arch x86_64
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
-mmacosx-version-min=10.7
-o CMakeFiles/XMPCoreStatic.dir/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/source/XMPMeta-GetSet.cpp.o
-c /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/XMPCore/source/XMPMeta-GetSet.cpp
2.5 Compiler options used to build DumpFile sample Application
-----------------------------------------------------------
clang -x c++
-arch x86_64
-fmessage-length=0
-fdiagnostics-show-note-include-stack
-fmacro-backtrace-limit=0
-stdlib=libstdc++
-Wno-trigraphs
-fpascal-strings
-O3
-Wno-missing-field-initializers
-Wno-missing-prototypes
-Wno-return-type
-Wno-non-virtual-dtor
-Wno-overloaded-virtual
-Wno-exit-time-destructors
-Wno-missing-braces
-Wparentheses
-Wswitch
-Wno-unused-function
-Wno-unused-label
-Wno-unused-parameter
-Wno-unused-variable
-Wunused-value
-Wno-empty-body
-Wno-uninitialized
-Wno-unknown-pragmas
-Wno-shadow
-Wno-four-char-constants
-Wno-conversion
-Wno-constant-conversion
-Wno-int-conversion
-Wno-bool-conversion
-Wno-enum-conversion
-Wno-shorten-64-to-32
-Wno-newline-eof
-Wno-c++11-extensions
-DCMAKE_INTDIR=\"Release\"
-DMAC_ENV=1
-DXMP_64=1
-DXMP_StaticBuild=1
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-fasm-blocks
-fstrict-aliasing
-Wdeprecated-declarations
-Winvalid-offsetof
-mmacosx-version-min=10.9
-g
-Wno-sign-conversion
-Wno-infinite-recursion
-Wno-move
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/target/macintosh/intel_64/Release/include
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/../../..
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/../../../public/include
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/DerivedSources/x86_64
-I/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/DerivedSources
-Wmost
-Wno-four-char-constants
-Wno-unknown-pragmas
-F/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/target/macintosh/intel_64/Release
-F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks
-funsigned-char
-fshort-enums
-fno-common
-Wall
-Wextra
-Wno-missing-field-initializers
-Wno-shadow
-Wno-reorder
-std=c++11
-Wnon-virtual-dtor
-Woverloaded-virtual
-Wno-unused-variable
-Wno-unused-function
-Wno-unused-parameter
-fstack-protector
-DNDEBUG=1
-D_NDEBUG=1
-MMD
-MT dependencies
-MF /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile.d
--serialize-diagnostics /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile.dia
-c /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/source/common/DumpFile.cpp
-o /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile.o
2.6 Linker Options used to link DumpFile
----------------------------------------
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
-arch x86_64
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
-L/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/target/macintosh/intel_64/Release
-F/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/target/macintosh/intel_64/Release
-F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks
-filelist /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile.LinkFileList
-mmacosx-version-min=10.9
-Xlinker
-object_path_lto
-Xlinker
/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile_lto.o
-stdlib=libstdc++
-Wl,-search_paths_first
-Wl,-headerpad_max_install_names
/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/../../../public/libraries/macintosh/intel_64/Release/libXMPCoreStatic.a
/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/../../../public/libraries/macintosh/intel_64/Release/libXMPFilesStatic.a
-framework Cocoa
-Xlinker
-dependency_info
-Xlinker
/Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/build/cmake/DumpFile/build_x64/XMP_Samples_64.build/Release/DumpFile.build/Objects-normal/x86_64/DumpFile_dependency_info.dat
-o /Users/rmills/gnu/github/exiv2/xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/samples/target/macintosh/intel_64/Release/DumpFile
==========================================================================
3 Features to Add to the CMake code
* Auto usage of ccache on Linux and Mac.
* Remove the trick to compile the documentation using the configure script. Native documentation generation on CMake.
==========================================================================
4 Build design for v0.26.1
Added : 2017-08-18
Modified: 2017-08-23
The purpose of the v0.26.1 is to release bug fixes and
experimental new features which may become defaults with v0.27
4.1 Backwards compatibility
This feature is being added to v0.26.1 DOT release and therefore existing
behaviour of ./configure, CMake and msvc does not change.
./configure --enable-xmp (default is true) will build a library libxmpsdk.a
using the code in xmpsdk/src. This library is linked into libexiv2.26.dylib
4.2 New Options for ./configure
./configure --with-std=c++11
The default behaviour is the same as v0.26.0 --with-std=-std=c++98
If the user requests --with-adobe, --with-std=c++11 is the default
./configure --with-adobe[=2016|2014]
This causes the Adobe XMPsdk to be downloaded and built
into the directory xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/libXMPCore.a
The dynamic library libexiv2.26.dylib will not have libXMPCore.a linked
All applications (such as exiv2, exifprint etc) will require:
-Lxmpsdk/Adobe/XMP-Toolkit-SDK-CC201607 -lXMPCore
This is an important architectural change in libexiv2.26.dylib
However this is the way to achieve architecture independence between
exiv2 and Adobe XMPsdk. A longer term goal is to totally remove
all Adobe code from the exiv2 tarball.
Although --with-adobe causes the default compiler to be --with-std=c++11,
the code will with --with-std=c++98 The Adobe XMPsdk is built using the script
xmpsdk/buildXMPsdk which downloads and builds libXMPCore.a using Adobe provided
CMake code. For the 2016 SDK, libXMPCore.a is built with -std=c++11
4.3 New Options for msvc
This work has not been started. There are at least two possibilities:
1) To provide a new solution msvc/exiv2-adobe.sln
This is the model used by msvc/exiv2-webready.sln
2) To update msvc/configure.py to support options similar to
./configure --with-std= and ./config --with-adobe
4.4 New Options for CMake
Luis is adopting a two stage approach:
1) Cleaning up and refactoring the existing CMake scripts (*.cmake and CMakeLists.txt files)
This work is proceeding well.
2) Adding new EXIV2_ENABLE_XXXXs equivalent to ./configure --with-std and --with-adobe
This work is waiting for Robin to complete the ./configure modifications.
It is expected that the CMake files will execute the script xmpsdk/buildXMPsdk.{sh|cmd}
to perform the build of libXMPCore.a
4.5 New Options for contrib/cmake/msvc
To be written. Luis' great new work may make this obsolete.
4.6 Scripts xmpsdk/buildXMPsdk.sh and xmpsdk/buildXMPsdk.cmd
These scripts download and build the Adobe XMPsdk. Minor modification of the SDK
is necessary to enable the code to build. For example, Adobe does not support Cygwin
and minor change are necessary to build on that platform.
Changes to Adobe code will be minimal and documented in xmpsdk/buildXMPsdk.{sh|cmd}
4.7 Support for 32/64 bit builds, Release/Debug, Shared/Static
1) v0.26.1 will build exactly as v0.26 and support linking static libxmpsdk.a
32/64 static/shared debug/release are supported on all platforms
2) The new features (--with-std and --with-adobe) will only be supported for
a static/release build of libXMPCore.a
msvc will continue to supports 32/64 ReleaseDLL/Release/DebugDLL/Debug of libexiv2
which will include the static libXMPCore.a or libxmpsdk.a
4.8 Support for MinGW
MinGW msys/1.0 was deprecated when v0.26 was released.
No support for MinGW msys/1.0 will be provided.
It's very likely that the MinGW msys/1.0 will build.
I will not provide any user support for MinGW msys/1.0 in future.
MinGW msys/2.0 might be supported as "experimental" in Exiv2 v0.26.2
4.9 Building and linking XMPsdk with libexiv2
Exiv2 v0.26 (and many earlier editions) contain Adobe XMPsdk code.
The precise version of this code is unknown.
The xmpsdk/ directory has files in src/*.cpp and include/*.hpp
The xmpsdk/ has its own Makefile, CMakeLists.txt and msvc/xmpsdk/project files.
Exiv2 v0.26 builds a static library libxmpsdk.a which is linked into libexiv2.a
None of this will change in Exiv2 v0.26.1
Exiv2 v0.26.1 will support a new option to download and build "pure Adobe" XMPsdk.
Three flavours will be supported:
2013 http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC-201306.zip
2014 http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC201412.zip
2016 http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC201607.zip
As Adobe appear to release new editions every 18 months, I guess that Adobe will release
a new version in Spring 2018.
The autotools build supports new ./configure options:
--with-adobe builds and links 4089256 Aug 23 11:19 ./xmpsdk/Adobe/XMP-Toolkit-SDK-CC201306/libXMPCore.a
--with-adobe=2016 as --with-adobe
--with-adobe=2014 4108792 Aug 23 11:18 ./xmpsdk/Adobe/XMP-Toolkit-SDK-CC201412/libXMPCore.a
--with-adobe=2013 22242448 Aug 23 10:25 ./xmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/libXMPCore.a
The build of the Adobe XMPsdk is "pure Adobe". We download the code and build it without modification.
The scripts to download and build the Adobe code is:
xmpsdk/buildXMPsdk.sh
xmpsdk/buildXMPsdk.cmd
Adobe build the 2016 SDK with -std=c++11. This links successfully with the libexiv2 -std=c++98 build.
I haven't investigated which compiler Adobe use to build the 2013 and 2014 editions.
Mac Build
Following the analysis of Adobe's build (Topics 1 and 2 in this report), it has been realised that:
1 Adobe build the SDK with (both linker and compiler) -stdlib=libstdc++
2 Visibility should be disabled in the compiler
CPPFLAGS = -I/usr/local/include -I/usr/local/include -std=c++98 -fvisibility=hidden -fvisibility-inlines-hidden \
-stdlib=libstdc++ -Wno-deprecated -Ixmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/public/include
LDFLAGS = -L/usr/local/lib -lpthread -stdlib=libstdc++ -Wno-deprecated -Lxmpsdk/Adobe/XMP-Toolkit-SDK-CC201607
LIBS = -lz -lXMPCore
CXX = g++
Linux Build
CPPFLAGS = -I/usr/local/include -std=c++98 -Ixmpsdk/Adobe/XMP-Toolkit-SDK-CC201607/public/include
LDFLAGS = -L/usr/local/lib -ldl -lpthread -Lxmpsdk/Adobe/XMP-Toolkit-SDK-CC201607
LIBS = -lz -lm -lXMPCore
CXX = g++
When 2013 or 2014 are selected, the appropriate -Ipath and -Lpath should be set.
CMake Build
A new option -DEXIV2_ENABLE_ADOBE[=2016|=2014|=2013] should be added.
The build of XMPsdk should be performed with xmpsdk/buildXMPsdk.sh (or xmpsdk/buildXMPsdk.cmd for Visual Studio)
The build should set same LDFLAGS and CPPFLAGS as the autotools build.
When the options -DEXIV2_ENABLE_ADOBE is not used, we should compile the code in xmpsdk/src
to create and link libxmpsdk.a
==========================================================================
5 Refactoring the Tiff Code
Added : 2017-09-24
Modified: 2017-09-24
5.1 Background
Tiff parsing is the root code of a metadata engine.
The Tiff parsing code in Exiv2 is very difficult to understand and has major architectural shortcomings:
1) It requires the Tiff file to be totally in memory
2) It cannot handle BigTiff
3) The parser doesn't know the source of the in memory tiff image
4) It uses memory mapping on the tiff file
- if the network connection is lost, horrible things happen
- it requires a lot of VM to map the complete file
- BigTiff file can be 100GB+
- The memory mapping causes problems with Virus Detection software on Windows
5) The parser cannot deal with multi-page tiff files
6) It requires the total file to be in contiguous memory and defeats 'webready'.
The Tiff parsing code in Exiv2 is ingenious. It's also very robust. It works well. It can:
1) Handle 32-bit Tiff and Many Raw formats (which are derived from Tiff)
2) It can read and write Manufacturer's MakerNotes which are (mostly) in Tiff format
3) It probably has other great features that I haven't discovered
- because the code is so hard to understand, I can't simply browse and read it.
4) It separates file navigation from data analysis.
The code in image::printStructure was originally written to understand "what is a tiff?"
It has problems:
1) It was intended to be a single threaded debugging function and has security issues.
2) It doesn't handle BigTiff
3) It's messy. It's reading and processing metadata simultaneously.
The aim of this project is to
1) Reconsider the Tiff Code.
2) Keep everything good in the code and address known deficiencies
3) Establish a Team Exiv2 "Tiff Expert" who knows the code intimately.
5.2 How does Exiv2 decode the ExifData in a JPEG?
You can get my test file from http://clanmills.com/Stonehenge.jpg
808 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/build $ exiv2 -pS ~/Stonehenge.jpg
STRUCTURE OF JPEG FILE: /Users/rmills/Stonehenge.jpg
address | marker | length | data
0 | 0xffd8 SOI
2 | 0xffe1 APP1 | 15288 | Exif..II*......................
15292 | 0xffe1 APP1 | 2610 | http://ns.adobe.com/xap/1.0/.<?x
17904 | 0xffed APP13 | 96 | Photoshop 3.0.8BIM.......'.....
18002 | 0xffe2 APP2 | 4094 | MPF.II*...............0100.....
22098 | 0xffdb DQT | 132
22232 | 0xffc0 SOF0 | 17
22251 | 0xffc4 DHT | 418
22671 | 0xffda SOS
809 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/build $
Exiv2 calls JpegBase::readMetadata which locates the APP1/Exif segment.
It invokes the ExifParser:
ExifParser::decode(exifData_, rawExif.pData_, rawExif.size_);
This is thin wrapper over:
TiffParserWorker::decode(....) in tiffimage.cpp
What happens then? I don't know. The metadata is decoded in:
tiffvisitor.cpp TiffDecoder::visitEntry()
The design of the TiffMumble classes is the "Visitor" pattern
described in "Design Patterns" by Addison & Wesley. The aim of the pattern
is to separate parsing from dealing with the data.
The data is being stored in ExifData which is a vector.
Order is important and preserved.
As the data values are recovered they are stored as Exifdatum in the vector.
How does the tiff visitor work? I think the reader and processor
are connected by this line in TiffParser::
rootDir->accept(reader);
The class tree for the decoder is:
class TiffDecoder : public TiffFinder {
class TiffReader ,
class TiffFinder : public TiffVisitor {
class TiffVisitor {
public:
//! Events for the stop/go flag. See setGo().
enum GoEvent {
geTraverse = 0,
geKnownMakernote = 1
};
void setGo(GoEvent event, bool go);
virtual void visitEntry(TiffEntry* object) =0;
virtual void visitDataEntry(TiffDataEntry* object) =0;
virtual void visitImageEntry(TiffImageEntry* object) =0;
virtual void visitSizeEntry(TiffSizeEntry* object) =0;
virtual void visitDirectory(TiffDirectory* object) =0;
virtual void visitSubIfd(TiffSubIfd* object) =0;
virtual void visitMnEntry(TiffMnEntry* object) =0;
virtual void visitIfdMakernote(TiffIfdMakernote* object) =0;
virtual void visitIfdMakernoteEnd(TiffIfdMakernote* object);
virtual void visitBinaryArray(TiffBinaryArray* object) =0;
virtual void visitBinaryArrayEnd(TiffBinaryArray* object);
//! Operation to perform for an element of a binary array
virtual void visitBinaryElement(TiffBinaryElement* object) =0;
//! Check if stop flag for \em event is clear, return true if it's clear.
bool go(GoEvent event) const;
}
}
}
The reader works by stepping along the Tiff directory and calls the visitor's
"callbacks" as it reads.
There are 2000 lines of code in tiffcomposite.cpp and, to be honest,
I don't know what most of it does!
Set a breakpoint in src/exif.cpp#571.
That’s where he adds the key/value to the exifData vector.
Exactly how did he get here? That’s a puzzle.
void ExifData::add(const ExifKey& key, const Value* pValue)
{
add(Exifdatum(key, pValue));
}
5.3 How is metadata organized in Exiv2
section.group.tag
section: Exif | IPTC | Xmp
group: Photo | Image | MakerNote | Nikon3 ....
tag: YResolution etc ...
820 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ exiv2 -pa ~/Stonehenge.jpg | cut -d' ' -f 1 | cut -d. -f 1 | sort | uniq
Exif
Iptc
Xmp
821 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ exiv2 -pa --grep Exif ~/Stonehenge.jpg | cut -d'.' -f 2 | sort | uniq
GPSInfo
Image
Iop
MakerNote
Nikon3
NikonAf2
NikonCb2b
NikonFi
NikonIi
NikonLd3
NikonMe
NikonPc
NikonVr
NikonWt
Photo
Thumbnail
822 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ 533 rmills@rmillsmbp:~/Downloads $ exiv2 -pa --grep Exif ~/Stonehenge.jpg | cut -d'.' -f 3 | cut -d' ' -f 1 | sort | uniq
AFAperture
AFAreaHeight
AFAreaMode
...
XResolution
YCbCrPositioning
YResolution
534 rmills@rmillsmbp:~/Downloads $
823 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $
The data in IFD0 of is Exiv2.Image:
826 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ exiv2 -pR ~/Stonehenge.jpg | head -20
STRUCTURE OF JPEG FILE: /Users/rmills/Stonehenge.jpg
address | marker | length | data
0 | 0xffd8 SOI
2 | 0xffe1 APP1 | 15288 | Exif..II*......................
STRUCTURE OF TIFF FILE (II): MemIo
address | tag | type | count | offset | value
10 | 0x010f Make | ASCII | 18 | 146 | NIKON CORPORATION
22 | 0x0110 Model | ASCII | 12 | 164 | NIKON D5300
34 | 0x0112 Orientation | SHORT | 1 | | 1
46 | 0x011a XResolution | RATIONAL | 1 | 176 | 300/1
58 | 0x011b YResolution | RATIONAL | 1 | 184 | 300/1
70 | 0x0128 ResolutionUnit | SHORT | 1 | | 2
82 | 0x0131 Software | ASCII | 10 | 192 | Ver.1.00
94 | 0x0132 DateTime | ASCII | 20 | 202 | 2015:07:16 20:25:28
106 | 0x0213 YCbCrPositioning | SHORT | 1 | | 1
118 | 0x8769 ExifTag | LONG | 1 | | 222
STRUCTURE OF TIFF FILE (II): MemIo
address | tag | type | count | offset | value
224 | 0x829a ExposureTime | RATIONAL | 1 | 732 | 10/4000
236 | 0x829d FNumber | RATIONAL | 1 | 740 | 100/10
827 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ exiv2 -pa --grep Image ~/Stonehenge.jpg
Exif.Image.Make Ascii 18 NIKON CORPORATION
Exif.Image.Model Ascii 12 NIKON D5300
Exif.Image.Orientation Short 1 top, left
Exif.Image.XResolution Rational 1 300
Exif.Image.YResolution Rational 1 300
Exif.Image.ResolutionUnit Short 1 inch
Exif.Image.Software Ascii 10 Ver.1.00
Exif.Image.DateTime Ascii 20 2015:07:16 20:25:28
Exif.Image.YCbCrPositioning Short 1 Centered
Exif.Image.ExifTag Long 1 222
Exif.Nikon3.ImageBoundary Short 4 0 0 6000 4000
Exif.Nikon3.ImageDataSize Long 1 6173648
Exif.NikonAf2.AFImageWidth Short 1 0
Exif.NikonAf2.AFImageHeight Short 1 0
Exif.Photo.ImageUniqueID Ascii 33 090caaf2c085f3e102513b24750041aa
Exif.Image.GPSTag Long 1 4060
828 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $
The data in IFD1 is Exiv2.Photo
The data in the MakerNote is another embedded TIFF (which more embedded tiffs)
829 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $ exiv2 -pa --grep MakerNote ~/Stonehenge.jpg
Exif.Photo.MakerNote Undefined 3152 (Binary value suppressed)
Exif.MakerNote.Offset Long 1 914
Exif.MakerNote.ByteOrder Ascii 3 II
830 rmills@rmillsmbp:~/gnu/github/exiv2/exiv2/src $
The MakerNote decodes them into:
Exif.Nikon1, Exiv2.NikonAf2 and so on. I don't know exactly it achieves this.
However it means that tag-numbers can be reused in different IFDs.
Tag 0x0016 = Nikon GPSSpeed and can mean something different elsewhere.
5.4 Where are the tags defined?
There's an array of "TagInfo" data structures in each of the makernote decoders.
These define the tag (a number) and the tag name, the groupID (eg canonId) and the default type.
There's also a callback to print the value of the tag. This does the "interpretation"
that is performed by the -pt in the exiv2 command-line program.
TagInfo(0x4001, "ColorData", N_("Color Data"), N_("Color data"), canonId, makerTags, unsignedShort, -1, printValue),
5.5 How do the MakerNotes get decoded?
I don't know. It has something to do with this code in tiffcomposite.cpp#936
TiffMnEntry::doAccept(TiffVisitor& visitor) { ... }
Most makernotes are TiffStructures. So the TiffXXX classes are invoked recursively to decode the maker note.
#0 0x000000010058b4b0 in Exiv2::Internal::TiffDirectory::doAccept(Exiv2::Internal::TiffVisitor&) at /Users/rmills/gnu/github/exiv2/exiv2/src/tiffcomposite.cpp:916
This function iterated the array of entries
#1 0x000000010058b3c6 in Exiv2::Internal::TiffComponent::accept(Exiv2::Internal::TiffVisitor&) at /Users/rmills/gnu/github/exiv2/exiv2/src/tiffcomposite.cpp:891
#2 0x00000001005b5357 in Exiv2::Internal::TiffParserWorker::parse(unsigned char const*, unsigned int, unsigned int, Exiv2::Internal::TiffHeaderBase*) at /Users/rmills/gnu/github/exiv2/exiv2/src/tiffimage.cpp:2006
This function creates an array of TiffEntries
#3 0x00000001005a2a60 in Exiv2::Internal::TiffParserWorker::decode(Exiv2::ExifData&, Exiv2::IptcData&, Exiv2::XmpData&, unsigned char const*, unsigned int, unsigned int, void (Exiv2::Internal::TiffDecoder::* (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, Exiv2::Internal::IfdId))(Exiv2::Internal::TiffEntryBase const*), Exiv2::Internal::TiffHeaderBase*) at /Users/rmills/gnu/github/exiv2/exiv2/src/tiffimage.cpp:1900
#4 0x00000001005a1ae9 in Exiv2::TiffParser::decode(Exiv2::ExifData&, Exiv2::IptcData&, Exiv2::XmpData&, unsigned char const*, unsigned int) at /Users/rmills/gnu/github/exiv2/exiv2/src/tiffimage.cpp:260
#5 0x000000010044d956 in Exiv2::ExifParser::decode(Exiv2::ExifData&, unsigned char const*, unsigned int) at /Users/rmills/gnu/github/exiv2/exiv2/src/exif.cpp:625
#6 0x0000000100498fd7 in Exiv2::JpegBase::readMetadata() at /Users/rmills/gnu/github/exiv2/exiv2/src/jpgimage.cpp:386
#7 0x000000010000bc59 in Action::Print::printList() at /Users/rmills/gnu/github/exiv2/exiv2/src/actions.cpp:530
#8 0x0000000100005835 in Action::Print::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) at /Users/rmills/gnu/github/exiv2/exiv2/src/actions.cpp:245
5.6 How do the encoders work?
I understand writeMetadata() and will document that soon.
I still have to study how the TiffVisitor writes metadata.
6 Using external XMP SDK via Conan
Section 1 describes how to compile the newer versions of XMP SDK with a bash script. This
approach had few limitations:
1) We had to include sources from other projects into the Exiv2 repository: Check the folder
xmpsdk/third-party.
2) Different scripts for compiling XMP SDK on Linux, Mac OSX and Windows.
3) Lot of configuration/compilation issues depending on the system configuration.
Taking into account that during the last months we have done a big effort in migrating the
manipulation of 3rd party dependencies to Conan, we have decided to do the same here. A conan recipe
has been written for XmpSdk at:
https://github.com/piponazo/conan-xmpsdk
And the recipe and package binaries can be found in the piponazo's bintray repository:
https://bintray.com/piponazo/piponazo
This conan recipe provides a custom CMake finder that will be used by our CMake code to properly
find XMP SDK in the conan cache and then be able to use the CMake variables: ${XMPSDK_LIBRARY} and
${XMPSDK_INCLUDE_DIR}.
These are the steps you will need to follow to configure the project with the external XMP support:
# Add the conan-piponazo remote to your conan configuration (only once)
conan remote add conan-piponazo https://api.bintray.com/conan/piponazo/piponazo
mkdir build && cd build
# Run conan to bring the dependencies. Note that the XMPSDK is not enabled by default and you will
# need to enable the xmp option to bring it.
conan install .. --options xmp=True
# Configure the project with support for the external XMP version. Disable the normal XMP version
cmake -DCMAKE_BUILD_TYPE=Release -DEXIV2_ENABLE_XMP=OFF -DEXIV2_ENABLE_EXTERNAL_XMP=ON -DBUILD_SHARED_LIBS=ON ..
Note that the usage of the newer versions of XMP is experimental and it was included in Exiv2
because few users has requested it.