Unit File Migration Tool added to build.gradle.

Fixed 2 data issues with bad unicode chars.

megamek/build.gradle

@@ -400,3 +400,10 @@ task techLevelCompareTool(type: JavaExec, dependsOn: jar) {
     classpath = sourceSets.main.runtimeClasspath
     mainClass = 'megamek.utilities.TechLevelCompareTool'
 }
+
+task unitFileMigrationTool(type: JavaExec, dependsOn: jar) {
+    description = 'Tech Level Comparison Tool'
+    group = 'utility'
+    classpath = sourceSets.main.runtimeClasspath
+    mainClass = 'megamek.utilities.UnitFileMigrationTool'
+}

megamek/data/mechfiles/mechs/3058Uu/Dragon Fire DGR-3F.mtf

@@ -161,7 +161,7 @@ capabilities:<p>Based on the original design blueprints, the Dragon Fire was dev
 
 deployment:A basic model featuring a combination of large laser and medium pulse lasers, making it a reliable choice for many combat scenarios.
 
-history:The Dragon Fire walked off the JBBMI production lines in late 3058, with the first company going directly to the Draconis Combine. It proved itself in Operation Bulldog and the subsequent Great Refusal, which caused orders for the design to surge among units facing the Clan threat. A subsidiary contract with the Word of Blake in the mid-3060s incorporated the Word�s improved C3 technology, helping JBBMI achieve solid profit goals. In early 3068, JBBMI encountered a financial crisis when three stockholders gained over sixty percent of the corporation, leading to the closure of most production facilities in the Alliance, the Combine, and the Suns, with massive layoffs. Unconfirmed reports suggest new production facilities have been opened on Epsilon Eridani, Caph, and New Earth.
+history:The Dragon Fire walked off the JBBMI production lines in late 3058, with the first company going directly to the Draconis Combine. It proved itself in Operation Bulldog and the subsequent Great Refusal, which caused orders for the design to surge among units facing the Clan threat. A subsidiary contract with the Word of Blake in the mid-3060s incorporated the Word's improved C3 technology, helping JBBMI achieve solid profit goals. In early 3068, JBBMI encountered a financial crisis when three stockholders gained over sixty percent of the corporation, leading to the closure of most production facilities in the Alliance, the Combine, and the Suns, with massive layoffs. Unconfirmed reports suggest new production facilities have been opened on Epsilon Eridani, Caph, and New Earth.
 
 manufacturer:J.B. BattleMechs Inc
 primaryfactory:Storfors
@@ -170,4 +170,3 @@ systemmanufacturer:ENGINE:Pitban 300 XL
 systemmanufacturer:ARMOR:ArcShield Maxi II
 systemmanufacturer:COMMUNICATIONS:CeresCom Model 21-Rs with Guardian ECM
 systemmanufacturer:TARGETING:FireScan with IndirecTrack
-

megamek/data/mechfiles/mechs/Dark Age/Pack Hunter II (Isis).mtf

@@ -159,7 +159,7 @@ Jump Jet
 overview:The Pack Hunter II is a light BattleMech produced by Clan Wolf-in-Exile on Arc-Royal during the Jihad.
 capabilities:The 'Mech's legs are protected from possible Battle Armor attack by having a B-Pod mounted in each leg. The 'Mech's weapons and movement heat is managed by 10 Double Heat Sinks, but it's not adequate enough to handle the 'Mech's heat load.
 deployment:The personal Mech of Galaxy Commander Isis Bekker of Clan Ghost Bear's Omega Galaxy in 3136, this Pack Hunter II variant removes the standard weaponry of the Pack Hunter II. In its place, the 'Mech carried a Medium Pulse Laser in each arm, twin ER Micro Lasers in the right and left torso. To provide additional punch, an Artemis IV enhanced SRM-6 was mounted in the right torso. This version also carries slightly less armor than a standard Pack Hunter II.
-history:The 'Mech would first appear in combat in 3077 after a considerable amount of delays in its development. The design was originally seen as the OmniMech successor to the original Pack Hunter, however the Blakist attack against Arc-Royal and other priorities caused the delays. The Pack Hunter II improves on its predecessor�s qualities. Among the things the 'Mech has been fitted with, the designers made a substantial improvement in armor and weaponry while retaining the original's speed. In a bid to improve survival rates of pilots, the design also includes a Full-Head Ejection System to prevent the loss of a warrior in combat. The design had been seen deployed by 3085 with toumans of Clan Hell's Horses, the Draconis Combine and the Republic of the Sphere
+history:The 'Mech would first appear in combat in 3077 after a considerable amount of delays in its development. The design was originally seen as the OmniMech successor to the original Pack Hunter, however the Blakist attack against Arc-Royal and other priorities caused the delays. The Pack Hunter II improves on its predecessor's qualities. Among the things the 'Mech has been fitted with, the designers made a substantial improvement in armor and weaponry while retaining the original's speed. In a bid to improve survival rates of pilots, the design also includes a Full-Head Ejection System to prevent the loss of a warrior in combat. The design had been seen deployed by 3085 with toumans of Clan Hell's Horses, the Draconis Combine and the Republic of the Sphere
 manufacturer:WC Site 3
 primaryfactory:Arc-Royal
 systemmanufacturer:CHASSIS:Type AR1 Endo Steel
@@ -168,4 +168,3 @@ systemmanufacturer:ARMOR:Royal-7a Ferro-Fibrous
 systemmanufacturer:JUMPJET:Leaper Model, L5
 systemmanufacturer:COMMUNICATIONS:Trueborn Ultra 945B
 systemmanufacturer:TARGETING:Hunter 5 Dedicated TTS
-

megamek/src/megamek/utilities/UnitFileMigrationTool.java

@@ -20,48 +20,66 @@
 
 import java.io.File;
 import java.io.IOException;
+import java.nio.charset.MalformedInputException;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.util.List;
 
 import megamek.common.MechSummary;
 import megamek.common.MechSummaryCache;
 import megamek.common.loaders.MtfFile;
+import megamek.logging.MMLogger;
 
 /**
  * This is not a functional tool, just some template code to use when changing
  * unit files programmatically. Last used to move the unit roles into the unit
  * files. I leave this in so I don't have to reinvent the wheel.
  */
 public class UnitFileMigrationTool {
+    private static final MMLogger logger = MMLogger.create(UnitFileMigrationTool.class);
 
     public static void main(String... args) throws IOException {
         MechSummaryCache cache = MechSummaryCache.getInstance(true);
         MechSummary[] units = cache.getAllMechs();
+
+        List<String> lines = null;
+
         for (MechSummary unit : units) {
             File file = unit.getSourceFile();
+
             if (file.toString().toLowerCase().endsWith(".mtf")) {
-                List<String> lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8);
+                try {
+                    lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8);
+                } catch (MalformedInputException exception) {
+                    String message = String.format("MalformedInputException for file %s", file);
+                    logger.error(exception, message);
+                    String rawText = new String(Files.readAllBytes(file.toPath()));
+                    lines.clear();
+                    lines.add(rawText);
+                    Files.write(file.toPath(), lines, StandardCharsets.UTF_8);
+                    continue;
+                }
+
                 if (lines.get(0).startsWith("Version:")) {
                     lines.remove(0);
                 } else {
-                    System.out.println(unit + " doesn't have Version");
                     continue;
                 }
+
                 if (!lines.get(0).contains(":")) {
                     String chassis = lines.remove(0);
                     lines.add(0, MtfFile.CHASSIS + chassis);
                 } else {
-                    System.out.println(unit + " doesn't have chassis without :");
                     continue;
                 }
+
                 if (!lines.get(1).contains(":")) {
                     String model = lines.remove(1);
                     lines.add(1, MtfFile.MODEL + model);
                 } else {
-                    System.out.println(unit + " doesn't have model without :");
                     continue;
                 }
+
                 Files.write(file.toPath(), lines, StandardCharsets.UTF_8);
             }
         }