Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Grisu2 algorithm in String::num_scientific to fix serializing #98750

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use Grisu2 algorithm in String::num_scientific to fix serializing #98750

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions COPYRIGHT.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,11 @@ Comment: Graphite engine
Copyright: 2010, SIL International
License: Expat

Files: ./thirdparty/grisu2/grisu2.h
Comment: Grisu2 float serialization algorithm
Copyright: 2009 Florian Loitsch and 2018-2023 The simdjson authors
License: Expat and Apache

Files: ./thirdparty/harfbuzz/
Comment: HarfBuzz text shaping library
Copyright: 2010-2022, Google, Inc.
Expand Down
18 changes: 16 additions & 2 deletions core/doc_data.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,24 @@
#include "doc_data.h"

String DocData::get_default_value_string(const Variant &p_value) {
if (p_value.get_type() == Variant::ARRAY) {
const Variant::Type type = p_value.get_type();
if (type == Variant::ARRAY) {
return Variant(Array(p_value, 0, StringName(), Variant())).get_construct_string().replace("\n", " ");
} else if (p_value.get_type() == Variant::DICTIONARY) {
} else if (type == Variant::DICTIONARY) {
return Variant(Dictionary(p_value, 0, StringName(), Variant(), 0, StringName(), Variant())).get_construct_string().replace("\n", " ");
} else if (type == Variant::INT) {
return itos(p_value);
} else if (type == Variant::FLOAT) {
// Since some values are 32-bit internally, use 32-bit for all
// documentation values to avoid garbage digits at the end.
const String s = String::num_scientific((float)p_value);
// Use float literals for floats in the documentation for clarity.
if (s != "inf" && s != "-inf" && s != "inf_neg" && s != "nan") {
if (!s.contains(".") && !s.contains("e")) {
return s + ".0";
}
}
return s;
} else {
return p_value.get_construct_string().replace("\n", " ");
}
Expand Down
36 changes: 15 additions & 21 deletions core/string/ustring.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
#include "core/variant/variant.h"
#include "core/version_generated.gen.h"

#include "thirdparty/grisu2/grisu2.h"

#ifdef _MSC_VER
#define _CRT_SECURE_NO_WARNINGS // to disable build-time warning which suggested to use strcpy_s instead strcpy
#endif
Expand Down Expand Up @@ -1849,28 +1851,20 @@ String String::num_scientific(double p_num) {
if (Math::is_nan(p_num) || Math::is_inf(p_num)) {
return num(p_num, 0);
}
char buffer[256];
char *last = grisu2::to_chars(buffer, p_num);
*last = 0;
return buffer;
}

char buf[256];

#if defined(__GNUC__) || defined(_MSC_VER)

#if defined(__MINGW32__) && defined(_TWO_DIGIT_EXPONENT) && !defined(_UCRT)
// MinGW requires _set_output_format() to conform to C99 output for printf
unsigned int old_exponent_format = _set_output_format(_TWO_DIGIT_EXPONENT);
#endif
snprintf(buf, 256, "%lg", p_num);

#if defined(__MINGW32__) && defined(_TWO_DIGIT_EXPONENT) && !defined(_UCRT)
_set_output_format(old_exponent_format);
#endif

#else
sprintf(buf, "%.16lg", p_num);
#endif

buf[255] = 0;

return buf;
String String::num_scientific(float p_num) {
if (Math::is_nan(p_num) || Math::is_inf(p_num)) {
return num(p_num, 0);
}
char buffer[256];
char *last = grisu2::to_chars(buffer, p_num);
*last = 0;
return buffer;
}

String String::md5(const uint8_t *p_md5) {
Expand Down
1 change: 1 addition & 0 deletions core/string/ustring.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,7 @@ class String {
String unquote() const;
static String num(double p_num, int p_decimals = -1);
static String num_scientific(double p_num);
static String num_scientific(float p_num);
static String num_real(double p_num, bool p_trailing = true);
static String num_real(float p_num, bool p_trailing = true);
static String num_int64(int64_t p_num, int base = 10, bool capitalize_hex = false);
Expand Down
12 changes: 11 additions & 1 deletion core/variant/variant_call.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1620,6 +1620,16 @@ StringName Variant::get_enum_for_enumeration(Variant::Type p_type, const StringN
register_builtin_method<Method_##m_type##_##m_method>(sarray(), m_default_args);
#endif

#ifdef DEBUG_METHODS_ENABLED
#define bind_static_methodv(m_type, m_name, m_method, m_arg_names, m_default_args) \
STATIC_METHOD_CLASS(m_type, m_name, m_method); \
register_builtin_method<Method_##m_type##_##m_name>(m_arg_names, m_default_args);
#else
#define bind_static_methodv(m_type, m_name, m_method, m_arg_names, m_default_args) \
STATIC_METHOD_CLASS(m_type, m_name, m_method); \
register_builtin_method<Method_##m_type##_##m_name>(sarray(), m_default_args);
#endif

#ifdef DEBUG_METHODS_ENABLED
#define bind_methodv(m_type, m_name, m_method, m_arg_names, m_default_args) \
METHOD_CLASS(m_type, m_name, m_method); \
Expand Down Expand Up @@ -1798,7 +1808,7 @@ static void _register_variant_builtin_methods_string() {
bind_string_method(hex_decode, sarray(), varray());
bind_string_method(to_wchar_buffer, sarray(), varray());

bind_static_method(String, num_scientific, sarray("number"), varray());
bind_static_methodv(String, num_scientific, static_cast<String (*)(double)>(&String::num_scientific), sarray("number"), varray());
bind_static_method(String, num, sarray("number", "decimals"), varray(-1));
bind_static_method(String, num_int64, sarray("number", "base", "capitalize_hex"), varray(10, false));
bind_static_method(String, num_uint64, sarray("number", "base", "capitalize_hex"), varray(10, false));
Expand Down
70 changes: 38 additions & 32 deletions core/variant/variant_parser.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1932,20 +1932,26 @@ Error VariantParser::parse(Stream *p_stream, Variant &r_ret, String &r_err_str,
//////////////////////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////////////////////

static String rtos_fix(double p_value) {
// These two functions serialize floats or doubles using num_scientific to ensure
// it can be read back in the same way (except collapsing -0 to 0, and NaN values).
static String serialize_real(float p_value) {
if (p_value == 0.0) {
return "0"; //avoid negative zero (-0) being written, which may annoy git, svn, etc. for changes when they don't exist.
} else if (isnan(p_value)) {
return "nan";
} else if (isinf(p_value)) {
if (p_value > 0) {
return "inf";
} else {
return "inf_neg";
}
} else {
return rtoss(p_value);
return "0"; // Avoid negative zero (-0) being written, which may annoy git, svn, etc. for changes when they don't exist.
}
if (isinf(p_value) && p_value < 0) {
return "inf_neg"; // TODO: Replace with -inf or -INF.
}
return String::num_scientific(p_value);
}

static String serialize_real(double p_value) {
if (p_value == 0.0) {
return "0"; // Avoid negative zero (-0) being written, which may annoy git, svn, etc. for changes when they don't exist.
}
if (isinf(p_value) && p_value < 0) {
return "inf_neg"; // TODO: Replace with -inf or -INF.
}
return String::num_scientific(p_value);
}

Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_string_func, void *p_store_string_ud, EncodeResourceFunc p_encode_res_func, void *p_encode_res_ud, int p_recursion_count, bool p_compat) {
Expand All @@ -1960,7 +1966,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
p_store_string_func(p_store_string_ud, itos(p_variant.operator int64_t()));
} break;
case Variant::FLOAT: {
String s = rtos_fix(p_variant.operator double());
String s = serialize_real(p_variant.operator double());
if (s != "inf" && s != "inf_neg" && s != "nan") {
if (!s.contains_char('.') && !s.contains_char('e')) {
s += ".0";
Expand All @@ -1977,47 +1983,47 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
// Math types.
case Variant::VECTOR2: {
Vector2 v = p_variant;
p_store_string_func(p_store_string_ud, "Vector2(" + rtos_fix(v.x) + ", " + rtos_fix(v.y) + ")");
p_store_string_func(p_store_string_ud, "Vector2(" + serialize_real(v.x) + ", " + serialize_real(v.y) + ")");
} break;
case Variant::VECTOR2I: {
Vector2i v = p_variant;
p_store_string_func(p_store_string_ud, "Vector2i(" + itos(v.x) + ", " + itos(v.y) + ")");
} break;
case Variant::RECT2: {
Rect2 aabb = p_variant;
p_store_string_func(p_store_string_ud, "Rect2(" + rtos_fix(aabb.position.x) + ", " + rtos_fix(aabb.position.y) + ", " + rtos_fix(aabb.size.x) + ", " + rtos_fix(aabb.size.y) + ")");
p_store_string_func(p_store_string_ud, "Rect2(" + serialize_real(aabb.position.x) + ", " + serialize_real(aabb.position.y) + ", " + serialize_real(aabb.size.x) + ", " + serialize_real(aabb.size.y) + ")");
} break;
case Variant::RECT2I: {
Rect2i aabb = p_variant;
p_store_string_func(p_store_string_ud, "Rect2i(" + itos(aabb.position.x) + ", " + itos(aabb.position.y) + ", " + itos(aabb.size.x) + ", " + itos(aabb.size.y) + ")");
} break;
case Variant::VECTOR3: {
Vector3 v = p_variant;
p_store_string_func(p_store_string_ud, "Vector3(" + rtos_fix(v.x) + ", " + rtos_fix(v.y) + ", " + rtos_fix(v.z) + ")");
p_store_string_func(p_store_string_ud, "Vector3(" + serialize_real(v.x) + ", " + serialize_real(v.y) + ", " + serialize_real(v.z) + ")");
} break;
case Variant::VECTOR3I: {
Vector3i v = p_variant;
p_store_string_func(p_store_string_ud, "Vector3i(" + itos(v.x) + ", " + itos(v.y) + ", " + itos(v.z) + ")");
} break;
case Variant::VECTOR4: {
Vector4 v = p_variant;
p_store_string_func(p_store_string_ud, "Vector4(" + rtos_fix(v.x) + ", " + rtos_fix(v.y) + ", " + rtos_fix(v.z) + ", " + rtos_fix(v.w) + ")");
p_store_string_func(p_store_string_ud, "Vector4(" + serialize_real(v.x) + ", " + serialize_real(v.y) + ", " + serialize_real(v.z) + ", " + serialize_real(v.w) + ")");
} break;
case Variant::VECTOR4I: {
Vector4i v = p_variant;
p_store_string_func(p_store_string_ud, "Vector4i(" + itos(v.x) + ", " + itos(v.y) + ", " + itos(v.z) + ", " + itos(v.w) + ")");
} break;
case Variant::PLANE: {
Plane p = p_variant;
p_store_string_func(p_store_string_ud, "Plane(" + rtos_fix(p.normal.x) + ", " + rtos_fix(p.normal.y) + ", " + rtos_fix(p.normal.z) + ", " + rtos_fix(p.d) + ")");
p_store_string_func(p_store_string_ud, "Plane(" + serialize_real(p.normal.x) + ", " + serialize_real(p.normal.y) + ", " + serialize_real(p.normal.z) + ", " + serialize_real(p.d) + ")");
} break;
case Variant::AABB: {
AABB aabb = p_variant;
p_store_string_func(p_store_string_ud, "AABB(" + rtos_fix(aabb.position.x) + ", " + rtos_fix(aabb.position.y) + ", " + rtos_fix(aabb.position.z) + ", " + rtos_fix(aabb.size.x) + ", " + rtos_fix(aabb.size.y) + ", " + rtos_fix(aabb.size.z) + ")");
p_store_string_func(p_store_string_ud, "AABB(" + serialize_real(aabb.position.x) + ", " + serialize_real(aabb.position.y) + ", " + serialize_real(aabb.position.z) + ", " + serialize_real(aabb.size.x) + ", " + serialize_real(aabb.size.y) + ", " + serialize_real(aabb.size.z) + ")");
} break;
case Variant::QUATERNION: {
Quaternion quaternion = p_variant;
p_store_string_func(p_store_string_ud, "Quaternion(" + rtos_fix(quaternion.x) + ", " + rtos_fix(quaternion.y) + ", " + rtos_fix(quaternion.z) + ", " + rtos_fix(quaternion.w) + ")");
p_store_string_func(p_store_string_ud, "Quaternion(" + serialize_real(quaternion.x) + ", " + serialize_real(quaternion.y) + ", " + serialize_real(quaternion.z) + ", " + serialize_real(quaternion.w) + ")");
} break;
case Variant::TRANSFORM2D: {
String s = "Transform2D(";
Expand All @@ -2027,7 +2033,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i != 0 || j != 0) {
s += ", ";
}
s += rtos_fix(m3.columns[i][j]);
s += serialize_real(m3.columns[i][j]);
}
}

Expand All @@ -2041,7 +2047,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i != 0 || j != 0) {
s += ", ";
}
s += rtos_fix(m3.rows[i][j]);
s += serialize_real(m3.rows[i][j]);
}
}

Expand All @@ -2056,11 +2062,11 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i != 0 || j != 0) {
s += ", ";
}
s += rtos_fix(m3.rows[i][j]);
s += serialize_real(m3.rows[i][j]);
}
}

s = s + ", " + rtos_fix(t.origin.x) + ", " + rtos_fix(t.origin.y) + ", " + rtos_fix(t.origin.z);
s = s + ", " + serialize_real(t.origin.x) + ", " + serialize_real(t.origin.y) + ", " + serialize_real(t.origin.z);

p_store_string_func(p_store_string_ud, s + ")");
} break;
Expand All @@ -2072,7 +2078,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i != 0 || j != 0) {
s += ", ";
}
s += rtos_fix(t.columns[i][j]);
s += serialize_real(t.columns[i][j]);
}
}

Expand All @@ -2082,7 +2088,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
// Misc types.
case Variant::COLOR: {
Color c = p_variant;
p_store_string_func(p_store_string_ud, "Color(" + rtos_fix(c.r) + ", " + rtos_fix(c.g) + ", " + rtos_fix(c.b) + ", " + rtos_fix(c.a) + ")");
p_store_string_func(p_store_string_ud, "Color(" + serialize_real(c.r) + ", " + serialize_real(c.g) + ", " + serialize_real(c.b) + ", " + serialize_real(c.a) + ")");
} break;
case Variant::STRING_NAME: {
String str = p_variant;
Expand Down Expand Up @@ -2397,7 +2403,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i]));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i]));
}

p_store_string_func(p_store_string_ud, ")");
Expand All @@ -2412,7 +2418,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i]));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i]));
}

p_store_string_func(p_store_string_ud, ")");
Expand Down Expand Up @@ -2442,7 +2448,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i].x) + ", " + rtos_fix(ptr[i].y));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i].x) + ", " + serialize_real(ptr[i].y));
}

p_store_string_func(p_store_string_ud, ")");
Expand All @@ -2457,7 +2463,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i].x) + ", " + rtos_fix(ptr[i].y) + ", " + rtos_fix(ptr[i].z));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i].x) + ", " + serialize_real(ptr[i].y) + ", " + serialize_real(ptr[i].z));
}

p_store_string_func(p_store_string_ud, ")");
Expand All @@ -2472,7 +2478,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i].r) + ", " + rtos_fix(ptr[i].g) + ", " + rtos_fix(ptr[i].b) + ", " + rtos_fix(ptr[i].a));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i].r) + ", " + serialize_real(ptr[i].g) + ", " + serialize_real(ptr[i].b) + ", " + serialize_real(ptr[i].a));
}

p_store_string_func(p_store_string_ud, ")");
Expand All @@ -2487,7 +2493,7 @@ Error VariantWriter::write(const Variant &p_variant, StoreStringFunc p_store_str
if (i > 0) {
p_store_string_func(p_store_string_ud, ", ");
}
p_store_string_func(p_store_string_ud, rtos_fix(ptr[i].x) + ", " + rtos_fix(ptr[i].y) + ", " + rtos_fix(ptr[i].z) + ", " + rtos_fix(ptr[i].w));
p_store_string_func(p_store_string_ud, serialize_real(ptr[i].x) + ", " + serialize_real(ptr[i].y) + ", " + serialize_real(ptr[i].z) + ", " + serialize_real(ptr[i].w));
}

p_store_string_func(p_store_string_ud, ")");
Expand Down
2 changes: 1 addition & 1 deletion doc/classes/Animation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -689,7 +689,7 @@
<member name="loop_mode" type="int" setter="set_loop_mode" getter="get_loop_mode" enum="Animation.LoopMode" default="0">
Determines the behavior of both ends of the animation timeline during animation playback. This is used for correct interpolation of animation cycles, and for hinting the player that it must restart the animation.
</member>
<member name="step" type="float" setter="set_step" getter="get_step" default="0.0333333">
<member name="step" type="float" setter="set_step" getter="get_step" default="0.033333335">
The animation step value.
</member>
</members>
Expand Down
4 changes: 2 additions & 2 deletions doc/classes/CharacterBody2D.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@
If [code]false[/code] (by default), the body will move faster on downward slopes and slower on upward slopes.
If [code]true[/code], the body will always move at the same speed on the ground no matter the slope. Note that you need to use [member floor_snap_length] to stick along a downward slope at constant speed.
</member>
<member name="floor_max_angle" type="float" setter="set_floor_max_angle" getter="get_floor_max_angle" default="0.785398">
<member name="floor_max_angle" type="float" setter="set_floor_max_angle" getter="get_floor_max_angle" default="0.7853982">
Maximum angle (in radians) where a slope is still considered a floor (or a ceiling), rather than a wall, when calling [method move_and_slide]. The default value equals 45 degrees.
</member>
<member name="floor_snap_length" type="float" setter="set_floor_snap_length" getter="get_floor_snap_length" default="1.0">
Expand Down Expand Up @@ -195,7 +195,7 @@
<member name="velocity" type="Vector2" setter="set_velocity" getter="get_velocity" default="Vector2(0, 0)">
Current velocity vector in pixels per second, used and modified during calls to [method move_and_slide].
</member>
<member name="wall_min_slide_angle" type="float" setter="set_wall_min_slide_angle" getter="get_wall_min_slide_angle" default="0.261799">
<member name="wall_min_slide_angle" type="float" setter="set_wall_min_slide_angle" getter="get_wall_min_slide_angle" default="0.2617994">
Minimum angle (in radians) where the body is allowed to slide when it encounters a slope. The default value equals 15 degrees. This property only affects movement when [member motion_mode] is [constant MOTION_MODE_FLOATING].
</member>
</members>
Expand Down
4 changes: 2 additions & 2 deletions doc/classes/CharacterBody3D.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@
If [code]false[/code] (by default), the body will move faster on downward slopes and slower on upward slopes.
If [code]true[/code], the body will always move at the same speed on the ground no matter the slope. Note that you need to use [member floor_snap_length] to stick along a downward slope at constant speed.
</member>
<member name="floor_max_angle" type="float" setter="set_floor_max_angle" getter="get_floor_max_angle" default="0.785398">
<member name="floor_max_angle" type="float" setter="set_floor_max_angle" getter="get_floor_max_angle" default="0.7853982">
Maximum angle (in radians) where a slope is still considered a floor (or a ceiling), rather than a wall, when calling [method move_and_slide]. The default value equals 45 degrees.
</member>
<member name="floor_snap_length" type="float" setter="set_floor_snap_length" getter="get_floor_snap_length" default="0.1">
Expand Down Expand Up @@ -186,7 +186,7 @@
<member name="velocity" type="Vector3" setter="set_velocity" getter="get_velocity" default="Vector3(0, 0, 0)">
Current velocity vector (typically meters per second), used and modified during calls to [method move_and_slide].
</member>
<member name="wall_min_slide_angle" type="float" setter="set_wall_min_slide_angle" getter="get_wall_min_slide_angle" default="0.261799">
<member name="wall_min_slide_angle" type="float" setter="set_wall_min_slide_angle" getter="get_wall_min_slide_angle" default="0.2617994">
Minimum angle (in radians) where the body is allowed to slide when it encounters a slope. The default value equals 15 degrees. When [member motion_mode] is [constant MOTION_MODE_GROUNDED], it only affects movement if [member floor_block_on_wall] is [code]true[/code].
</member>
</members>
Expand Down
Loading
Loading