-
Notifications
You must be signed in to change notification settings - Fork 922
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Introduce CompositeHttpHeadersBase
to replace HttpHeadersUtil.merge()
#5340
base: main
Are you sure you want to change the base?
Conversation
d3ab464
to
8a080c9
Compare
8a080c9
to
1c3776f
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5340 +/- ##
===========================================
+ Coverage 0 74.21% +74.21%
- Complexity 0 21119 +21119
===========================================
Files 0 1803 +1803
Lines 0 77283 +77283
Branches 0 9948 +9948
===========================================
+ Hits 0 57354 +57354
- Misses 0 15254 +15254
- Partials 0 4675 +4675 ☔ View full report in Codecov by Sentry. |
@Nullable | ||
private StringMultimap<IN_NAME, NAME> appender; | ||
@Nullable | ||
private List<StringMultimap<IN_NAME, NAME>> delegates; | ||
|
||
private final Set<IN_NAME> removed = new HashSet<>(); | ||
private final Map<IN_NAME, Integer> removedSizeCache = new HashMap<>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR, left a preliminary comment 🙇
} | ||
|
||
@Override | ||
public Iterator<Entry<NAME, String>> iterator() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question) To match the current merge logic, should this loop deduplicate such that only one key is returned?
i.e.
final CompositeStringMultimap<CharSequence, AsciiString> headers =
new CompositeHttpHeadersBase(HttpHeaders.of("k1", "v1"),
HttpHeaders.of("k1", "v2"));
for (Entry<AsciiString, String> entry: headers) {
System.out.println(entry); // ("k1", "v2") only should be printed
}
ditto for the other iterating methods (forEach
, forEachValue
, etc..)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
armeria/core/src/main/java/com/linecorp/armeria/internal/common/HttpHeadersUtil.java
Lines 46 to 48 in d328e27
public static ResponseHeaders mergeResponseHeaders(ResponseHeaders headers, | |
HttpHeaders additionalHeaders, | |
HttpHeaders defaultHeaders, |
armeria/core/src/main/java/com/linecorp/armeria/internal/common/HttpHeadersUtil.java
Lines 64 to 66 in d328e27
for (AsciiString name : defaultHeaders.names()) { | |
if (!isPseudoHeader(name) && !builder.contains(name)) { | |
defaultHeaders.forEachValue(name, value -> builder.add(name, value)); |
Oh you're right! based on current merge logic, we don't add duplicated key on mergeHeaders()
. I miss it 😅
I think CompositeHeaders
can support merge (remove duplicated key)
and just composite
options.
(or maybe we can explicitly introduce CompositeHeader
and MergedHeader
class separately,
or Iwe can just simply introduce additionalHeaders
, defaultHeaders
field on CompositeHeader
!)
Anyway, I'll think more about this design and share to you soon~! 🙇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// 1) Merge headers same with HttpHeadersUtil.merge(additionalHeaders, headers, defaultHeaders)
CompositeHeader additionals = new CompositeHeader(add1, add2, ..);
for (Name name : additional.names() {
if (isPsedoHeader(name) ||
ADDITIONAL_REQUEST_HEADER_DISALLOWED_LIST.contains(name)) {
additional.remove(name) // Only mark as removed, without deep copy
}
}
..
// Can composite default and internal header like this without deep copy
HttpHeaders internals = HttpHeaders.of(HttpHeaderNames.SERVER, ArmeriaHttpUtil.SERVER_HEADER);
CompositeHeader defaults = new CompositeHeader(default1, internals);
// 2) Also, can composite headers
CompositeHeader headers = new CompositeHeader(header1, header2, ..);
✅ To respect current HttpHeadersUtil.merge(additionals, headers, defaults)
logic,
I make CompositeHeader
can merge additional, header, default
too with same logic, without deep copy~!
a5d6074
to
b088bae
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about the late review. 😓
Basically, looks good and left some suggestions. 👍
CompositeStringMultimap(@Nullable CompositeStringMultimap<IN_NAME, NAME> additionals, | ||
List<StringMultimap<IN_NAME, NAME>> parents, | ||
@Nullable CompositeStringMultimap<IN_NAME, NAME> defaults, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It seems like the additionals
and defaults
CompositeStringMultimap
s are tightly coupled to the logic that merges response headers and I think they complicates this class.
How about removing those and just taking the list of StringMultimap
? The additionalHeaders
, headers
, and defaultHeaders
are specified in that order so that the headers in additionalHeaders
take precedence.
new CompositeHttpHeadersBase(additionalHeaders, headers, defaultHeaders)
armeria/core/src/main/java/com/linecorp/armeria/internal/common/HttpHeadersUtil.java
Lines 46 to 48 in dc2bdc6
public static ResponseHeaders mergeResponseHeaders(ResponseHeaders headers, | |
HttpHeaders additionalHeaders, | |
HttpHeaders defaultHeaders, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about removing those and just taking the list of StringMultimap?
✅ updated, nice suggestion! this way looks much simpler 👍
if (contentLength < 0) { | ||
contentLength = 0; | ||
} | ||
contentLength += length; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about aggregating all the Content-Length headers as it's unlikely that it accurately represents the actual content length. How about using the length of the first encountered header as we do for others?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about using the length of the
first
encountered header as we do for others?
✅ updated! agree with this way.
Similarly, I'm not sure how we should handle on isEndOfStream
case 🤔
- e.g. If only some of the parents are
endOfStream: true
, what should we return onisEndOfStream()
?
PTAL 🙇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good question.
Based on previous usage, simply using the flag of the first header may not be sufficient.
I see two potential options:
- If there's a headers with
isEndOfStream`` set to
true``, consider the composite headers as true as well. This approach seems reasonable, as the composite headers should reflect the end-of-stream status if any of them does. - Ignore the
isEndOfStream
flag in all headers objects and allow the user to specify it after composition.
What do you think of it?
Of course, others might have different opinions. cc @line/dx
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there's a headers with
isEndOfStream
set totrue
, consider the composite headers astrue
as well.
I think option1
looks better so updated code and test~! thank you! 🙇
core/src/main/java/com/linecorp/armeria/common/CompositeHttpHeadersBase.java
Outdated
Show resolved
Hide resolved
return true; | ||
} | ||
|
||
if (!(o instanceof HttpHeaderGetters)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this intentional not using CompositeHttpHeadersBase
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// CompositeStingMultimap
if (that instanceof CompositeStringMultimap ||
that instanceof StringMultimap) { // allow StringMultimap to compare too
for (NAME name : names()) {
if (!getAll((IN_NAME) name.toString())
.equals(that.getAll((IN_NAME) name.toString()))) {
return false;
}
}
Aha right. on CompositeStingMultimap#equal
, I think we may can compare CompositeStingMultimap
with StringMultimap
too.
So i intended to just check !(o instanceof HttpHeaderGetters)
here to allow both CompositeStingMultimap
and StringMultimap
.
Hmm should we allow only o instanceof CompositeHttpHeadersBase
to check equality here?
I'm not sure so please share your opinion~! thanks 🙇
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's okay because they are equal if they have the same headers no matter what the type is.
We have the similar case for ByteArrayHttpData
:
armeria/core/src/main/java/com/linecorp/armeria/common/ByteArrayHttpData.java
Lines 78 to 83 in 2aa149b
final HttpData that = (HttpData) o; | |
if (length() != that.length()) { | |
return false; | |
} | |
return Arrays.equals(array(), that.array()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got it! I added related test case :) (compare equality with CompositeHeader and just Header)
Really sorry for late, I'll update this PR within this weekend 🙇 |
3f7688f
to
1480260
Compare
* | ||
* @see CompositeHttpHeadersBase | ||
*/ | ||
abstract class CompositeStringMultimap<IN_NAME extends CharSequence, NAME extends IN_NAME> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think is should be Multi'M'ap
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, I don't know what I was thinking when I made that comment. Sorry. Let's just leave it as it is.
1480260
to
d291c12
Compare
bb5ef93
to
f35a37f
Compare
f35a37f
to
e9fcd38
Compare
super(from(parents), DEFAULT_APPENDER_SUPPLIER); | ||
} | ||
|
||
CompositeHttpHeadersBase(@Nullable List<HttpHeaderGetters> additionals, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Considering the current usage, which involves three different headers in HttpHeadersUtil, it seems that this constructor isn't necessary. Do you have a specific reason for adding it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in HttpHeadersUtil, it seems that this constructor isn't necessary.
armeria/core/src/main/java/com/linecorp/armeria/internal/common/HttpHeadersUtil.java
Lines 57 to 62 in c6b8906
for (AsciiString name : additionalHeaders.names()) { | |
if (!isPseudoHeader(name)) { | |
builder.remove(name); | |
additionalHeaders.forEachValue(name, value -> builder.add(name, value)); | |
} | |
} |
If we don't have this constructor, how can we pass additioanls
headers to CompositeStringMultimap
? 🤔
armeria/core/src/main/java/com/linecorp/armeria/common/AbstractHttpHeadersBuilder.java
Lines 32 to 40 in c6b8906
@Override | |
final HttpHeadersBase newSetters(int sizeHint) { | |
return new HttpHeadersBase(sizeHint); | |
} | |
@Override | |
final HttpHeadersBase newSetters(HttpHeadersBase parent, boolean shallowCopy) { | |
return new HttpHeadersBase(parent, shallowCopy); | |
} |
armeria/core/src/main/java/com/linecorp/armeria/common/DefaultHttpHeadersBuilder.java
Lines 24 to 30 in c6b8906
DefaultHttpHeadersBuilder(HttpHeadersBase parent) { | |
super(parent); | |
} | |
@Override | |
public HttpHeaders build() { | |
final HttpHeadersBase delegate = delegate(); |
Also, this constructor is needed for next PR, to add CompositeHttpHeaderBuilder
similar with HttpHeaderBuilder
!
It uses new HttpHeaderBase(..)
constuructor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't have this constructor, how can we pass additioanls headers to CompositeStringMultimap?
CompositeHttpHeadersBase(HttpHeaderGetters... parents) {...}
This constructor accepts multiple HttpHeaders
. Can't users simply specify the additional headers that take precedence as the first argument?
/**
* Merges the given {@link ResponseHeaders}. The headers have priority in the following order.
* <pre>{@code
* additional headers (highest priority) > headers > default headers > framework headers (lowest priority)
* }</pre>
*/
public static ResponseHeaders mergeResponseHeaders(ResponseHeaders headers,
HttpHeaders additionalHeaders,
HttpHeaders defaultHeaders) {
...
new CompositeHttpHeadersBase(additionalHeaders, headers, defaultHeaders);
...
}
this constructor is needed for next PR, to add CompositeHttpHeaderBuilder similar with HttpHeaderBuilder!
It uses new HttpHeaderBase(..) constructor
I might miss this. Could you share an example of how it will be used?
|
||
@Override | ||
public boolean isEndOfStream() { | ||
for (StringMultimapGetters<CharSequence, AsciiString> delegate : delegates()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we check the appender
first?
Additionally, we need to ensure that the appender isn't created if it's null.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#5340 (comment) If there's a headers with
isEndOfStream
set totrue
, consider the composite headers astrue
as well.
As we discussed like above, I think it's not mandatory to always check appender
first on isEndOfStream
!
Cause if one of headers is end-of-stream, we return composite headers' isEndOfStream
as true
.
For optimization, we can check appender
first 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, we need to ensure that the appender isn't created if it's null.
protected final StringMultimap<IN_NAME, NAME> appender() {
if (appender != null) {
return appender;
}
appender = requireNonNull(appenderSupplier.get(), "appender");
final ImmutableList.Builder<StringMultimapGetters<IN_NAME, NAME>> builder = ImmutableList.builder();
// 👈👈 If appender is created, add appender on delegates.
delegates = builder.addAll(delegates())
.add(appender)
.build();
return appender;
}
protected final List<StringMultimapGetters<IN_NAME, NAME>> delegates() {
if (delegates != null) { // 👈👈 if appender is created or delegates() is called before, return it.
return delegates;
}
}
Nice point! I considered this case and on delegates()
, it doesn't contain appender
if appender is not created yet~!
So we don't have to check appender is null or not when using delegates()
!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we discussed like above, I think it's not mandatory to always check appender first on isEndOfStream !
I apologize for my previous comment. It doesn't contain all the information
I thought we still needed to check the appender if endOfStream
is set after the CompositeHttpHeadersBase
is created.
I imagined a case something like:
CompositeHttpHeadersBase headers =
new CompositeHttpHeadersBase(additionalHeaders, endOfStreamTrueHeaders);
headers.endOfStream(false);
Additionally, we need to ensure that the appender isn't created if it's null.
What I meant was that we need to ensure that the appender is not created if we change the logic to check for the appender first. 😉
CompositeHttpHeadersBase
to replace HttpHeadersUtil.merge()CompositeHttpHeadersBase
to replace HttpHeadersUtil.merge()
Really sorry for late, I think we can reduce complexity of CompositeHeader by supporting only
I found that spring has similar ✅ So, I'll check |
related issue: #5146
Motivation:
armeria/core/src/main/java/com/linecorp/armeria/internal/common/HttpHeadersUtil.java
Line 46 in 3595dac
HttpHeadersUtil.mergeHeaders(..)
occupies a large part of CPU cycles due to expensive deep-copy operations📌 PRs Overview:
CompositeHttpHeadersBase
to replaceHttpHeadersUtil.mergeHeaders(..)
CompositeHttpBeaderBuilder
builderDefaultCompositeRequest, ResponseHeaders
HttpHeadersUtil.mergeHeaders(..)
toComposite*Headers
Modifications:
CompositeHttpHeadersBase
andCompositeStringMultiMap
to replace HttpHeadersUtil.merge()Result:
CompositeHttpHeadersBase
so can implementDefaultCompositeRequest, ResponseHeaders