Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dynamic field type(flat_object) support #3067

Open
kedbirhan opened this issue Oct 10, 2024 · 5 comments
Open

dynamic field type(flat_object) support #3067

kedbirhan opened this issue Oct 10, 2024 · 5 comments
Labels
catch-all acknowledged Marks issues caught in catch-all triage for searchability enhancement New feature or request

Comments

@kedbirhan
Copy link

dynamic field type(flat_obejct) field are missing from returned query result. For instance requestParameter and responseElement field of cloudtrail logs are commonly mapped to flat_object field type to avoid mapping explosion but when doing so the query result returned completely ignores this field on top of that we can't even use this dynamic nested fields for querying this is a big problem for storing cloud native logs that may have dynamic field values,

@kedbirhan kedbirhan added bug Something isn't working untriaged labels Oct 10, 2024
@kedbirhan kedbirhan changed the title dynamic field type(flat_obejct) field missing dynamic field type(flat_obejct) value missing Oct 11, 2024
@dblock dblock removed the untriaged label Oct 28, 2024
@dblock
Copy link
Member

dblock commented Oct 28, 2024

[Catch All Triage - 1, 2, 3]

@kedbirhan
Copy link
Author

Query

SELECT requestParameters
from logs-cloudtrail WHERE 
requestParameters.nextToken is not null
LIMIT 25

Response

400 {error={reason=There was internal problem at backend, details=FLAT_OBJECT fieldName types are currently not supported., type=IllegalArgumentException}, status=500}

@kedbirhan kedbirhan changed the title dynamic field type(flat_obejct) value missing dynamic field type(flat_obejct) support Nov 13, 2024
@kedbirhan
Copy link
Author

SELECT * FROM logs-cloudtrail
WHERE (
        (eventName = 'CreateInstanceExportTask' AND eventSource = 'ec2.amazonaws.com') 
        AND NOT (
                (errorMessage LIKE '%%%' ) 
                OR (errorCode LIKE '%%%') OR (responseElements LIKE '%Failure%' )))
400 Bad Request: "{<EOL> "error": {<EOL> "reason": "Invalid SQL query",<EOL> "details": "can't resolve Symbol(namespace=FIELD_NAME, name=responseElements) in type env",<EOL> "type": "SemanticCheckException"<EOL> },<EOL> "status": 400<EOL>}"

@acarbonetto acarbonetto changed the title dynamic field type(flat_obejct) support dynamic field type(flat_object) support Jan 2, 2025
@acarbonetto
Copy link
Collaborator

Related issue: #1604

@acarbonetto acarbonetto added enhancement New feature or request and removed bug Something isn't working labels Jan 2, 2025
@acarbonetto
Copy link
Collaborator

@kedbirhan would you consider mapping the fields you'd like to search to an object? Searches on flat_objects are limited.

@Swiddis Swiddis added the catch-all acknowledged Marks issues caught in catch-all triage for searchability label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
catch-all acknowledged Marks issues caught in catch-all triage for searchability enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants