Skip to content

v0.30.0
Compare
Choose a tag to compare
@philrz philrz released this 13 Sep 19:10
· 1665 commits to main since this release
v0.30.0
f502483
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Quick Start

Visit the Brim Download page to find the package for your OS platform.

Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.30.0/zed-v0.30.0.linux-amd64.zip
unzip zed-v0.30.0.linux-amd64.zip

Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.

Release Notes

As you can see below, there's been many changes since the last Zed GA release! Highlights include:

  • The introduction of Zed lakes for data storage, which include powerful Git-like branching. See the Zed lake README for details.
  • Enhancements to the Zed language to unify search and expression syntax, introduce new operators and functions for data exploration and shaping, and more! Review the Zed language docs for details.

The exhaustive set of changes is listed below. Come talk to us on Slack if you have additional questions.

  • Revise Zed language to unify search and expression syntax (#2072, #2152, #2252, #2304, #2294)
  • Add join() and split() functions for use on strings (#2098)
  • Add array slice expressions (#2100)
  • Fix an issue with connection resets after several minutes when posting data to S3 (#2106)
  • Fix an issue with parsing IPv6 literals (#2112)
  • Make the fuse operator work on nested records (#2052)
  • Fix an issue where cut(.) could cause a slice bounds out of range panic (#2107)
  • Add is(), fields(), and exists() functions (#2131)
  • Add auto-detection of ZSON format (#2123)
  • Fix an issue where cut to the root would exit if the referenced field was missing from a record (#2121)
  • Fix an issue where put to the root would panic on a non-record field (#2136)
  • Add support for parsing map types in ZSON (#2142)
  • Add a fuse() aggregate function (#2115)
  • Remove backward compatibility with alpha ZNG format (#2158)
  • Simplify ZSON by dropping type decorators when a complex value is fully implied (#2160)
  • Add a switch operator to allow branched processing (#2087, #2364, #2318, #2336)
  • Add constants and type literals to the Zed language (#2181)
  • The -I option in zq is now used for file includes (and allows multiple files), while -z now used for compact ZSON output (#2180, #2208)
  • Add support for shaping arrays and sets (#2173)
  • Fix an issue where outer aliases were being lost when ZSON was read into ZNG (#2189)
  • Add the sample operator that returns an example value for a named field, or for each unique record type (#2200, #2211, #2623)
  • Make the current record (i.e., this or .) an implicit argument to shape() (#2199)
  • Begin deprecating current TZNG format in favor of ZSON (#2208, #2312, #2333, #2338, #2337, #2339, #2340, #2355, #2367, #2377, #2387, #2388, #2389, #2395, #2477, #2485, #2480, #2513, #2520)
  • Fix an issue where accidentally reading non-Zed binary data caused a zq panic (#2206)
  • Fix an issue where time-sorted aggregations were returning non-deterministic results (#2220)
  • Add canonical Zed and the summarize operator as an explicit keyword before invoking aggregate functions (#2217, #2378, #2430, #2698)
  • Add support for casting the duration type (#2194)
  • Extend join to support inner (now the default), left, and right variations (#2210)
  • Fix an issue where Zed would not compile on FreeBSD (#2233)
  • Add the zson_parse() function (#2242)
  • Fix an issue where filenames containing : could not be read (#2240)
  • Handle aliases and typedefs in shaper functions, which also fixes a panic (#2257)
  • Improve Zeek reader performance (#2265, #2268)
  • Fix an issue where const references were not honored during query execution (#2260)
  • Fix an issue where shapers did not handle aliases to different castable types (#2280)
  • Add an unflatten() function that turns fields with dot-separated names into fields of nested records (#2277)
  • Fix an issue where querying an index in a Zed lake did not return all matched records (#2273)
  • Accept type definition names and aliases in shaper functions (#2289)
  • Add a reference shaper for Zeek data (#2300, #2368, #2448, #2489, #2601)
  • Fix an issue where accessing a null array element in a by grouping caused a panic (#2310)
  • Add support for parsing timestamps with offset format ±[hh][mm] (#2297)
  • Remove cropping from shape() (#2309)
  • Apply a Zed shaper when reading Suricata EVE data, instead of legacy JSON typing (#2298, #2370, #2400)
  • Add support for reading comma-separated value (CSV) files (#2317, #2858, #2942, #2963)
  • Fix an issue where reading a Zeek TSV log line would cause a panic if it contained too few fields (#2325)
  • Add a shape operator, which is useful for cleaning up CSV inputs (#2327)
  • Fix an issue where querying a Zed lake index for a named field could cause a panic (#2319)
  • Make casting to time and duration types more flexible (#2334, #2442)
  • Fix an issue where null values were not output consistently in a group-by aggregation (#2363)
  • Fix an issue where the confirmation messages from adding an index were sometimes incomplete (#2361)
  • Finalize ZSON duration format to be an extension of durations in Prometheus (#2358, #2371, #2381, #2396, #2405)
  • Add functions missing(), has(), and nameof() (#2393, #2708)
  • Add prototype support for SQL expressions (#2392)
  • Allow type definitions to be redefined (#2386)
  • Fix an issue where casting to a named type caused the loss of the type definition name (#2384)
  • Add support for Parquet output and rework the Parquet reader (#2227)
  • Don't interpret the first zq argument as a query if there are no additional arguments (#2382)
  • Fix an issue that was preventing the reference in an expression to a field name containing a . (#2407)
  • Add support for ISO time literals and support durations and time literals in expressions (#2406)
  • Add support for complex literals (#2403)
  • Code/repo reorganization for phasing out "ZQL" or "Z" in favor of "Zed language", or just "Zed" if context allows (#2416, #2431, #2455, #2831)
  • Support in with the map data type (#2421)
  • Normalize map values created from Zed expressions (#2423)
  • Switch to function-style casting (e.g., int64(123) instead of 123:int64) (#2427, #2438)
  • Allow shapers to to refer to the contents of input records to determine the type to apply (#2426)
  • Fix an issue where referencing a non-existent table in a SQL query caused a panic (#2432)
  • Accept - (stdin) as a zapi argument for loading data (#2435)
  • Fix an issue where a single bad cast could cause input processing to halt (#2446)
  • Create the zed command with sub-commands like query and api, but shortcut commands (e.g., zq, zapi) still remain (#2450, #2465, #2466, #2463, #2624, #2620)
  • Rename ZAR_ROOT environment variable to ZED_LAKE_ROOT (#2469)
  • Revise the top-level Zed README to reflect reorganization of the repo and new/changed tools (#2461)
  • Remove the -P flag from zq in favor of using from in the Zed language (#2491)
  • Add casting of the net data type (#2493, #2496)
  • zq now reads its inputs sequentially rather than the prior merged behavior (#2492)
  • Extend the len() function to return the number of fields in a record (#2494)
  • Remove the -E flag in zed commands that displayed time values as epoch (#2495)
  • Add the Zed lake design README document (#2500, #2569, #2595, #2781, #2940, #3014, #3034, #3035)
  • Fix an issue where escaping quotes caused a parse error (#2510)
  • Fix an issue where multiple ZSON type definitions would be output when only the first was needed (#2511)
  • Use less buffer when decoding ZSON (#2515)
  • Allow aliases of all primitive types to be expressed in ZSON (#2519)
  • Revert the "auto-fuse CSV" behavior originally added in #1908 (#2522)
  • Add support for Git-style Zed lakes (#2548, #2556, #2562, #2563, #2564, #2566, #2571, #2577, #2580, #2616, #2613, #2738, #2763, #2806, #2808, #2811, #2816, #2860, #2861, #2931, #2944, #2954, #2960, #2976, #2994, #3007, #3013, #3020, #3023, #3024, #3026, #3030, #3031, #3039, #3046)
  • Add support for reading JSON format input data via -i json (#2573, #2608)
  • Remove the legacy approach for applying Zed types to NDJSON input, as this is now done via Zed shapers (#2587)
  • Fix a Go client issue where ZNG marshal of unexported struct fields caused a panic (#2589)
  • Show a warning rather than failing when an unset value tries to be cut to the root (#2591)
  • Standardize -h usage in Zed CLI tools for showing help text (#2596, #2618)
  • Fix an issue where type names that started with primitive type names caused parse errors (#2612)
  • Colorize zson -Z output (#2621)
  • Remove pcap-related code, as this functionality has been moved to Brimcap (#2632)
  • The role previously performed by zqd is now handled by zed lake serve (#2629, #2722)
  • Revise ZJSON to encode types and type values using JSON structure instead of ZSON type strings (#2526)
  • this can now be used to reference the current top-level record (formerly ., which may be deprecated in the future) (#2650)
  • Rework dataflow model and Zed compiler optimizations (#2669)
  • Add initial explode operator that can break values from complex fields out into separate records (#2673)
  • Fix an issue where including a particular time-typed field in a shaper script caused errors with shaping other fields (#2685)
  • Silently discard duplicate fields when reading NDJSON records, which works around Suricata bug 4016 (#2691)
  • Fix an issue where ZSON type values were output without parentheses (#2700)
  • Swallow single-backslash-escaped / when reading NDJSON, which allows for reading default Suricata EVE output (#2697)
  • Improve the error message shown when no Zed lake root is specified (#2701, #2739)
  • Require on in join syntax (#2698)
  • Add a typeunder() function that returns the concrete type underlying a named type (#2709)
  • Improve ZNG scanner performance via multi-threading (#2678, #2682)
  • Fix an issue where a shaper created a corrupt time-typed value from an invalid timestamp rather than rejecting it (#2705)
  • Simplify keyword search by requiring := for assignment, == for comparison, and using matches for regex & glob match (#2692, #2744, #2773)
  • Allow reading data from http:// and https:// targets (#2723, #2732)
  • Support for arbitrary pool keys in Zed lakes (#2729, #2752)
  • Add API docs for the Zed lake service (#2679)
  • Support from file in Zed language in zq, which is particularly useful with join (#2753)
  • Fix an issue where certain data could be queried successfully via zq but not if loaded into a Zed lake pool (#2755)
  • Revise Python client docs to show double quotes during pip install, since Windows needs that (#2758)
  • Fix an issue where a query was incorrectly parallelized by merging on the wrong key (#2760)
  • Fix an issue where len() of a null array was evaluating to something greater than zero (#2761)
  • Fix an issue where sort with no fields was ignoring alias types and nested fields when picking a sort field (#2762)
  • Fix an issue where unexpected cut: no record found warnings were returned by zed lake query but not when the same data was queried via zq (#2764)
  • Move and extend the Zeek interoperability docs (#2770, #2782, #2830)
  • Create endpoints in the Zed lake service API that correspond to underlying Zed lake operations, and expose them via zapi commands (#2741, #2774, #2786, #2775, #2794, #2795, #2796, #2920, #2925, #2928)
  • Fix an issue where zq would surface a syntax error when reading ZSON it had sent as output (#2792)
  • Add an /events endpoint to the API, which can be used by clients such as the Brim app to be notified of pool updates (#2791)
  • Simplify the ZSON enum type by removing the values from the list of symbols (#2820)
  • Add Zed language documentation for the join operator (#2836)
  • Fix an issue where reading ZNG input with more than 222 type definitions triggered a zng type ID out of range error (#2847)
  • Have put only return the a referenced field is missing error on first occurrence (#2843)
  • Fix an issue where a zed lake query triggered a send on closed channel panic (#2842)
  • Allow casting to bool type (#2840)
  • Fix an issue where zq would surface an error when reading ZST it had sent as output (#2854)
  • Fix an issue where backend errors triggered by zapi query were not being surfaced (#2859)
  • Have the Python client use the /query endpoint for the Zed lake (#2869)
  • Minimize the amount of surrounding context shown when reporting parse errors (#2864)
  • Field assignments in join now behave like cut instead of pick (#2868)
  • Add more background/context to Zed top-level language README (#2866 #2878, #2901)
  • Unify from, split, and switch syntax to the forms shown here (#2871, #2896)
  • Shapers can now cast values of the null type to any type (e.g., arrays or records) (#2882)
  • Fix an issue where join was failing to match on values of comparable types (e.g., string and bstring) (#2880, #2884)
  • Shapers can now cast a value to a union type (#2881)
  • Introduce alternate switch syntax (#2888, #3004)
  • When fuse encounters a field with the same name but different types, it now creates one field of union type rather than separate, uniquely-named fields (#2885, #2886)
  • Fix an issue where fuse would consume too much memory when fusing many types (#2897, #2899)
  • Emphasize in the sort documentation that its output can be non-deterministic in the absence of an explicit field list (#2902)
  • Remove the space separator before decorator in ZSON -z output (#2911)
  • Fix an issue where handling of record alises caused a failure to shape Zeek NDJSON data (#2904)
  • Fix an issue where posting garbage input data to a pool caused an HTTP 500 response (#2924)
  • Fix an issue where reading a ZNG file and outputting as CSV caused a deadlock (#2929)
  • In a from clause, range is now used instead of over to specify a range scan over a data source (#2943)
  • Fix a Zed language issue with parsing parenthesized search terms (#2951)
  • Column headers in -f table outputs now reflect the case of the field name rather than always being uppercase (#2964)
  • Reserved words in the Zed language can now be used in more places (e.g., field name references) without risk of collisions that would require escaping (#2968)
  • Zed CLI tools now send human-readable ZSON by default if output is to a terminal, otherwise binary ZNG (#2979, #2985)
  • Temporary directories for spill-to-disk operations now are prefixed with zed-spill- rather than zq-spill- (#2980)
  • The put operator keyword is now optional (e.g., can write x:=1 instead of put x:=1) (#2967, #2986, #3043)
  • Fix an issue where a put on a nested record with an alias triggered a panic (#2990)
  • Fix an issue where temporary spill-to-disk directories were not being deleted upon exit (#3009, #3010)
  • Fix a ZSON issue with union types with alias decorators (#3015, #3016)
  • The ZSON format has been changed such that integer type IDs are no longer output (#3017)
  • Update the reference Zed shaper for Zeek (shaper, docs) to reflect changes in Zeek release v4.1.0 (#3021)
  • Fix an issue where backslash escapes in Zed regular expressions were not accepted (#3040)
  • The ZST format has been updated to work for typedef'd outer records (#3047)
  • Fix an issue where an empty string could not be output as a JSON field name (#3054)
Assets 5
Loading