Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Otp 24 crypto fixes #83

Merged
merged 11 commits into from
Aug 20, 2021
Merged

Otp 24 crypto fixes #83

merged 11 commits into from
Aug 20, 2021

Conversation

kgautreaux
Copy link
Collaborator

@kgautreaux kgautreaux commented Jun 10, 2021

Sorry for the delay pushing the branch.

As you can see the actual fix for the removed :crypto.block_encrypt and :crypto.block_decrypt in OTP 24 is literally two lines in aes.ex, but the fixes caused a cascade of errors in the test suite.

I finally got some time the last few days to spelunk into the Erlang :crypto module and the NIF's in aead.c. I relate the story here because I think it is interesting.

The test suite instructions here currently recommend running the test suite with manually set env vars (ENCRYPTION_KEYS="key1,key2" SECRET_KEY_BASE="key" mix test), which must have worked fine in the :crypto.block_encrypt days, but after a lot of trial, error, and pretending I know how to read C I realized that in the OTP 24 era the cipher selection depends both on the atom passed to :crypto.crypto_one_time_aead and the key size.

So if you pass :aes_256_gcm as the cipher but you pass the binary "key1" as the key then your key size is only 24 bytes which corresponds to :aes_192_gcm not the :aes_256_gcm that you passed as your selected cipher and Erlang gives up and throws the:badarg exception saying that you have an unknown cipher.

So to fix the test suite failures we just have to make sure we pass a 32 byte key, but it seemed more ergonomic to me to try to inject the key as a dependency at run-time during the tests rather than rely on the tester passing the key with the right length as an env var.

That is why I changed the module attribute key extraction to a function so that it could be evaluated at run-time instead of compile time, and in the future an arbitrary key could be passed in during testing.

I think if I knew more about ExUnit this would have been a great idea, but instead I blundered my way into making a ExUnit.CaseTemplate using module as the basis for the testing so that common setup for the generated keys could be injected into each test module. I'm still not sure this is a good solution, but here we are.

After all of that I figured I better actually commit something so I @tag :skip-ed the failing hash tests, which have hardcoded binary's for the comparison tests and don't tolerate the custom setup I introduced.

I apologize for the length I didn't have time to make it shorter.

…EYS and SECRET_KEY_BASE that are adequately sized bitstrings to ensure that the cipher selection in aead.c in the Erlang :crypto module can find the :aes_256_gcm cipher. Cipher selection depends upon both the atom given to the Erlang NIF and the key size.
@kgautreaux kgautreaux requested a review from nelsonic June 10, 2021 04:40
@kgautreaux kgautreaux linked an issue Jun 10, 2021 that may be closed by this pull request
@nelsonic nelsonic self-assigned this Jun 10, 2021
@kgautreaux
Copy link
Collaborator Author

I had this CI problem locally as well. Fields.TestCase module wouldn't be available to use in the test files because it never got compiled to .beam code. I thought it might be because it wasn't postfixed with _test.exs but changing that didn't cause it to get picked up by the compiler. Eventually I had to elixirc the .ex file and move it into the test /ebin dir which I'm sure is a terrible hack because I don't understand the BEAM build structure like I should. I would love to learn why elixir scripts in the test dir aren't being added to the module hierarchy during a test run.

@nelsonic
Copy link
Member

Code changes LGTM. 👍
Having a bit of a roadblock installing Elixir @ 1.12.1 with OTP 24 on Mac. 🙄
Will take a look with a fresh pair of eyes.

@codecov
Copy link

codecov bot commented Aug 3, 2021

Codecov Report

Merging #83 (1dccb1b) into master (4297289) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##            master       #83   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           23        23           
  Lines          100       128   +28     
=========================================
+ Hits           100       128   +28     
Impacted Files Coverage Δ
lib/aes.ex 100.00% <100.00%> (ø)
lib/helpers.ex 100.00% <100.00%> (ø)
lib/url.ex 100.00% <0.00%> (ø)
lib/hash.ex 100.00% <0.00%> (ø)
lib/name.ex 100.00% <0.00%> (ø)
lib/address.ex 100.00% <0.00%> (ø)
lib/password.ex 100.00% <0.00%> (ø)
lib/postcode.ex 100.00% <0.00%> (ø)
lib/encrypted.ex 100.00% <0.00%> (ø)
lib/html-body.ex 100.00% <0.00%> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4297289...1dccb1b. Read the comment docs.

@th0mas
Copy link
Contributor

th0mas commented Aug 3, 2021

@kgautreaux Ideally you'd want to use a config file to change these values, but your solution works.

You're right in that Fields.TestCase needs to be in an .ex file - you need to add it to your compile path by changing mix.exs:

@@ -10,6 +10,7 @@ defmodule Fields.MixProject do
       start_permanent: Mix.env() == :prod,
       deps: deps(),
       package: package(),
+      elixirc_paths: elixirc_paths(Mix.env()),
 
@@ -28,6 +29,9 @@ defmodule Fields.MixProject do

+  defp elixirc_paths(:test), do: ["lib", "test/support"]
+  defp elixirc_paths(_), do: ["lib"]

edit: Didn't realise you were setting environment variables in the TestCase

@th0mas
Copy link
Contributor

th0mas commented Aug 3, 2021

Added the environment variables to a config file and re-used the previous testing values.

@nelsonic Should be good to merge now 👍

@th0mas th0mas linked an issue Aug 3, 2021 that may be closed by this pull request
lib/aes.ex Show resolved Hide resolved
Copy link
Member

@nelsonic nelsonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kgautreaux thank you so much for your efforts on this update. 🥇
@th0mas thanks for helping get the PR over the line. ⭐

@nelsonic nelsonic merged commit 3ef7c2b into master Aug 20, 2021
@nelsonic nelsonic deleted the otp-24-crypto-fixes branch August 20, 2021 04:12
@nelsonic
Copy link
Member

attempting to run mix hex.publish ...

getting the following error:

Building docs...
Generating docs...
** (FunctionClauseError) no function clause matching in ExDoc.Retriever.doc_ast/3

    The following arguments were given to ExDoc.Retriever.doc_ast/3:

        # 1
        "text/markdown"

        # 2
        %{}

        # 3
        [file: "lib/helpers.ex", line: 2]

    Attempted function clauses (showing 4 out of 4):

        defp doc_ast(_, :none, _options)
        defp doc_ast(_, :hidden, _options)
        defp doc_ast("text/markdown", %{"en" => doc}, options)
        defp doc_ast(other, %{"en" => _}, _)

    (ex_doc 0.22.5) lib/ex_doc/retriever.ex:167: ExDoc.Retriever.doc_ast/3
    (ex_doc 0.22.5) lib/ex_doc/retriever.ex:214: ExDoc.Retriever.get_module_docs/2
    (ex_doc 0.22.5) lib/ex_doc/retriever.ex:133: ExDoc.Retriever.do_generate_node/3
    (ex_doc 0.22.5) lib/ex_doc/retriever.ex:124: ExDoc.Retriever.generate_node/3
    (elixir 1.12.2) lib/enum.ex:3894: Enum.flat_map_list/2
    (elixir 1.12.2) lib/enum.ex:3895: Enum.flat_map_list/2
    (ex_doc 0.22.5) lib/ex_doc/retriever.ex:43: ExDoc.Retriever.docs_from_modules/2
    (ex_doc 0.22.5) lib/ex_doc.ex:26: ExDoc.generate_docs/3

Investigating ... 🔍

@nelsonic
Copy link
Member

For reference: elixir -v:

Erlang/OTP 24 [erts-12.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit] [dtrace]

Elixir 1.12.2 (compiled with Erlang/OTP 24)

Tried:

mix deps.clean --all
mix deps.get

From reading https://github.com/elixir-lang/ex_doc/blob/master/lib/ex_doc.ex#L2
Looks like we just need to add line to lib/helpers.ex to inform ExDoc that the file doesn't have moduledoc ...

@nelsonic
Copy link
Member

Looks like the @moduledoc false line addition worked. ✅
Now I just need to reset my hex.pm local password ... https://medium.com/@wiserfirst/reset-forgotten-hex-local-password-67d88c0c5dc4 🙄

@nelsonic
Copy link
Member

@kgautreaux [email protected] contains your code: https://hex.pm/packages/fields/2.8.0 thanks again! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to run mix test Update to Elixir v1.12 with OTP 24
3 participants