[stdlib] Move StringRef find() implementation to Span

[martinvuyk]

Move StringRef find() implementation to Span

[JoeLoser]

Question It seems a bit weird from an API design perspective to have this "find" function in span, can you help me understand why we'd want this? It feels more coupled to string algorithms, and Span isn't just for a view over string data.

[martinvuyk]

It can be used if you want to find occurrences of a specific scalar value sequence. My main thought was that we'll lose the ability to do that once StringSlice.find() is fixed to use unicode codepoints and it introduces quite some overhead for algorithms that are faster using raw bytes. We could also make this private WDYT?

Also, once PR's #3577 DType.get_dtype() gets merged, List.index() can also delegate to this

[martinvuyk]

@JoeLoser gentle ping and also to add something, since it seems like we are moving toward unifying Python's bytes into Span[UInt8]in PR #3636, request #3634 and many other places. Python's bytes.find is actually a method, and many other things that we put in StringSlice and StringRef are actually operations from bytes.

[msaelices]

I think this is a great change for overall performance when manipulating slices. It opens a lot of possibilities.

[owenhilyard]

@JoeLoser per discussion on discord, Span[T].find is intended to be memchr or memmem as appropriate, but that the subspan of bytes must be aligned to alignof[T](). We can probably expand it to any T: Trivial once we have a Trivial trait.

The proper string version likely needs be specialized for UTF-8 comparisons. @martinvuyk and I have discussed an "AsciiString" which can more aggressively optimize for using byte-oriented intrinsics since UTF-8 is annoying to do with SIMD.

[martinvuyk]

@JoeLoser would it be possible to land this before the next stable release? I'd like to use this to be able to bypass string's .find() for algorithms which work on raw bytes. And using that bypass, I'd like to make the breaking change to switch over to full unicode:

• String.__len__() should return unicode length and not byte length
• String.find() should work on unicode offsets
• String.__getitem__() should work on unicode offsets

I would like to make the change as soon as possible because it will break a lot of projects, and many more are coming because the community is growing.