Converting collections to InlineString

[bkamins]

Currently the following is challenging:

julia> x = ["1"^i for i in 1:20]
20-element Vector{String}:
 "1"
 "11"
 "111"
 "1111"
 "11111"
 "111111"
 "1111111"
 "11111111"
 "111111111"
 "1111111111"
 "11111111111"
 "111111111111"
 "1111111111111"
 "11111111111111"
 "111111111111111"
 "1111111111111111"
 "11111111111111111"
 "111111111111111111"
 "1111111111111111111"
 "11111111111111111111"

julia> y = InlineString.(x)
20-element Vector{InlineString}:
 "1"
 "11"
 "111"
 "1111"
 "11111"
 "111111"
 "1111111"
 "11111111"
 "111111111"
 "1111111111"
 "11111111111"
 "111111111111"
 "1111111111111"
 "11111111111111"
 "111111111111111"
 "1111111111111111"
 "11111111111111111"
 "111111111111111111"
 "1111111111111111111"
 "11111111111111111111"

julia> typeof.(y)
20-element Vector{DataType}:
 InlineString1
 InlineString3
 InlineString3
 InlineString7
 InlineString7
 InlineString7
 InlineString7
 InlineString15
 InlineString15
 InlineString15
 InlineString15
 InlineString15
 InlineString15
 InlineString15
 InlineString15
 InlineString31
 InlineString31
 InlineString31
 InlineString31
 InlineString31

Ideally we should provide a convenient way to automatically get a correct "widest" type for the container. I am not sure how to best handle it (maybe add a function that for a given collection of strings returns the InlineString* type that can store them all?).

[quinnj]

We could perhaps overload InlineString.(vec) explicitly, as well as collect(::Type{InlineString}, itr).

[bkamins]

I also thought of handling special cases explicitly.

Also this is problematic (just to keep track of it):

julia> x = ["1", "11111111"]
2-element Vector{String}:
 "1"
 "11111111"

julia> [InlineString(s) for s in x]
2-element Vector{InlineString}:
 "1"
 "11111111"

julia> map(InlineString, x)
2-element Vector{InlineString}:
 "1"
 "11111111"

Finally the following requires fixing by adding missing promotion rules:

julia> vcat(InlineString("1"), InlineString("1111111"))
ERROR: MethodError: promote_type(::Type{InlineString7}, ::Type{Union{}}) is ambiguous. Candidates:
  promote_type(::Type{T}, ::Type{Union{}}) where T in Base at promotion.jl:223
  promote_type(::Type{T}, ::Type{S}) where {T<:InlineString, S<:InlineString} in WeakRefStrings at C:\Users\bogum\.julia\packages\WeakRefStrings\a3jYm\src\inlinestrings.jl:26
Possible fix, define
  promote_type(::Type{T}, ::Type{Union{}}) where T<:InlineString

[nalimilan]

The fact that map and broadcast do not use promotion is really annoying for data handling in general. At least we can mitigate this problem by using promotion as much as we can in DataFrames (as we already do).

[quinnj]

@bkamins, @nalimilan, are either of you aware of patterns used in DataFrames.jl that we'd want to explicitly overload in InlineStrings.jl to make things nicer? I'm wondering if we should specialize Base.append!(A::AbstractVector{<:InlineString}, B::AbstractVector{<:InlineString}) to promote? Or vcat? I've got a PR coming w/ a useful inlinestrings function that will promote, but just want to know in what cases we want to overload and use it.

[bkamins]

The crucial things are:

1. to make sure that ByRow works correctly:

(f::ByRow)(cols::AbstractVector...) = map(f.fun, cols...)
(f::ByRow)(table::NamedTuple) = [f.fun(nt) for nt in Tables.namedtupleiterator(table)]

2. to make sure that broadcasting works correctly, i.e. in fun.(col) where fun could return different concrete InlineStrings - that the resulting container has a proper eltype.

For push!, append!, vcat, and outerjoin we use promote_type:
https://github.com/JuliaData/DataFrames.jl/blob/main/src/dataframe/dataframe.jl#L1530
https://github.com/JuliaData/DataFrames.jl/blob/main/src/dataframe/dataframe.jl#L1367
https://github.com/JuliaData/DataFrames.jl/blob/main/src/abstractdataframe/abstractdataframe.jl#L1848
https://github.com/JuliaData/DataFrames.jl/blob/main/src/join/composer.jl#L12

[nalimilan]

Maybe we should change ByRow so that instead of vanilla map it uses a copy of map which computes the eltype via promote_type rather than promote_typejoin?

[bkamins]

which computes the eltype via promote_type rather than promote_typejoin?

In what practical cases do you think this is relevant? I would expect that ByRow is most often used in scenarios when a homogenous type is retuned, possibly unioned with Missing. I guess you are referring to the cases of e.g.:

julia> Base.promote_type(Int, Float64)
Float64

julia> Base.promote_typejoin(Int, Float64)
Real

but the problem is that changing this would be breaking in corner cases.

Consider for example such a case:

julia> x = [Float16(10), 10^9]
2-element Vector{Float16}:
 10.0
 Inf

I know Julia Base does it, but I do not think I like it.

[nalimilan]

I was thinking about an operation over a Vector{<:InlineString}, for example:

julia> df = DataFrame(x = String1.(["a", "b", "c"]), y = String3.(["aaa", "bbb", "ccc"]))
3×2 DataFrame
 Row │ x        y       
     │ String1  String3 
─────┼──────────────────
   1 │ a        aaa
   2 │ b        bbb
   3 │ c        ccc

julia> select(df,
              :x => ByRow(x -> x == "a" ? String1("z") : x),
              [:x, :y] => ByRow((x, y) -> x == "a" ? y : x))
3×2 DataFrame
 Row │ x_function  x_y_function 
     │ String1     InlineString 
─────┼──────────────────────────
   1 │ z           aaa
   2 │ b           b
   3 │ c           c

If we used type promotion we would get String and String3 columns, which is much cleaner and efficient.

The fact that promote_type(Int, Float16) === Float16 is problematic indeed. I would prefer promotion to return a wider type (Float64 in that case), maybe with the exception that promote_type(Int, Float64) === Float64, which is mostly OK as the precision loss only happens for very large integers and is relatively limited.