Usage of OneHotMatrix for input to neural network is very slow.

[racinmat]

Multiplication of OneHotMatrix by dense layer could be more optimized, e.g. by

using LinearAlgebra
function Base.:*(A::AbstractMatrix, B::Flux.OneHotMatrix)
	m = size(A,1)
	Y = similar(A, m, size(B,2))
	for (j,ohv) in enumerate(B.data)
		ix = ohv.ix
		for i in 1:m
			@inbounds Y[i,j] = A[i,ix]
		end
	end
	Y
end
function Base.:*(A::AbstractMatrix, B::Adjoint{Bool,<: Flux.OneHotMatrix})
	m = size(A,1)
	Y = similar(A, m, size(B,2))
	Y .= 0
	BT = B'
	for (j,ohv) in enumerate(BT.data)
		ix = ohv.ix
		for i in 1:m
			@inbounds Y[i,ix] += A[i,j]
		end
	end
	Y
end

should I make PR for this?

[DhairyaLGandhi]

Yes that would be awesome. Somewhat like Zeros, we can have a method dispatch to get the correct columns from the hot neuron.

[racinmat]

Could you elaborate more what exactly do you have in mind?

[CarloLucibello]

@racinmat did you submit a PR for this?

[racinmat]

Not yet, but I plan to do it.

[ToucheSir]

I think this is resolved by #1448 and #1424?

[racinmat]

In the end it looks like it has not been resolved, here is benchmark: https://github.com/racinmat/flux_benchmarks/blob/master/results_slurm_1.md
I'll make new PR for it.

[DhairyaLGandhi]

What does the tricks imply there?

[racinmat]

That's the faster implementation: https://github.com/racinmat/flux_benchmarks/blob/master/0_12_7_tricks/main.jl#L6-L28
although I guess I should modify it for the PR so it would dispatch only on OneHotArray of dimension 2, right?

[ToucheSir]

The problem with the linked implementations is that they are decidedly not GPU-friendly. Given that the current code path is literally a property lookup and vectorized index, I think it would be fruitful to profile that first and see where the bottlenecks are.

[racinmat]

That's true, we would need to dispatch to different implementation for CuArrays, right?
It would definitely be fuitful to see why current implementations are that slow.

[ToucheSir]

Well, that's the thing. The current implementation should be more than fast enough since it theoretically does less work than the custom dense CPU array only version with a loop. That it doesn't warrants an investigation. Ideally, we'd like to avoid writing custom kernels for something so trivial.

[DhairyaLGandhi]

Right, maybe the we can find out where the bottleneck is in the current implementation instead

[racinmat]

I might have a culprit, https://github.com/FluxML/Flux.jl/blob/master/src/onehot.jl#L229 dispatches on OneHotVector, not OneHotMatrix, and multiplication by OneHotMatrix defaults to https://github.com/JuliaLang/julia/blob/v1.6.3/stdlib/LinearAlgebra/src/matmul.jl#L151-L154