Implementing Neural Network
All the features of input are weighted to obtain one output, if we want more outputs we need to increase weight accordingly, so if we have D features (as row/width) in input number of weights will be D (the column/height of Weight Matrix if the implementation is considered). If we want more outputs we need to add more columns.
https://www.draw.io/ File
Depending on where you keep your input or structure it, output weight matrix dimension or structure becomes accordingly.
There are two cases:
Case 1: Y = X * W [This is the case the above figure considers]
Where each row of X is a single data and thus number of columns of X denotes the number of input features.
Then,
Size of Weight Matrix = Feature Length of Input x Feature Length of Output
W = Input x Output
So in the output i.e. Y, each row is a single data, and number of output features = number of columns.
Case 2: Y = W * X [https://medium.com/usf-msds/deep-learning-best-practices-1-weight-initialization-14e5c0295b94]
Where each column of X is a single data and thus number of rows of X denotes the number of features.
Then
Size of Weight Matrix = Feature Length of Output x Feature Length if Input
W = Output x Input
So in the output i.e. Y, each column is a single data, and number of output features = number of rows.
Think of Case 2 as a transpose of Case 1.
Both cases are viable.